4 - 1 Information Assurance Foundations - SANS ©2001 1 Incident Handling Foundations Security Essentials The SANS Institute Hello. The material we are going to cover this next hour is central to understanding the theory and practice of information security. This is a foundational course, developed for the SANS Security Essentials program. When you complete this course there will be a quiz available from the SANS web page to help reinforce the material and ensure your mastery of it. So many companies and people worry about their network or computer systems being compromised, but few address what they would do if they were compromised. If a company is connected to the Internet they will never be able to prevent all attacks. The motto I like to use is “prevention is ideal but detection is a must.” Being able to detect and react to an attack in a timely matter is key. This module covers the fundamentals of incident handling and shows what a company needs to do to properly address an incident. 4 - 2 Incident Handling Foundations - SANS ©2001 2 Agenda • What is incident handling? • Why is it important? • What is an incident? • Fundamentals • The Six Step process On the Agenda slide for this module we are going to address various aspects of incident handling. We are going to start with the basics and look at what incident handling is and what it means to your company. We are then going to cover why it is important and why a company needs to be concerned and have proper procedures for dealing with an incident. Being able to identify an incident in a timely manner and react is very important. Just as important is knowing what is not an incident so a company does not have to waste any of their time. The fundamentals of incident handling will also be covered along with the 6 step process for dealing with an incident. The six step process is taken from the “Incident Handling, Step-by-Step guide” published by the SANS Institute. For additional details on how to handle an incident, the Step-by-Step guide is recommended along with a full day course offered by the SANS Institute on Incident Handling. 4 - 3 Incident Handling Foundations - SANS ©2001 3 Incident Handling • Incident Handling is an action plan for dealing with intrusions, cyber- theft, denial of service, fire, floods, and other security-related events • Having proper procedures in place so you know what to do when an incident occurs As stated on the slide Incident Handling, incident handling is an action plan for dealing with intrusions, cyber-theft, denial of service, fire, floods, and other security-related events. This slide makes it clear that the scope of incident handling is greater than just intrusions, it covers insider crime, and intentional and unintentional events that cause a loss of availability. In fact, fires and floods are every bit as much an incident as a hacker attack. A lot of people only think an incident is a hacker attacker but it is a lot more than that. The other key point of the definition is the notion of action. Sitting there watching is not incident handling. You do not want to move too fast, but you do need to get in motion in an incident! Identifying an incident is important but you must act on that information to secure your systems in a timely manner. The best way to act on an incident and minimize your chance of a mistake is by having proper procedures in place. Well-documented procedures make sure that you know what to do when an incident occurs and minimizes the chances that you will forget something. 4 - 4 Incident Handling Foundations - SANS ©2001 4 Why is it Important? • Sooner or later an incident is going to occur. Do you know what to do? • It is not a matter of “if” but “when” • Planning is everything • Similar to backups – You might not use it every day, but if a major problem occurs you are going to be glad that you did It does not matter how big your company is or what type of business you are in, sooner or later you are going to have an incident. Companies of all sizes and types have had incidents and those that were not prepared and did not handle it correctly in some cases are no longer around to talk about it. When it comes to having to deal with an incident, it is not a matter of IF an incident is going to occur but WHEN is it going to occur. Another important point is the way some companies choose to deal with an incident is by ignoring it, but as you can image this is very risky to do. I bring this up because some companies I talk to say, “I have never had an incident in 2 years, why do I have to worry about it?” In this case the truth of the matter is they probably have had several incidents, but since they failed to detect them they took a stance of ignoring each incident. As we stated, this is very dangerous and it is only a matter of time until this catches up with you. One of the main reasons for a module on incident handling is, planning is everything. If you are prepared and know what to do, dealing with an incident can be fairly straightforward. On the other hand, if it catches you off-guard, there can be many sleepless nights. The key thing with incident handling is planning is very important but not to get discouraged if you do all of this planning and do not use it right away. Do not say, “I have done this planning and have not had an incident in 3 months.” Think of it as backups, you might not need to use them every day but if a problem ever occurs, and it will, you will be so glad that you did it. 4 - 5 Incident Handling Foundations - SANS ©2001 5 What is an Incident? • An “incident” is an adverse event in an information system, and/or network, or the threat of the occurrence of such an event. • Examples of incidents include: – unauthorized use of another user’s account – unauthorized use of system privileges – execution of malicious code that destroys data • Incident implies harm, or the attempt to do harm – Incident handler reduces or minimizes harm The slide “What is an Incident?” is for the purpose of defining what we mean when we use a word like incident or event. Incident, as we are using it, refers to harm or the significant threat of harm. There are several important points for an incident handler that flow from this definition. • Since we are dealing with harm or potential harm, our task is to limit the damage. We want to be careful to choose courses of action that do not cause further harm. • If the incident is not what is termed an act of God, your organization may well have a legal right. In either case, the incident handler should proceed in a manner that does not preclude using the evidence gathered in a court setting. Some examples of incidents are: •unauthorized use of another user’s account •unauthorized use of system privileges •execution of malicious code that destroys data Notice the key word in several of these examples, “unauthorized”. If a user openly and willingly gives their account information to another user with the intent that they will use the account to access the network, it is not an incident. It is only when someone uses that account without the permission of the owner in an unauthorized manner. 4 - 6 Incident Handling Foundations - SANS ©2001 6 What is an Event? • An “event” is any observable occurrence in a system and/or network • Examples of events include: – the system boot sequence – a system crash – packet flooding within a network • These observable events compose an incident • All incidents are composed of events, but not all events are incidents Since an incident is composed of events, lets look at what an event is. An event is something that happened in time that you either directly experienced or that you can show actually occurred. An event is something that you saw flash on the screen or that you heard. It can also be something that you know occurred because it was collected in a log or audit file. As part of the incident handling process, you should create forms that you use to record events. These forms can help you write down the information that should be documented. They can help you to be alert for the things you should be looking for. If you need a starting point, the SANS Incident Handling Step-by-Step guide has sample forms you can use and they are not copyrighted. Make all the copies you want and if you have suggestions for improvement, please email these to intrusion@sans.org. If there is any chance of the incident ending in a court case, having correlating information is better than a single source claiming the event happened. For instance, if two people saw a message flash on a screen, that will likely have more validity in court than if one person saw it. Further, attackers sometimes use tools to alter or delete their traces in log files. If you can produce two independent sources for the information, this has more validity. This is one reason we really push intrusion analysts to become familiar with a large number of log formats. The key thing to remember when looking at what an incident is, versus an event, is all incidents are composed of events but not all events are considered incidents. For example, an unauthorized logon is considered an incident, while an authorized logon is not considered an incident, yet both of these are network events. 4 - 7 Incident Handling Foundations - SANS ©2001 7 Examples of an Incident • Which of the following is an incident: 1. An attacker running NetBIOS scans against a Unix system. 2. An attacker exploiting Sendmail on a Unix system. 3. A backup tape containing sensitive information is missing. On the slide “Examples of an Incident,” lets look at a couple of examples to see what is and what is not an incident. The following are three events. Which ones would you consider to be an incident? 1) An attacker running NETBios scans against a Unix system. 2) An attacker exploiting Sendmail on a Unix system. 3) A backup tape containing sensitive information is missing. Actually the correct answer is all 3 would be considered incidents. In the first example, some might not consider it an incident because an attacker running an NT exploit against a Unix system would not be successful and therefore you do not have to worry about it. Remember our definition of what an incident is, a threat or occurrence of an event. So even though this attack was not successful, it should still be taken as a threat and the next time you might not be so lucky. Remember a lot of these attackers are running scripts against a wide range of systems, so in this case you were lucky because the attack was not successful but what if they ran this NT exploit against an NT system that you have or what if they come back and run a Unix exploit against your Unix system? Wouldn’t you rather deal with an incident when it is not successful than when it is? The second example is a successful attack against a Unix system and should be fairly obvious that this is an example of an incident. Someone successfully compromised your system without authorization or permission. The third example is also an incident because a tape containing all of your company’s data has been compromised. This has the same net impact as if someone broke into your system over the Internet and stole all of your information. Even though stealing a tape is not as glamorous as a hacking attack, it is still considered an incident and must be acted upon. Now that we have a good idea of what an incident is and looked at some examples, lets look at the incident handling process in the next slide. 4 - 8 Incident Handling Foundations - SANS ©2001 8 Overview of the Incident Handling Process Incident Handling is similar to first aid. The caregiver tends to be under pressure and mistakes can be very costly . A simple, well- understood approach is best. Keep the six stages, (preparation, detection, containment, eradication, recovery, and follow-up) in mind. Use pre-designed forms, and call on others for help. A good way to get an overview of the incident handling process is to compare it to first aid. In both cases, time is not on your side. You are under a lot of pressure and mistakes are very costly. To give you an example, my first real job out of college was working at a Defense Mapping Agency. At that time, they wanted to have an internal rescue squad. I volunteered and completed the training and was pretty excited for my first call. They gave me a pager to carry out so I could be notified when there was a problem. Remember this is back in the day when pagers and cell phones were not that popular, so to carry one around was very impressive. I could not wait for my pager to go off. Finally my pager went off and I was racing to the scene. I passed the chief of the squad and he reached out and grabbed me. He said, “Son, if you hurt someone else or yourself, when you get to the scene, you will not be any good to anyone. Now let’s walk down together”. Now, many years later, I am the grizzled old veteran and I want to pass this advice on to you. Law enforcement agents tell story after story of the well-meaning system administrator that ruined the evidence and usually just a couple minutes after the incident. You do need to act, but take time to think. There is a crucial point to this story. No one can run so fast they can outrun a computer with a 650Mhz Pentium III chip attached to a 100Mb Ethernet network. More importantly, when one is working as root, or administrator, or supervisor, there are many operations that do not have an “undo”. Several times we will draw the analogy between incident handling and first aid. It is a solid analogy. In some sense, first aid is a form of incident handling. So to review this slide the three things you have to remember when dealing with an incident is that it will be very stressful. Every minute will count and mistakes need to be minimized. Putting these three things together means you need to work but not so quick that you make matters worse. Remember the saying, “If you do not have enough time to do it correctly the first time, how will you have enough time to do it again?” 4 - 9 Incident Handling Foundations - SANS ©2001 9 Incident Handling – 6 Steps •Preparation • Identification • Containment •Eradication • Recovery • Lessons Learned On the slide “Incident Handling - 6 Steps,” we list the six steps in incident handling, which are preparation, identification, containment, eradication, recovery, and lessons learned. The steps serve you, the handler, as a compass or a roadmap, a way to keep in mind what you should be trying to do and the things you need to do next. The key thing with this list is, in order to be successful, you must follow all 6 steps. Some people think if they follow only some of the steps they will be in good shape, but in order to be successful at incident handling, this requires following all of the steps. Now each of the steps need to be customized to a particular company and the industry they work in and the following slides will help you do that. 4 - 10 Incident Handling Foundations - SANS ©2001 10 Preparation • Planning is everything • Policy – Organizational approach – Inter-organization • Obtain management support • Select team members • Identify contacts in other organizations (legal, law enforcement) When it comes to incident handling, planning is everything and preparation plays a key role. It is very important that you have a policy in place that covers an organization’s approach to dealing with an incident. Things that it needs to cover is whether a company is going to notify law enforcement agencies or run silent or whether a company is going to contain and clean an incident or watch and learn. One thing you really want to avoid is having an incident happen and finding yourself in a debate about whether to contain the incident and clean up, or to watch the attackers and try to gather more evidence. The time to make these (career affecting) decisions is before the incident, keeping senior management and your legal staff apprised. The policy should also contain what the policy is for inter organization approach and how a company works with other companies on an incident. It is very important that an incident handling team has management support and buy-in. The last thing a company wants is for senior management to be questioning or doubting the decisions that were made during an incident. Not everybody makes a good incident handler. There are some very smart people that I have worked with whose personalities do not lend themselves to being a good incident handler. People that like to work solo and be heroes usually do not make good team members. You want someone who works well in a team environment and thinks out solutions and not make rash decisions. [...]... jump bag with everything you need to handle an incident Incident Handling Foundations - SANS ©2001 11 During preparation, a company needs to make sure they update their organization’s disaster recovery plan to include computer incident handling A large organization with over ten thousand computers is going to rack up some incidents This can cause the incident handlers to burn out It is kind of interesting;... handler • Determine whether an event is an incident • Maintain a provable chain of custody • Make a clean backup of the system Incident Handling Foundations - SANS ©2001 14 When it comes to handling an incident, if one person isn’t in charge, no person is in charge For smaller incidents, often of the “would you check this out” category, there is no need to send core incident handlers It is recommended practice... handle • Learn from the past and keep improving your incident handling procedures Incident Handling Foundations - SANS ©2001 22 Incident handling is not easy and very prone to error due to the conditions that you are working under Things that normally are very easy to perform become very difficult when you increase the stress and pressure that an incident handler is under To give you an idea of the... Putting the Steps Together • Steps must be customized for your environment • Every incident is different • Planning is everything • Make things simple with checklists and tested procedures Incident Handling Foundations - SANS ©2001 19 We just covered the general 6 steps for handling an incident, but since every company and every incident is different, they must be customized for your particular environment... helps ensure that mistakes are minimized 4 - 19 Key Mistakes in Incident Handling • • • • • • • Failure to report or ask for help Incomplete/non-existent notes Mishandling/destroying evidence Failure to create working backups Failure to contain or eradicate Failure to prevent re-infection Failure to apply lessons learned Incident Handling Foundations - SANS ©2001 20 On this slide, we have a list in probably... like law enforcement and legal people By doing this, everyone can do what they do best and increase your chance of success Incident handling is not easy, so learn from each event and follow the 6-step process during an incident 4 - 22 Course Revision History Incident Handling Foundations - SANS ©2001 v1.0 – E Cole – April 2001 v1.1 – edited/formatted by J Kolde – 8 May 2001 v1.2 – edited and audio... home, get that shower, and jump into bed several hours earlier 4 - 20 Common Incidents You Might See • Port scanning • Denial of service attacks • Unauthorized access to accounts • Viruses Incident Handling Foundations - SANS ©2001 21 There are a lot of attacks occurring on the Internet but the following are some of the more common incidents you might see: Port scanning Denial of service attacks Unauthorized... • System reboots • Poor performance Incident Handling Foundations - SANS ©2001 13 There are many different signs of an incident, but this slide lists some of the more common ones •IDS tool has an alert •Unexplained entries in a log file •Failed events, such as logon •Unexplained events (new accounts) •System reboots •Poor performance Some additional signs of an incident are: Suspicious entries in system... symptom • Improve defenses • Perform vulnerability analysis Incident Handling Foundations - SANS ©2001 16 Before the system goes back online, an incident handler must make sure that they fix the problem or the vulnerability that the attacker used to compromise the system Nuking the operating system from high orbit may be considered a shortcut in the handling process While it is certainly true that total... for an executive summary to cover? How much the organization saved by having an effective incident handling procedure! During every incident, mistakes occur We learn from these, improve our process for the future, and move on Sometimes, we run into policy or other organizational problems that hinder bringing the incident to a close We note these and submit them to management for their consideration Follow-up . process in the next slide. 4 - 8 Incident Handling Foundations - SANS ©2001 8 Overview of the Incident Handling Process Incident Handling is similar to first. the fundamentals of incident handling and shows what a company needs to do to properly address an incident. 4 - 2 Incident Handling Foundations - SANS