CHAPTER 14 Intrusion Detection 261 Copyright 2001 The McGraw-Hill Companies, Inc. Click Here for Terms of Use. 262 Network Security: A Beginner’s Guide I ntrusion detection is another tool for security staff to use to protect an organization from attack. Intrusion detection is a reactive concept that tries to identify a hacker when a penetration is attempted. Ideally, such a system will only alarm when a suc - cessful attack is made. Intrusion detection can also assist in the proactive identification of active threats by providing indications and warnings that a threat is gathering informa - tion for an attack. In reality, as we will see in the following pages, this is not always the case. Before we discuss the details of intrusion detection, let’s define what it actually is. Intrusion detection systems (IDS) have existed for a long time. Some of the earliest forms included night watchmen and guard dogs. In this case, the watchmen and guard dogs served two purposes: they provided a means of identifying that something bad was happening and they provided a deterrent to the perpetrator. Most thieves were not inter - ested in facing a dog so they were unlikely to attempt to rob a building with dogs. The same is true for a night watchman. Thieves did not want to be spotted by a watchman who might have a gun or who would call the police. Burglar and car alarms are also forms of IDS. If the alarm system detects an event that it is programmed to notice (such as the breaking of a window or the opening of a door), lights go on, an alarm sounds, or the police are called. The deterrent function is provided by a window sticker or a sign in the front yard of the house. Cars often have a red light visible on the dashboard to give an indication that an alarm is active. All of these examples share a single, principal aim: detect any attempt to penetrate the security perimeter of the item (business, building, car, and so on) being protected. In the case of a building or car, the security perimeter is easy to identify. The walls of the build- ing, a fence around the property, or the doors and windows of the car clearly define the security perimeter. Another characteristic that all of these examples have in common is well-defined criteria for what constitutes a penetration attempt and what constitutes the security perimeter. If we translate the concept of the alarm system into the computer world, we have the base concept of an IDS. Now we must define what the security perimeter of our computer system or network actually is. Clearly, the security perimeter does not exist in the same way as a wall or fence. Instead, the security perimeter of a network refers to the virtual perimeter surrounding an organization’s computer systems. This perimeter can be de - fined by firewalls, telecom demarcation points, or desktop computers with modems. It may also be extended to include the home computers of employees who are allowed to telecommute or a business partner that is allowed to connect to the network. A burglar alarm is designed to detect any attempted entry into a protected area dur - ing times of non-occupancy. An IDS is designed to differentiate between an authorized entry and a malicious intrusion, which is much more difficult. A good analogy to further explain this is a jewelry store with a burglar alarm. If anyone, even the owner, opens the door, the alarm sounds. The owner must then notify the alarm company that he has opened his store and all is well. An IDS is more like the guard at the front door watching every patron of the store and looking for malicious intent (carrying a gun for example). Unfortunately, in the virtual world the gun is very often invisible. Chapter 14: Intrusion Detection 263 The second issue that must be dealt with is the definition of what events constitute a violation of the security perimeter. Is an attempt to identify live systems such an event? What about the use of a known attack against a system on the network? As these ques - tions are asked, it becomes clear that the answers are not black and white. Instead, they depend upon other events and the state of the target system. TYPES OF INTRUSION DETECTION SYSTEMS There are two primary types of IDS: host-based (H-IDS) and network-based (N-IDS). An H-IDS resides on a particular host and looks for indications of attacks on that host. An N-IDS resides on a separate system that watches network traffic, looking for indications of at - tacks that traverse that portion of the network. Figure 14-1 shows how the two types of IDS may exist in a network environment. Host-Based IDS An H-IDS exists as a software process on a system. Traditionally, H-IDS systems have ex- amined log entries for specific information. On Unix systems, the logs that are normally examined include Syslog, Messages, Lastlog, and Wtmp. On Windows systems, the Sys- tem, Application, and Security Event Logs are examined. Periodically, the H-IDS process looks for new log entries and matches them up to pre-configured rules. If a log entry matches a rule, the H-IDS will alarm. If the H-IDS is to function properly, the necessary information must appear in the logs. Therefore, if the information that is most interesting is generated by an application, the application must place that information into the stan- dard logs on the system or the H-IDS must be capable of examining the application logs. More recently, a new form of H-IDS has been created that examines calls to the oper- ating system kernel. This type of H-IDS is programmed with known attack signatures and will alarm if a system call matches any of the signatures. Both types of H-IDS are capable of checking files on the system for modification. This is done by performing a cryptographic checksum on the file using a hashing function such as MD5 (see Chapter 12). This value is then stored and used as a comparison against periodic checksums of the file. If the checksums do not match, the file has been altered and the H-IDS will report this information. There are three primary advantages to an H-IDS system: ▼ The H-IDS will not miss attack traffic that is directed at a system as long as the attack generates a log message (or a system call). ■ The H-IDS can determine if an attack was successful by examining log messages or other indications on the system (such as the modification of key system binaries or configuration files). ▲ The H-IDS can be used to identify unauthorized access attempts by legitimate system users. 264 Network Security: A Beginner’s Guide There are three disadvantages to an H-IDS system: ▼ The H-IDS process may be identified and disabled by an attacker. ■ The H-IDS system can only alarm if the log entries or system calls match pre-configured rules or signatures. ▲ Certain H-IDS systems may impact support and maintenance agreements on operating system software. This is primarily associated with an H-IDS that examines system calls. Figure 14-1. Examples of IDS placement in a network environment Network-Based IDS An N-IDS exists as a software process on a dedicated hardware system. The N-IDS places the network interface card on the system into promiscuous mode, meaning that the card passes all traffic on the network (rather than just traffic destined for that system) to the N-IDS software. The traffic is then analyzed according to a set of rules and attack signa - tures to determine if it is traffic of interest. If it is, an event is generated. At this time, N-IDS systems are primarily signature-based. This means that a set of at - tack signatures has been built into the systems and these are compared against the traffic on the wire. If an attack is used that is not in the signature file, the N-IDS will not pick it up. N-IDS systems also have the capability to specify traffic of interest based on the source address, destination address, source port, or destination port. This allows organi - zations to define traffic to watch for that is outside of the attack signatures. The most common configuration for an N-IDS is to use two network interface cards. One card is used to monitor a network. This card is placed in a “stealthy” mode so that it does not have an IP address and, therefore, does not respond to incoming connections. The stealthy card does not have a protocol stack bound to it so that it cannot respond to probes such as a ping. The second card is used to communicate with the IDS management system and to send alarms. This card is attached to an internal network that is not visible to the network being monitored. Advantages of an N-IDS include ▼ The N-IDS can be completely hidden on the network so an attacker will not know that he is being monitored. ■ A single N-IDS can be used to monitor traffic to a large number of potential target systems. ▲ The N-IDS can capture the contents of all packets traveling to a target system. Disadvantages of an N-IDS system include ▼ The N-IDS system can only alarm if the traffic matches pre-configured rules or signatures. ■ The N-IDS can miss traffic of interest due to high bandwidth utilization or alternate routes. ■ The N-IDS cannot determine if the attack was successful. ■ The N-IDS cannot examine traffic that is encrypted. ▲ Switched networks (as opposed to shared media networks) require special configurations so that the N-IDS can see all the traffic. Is One Type of IDS Better? Is one type of IDS better? It depends. Both types have their advantages and disadvan - tages as we have seen. While an N-IDS may be more cost-effective (a single N-IDS can Chapter 14: Intrusion Detection 265 266 Network Security: A Beginner’s Guide monitor traffic to a large number of systems), an H-IDS may be more appropriate for or - ganizations that are more concerned about legitimate users than about external hackers. Another way to say this is that the choice of which type of IDS to use depends upon the primary threats to the organization. SETTING UP AN IDS In order to get the most out of an IDS, a lot of planning must be done beforehand. Even before an appropriate policy can be created, information must be gathered, the network must be analyzed, and executive management must be involved. As with most complex systems, the policy must be created, validated, and tested prior to deployment. The spe - cific steps in creating an IDS policy are 1. Define the goals of the IDS. 2. Choose what to monitor. 3. Choose the response. 4. Set thresholds. 5. Implement the policy. Defining the Goals of the IDS The goals of the IDS provide the requirements for the IDS policy. Potential goals include ▼ Detection of attacks ■ Prevention of attacks ■ Detection of policy violations ■ Enforcement of use policies ■ Enforcement of connection policies ▲ Collection of evidence Keep in mind that goals can be combined and that the actual goals for any IDS depend on the organization that is deploying it. This is by no means a comprehensive list. The IDS can allow an organization to detect when an attack starts and may allow for the collection of evidence or the prevention of additional damage by terminating the incident. Of course, that is not the only purpose that an IDS can serve. Since the IDS will gather de - tailed information on many events taking place on the network and computer systems of an organization, it can also identify actions that violate policy and the real usage of net - work resources. . extended to include the home computers of employees who are allowed to telecommute or a business partner that is allowed to connect to the network. A burglar alarm is designed to detect any attempted. are two primary types of IDS: host-based (H-IDS) and network-based (N-IDS). An H-IDS resides on a particular host and looks for indications of attacks on that host. An N-IDS resides on a separate