1 UNIX/Linux Auditing – SANS GIAC LevelOne © 2000, 2001 1 Basic UNIX and Linux Auditing John Green jegreen@crosslink.net Hello, my name is John Green. I consider it a privilege to be able to talk to you about a security issue that is vitally important, and yet often overlooked. That topic is basic UNIX auditing. (Editor’s note: Please note that your mileage may vary. Although we have tried to demonstrate widely available techniques, it is possible that some of the tools shown here may not exist on your particular flavor of UNIX or Linux, or may work slightly differently. - SE) With so many sophisticated security tools on the market, it is easy to understand why administrators are bypassing the basics in favor of these tools and the hype that surrounds them. Intrusion detection systems, firewalls, and other perimeter defense systems promote the ease of single point protection. But what happens when these countermeasures fail? How do we protect our systems and detect intrusions once an intruder has breached our bulkhead? How can we assess the damage that has occurred as a result? The short answer is that we can’t, unless we have taken the time to install and configure some basic auditing tools. Many of these tools come with the base operating system and those that don’t can be downloaded from the Internet for free. On the next slide, titled “Introduction”, I’d like to explain what I mean by basic UNIX auditing and take a look at the areas that get a return on the investment of time taken to properly configure auditing at the host level. 2 UNIX/Linux Auditing – SANS GIAC LevelOne © 2000, 2001 2 Introduction • Auditing is the process of keeping a recorded history of events of interest on a system • UNIX accomplishes this through system processes, log files, and manual labor • Audit trails serve a multitude of purposes – Identifying system errors – Detection of attempted intrusion – Break-in discovery – Damage assessment Every UNIX system provides the capability to record information about various system events. Boot-up messages, user activity, and network connections are only a few of the events that can be recorded in system log files. Auditing is the process of recording this information and, perhaps more importantly, reviewing it for signs of unexpected activity. Since audit trails are useless if no one is looking at them, procedures must be put in place to make this easy to accomplish on a daily basis. Throughout this presentation, I’ll provide tips that can be implemented to ease the burden of audit trail review. Think about the concept of ROI, or return on investment. The proper configuration of system auditing processes only takes a little while, but the payoff can be significant. The information found in log files is extremely valuable, and can be used for a lot of different things. To the system administrator, log files can be a source of information about system errors and misconfigurations that might otherwise have gone unnoticed. To the information system security officer, log files provide a means to detect attempted intrusions and subsequently verify system integrity. If a break-in has occurred, good audit trails help incident handlers assess the extent of the damage that may have occurred. On the next slide, I outline the topics that are covered in this course, and discuss why it is important that they are understood. 3 UNIX/Linux Auditing – SANS GIAC LevelOne © 2000, 2001 3 During This Course, The Student Will: • Learn how to discover suspicious account activity by reviewing log files • Understand syslog and how to configure it for optimal system reporting • Discover the importance of TCP Wrappers as an access control device and how to integrate it with syslog • Be able to find programs that run as root and look for hidden files • Verify and monitor network services and Internet connection In my experience as the leader of a small DoD intrusion detection team, I have found that one of the best intrusion detection sensors is a knowledgeable system administrator who reviews audit trails daily. As an example, a few weeks ago our whole class B address space was scanned on port 21 for ftp. More than a dozen departmental administrators reported that they were seeing unauthorized activity. Even if we had not had a single perimeter IDS in place, we still would have been aware of the scan. This was a great confidence boost! This type of host-based detection is possible because our system administrators take basic auditing seriously. We have built it into the host accreditation process and made the system administrators accountable. During this course, I will explore the basic auditing functions provided by UNIX, as well as some freely available packages that no system should be without. I will demonstrate how to configure and tune various logging functions to provide the maximum benefit to the information security professional and system administrator alike. At the conclusion of this course, the student will be able to discover suspicious user account activity, configure the syslog facility for optimal reporting, and integrate TCP Wrappers with syslog to provide access control and network connection auditing. The student will also be able to identify potentially troublesome files and network connections and understand how to audit these for intrusion detection or incident handling purposes. Please go on to the next slide titled “Auditing User Activity”. 4 UNIX/Linux Auditing – SANS GIAC LevelOne © 2000, 2001 4 Auditing User Activity utmp wtmp btmp lastlog messages The easiest way to get started in auditing is to keep an eye on user account activity. Repeated bad login attempts, or logins at strange times or from unexpected places, are often an indication of an intrusion. Some account activity is logged by default, and some must be “turned on”. This section of the course will focus on where to find these logs, their content, and the commands necessary to examine them. On the next slide titled “UNIX Log Files”, we’ll take a look at some examples and their locations in the UNIX file system. 5 UNIX/Linux Auditing – SANS GIAC LevelOne © 2000, 2001 5 UNIX Logfiles Logfile /var/run/utmp /var/log/wtmp /var/log/btmp /var/log/messages /var/log/secure Contains Current login “snapshot” Login-logout history Bad login history Messages from the syslog facility Access and authentication Log files are the mechanism that UNIX employs to record audit trails. There are five major log files that the administrator should keep a watchful eye on for signs of unauthorized activity. For the most part, these files can be found in the /var/log directory of the Linux filesystem. The notable exception is utmp which can be found in /var/run. It is important to understand the contents of each of these log files and how one can exploit the information to achieve a more secure system. Starting with the next slide titled “The utmp File”, I will discuss the details of each. 6 UNIX/Linux Auditing – SANS GIAC LevelOne © 2000, 2001 6 The utmp File • UNIX keeps a snapshot of users currently logged into the system in /var/run/utmp • Special binary file used by programs – who, users, finger •Contents are ephemeral • For each successful login, utmp contains: – Username, terminal, login time, remote host I used to work in a secure facility. Our employee badges had our identity and other pertinent information encoded on a strip on the back of the badge. People authorized to enter a building could do so by swiping their badges through a badge reader. This would record information and unlock the door. The exit process was exactly the same. At any given time, physical security personnel could call up a list of people currently in the building. The utmp file works in a similar fashion. As users log in, an entry for them is made in the /var/run/utmp file. Upon logout, that entry is removed. The result is a file that contains a current user snapshot at any given point in time. As such, the contents of this file are short lived and always changing. For every successful login, utmp records, among other things, the username, device name, time, and origin. Unfortunately, utmp is a binary file so it cannot be viewed with a text editor or other ascii based tool. However, programs like who, users, and finger read the utmp file and display its contents. On the next slide titled “Examining utmp With who” we will take a look at the contents of a sample utmp file. 7 UNIX/Linux Auditing – SANS GIAC LevelOne © 2000, 2001 7 Examining utmp With who # who idle heading USER LINE LOGIN-TIME IDLE FROM norbert pts/0 Aug 16 12:49 14:48 (:0) norbert pts/1 Aug 16 12:49 . (:0) dagget pts/2 Aug 16 14:48 . (okhost.mycom.com) root pts/3 Aug 17 02:47 . (bad.org) Here, we see the usage of the who command. Several options have been given on the command line which control the output format, including how long a particular login has been idle. As advertised, who shows that users norbert, dagget, and root are all currently logged in. We can see the terminal lines that were used, as well as the actual login time. Notice that the first entry for norbert indicates that the session has been idle for more than 14 hours, but all of the other sessions are currently active. The final column shows the origin of the login. The :0 indicates that the login occurred on the console itself, so we can assume that norbert sits at this computer. Dagget has logged on from okhost.myhost.com, and root is logged in from the remote host bad.org. It doesn’t take a computer scientist to see that something terrible is going on. Someone from an unexpected domain has logged on to our host as root. The initial login occurred at a strange time and the . in the idle column indicates that whoever has assumed root is still actively issuing commands. Admittedly, typing who every now and then can be a pain, assuming that I even remember to do it. So, what I’ve done is put the who command in my .tcshrc file. This way, whenever I open a new window, I see who is logged on without any further action on my part. Utmp doesn’t require any action on the administrator’s part for activation; it runs by default. However, there are some security issues to be aware of. I’ll talk briefly about these on the next slide titled “A Note On Securing utmp”. 8 UNIX/Linux Auditing – SANS GIAC LevelOne © 2000, 2001 8 A Note On Securing utmp • utmp is sometimes distributed “out of the box” as a world-writeable file • Malicious users can corrupt the file or hide themselves • Using the ls -la command, ensure that utmp’s file permissions are set to -rw-r r Sometimes utmp comes installed with world or group write permission by default. As a result, a malicious user can erase or otherwise corrupt the utmp file. The savvy hacker can hide himself or fake entries, diverting attention from himself to others. Knowing this, be slow to react to the contents of utmp. Consider it a piece of the puzzle and treat it appropriately. At configuration time, always check the mode of log files to make sure that only root can write to them. The ls command with the -la arguments can help you here. On the next slide titled “The wtmp File” , we’ll discuss a file that is very similar to utmp in many ways. 9 UNIX/Linux Auditing – SANS GIAC LevelOne © 2000, 2001 9 The wtmp File • UNIX keeps a record of users’ login and logout activity in /var/log/wtmp • Binary file, similar to utmp , used by programs –who, last • Contents are semi-permanent • Like utmp, wtmp also contains the following information for login or logout actions: – Username, terminal, login time, remote host Wtmp can be found in /var/log, and is the same as utmp in terms of file type and format. It records the username, device, event time, and connection origin as a binary file. The major difference in file content lies in the fact that wtmp keeps a history of all logins, logouts, and system events. This provides a formal audit trail of user account access and host booting. Instead of being a user snapshot, wtmp is a running account and system history. This information is critical to intrusion detection and incident investigation efforts. The contents of wtmp can be displayed with the who command, but the last command provides much more information. Administrators can prune the results of the who command to display only the last N events, or to show the account events for a particular user. On the next slide titled “Examining wtmp With last” I will show the last command in action on a wtmp file. 10 UNIX/Linux Auditing – SANS GIAC LevelOne © 2000, 2001 10 Examining wtmp With last [root@myhost log]# last root pts/3 Tue Aug 15 07:15 still logged in bad.org norbert pts/2 Tue Aug 15 07:01 still logged in bad.org norbert pts/1 Tue Aug 14 06:56 still logged in :0 dagget pts/1 Mon Aug 13 10:24 - 10:24 (00:00) ok.com seap pts/0 Thu Aug 12 15:07 - 15:07 (00:00) :0 runlevel (to lvl 5) Thu Aug 12 09:54 - 08:01 (4+22:06) reboot system boot Thu Aug 12 09:54 norbert pts/0 Thu Aug 12 09:17 - crash (00:36) :0 dagget pts/0 Thu Aug 12 09:09 - 09:16 (00:07) ok.com norbert pts/0 Wed Aug 4 11:19 - 09:45 (6+22:25) :0 wtmp begins Mon Aug 2 14:57:03 1999 When using last, the first thing to be aware of is that it displays the contents of the wtmp file in a most- recent to least-recent time order. Said differently, entries at the top of the output occurred most recently. The start date of the file is on the very last line. This wtmp began on August 2nd at almost 3 PM. Scanning left to right, we see the username, the tty, the login and logout time and the host of origin. Elapsed activity time is displayed in parentheses to the right of event times. Also, for those users still logged in, a corresponding message is displayed in place of the logout time. If we examine the output from the bottom up, we can read a history of the activity on the host. The entries look fairly normal until norbert’s entry on August 12th at 9:17. Apparently, while norbert was logged on, the system crashed. For some reason, the system rebooted at 9:54 and reached runlevel 5 less than a minute later. Unexpected reboots should always be investigated. At the top of the output, we see that norbert logged in from the console at 6:56 on August 14th and was still logged in when the last command was run. The frightening lines are the top two. Apparently someone is using norbert’s account from the host bad.org. In this example, the hacker also acquired root 14 minutes later. Could a sniffed password have been the vehicle for a local compromise? As you can see, the wtmp file is a necessity for reconstructing system events in an investigation, but be warned! Wtmp must be activated or no such logs are collected. The next slide titled “Activating The wtmp Logfile” shows what must be done to record this audit trail. [...]... attempts • Backup log files UNIX/ Linux Auditing – SANS GIAC LevelOne © 2000, 2001 15 Whenever I set up a new UNIX system, the first thing that I do is ensure that user account auditing is activated and secured I do this by following these simple steps: First, I check to see if wtmp and btmp were installed by default If not, then I create them using the touch command I use the ls command to check the mode... good backup plan in place, all of the basics have been covered with respect to UNIX log files and the role that they play in auditing On the next slide titled Auditing Account Activity: Step By Step” I will review the procedures that administrators should perform to insure that account auditing is in place 14 Auditing Account Activity: Step By Step • Make sure wtmp and btmp files exist • Run who to look... Configuration file is /etc/syslog.conf UNIX/ Linux Auditing – SANS GIAC LevelOne © 2000, 2001 17 Syslog provides a centralized utility for managing and logging system messages Both system routines and user programs can send messages to the syslog facility for handling Syslog is important because it provides a single utility from which any and all messages can be controlled and distributed This significantly... advanced options • If everything “checks out”, tcpd runs the appropriate server program and goes away UNIX/ Linux Auditing – SANS GIAC LevelOne © 2000, 2001 28 On a UNIX operating system, the Internet daemon, inetd, listens for connections and services them upon arrival It does this by starting the requested server and then handing over the connection for processing TCP Wrapper works by tricking the inetd... ftp://ftp.porcupine.org/pub/security/index.html • Make the executable and install it – Note: leave the real daemons in place • Edit the inetd.conf file to call tcpd • Create hosts.allow and hosts.deny • Check /var/log/secure and /var/log/messages to verify that TCP Wrapper is logging • Validate with tcpdchk and tcpdmatch UNIX/ Linux Auditing – SANS GIAC LevelOne © 2000, 2001 33 If TCP Wrapper didn’t come with your UNIX distribution, it can be... and Internet service auditing in place, it is time to move on to auditing file integrity The next slide titled Auditing Important Files” introduces this concept 33 Auditing Important Files Suid Root Files find lsof UNIX/ Linux Auditing – SANS GIAC LevelOne © 2000, 2001 34 No auditing capability is complete without the ability to track and verify important files Often when hackers successfully exploits... hosts.allow and hosts.deny files 29 Access Control File Format • daemon_list : client_list [ : shell_command ] daemon_list is a list of one or more daemon process names or wildcards client_list is a list of one or more host names, host addresses, patterns or wildcards that will be matched against the client host name or address shell_command is a command to be executed when the rule fires UNIX/ Linux Auditing. .. Wrapper” introduces this tool and talks about some of the features that it provides 26 What Is TCP Wrapper? • Used to monitor incoming requests for Internet services, e.g – ftp, telnet, finger, ssh, etc • Provides access control and syslog logging of service requests • Protects against various forms of spoofing UNIX/ Linux Auditing – SANS GIAC LevelOne © 2000, 2001 27 The basic UNIX kernel provides little... syslogd is started with the -r argument on syslog server and clients UNIX/ Linux Auditing – SANS GIAC LevelOne © 2000, 2001 24 When setting up basic auditing via syslog, the first thing that I do is to identify a host that can be a syslog server for the network in question I make sure that an entry for syslog exists in the /etc/services file, and that the syslogd daemon is started with the -r argument... notice info debug (lowest) UNIX/ Linux Auditing – SANS GIAC LevelOne © 2000, 2001 20 At first glance, selectors can seem a little confusing A selector has two parts, a facility definition and a priority definition joined by a dot The facility is the program that generates the message and the priority is the alert level Every message has a priority Debug is the lowest priority and emerg, or emergency is . 1 UNIX/ Linux Auditing – SANS GIAC LevelOne © 2000, 2001 1 Basic UNIX and Linux Auditing John Green jegreen@crosslink.net Hello,. the next slide titled UNIX Log Files”, we’ll take a look at some examples and their locations in the UNIX file system. 5 UNIX/ Linux Auditing – SANS GIAC