Securing and Auditing Unix Examples tested on a Red Hat Linux 6.1 (Hedwig) build Secure System Administration - SANS GIAC © 2000, 2001 Welcome to Unix and Linux, security for these operating systems is a complete paradigm shift from Windows Unix has been around a lot longer The source code for Linux is freely available, so would be attackers are free to examine it and test it for holes such as buffer overflows and deadlock conditions Linux is different than Unix Has the source code been available for Unix? Certainly, you used to be able to license source for both the ATT and BSD versions of Unix It is rumored the Sun source code was stolen once via a workstation with a modem connection This means that we are dealing with a lot more “knowns” than with Windows Well, at least that was true until October 2000 and critical Microsoft source code was stolen From now on the rules of the game are “who knows the most wins” Let’s start our discussion with the notion of a firm foundation Nothing is certain, but if we can start with a clean build we have a better chance of ending up with a secure system A Clean Build is a Happy Build • Load • Load • Load • Load from CD from Net from another system?????? from tape?????? Secure System Administration - SANS GIAC © 2000, 2001 Windows loads from CDs They are licensed and controlled Though it is possible there could be a compromised version, it is unlikely The initial load of Unix can be a whole new ballgame Many Unix systems make it easy to clone a system from an existing system You plug in your Ethernet cable, power on the system, and the new computer looks for a system to boot from This is great…unless the system you boot from is already compromised The same thing goes for backup tapes This is a big problem in incident handling How you know you are loading from a clean operating system? For Linux, it is possible to load from an Internet site, but in general, it is best to load from the CDROM If you start out compromised, you may never get control So we send a pristine system out into the cold, cruel world If it is compromised, or we suspect it is compromised, what we do? In this section we will learn commands to evaluate Unix systems, but if source code has been available, attackers can replace system commands with theirs We’ll also cover some basic ways you can make it more difficult for the attacker to compromise the system, as well as some tools you can use to aid in detection and recovery Oh yes, Unix and Linux have the equivalent of Service Packs and hotfixes – they are called patches and it is important to keep track of the patch status for these computers as well Plan on needing to an update at least quarterly – more frequently as advisories get issued The Basics • Startup: boot, rc, inetd.conf • Similarities • Differences • Some basic commands Secure System Administration - SANS GIAC © 2000, 2001 Just as we’ve seen with other operating systems, Unix and Linux have an orderly start up sequence Much of what gets the computer going can be found in the “rc” (run command) files in the /etc directory and the inted.conf file (also located in /etc) The inetd.conf file is the Internet Daemon configuration file that specifies which daemons are accessible via the network (such as telnetd and ftpd etc.) Other daemons such as email (smtpd) get started at bootup This is a good time to review some similarities and differences between MS-DOS (as would be used in a DOS command window in Microsoft Windows) and Unix commands and conventions For example, the directories in a DOS path are separated by a “ \ ” (backslash); those in Unix are separated by a “ / ” (forward slash) In DOS, file names are case-INsensitive while in Unix they are case-sensitive The “ ” character is used as a separator between the file name and extension in DOS, but that syntax does not have the same context in Unix (older DOS allows only one “ ”; Unix permits more than one) Both operating systems support the concept of a “pipe” (the vertical bar | ) that can be used to ‘connect’ commands, such as type file.txt | more (DOS) to show the file named file.txt one screen at a time Both operating systems support the use of “ * ” as a wild card character Regarding floppy disks, DOS (and Windows) use a: to refer to the first floppy disk drive in the system while Unix treats everything as a file and uses a name such as /dev/fd0 to refer to the floppy disk In general, Unix command equivalents have a broader functionality than the DOS cousins This is somewhat of an oversimplification, but it will meet our needs for this introductory module You need to be familiar with basic Unix commands to be ready for Security Essentials where you will gain the fundamental skills to enable you to handle an incident involving a Unix system More Basics • List files: ls -lart • Show the file on the screen: cat, more • Display system processes: ps -ef, ps ax, ps -ewf • Display network information: netstat -a • Verify a system file is not corrupt rpm -V filename (no news is good news) Secure System Administration - SANS GIAC © 2000, 2001 Every Unix variant is different and many of the commands listed have multiple options We encourage you to become familiar with the operating systems that are used in your organization A recommended practice is to print the man (manual) pages ahead of time for each OS to create a reference notebook If you are having trouble printing the man pages, you might find some variation of “nroff - man filename | lp” (meaning: new runoff, manual , pipe to line printer) helpful To determine the exact syntax to print the man pages using nroff, type man nroff Use a highlighter for the options you feel may help you meet your needs Sometimes man pages may be a bit cryptic so you may want to test these commands ahead of time and write in examples with the options that work Some of these will be a review for you, but we will go quickly ls for Unix is like dir for Windows, ls -lart lists all files, hidden or not, in time order, with the most recent change on the bottom and is very helpful We can use the cat (for concatenate) command to view the contents of a file on the screen For example cat /etc/passwd will display to standard output (the screen) the contents of the file passwd in the /etc directory The ps (for list process status) command lists active processes: ps without any options lists the processes that are yours The options needed to list ALL processes running varies by operating system; learn what works on yours Commonly ps –ax and ps –ef will work netstat will display network connections, routing tables, interface statistics, masquerade connections, netlink messages, and multicast memberships For Linux, the Redhat Package Manager (RPM) can verify a file has not been modified or compromised with the -v option Still More Basics • Rename or move a file: mv • Copy a file: cp • Looking for text in a “binary” file: strings • Examine the bits: od, od -x • Are these two files the same? diff Secure System Administration - SANS GIAC © 2000, 2001 The tools on this slide help you manipulate and inspect files mv allows you to move a file from one place to another so it is similar to ren (rename) mv a.txt b.txt would rename a.txt to b.txt in the same way ren a.txt b.txt would in a Windows command prompt However, you can move whole directories with mv, change the name or the location copy in Windows is similar to cp in Unix, but there are a number of powerful options One to know is cp -p (for preserve the date) If you want to edit a system file it is a good idea to make a backup file For instance, if you edit the Internet Daemon configuration file, inetd.conf, you might first: cp -p inetd.conf inetd.conf.22OCT00 This will preserve the files date so if you list the /etc directory with ls -lart the original file’s age will be preserved and the edited file will be obvious The strings command will print the displayable ASCII strings of printable characters in files For each file given, strings prints the printable character sequences that are at least characters long and are followed by an unprintable character By default, it only prints the strings from the initialized and loaded sections of object files; for other types of files, it prints the strings from the whole file od (octal dump) dumps (lists) the file in binary mode; od -h and od –x will each dump a file in hex diff displays the differences between two files and is the programmer’s friend Finding things – find & grep • look for suid and sgid files find / -perm +2000 -o -perm +4000 -print • look for files modified today find / -mtime -1 –print • look for files with a pl extension find -name *.pl –print • print all lines in file.txt containing “aaa” grep aaa file.txt Secure System Administration - SANS GIAC © 2000, 2001 The find command is very powerful, but takes some serious practice To get started type “man find” to get a sense of what you can I strongly recommend that you try a few incantations and then write them down exactly so that you have commands that can search for the things you would need The slide above has but three of MANY examples The grep utility enables you to search through a set of files for all lines that match a specified pattern You can use grep as a filter to extract the matching lines from standard input and send those lines to standard output The last example in the slide: grep aaa file.txt will print all lines in file.txt which contain the text “aaa” Other examples include the following: grep -i dude file.txt - Print all lines with “dude” in file.txt, case insensitive grep -v dude file.txt - Print all lines in file.txt without “dude” You now have the basic commands needed to make your way through a Unix system We have chosen commands that are oriented towards auditing and forensics so you can make sure your system is secure Now that you know how to manipulate files, let’s find out where they are What to Look For - File System • what file systems are mounted df -a will list pseudo file systems as well • file system table can provide information about mountable systems cat /etc/fstab, cat /etc/mtab Secure System Administration - SANS GIAC © 2000, 2001 So far we have mostly been concerned with files and files are kept in file systems Linux supports a large number of file systems that can be mounted On this slide we see the command df -a which lists all information df displays the amount of disk space available on the file system containing each file name argument If no file name is given, the space available on all currently mounted file systems is shown To view the contents of the /etc/fstab file we can use the cat command as shown in the slide You can also type: more /etc/fstab and more /etc/mtab to view what the system thinks is mounted and mountable (respectively) The more command causes the output to be displayed one screenful at a time Next we will go on a quick tour of the way the Unix file system is organized This is so you know where you can expect to find certain files There are variations among Unix file systems so you will want to check to see if things are the same for your file system But since we will be following the File System Standard, we should be pretty close to what you have What to Look For - FSSTND • The following directories are common from /, /bin, /boot, /dev, /etc, /home, /lib, /lost+found, /mnt, /proc, /sbin, /tmp, /usr, /var • /, /usr, /usr/sbin, /bin, /mnt, /etc Secure System Administration - SANS GIAC © 2000, 2001 This is the file system according to the File System Standard FSSTND and gives some insight into a typical file system layout We can investigate further using ls The “ls” and “ls -lart” commands are safe and will not break anything Start with: ls / This will list the files in the top or root directory of your Unix system Now look again with ls -l The command cd /usr (change directory) will take you to the /usr file system This is where unchanging files are kept, including commands, documentation, and libraries Of special interest is /usr/sbin This contains system administrator commands Many may be unfamiliar, but one to know is list open files, lsof Where are the commands that we have been learning? The command whereis mv will give you the location of the mv command and it is in /bin cd /bin and take a look The /bin directory contains many of the commands we have been learning There are three more places we need to be familiar with in order to know our Unix system: ls -l /mnt, will show the files in the mount directory My system has a mount point for floppy, CD-ROM, and Zip These are the removable media I use When I put the removable media in the system, I have to mount it As a convenience, I can list and copy files between Linux and Windows with mcopy a: and mdir a: and avoid having to use the mount command ls -l /etc will show the location of the configuration files for this system more /etc/inetd.conf is a safe way to see what network daemons are started at bootup If a line in the file begins with a # symbol, the daemon(s) listed on that line are commented out and will not be run at startup You can view the files that end with conf and cf in general /tmp is a directory for temporary files, and since anyone can write there, it is a target for attackers Most systems are configured to delete all the files in /tmp at bootup If you create a directory in /tmp most systems will not delete it Auditing • History files • Several Unix systems such as Solaris have additional logging that can be turned on • Host based intrusion detection system vendors may have logging agents available For either of these to be successful they would have to be implemented well before the incident Secure System Administration - SANS GIAC © 2000, 2001 The history command performs one of several operations related to recently-executed commands recorded in a history list Each of these recorded commands is referred to as an ``event'' It is amazing how many attackers have left the history files intact A shell with history should always be the default for the user root Try the following commands to see if you have a history, (this is pronounced “dot history”) cd ls -la history The user root sometimes has a history file Try: cd / (You may need to type cd /root) ls -la history Some Unix systems have additional auditing and logging features that can be enabled Note that every choice has a cost - turning on a lot of auditing can eat up disk space and cause degradation in system performance On the other hand, a root level compromise may not be a lot of fun to try to recover from with no indication as to what happened In addition, there are host based intrusion detection systems that can record information as well As the slide says, these features must be implemented before the incident to be effective More Auditing • • • • syslog messages dmesg application logs Secure System Administration - SANS GIAC © 2000, 2001 10 Unix and Linux operating systems come with a built-in logging system The syslog daemon gets started in the “rc” files at bootup The events are recorded in the syslog file written to the loghost specified in the /etc/hosts file It’s easy to send the information on one computer to another by specifying that other server as the loghost It’s not completely secure, but it does get the information recorded in another location – very useful in the event of suspicious activity or compromise Another location that contains useful clues is the messages file in /var/adm/messages This file rotates through (typically) six versions on a weekly basis (a new file each week) before overwriting the earliest version The messages file keeps track of what is happening (primarily at the system hardware level) and records useful information such as bus errors and other data of a similar nature Another useful auditing tool is the information available from dmesg It begins collecting information on startup and keeps track of events until the computer is up and running The program helps users to print out their bootup messages Instead of copying the messages by hand, the user need only type: dmesg > boot.messages and mail the boot.messages file to whoever can debug their problem Finally, we have application logs to help sort out what is happening If we are running a web server, the logs from that application can be configured to will tell us which computers have accessed which files in the web tree 10 Bring Your Own Tools • Mounting a floppy drive – You can have common binaries as discussed in this section on the floppy # mount /dev/fd0 -t vfat /mnt/floppy • Set path or explicitly execute all binaries # $PATH=/mnt/floppy:$PATH # set (to check) Secure System Administration - SANS GIAC © 2000, 2001 11 Because attackers have access to source code, they can easily build tools that act like the real tools, but have slight differences that help the attackers This means we need to keep a copy of known good binaries on removable media I prefer to burn CD-ROMs Burners are cheap and available and CDs are less than a dollar In the example below, we mount a floppy disk with a set of binaries Then we look at them with the ls command Next we execute them to ensure they are working See the notes for details of the commands we’ll need [root@sleuth /root]# mount /dev/fd0 -t vfat /mnt/floppy [root@sleuth /root]# ls /mnt/floppy cp egrep kill ls mount mv netstat [root@sleuth /root]# cd /mnt/floppy [root@sleuth floppy]# /ls cp egrep kill ls mount mv netstat ps ps For CDs try /mnt/cdrom! 11 Tools and Commands Summary • ls, ls -a, ls lart • mount, fstab • mv, cp, cp -p • df, fdisk • rm • • • • strings od diff grep, grep -v, -i • find Secure System Administration - SANS GIAC © 2000, 2001 12 This slide contains a summary of the Unix commands that we have learned in this section and also during the course Please take time to make sure you are familiar with how each of these works As a closing note on Unix, as long as you are not superuser or root, most of the tools are quite safe I would stay away from “rm -rf” which is a recursive delete of all files you have authority to write to and would be pretty destructive in your home directory Other than that, go for it! The notes portion has a list of some of the common commands used in Unix and the similar functionality provided in a DOS window under Microsoft Windows DOS Unix Command Show files in a directory dir ls Copy a file copy cp Rename a file ren mv Move a file cp,del mv Partition a hard disk fdisk fdisk Look for text in a file find grep Identify differences between two files fc diff Find strings of text in a file find strings Delete a file del rm Display a file to the screen (standard output) type cat Display information one screenful at a time more more Well, we have learned a number of fundamental commands and have introduced the Unix file system layout As always, we hope you will take some time to really experiment with these Everything you have learned during this course will be built on as you progress in your GIAC training Take the time and build a strong foundation, it is a good investment This concludes our time together, I hope you have learned a lot and are ready to go give things a try! Thank you 12 Course Revision History Secure System Administration - SANS GIAC © 2000, 2001 13 v1.5 – S Northcutt – 25 Jun 2000 v1.6 – edited by C Wendt – Jul 2000 v1.7 – edited by J Kolde – 27 Jul 2000 v1.8 – edited by S Northcutt – 28 Jul 2000 v1.9 – updated by F Kerby - 27 October 2000 v2.0 – edited by S Northcutt and retaped 29 October 2000 v2.1 – edited by J Kolde, format grayscale for b/w printing – 23 Nov 2000 13 ... some tools you can use to aid in detection and recovery Oh yes, Unix and Linux have the equivalent of Service Packs and hotfixes – they are called patches and it is important to keep track of the... command window in Microsoft Windows) and Unix commands and conventions For example, the directories in a DOS path are separated by a “ \ ” (backslash); those in Unix are separated by a “ / ” (forward... without “dude” You now have the basic commands needed to make your way through a Unix system We have chosen commands that are oriented towards auditing and forensics so you can make sure your system