Forensic Analysis Tools 359 4072 WCESMgr -> 999 TCP C:\Program Files\Microsoft ActiveSync\WCESMgr.exe 1032 svchost -> 1025 TCP C:\WINDOWS\System32\svchost.exe 1032 svchost -> 1031 TCP C:\WINDOWS\System32\svchost.exe 1032 svchost -> 1034 TCP C:\WINDOWS\System32\svchost.exe 4 System -> 1042 TCP 4072 WCESMgr -> 2406 TCP C:\Program Files\Microsoft ActiveSync\WCESMgr.exe 2384 websearch -> 3008 TCP C:\Program Files\websearch\ websearch.exe 1144 -> 54321 TCP C:\Temp\cmd.exe 4072 WCESMgr -> 5678 TCP C:\Program Files\Microsoft ActiveSync\WCESMgr.exe 2384 websearch -> 8755 TCP C:\Program Files\websearch\ websearch.exe 136 javaw -> 8765 TCP C:\WINDOWS\System32\javaw.exe 1348 WCESCOMM -> 123 UDP C:\Program Files\Microsoft ActiveSync\WCESCOMM.EXE 2384 websearch -> 123 UDP C:\Program Files\websearch\ websearch.exe 940 svchost -> 135 UDP C:\WINDOWS\system32\svchost.exe 1144 -> 137 UDP 1032 svchost -> 1026 UDP C:\WINDOWS\System32\svchost.exe By looking at this listing, you can see what appear to be normal services and programs running, until about half way down where you can see that cmd.exe is running from the temp directory. This is the command prompt binary and it has no business being in a temp directory. Also, the fact that the service has no name should arouse suspicion. Finally, the incoming port number doesn’t match any known services. In fact, if you look it up in a database of known Trojan horses on the Internet (www.simovits.com/trojans/ Howlett_CH11.fm Page 359 Friday, June 25, 2004 12:33 AM 360 Chapter 11 • Forensic Tools trojans.html), it matches the port number of a documented Trojan horse. There is strong evidence that this system has been exploited. At this point, you have to decide if it is worth taking the system down to do further forensic analysis of the system. Table 11.1 lists a few options you can run with Fport to sort the output. You can also use the –h option to display short help descriptions. If you have a lot of processes, you can use these switches to look at all the high port numbers running, which is typically where malware runs. You can also sort by application path or name to find nonstandard applications running. This tool is similar to the Fport tool for Windows just discussed. The lsof tool (LiSt Open Files) associates open files with processes and users. It is like the netstat command, but in addition it reports the network port the service is using. This is important when try- ing to track down an active program on the network. Often the only way to find these elusive bugs is to watch for what network ports they open up. The lsof tool is being preinstalled on some UNIX and Linux distributions and is avail- able in RPM form on the installation disks of others such as Mandrake and RedHat Linux. To see if you have it preinstalled, type lsof and see if you get any response. Table 11.1 Fport Sorting Options Options Descriptions -a Sorts the output by application name. -ap Sorts the output by application path. -i Sorts the output by Process ID (PID). -p Sorts the output by port. lsof: A Port and Process Identification Tool for UNIX lsof Author/primary contact: Ray Shaw Web site http://freshmeat.net/projects/lsof/ Platforms: Linux and most UNIX License: GPL Version reviewed: 4.68 Mirror sites (these allow anonymous FTP without reverse DNS): thewiretapped.net/pub/security/host-security/lsof ftp.tau.ac.il/pub/unix/admin/ Howlett_CH11.fm Page 360 Friday, June 25, 2004 12:33 AM Forensic Analysis Tools 361 Installing lsof 1. Download the tar file from the book’s CD-ROM or the official Web site. If the IP address you are downloading from doesn’t have a reverse DNS record, the main FTP site will not allow you to connect to it. Try one of the alternate mirror sites listed. 2. Unzip the tar file. 3. You will see some text files and another tar file, something like lsof_4.68_src. This file has the sources in it. Untar this file and enter that directory. 4. Before you start the compilation process, you need to know the abbreviation code for your UNIX dialect. Since the lsof program is designed to be portable to just about any version of UNIX, you must tell it what flavor of UNIX you are running so the configure routine can set it up for your system. To find out the codes for the different versions of UNIX, type ./configure –h For example, the code for Linux is linux (easy enough, right?). 5. When you are ready, type the following command: ./Configure unix_dialect_code where you replace unix_dialect_code with the code for your specific system, for example, linux . This configures the program for compilation. 6. When the configuration is finished, type: make 7. This finishes the build process. You are now ready to start using lsof. Using lsof The lsof program has many uses, and has extensive man pages and several README files for the different applications. However, this section concentrates only on a few specific commands that are useful for forensic research. If you want to see all of the open files on your system at any given moment and the processes associated with them, type: lsof -n The -n option tells lsof not to attempt to do a DNS record on any IP addresses con- necting to your machine. This speeds up the process considerably. The output will look something like Listing 11.2 Howlett_CH11.fm Page 361 Friday, June 25, 2004 12:33 AM 362 Chapter 11 • Forensic Tools Listing 11.2 lsof –n output COMMAND PID USER FD TYPE DEVICE SIZE NODE xfs 903 xfs 0r DIR 3,1 4096 2 atd 918 daemon rtd DIR 3,1 4096 2 atd 918 daemon txt REG 3,6 14384 273243 /usr/sbin/atd sshd 962 root cwd DIR 3,1 4096 2 sshd 962 root rtd DIR 3,1 4096 2 sshd 962 root txt REG 3,6 331032 274118 /usr/sbin/sshd dhcpcd 971 root cwd DIR 3,1 4096 2 dhcpcd 971 root rtd DIR 3,1 4096 2 dhcpcd 971 root txt REG 3,1 31576 78314 /sbin/dhcpcd xinetd 1007 root cwd DIR 3,1 4096 2 5u IPv4 1723 TCP 127.0.0.1:1024 (LISTEN) xinetd 1007 root 8u unix 0xc37a8540 1716 rwhod 1028 root cwd DIR 3,1 4096 61671 /var/spool/rwho rwhod 1028 root rtd DIR 3,1 4096 61671 /var/spool/rwho rwhod 1028 tim cwd DIR 3,1 4096 61671 /var/spool/rwho crond 1112 root cwd DIR 3,1 4096 14 /var/spool crond 1112 root 1w FIFO 0,5 1826 1112 root 2w FIFO 0,5 1827 pipe nessusd 1166 root cwd DIR 3,1 4096 2 nessusd 1166 root rtd DIR 3,1 4096 2 nessusd 1166 root txt REG 3,6 1424003 323952 init 1 root cwd DIR 3,1 4096 2 init 1 root rtd DIR 3,1 4096 2 init 1 root txt REG 3,1 31384 75197 The connections in this listing look normal. The connection via the rwho service might give you pause. You would want to make sure that a valid user on your system is using this command legitimately. If this account belonged to a nontechnical secretary type, you might want to investigate this further. You can also use lsof to look for a specific file. If you want to see if anyone was accessing your password file, you could use the following command: lsof path/filename Replace path/filename with the specific path and filename you are interested in, in this case, /etc/passwd. You have to give lsof the whole path for it to find the file. Howlett_CH11.fm Page 362 Friday, June 25, 2004 12:33 AM Reviewing Log Files 363 Another way to use lsof is to have it list all the open socket files. This shows if there is a server listening that you don’t know about. The format of this command is: lsof –i This produces output similar to Listing 11.3. You can see all the programs you are running, including sshd and nessusd, which are the daemons for Nessus and SSH. You can even see the individual connections to these services. It looks like someone is using the Nessus server at the moment. Checking the IP address, you can see that it is an internal user. In fact, it is your own machine! So there is nothing to worry about this time. Listing 11.3 lsof –i Output COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME portmap 733 rpc 3u IPv4 1417 UDP *:sunrpc portmap 733 rpc 4u IPv4 1426 TCP *:sunrpc (LISTEN) sshd 962 root 3u IPv4 1703 TCP *:ssh (LISTEN) xinetd 1007 root 5u IPv4 1728 TCP localhost.localdomain:1024 (LISTEN) rwhod 1028 root 3u IPv4 1747 UDP *:who nessusd 1166 root 4u IPv4 1971 TCP *:1241 (LISTEN) nessusd 1564 root 5u IPv4 1972 TCP 192.168.1.101:1241- >192.168.1.2:1994 (ESTABLISHED) You can specify a particular IP address or host to look for by putting an @ (at sign) and the address after the -i switch. For example: lsof -i@192.168.1.0/24 shows any connections coming from within your network, assuming your internal network is 192.168.1.0/24. Reviewing Log Files You should also peruse your log files when you are looking for signs of trouble. The Win- dows log files can be found under Event Viewer in Administrative Tools. Under Linux and BSD-variant UNIX, log files are found in the /var/log/ directory. Other UNIX variants may have these files also though their location may be different. Table 11.2 lists the major Unix log files and their functions. These files may be located in a slightly different location or may not exist on other versions of UNIX. Also, programs often create their own log files, which may be kept in the /var directory. You can use a text editor to view these files and search for certain text strings or number (such as IP addresses and user names). Table 11.3 lists several operating system-level commands you can use on Linux and UNIX systems to scan these files quickly. Howlett_CH11.fm Page 363 Friday, June 25, 2004 12:33 AM 364 Chapter 11 • Forensic Tools Listing 11.4 Output from the last command tony pts/0 10.1.1.1 Sun Sep 5 23:06 still logged in tony pts/0 10.1.1.1 Sun Sep 5 22:44 - 23:04 (00:20) tony pts/0 10.1.1.1 Sun Sep 5 21:08 - 21:16 (00:07) tony pts/0 10.1.1.1 Sun Sep 5 20:20 - 20:36 (00:16) reboot system boot 2.4.18-14 Sun Sep 5 17:32 (05:34) Table 11.2 UNIX Log Files Log Files Descriptions /var/log/messages Stores general system messages. /var/log/secure Stores authentication and security messages. /var/log/wtmp Stores a history of past logins and logouts. /var/run/utmp Stores a dynamic list of who is currently logged in. /var/log/btmp For Linux only. Stores any failed or bad logins. Table 11.3 Linux and UNIX Scanning Commands Commands Descriptions users Shows the users currently on the system from the utmp file. w Shows users on the system with details such as where they logged in from (local or remote), IP address if they logged in remotely, and what commands they are executing. This command is highly useful for catching intruders in the act. last Shows the most recent contents of the wtmp file. This can also be quite useful in seeing who is logging onto your system, at what hours, and for how long. Listing 11.4 shows an example of this output. lastb For Linux only. This does the same thing as last but for btmp, the bad login log file. This can be the first place an intruder shows up with multiple failed login attempts. Howlett_CH11.fm Page 364 Friday, June 25, 2004 12:33 AM Making Copies of Forensic Evidence 365 tony tty1 Sun Sep 5 17:29 - down(00:01) tony pts/2 10.1.1.1 Sat Sep 4 23:02 - 23:34 (00:32) tony pts/2 10.1.1.1 Sat Sep 4 22:36 - 22:36 (00:00) hank pts/0 10.1.1.200 Sat Sep 4 12:13 - 12:22 (00:08) hank pts/0 adsl-66-141-23-1 Fri Sep 3 23:53 - 23:53(00:00) hank pts/0 192.168.1.100 Fri Sep 3 14:47 - 14:47(00:00) tony pts/3 192.168.1.139 Fri Sep 3 09:59 - down (00:01) larry pts/3 adsl-65-67-132-2 Thu Sep 2 22:59 - 23:11(00:12) tony pts/3 10.1.1.1 Thu Sep 2 21:33 - 21:49 (00:16) brian pts/3 adsl-65-68-90-12 Thu Sep 2 18:23 - 18:31(00:07) hank pts/5 192.168.1.139 Thu Sep 2 14:29 - 15:35 (01:06) sam pts/ dialup-207-218-2 Wed Sep 1 22:24 - 00:40(02:16) Keep in mind that if your system has been compromised, these programs may have been replaced with trojanized copies. A program like Tripwire (see Chapter 7) can help you determine if your system binaries have been tampered with. You should make known good copies of these binary files so you can execute from secure boot media instead of using the ones on the system. Also, remember that attackers will often selectively edit your log files to remove any trace of their actions. However, if they simply delete the log file, you may be able to recover it. You should also check all the log files as some neo- phytes only delete some of them. Making Copies of Forensic Evidence If you have verified that your system has been attacked or exploited, the first thing to do is take immediate action to stop the attack or limit that machine’s exposure. Ideally, this would mean disconnecting the machine from the network to conduct further analysis. If this is not possible, you will still want to disable any suspect accounts, kill any rogue pro- cesses, and possibly block offending IP addresses at the firewall while you figure out what is going on. Once you have eliminated the immediate danger, you should make a copy of any important data to look at offline per the tenet of good forensic analysis described earlier. You don’t want to use your tools on live data. To do this, make a perfect copy of the data. This requires creating an image of the data rather than just copying it. You don’t want to use the operating system’s built-in copy functions because this might change file dates and insert other unwanted information. There are special tools for making these mirror-image copies. Unfortunately, there are not any good open source alternatives for the Windows platform right now (anyone want to sign up for a good Windows open source project?). The most popular program for Windows is Norton Ghost by Symantec, which retails for about $50.00. Under UNIX, there is an excellent open source program for doing this: dd, which stands for data dump. Howlett_CH11.fm Page 365 Friday, June 25, 2004 1:05 PM 366 Chapter 11 • Forensic Tools You can use the dd tool to literally read blocks of data right off the hard disk and make exact copies of it. It goes directly to the media rather than using the file system, so it can capture deleted data and other things that a file system can’t see. It can be used to make bit-wise copies of your data on a UNIX file system. Because UNIX treats devices as files, you can take a whole hard drive and replicate it this way by simply copying the device file with a tool like dd. Installing dd You shouldn’t have to install dd on most UNIX operating systems because it is a part of any UNIX file system. Type man dd to verify that you have it. If for some reason you don’t have it, you can get it from the book’s CD-ROM or as part of the GNU file utilities at the site above. Using dd There are two ways to use dd. One way is to make a bit-wise copy, that is, copy the data bit by bit. This creates a mirror image of the data on another hard disk or partition. The other way is to create a single large file. This is sometimes convenient for analysis and portability purposes. You can easily make a hash of the file for verification purposes. This file is often referred to as an evidence file , and many forensic programs are designed to use these files as input. The basic format of the dd command is as follows: dd –if= input_file –of= output_file options where you replace input_file with the device file you want to copy, output_file with the filename you want to copy it to, and options with any dd options you want to use. The dd tool has many options, and Table 11.4 lists the basic ones. So, if you want to copy the hard drive device /dev/hdc onto another hard drive, device hdd using dd, you could use the following command: dd –if=dev/hdc of=/dev/hdd bs=1024 conv=noerror,notrunc,sync dd: A Disk and File Replication T ool dd Authors/primary contacts: Paul Rubin, David MacKenzie, and Stuart Kem Web site: http://mirrors.kernel.org/gnu/fileutils/ Platforms: Most Linux and UNIX License: GPL Version reviewed: N/A Other resources: Type man dd at the command prompt. Howlett_CH11.fm Page 366 Friday, June 25, 2004 12:33 AM Making Copies of Forensic Evidence 367 This copies the contents of the device at /dev/hdc (probably your primary hard drive) to the device at /dev/hdd (probably your secondary hard drive). Make sure you understand which drives relate to which devices. As the sidebar on dd explains, a mistake here can be very costly! Flamey the Tech Tip: Be Very Careful with dd! Do not use a low-level disk tool like dd lightly. One wrong command could easily erase your whole hard drive. Be particularly careful about the input and the output sources. Getting them mixed up can mean overwriting your evidence—or worse. Don’t play with dd unless you have at least a basic understanding of hard-disk terms like blocks and sectors. Unlike user-friendly Windows, dd won’t prompt you twice if you are about to do something stupid. So, like a good carpenter, read the manual twice . . . execute once. . . . If you want to create a single big evidence file instead, you can use the following command to copy the file onto a new device. dd if=/dev/hdc of=/mnt/storage/evidence.bin Table 11.4 Basic dd Options Options Descriptions bs= Block size. The size of the blocks, in bytes, to copy at a time. count= Block count. How many blocks to copy. This is useful if you don’t want to copy the whole file system if you have a very large hard drive or partition or limited space on your target media. skip= Skip x number of blocks before starting the copy. Again, this is useful for copying only a part of a file system. conv= Specifies any of several suboptions: notrunc—Won’t truncate the output if an error occurs. This is recommended in most cases. noerror—Won’t stop reading the input file in case of an error such as prob- lems with the physical media. Also recommended. sync—Requires the noerror command before it. If an error occurs, this will place zeros in its place, maintaining the sequential continuity of the data. Howlett_CH11.fm Page 367 Friday, June 25, 2004 12:33 AM 368 Chapter 11 • Forensic Tools You will probably want to mount a new device to capture this file. It should preferably be brand new media so as not to taint the evidence with old data. Remember, even deleted data will show up with these tools. If you can’t use fresh media, make sure it is truly wiped clean with a disk utility. The dd tool has this capability. Read the man pages for more information on this option. When you have all your evidence gathered, you are ready to analyze it further with a forensic toolkit. There are many excellent, professional-grade commercial toolkits. There are also some very good free toolkits available both for Windows and UNIX. The Sleuth Kit by Brian Carrier is a compilation of various forensic tools that run under UNIX. It includes parts of the popular Coroner’s Toolkit by Dan Farmer as well as other contributions, and works with the Autopsy Forensic Browser, which is a nifty Web interface for Sleuth Kit. It is designed to work with data files such as those output by disk utilities like dd. It is quite feature rich; in fact, it has more depth than some of the commer- cial programs available. Some of the key functions are: The Sleuth Kit/Autopsy Forensic Browser: A Collection of Forensic Tools for UNIX The Sleuth Kit and Autopsy Forensic Browser Author/primary contact: Brian Carrier Web site: www.sleuthkit.org/sleuthkit/index.php Platforms: Most UNIX License: IBM Public License Version reviewed: 1.70 Mailing lists: The Sleuth Kit User’s list General questions and discussion on Sleuth Kit. Subscribe at http://lists.sourceforge.net/lists/listinfo/sleuthkit-users The Sleuth Kit Informer list A monthly newsletter with news and tricks and tips. Subscribe at www.sleuthkit.org/informer/index.php. The Sleuth Kit Developer’s list For developers’ questions and discussion. Subscribe at: http://lists.source- forge.net/lists/listinfo/sleuthkit-developers. The Sleuth Kit Announcement list A read-only list with major announcements or releases of Sleuth Kit and Autopsy Forensic Browser. Subscribe at http://lists.sourceforge.net/lists/listinfo/sleuthkit-announce. The Coroner’s Toolkit (TCT) list Information on TCT, which Sleuth Kit is based on. Subscribe at www.porcupine.org/forensics/tct.html#mailing_list. Howlett_CH11.fm Page 368 Friday, June 25, 2004 12:33 AM . 2 2 2:5 9 - 2 3:1 1(0 0:1 2) tony pts/3 10.1.1.1 Thu Sep 2 2 1:3 3 - 2 1:4 9 (0 0:1 6) brian pts/3 adsl-65-68-90-12 Thu Sep 2 1 8:2 3 - 1 8:3 1(0 0:0 7) hank pts/5 192.168.1. 139 Thu Sep 2 1 4:2 9 - 1 5:3 5 (0 1:0 6) sam. 1 2:1 3 - 1 2:2 2 (0 0:0 8) hank pts/0 adsl-66-141-23-1 Fri Sep 3 2 3:5 3 - 2 3:5 3(0 0:0 0) hank pts/0 192.168.1.100 Fri Sep 3 1 4:4 7 - 1 4:4 7(0 0:0 0) tony pts/3 192.168.1. 139 Fri Sep 3 0 9:5 9 - down (0 0:0 1) larry. Sun Sep 5 2 3:0 6 still logged in tony pts/0 10.1.1.1 Sun Sep 5 2 2:4 4 - 2 3:0 4 (0 0:2 0) tony pts/0 10.1.1.1 Sun Sep 5 2 1:0 8 - 2 1:1 6 (0 0:0 7) tony pts/0 10.1.1.1 Sun Sep 5 2 0:2 0 - 2 0:3 6 (0 0:1 6) reboot