1-9 Semester 8 Internetwork Troubleshooting v1.0 - Lab 2.3.8 Copyright 2001, Cisco Systems, Inc. Lab 2.3.8: DebugOptions SanJose1 SanJose2 #2#1 S0/0 S0/0 192.168.1.10 192.168.2.10 Objective This lab will review using the various debug commands and options to analyze the routers performance. You will find the debug command options useful in the remaining troubleshooting labs. This lab covers: • Debug basics and optional parameters • The service timestamps debug uptime command • Using access control lists to filter debug activity • Console Terminal • Using a Syslog Daemon Warning: The debug command because of its heavy use of CPU cycles can be devastating to a production router’s performance. It is possible that a command such as debug IP packet running during a moderate to heavy traffic period could literally consume all CPU cycles and effectively stop routing resulting in discarded frames. This discussion is included primarily as a tool to help you visualize how and why certain network processes occur. We will also look at options that can reduce the impact of the debug commands. Scenario You have been asked to consult on a small network and offer suggestions on how performance might be improved. After running other processes like the Protocol Inspector you have some questions about why the network is behaving as it is. 2-9 Semester 8 Internetwork Troubleshooting v1.0 - Lab 2.3.8 Copyright 2001, Cisco Systems, Inc. Note: The configuration file used for this lab will be used for other module 2 labs, so please do not change any configuration settings. The configuration contains several components for testing purposes and is not intended to represent a good production configuration. If the lab is done in pairs, each person can run the lab steps albeit each host may display slightly different results. It might be beneficial to coordinate your efforts and compare results. Step 1 Cable the lab as shown in the diagram. Load the configuration files Lab2-3-8-SanJose1Config.txt and Lab2-3-8- SanJose2Config.txt into the appropriate routers. Configure the workstations as follows (same as the last lab): Host #1 Host #2 IP Address: 192.168.1.10 IP Address: 192.168.2.10 Subnet mask: 255.255.255.0 Subnet mask: 255.255.255.0 Default Gateway: 192.168.1.1 Default Gateway: 192.168.2.1 Step 2 The debug ip packet command. This command while a heavy user of CPU resources, documents the IP traffic activity on the router. Using HyperTerminal and a console connection to either router, type the debug ip packet command and watch the output for a couple (60-90) seconds. Don’t worry that it may be scrolling by too fast to read. Type u all (undebug all) to stop the output. The following is a short sample of output from SanJose1: SanJose1#debug ip packet IP packet debugging is on SanJose1# 03:44:33: IP: s=192.168.0.2 (Serial0), d=224.0.0.5, len 68, rcvd 0 03:44:33: IP: s=192.168.0.2 (Serial0), d=224.0.0.10, len 60, rcvd 2 SanJose1# 03:44:34: IP: s=192.168.0.1 (local), d=224.0.0.10 (Serial0), len 60, sending bro ad/multicast 03:44:34: IP: s=192.168.40.1 (local), d=224.0.0.10 (Loopback3), len 60, sending broad/multicast 03:44:34: IP: s=192.168.40.1 (Loopback3), d=224.0.0.10, len 60, rcvd 2 SanJose1# 03:44:36: IP: s=192.168.1.1 (local), d=255.255.255.255 (Ethernet0), len 132, sen ding broad/multicast 03:44:36: IP: s=192.168.4.1 (local), d=255.255.255.255 (Ethernet0), len 132, sen ding broad/multicas 03:44:38: IP: s=192.168.0.2 (Serial0), d=224.0.0.10, len 60, rcvd 2 03:44:38: IP: s=192.168.0.2 (Serial0), d=224.0.0.9, len 112, rcvd 2 03:44:39: IP: s=192.168.40.1 (local), d=224.0.0.10 (Loopback3), len 60, sending broad/multicast 3-9 Semester 8 Internetwork Troubleshooting v1.0 - Lab 2.3.8 Copyright 2001, Cisco Systems, Inc. 03:44:39: IP: s=192.168.40.1 (Loopback3), d=224.0.0.10, len 60, rcvd 2 SanJose1#u all All possible debugging has been turned off SanJose1# Without any additional information we know something about this traffic. The first highlighted entry originated from 192.168.0.2 and arrived through the Serial0 interface – the rcvd near the end tells us it was inbound. The d=224.0.0.5 reveals that it is an OSPF packet multicast to all OSPF routers. Remember that with multicast addresses add addresses from 224.0.0.0 to 224.0.0.255 are reserved. Similarly the third highlighted item is a RIPv2 update. A few multicast reserved addresses that we should remember are: Address Recipient 224.0.0.5 All OSPF Routers 224.0.0.6 All OSPF Designated Routers (DR) 224.0.0.9 All RIPv2 Routers 224.0.0.10 All EIGRP Routers By comparison the second highlighted line was sent using a broadcast (d=255.255.255.255). While we can’t be sure with the limited capture it and the following entry could be RIPv1 activity. Look over the output by scrolling your HyperTerminal output if you can and look for patterns of similar entries that might give you a clue as to their nature. Step 3 The service timestamps debug uptime command. Looking at the output from the last few minutes, assuming the router is using an IOS version 12.0 or greater, some numbers (highlighted below) appear at the beginning of each line of debug output. This is a timestamp that could reflect the hour:minute:second of the output. If the router clock has not been set it is actually the amount of time since the router was last powered up or since a reload command was executed. SanJose1#debug ip packet IP packet debugging is on SanJose1# 03:44:33: IP: s=192.168.0.2 (Serial0), d=224.0.0.5, len 68, rcvd 0 03:44:33: IP: s=192.168.0.2 (Serial0), d=224.0.0.10, len 60, rcvd 2 SanJose1# 03:44:34: IP: s=192.168.0.1 (local), d=224.0.0.10 (Serial0), len 60, sending bro ad/multicast 03:44:34: IP: s=192.168.40.1 (local), d=224.0.0.10 (Loopback3), len 60, sending broad/multicast 03:44:34: IP: s=192.168.40.1 (Loopback3), d=224.0.0.10, len 60, rcvd 2 SanJose1# 03:44:38: IP: s=192.168.0.2 (Serial0), d=224.0.0.10, len 60, rcvd 2 03:44:38: IP: s=192.168.0.2 (Serial0), d=224.0.0.9, len 112, rcvd 2 4-9 Semester 8 Internetwork Troubleshooting v1.0 - Lab 2.3.8 Copyright 2001, Cisco Systems, Inc. or 7w2d: IP: s=192.168.0.2 (Serial0/0), d=224.0.0.9, len 112, rcvd 2 03:44:39: IP: s=192.168.40.1 (local), d=224.0.0.10 (Loopback3), len 60, sending broad/multicast 03:44:39: IP: s=192.168.40.1 (Loopback3), d=224.0.0.10, len 60, rcvd 2 SanJose1#u all All possible debugging has been turned off SanJose1# Notice that several lines of the output wrapped around to the next row (see the highlighted entry). If we look at the length of the timestamp and how much text wraps over, it is clear the timestamp may be contributing to the effect but is not the cause. Type a show run command and look at the first few lines of output. The highlighted row shows the command that causes the time stamp to appear. This is the default in version 12.0 and above. SanJose1#sho run Building configuration . Current configuration: ! version 12.0 service timestamps debug uptime service timestamps log uptime no service password-encryption ! hostname SanJose1 Select the line, as highlighted above, and perform a copy. Press any key except the SPACEBAR or ENTER to break out of the show run command. Go to global configuration mode and type no and then paste your copy to complete the no service timestamps debug uptime. Run the debug ip packet command again for 60-90 seconds. The following is a sample output from SanJose1 SanJose1#conf t SanJose1(config)#no service timestamps debug uptime SanJose1(config)#^Z SanJose1# 03:47:37: %SYS-5-CONFIG_I: Configured from console by console SanJose1#debug ip packet IP packet debugging is on SanJose1# IP: s=192.168.40.1 (local), d=224.0.0.10 (Loopback3), len 60, sending broad/mult icast IP: s=192.168.40.1 (Loopback3), d=224.0.0.10, len 60, rcvd 2 SanJose1# IP: s=192.168.0.2 (Serial0), d=224.0.0.10, len 60, rcvd 2 IP: s=192.168.0.1 (local), d=224.0.0.10 (Serial0), len 60, sending broad/multica st IP: s=192.168.0.2 (Serial0), d=224.0.0.9, len 112, rcvd 2 IP: s=192.168.0.2 (Serial0), d=224.0.0.10, len 60, rcvd 2 IP: s=192.168.0.2 (Serial0), d=224.0.0.5, len 68, rcvd 0 IP: s=192.168.0.1 (local), d=224.0.0.10 (Serial0), len 60, sending broad/multica St SanJose1#u all All possible debugging has been turned off SanJose1# 5-9 Semester 8 Internetwork Troubleshooting v1.0 - Lab 2.3.8 Copyright 2001, Cisco Systems, Inc. Notice that without the timestamps it’s harder to see how frequent the output occurred and the intervals of repetitive activities like routing updates. After comparing the results, turn the feature back on. You should also see that removing the time stamp did not eliminate the line wrapping (see the highlighted lines above). To confirm whether the timestamp is the actual time or elapsed router operation time, the show clock command will confirm your suspicions. The following sample shows that the clock has not been set SanJose1#show clock *01:10:15.199 UTC Mon Mar 1 1993 SanJose1# Take a moment and put the service timestamps debug uptime command back into the configuration. Step 4 The terminal monitor command. Debug output does not display to the screen if you are in a telnet session. To see the debug output use the terminal monitor command. The output should start appearing. Telnet to the other router (192.168.2.1 for SanJose2 / 192.168.1.1 for SanJose1) and try it for yourself. After issuing the debug ip packet command wait a minute to see if output appears – it shouldn’t. Use the terminal monitor command and things should start happening. The following output shows a typical result. SanJose1#telnet 192.168.2.1 Trying 192.168.2.1 . Open User Access Verification Password: SanJose2>en Password: SanJose2#debug ip packet IP packet debugging is on SanJose2#terminal monitor SanJose2# 00:53:05: IP: s=192.168.0.1 (Serial0), d=192.168.2.1, len 42, rcvd 4 00:53:05: IP: s=192.168.2.1 (local), d=192.168.0.1 (Serial0), len 42, sending 00:53:06: IP: s=192.168.0.2 (local), d=224.0.0.5 (Serial0), len 68, sending broa d/multicast 00:53:08: IP: s=192.168.0.2 (local), d=224.0.0.10 (Serial0), len 60, sending bro ad/multicast 00:53:08: IP: s=192.168.80.1 (local), d=224.0.0.10 (Loopback2), len 60, sending broad/multicast 00:53:08: IP: s=192.168.80.1 (Loopback2), d=224.0.0.10, len 60, rcvd 2 SanJose2#u all All possible debugging has been turned off SanJose2# 6-9 Semester 8 Internetwork Troubleshooting v1.0 - Lab 2.3.8 Copyright 2001, Cisco Systems, Inc. 00:53:13: IP: s=192.168.0.2 (local), d=224.0.0.10 (Serial0), len 60, sending bro ad/multicast SanJose2# After you use the u all (undebug all) command you will notice that the output continues for several lines until the current buffers are empty. Step 5 (Optional Syslog lab.) The purpose of a Syslog Daemon (server) is to capture the various “log” messages that programs like the router’s IOS generates. As long as the host with the Syslog software running can be reached from the router or switch, debug, error and log messages can all be sent to it. If you do not already have a copy of Kiwi’s Syslog Daemon (or something comparable) running on one of the lab hosts, consider going out to the Internet and downloading it. The site is: http://www.kiwi- enterprises.com/infor_syslogd.htm. The software is free and runs on Win9X, WinNT, Win2000 and XP. The download is 3+ Mbytes in size. The software is very easy to install. After installing make sure that the Syslog is running. Pressing the control key and the letter “T” at the same time will send a test message that you should be able to read in the Syslog window. The following graphic shows the Syslog with a sample entry. For our example the software is loaded and running on the host attached to SanJose1. So next you need to tell either router that you want to log to the Syslog server. The commands are: SanJose1#conf t SanJose1(config)#logging 192.168.1.10 SanJose1(config)#logging trap debugging SanJose1# The first command told it where, the second told it what you want sent there. Turn on the debug ip packet command for a few minutes. If you look at the Syslog server window you will see output scrolling away. 7-9 Semester 8 Internetwork Troubleshooting v1.0 - Lab 2.3.8 Copyright 2001, Cisco Systems, Inc. Be sure to turn off the debug after you have captured some. You can use View | Clear Display to clear any entries. Choosing View | View Syslog Statistics will bring up the following devices that will display and let you manage some interesting counters. Experiment with the tool until you are comfortable with how it works. If you run the show logging command you can see the trap and where it is sending the logging results. 8-9 Semester 8 Internetwork Troubleshooting v1.0 - Lab 2.3.8 Copyright 2001, Cisco Systems, Inc. SanJose1#show logging Syslog logging: enabled (0 messages dropped, 0 flushes, 0 overruns) Console logging: level debugging, 1187 messages logged Monitor logging: level debugging, 0 messages logged Buffer logging: level debugging, 1187 messages logged Trap logging: level debugging, 106 message lines logged Logging to 192.168.1.10, 80 message lines logged Step 6 Using access lists to limiting debug output The debug ip packet command, which outputs information about IP packets sent and received by the router, is a particularly useful debug tool. Unfortunately as we saw earlier it is very easy to get buried in too much data. Fortunately the command has an option that allows you to create an access list to filter the data being watched. The syntax is debug ip packet [access-list-num]. The access list can be standard or extended, simple or complex. For our purposes we will use a simple example. Assume that a host on the other router is having trouble accessing your router. Both of you feel that the connection should work. It might be useful to know if the ping requests are getting to your router (maybe they can’t find their way back). We could use the debug ip icmp command but we would also like to try a telnet attempt. On your router type the debug ip packets command. Things should start happening soon. Have the host on the other router ping your LAN interface from their DOS / Command window. (SanJose1 would be 192.168.1.1 / SanJose2 would be 192.168.2.1). Can you see the packets in your debug output? They are there but they are buried in routing updates. Turn off debug for a few minutes. Assuming you are using SanJose1, type the following commands: SanJose1#conf t SanJose1(config)#access-list 40 permit 192.168.2.10 SanJose1(config)#exit SanJose1#debug ip packet 40 IP packet debugging is on for access list 40 SanJose1# The ACL allows the host for SanJose2 to get through the debug. If you were doing this on SanJose2 everything could be the same except change the IP address to 192.168.1.10. The ACL 40 is not currently used on either machine. You should notice that there is no debugging displayed. Try to telnet to the same interface from the DOS / Command window. You should get four lines of debug output. 9-9 Semester 8 Internetwork Troubleshooting v1.0 - Lab 2.3.8 Copyright 2001, Cisco Systems, Inc. SanJose1# 01:49:53: IP: s=192.168.2.10 (Serial0), d=192.168.1.1, len 100, rcvd 4 01:49:53: IP: s=192.168.2.10 (Serial0), d=192.168.1.1, len 100, rcvd 4 01:49:53: IP: s=192.168.2.10 (Serial0), d=192.168.1.1, len 100, rcvd 4 01:49:53: IP: s=192.168.2.10 (Serial0), d=192.168.1.1, len 100, rcvd 4 SanJose1# Have the try to telnet to the same interface from their DOS / Command window. You should see it happening. Remember to turn debugging off. The beauty is that if you had wanted to you could have written an extended ACL that would have captured the echo request and the echo reply packets. Or you could use several statements to debug only telnet and http traffic. . 2.3.8: Debug Options SanJose1 SanJose2 #2#1 S0/0 S0/0 192.168.1.10 192.168.2.10 Objective This lab will review using the various debug commands and options. the debug command options useful in the remaining troubleshooting labs. This lab covers: • Debug basics and optional parameters • The service timestamps debug