Troubleshooting Router Issues Using the show interface and show interfaces Commands 799 The first parameter (Serial0 is up) refers to the hardware layer and essentially reflects whether the interface is receiving the carrier detect signal from the other end (DCE). If the line is down, a problem with the cabling might exist in a back-to-back connection, with one end being “administratively down.” If the interface is administratively down, it has been disabled manually in the configuration. The show interfaces serial command also provides information to help diagnose other Layer 1 issues that are not as easy to determine. Example 18-3 show interface serial Command Cougars# show interface serial 0 Serial0 is up, line protocol is up Hardware is HD64570 MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation HDLC, loopback not set Keepalive set (10 sec) Last input never, output never, output hang never Last clearing of "show interface" counters 00:02:57 Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: weighted fair Output queue: 0/1000/64/0 (size/max total/threshold/drops) Conversations 0/0/256 (active/max active/max total) Reserved Conversations 0/0 (allocated/max allocated) Available Bandwidth 1158 kilobits/sec 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec 0 packets input, 0 bytes, 0 no buffer Received 0 broadcasts, 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 0 packets output, 0 bytes, 0 underruns 0 output errors, 0 collisions, 8 interface resets 0 output buffer failures, 0 output buffers swapped out 0 carrier transitions DCD=up DSR=up DTR=up RTS=up CTS=up chpt_18.fm Page 799 Tuesday, May 27, 2003 2:19 PM 800 Chapter 18: Basic Router Troubleshooting The following problems can be caused by an increasing number of carrier transitions counts on a serial link: ■ Line interruptions from the service provider network ■ Faulty switch, DSU, or router hardware If an increasing number of input errors appear in the show interfaces serial output, sev- eral possible sources of those errors exist. Some common problems related to Layer 1 are as follows: ■ Faulty telephone company equipment ■ Noisy serial line ■ Incorrect cable or cable length ■ Damaged cable or connection ■ Defective CSU or DSU ■ Defective router hardware Another area to examine is the number of interface resets. Interface resets are the result of too many missed keepalives. Layer 1 problems also could be caused by the following: ■ Bad phone line causing CD transitions ■ Possible hardware problem at the CSU, DSU, or switch The number of errors should be interpreted relative to the amount of traffic that the router has processed and the length of time that the statistics have been captured. The router tracks statistics that provide information about the interface. The statistics reflect router operation since it was started or since the last time the counters were cleared, as shown in Example 18-4. Example 18-4 Statistics Accumulation Cougars# show interface serial 0 Serial0 is up, line protocol is up Hardware is HD64570 MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation HDLC, loopback not set Keepalive set (10 sec) Last input never, output never, output hang never Last clearing of "show interface" counters 00:02:57 chpt_18.fm Page 800 Tuesday, May 27, 2003 2:19 PM Troubleshooting Routing Issues Using the show cdp neighbors Command 801 If the show interfaces output shows the last clearing of the counters as never, use the show version command to determine how long the router has been functional, as shown in Example 18-5. Use the clear counters command, as shown in Example 18-6, to reset the counters to 0. These counters always should be cleared after an interface problem has been corrected. Starting from 0 gives a better picture of the current status of the network and helps verify that the issue has been corrected. Troubleshooting Routing Issues Using the show cdp neighbors Command Cisco Discovery Protocol (CDP) advertises device information to its direct neighbors, including MAC and IP addresses and outgoing interfaces. Example 18-5 Router Uptime Cougars# show version Cisco Internetwork Operating System Software IOS (tm) 2600 Software (C2600-BNSY-L), Version 12.2(6h), RELEASE SOFTWARE (fc1) Copyright (c) 1986-2002 by cisco Systems, Inc. Compiled Mon 26-Aug-02 23:23 by kellythw Image text-base: 0x0303ED8C, data-base: 0x00001000 ROM: System Bootstrap, Version 11.0(10c), SOFTWARE BOOTLDR: 3000 Bootstrap Software (IGS-BOOT-R), Version 11.0(10c), RELEASE SOFTWA RE (fc1) Cougars uptime is 14 minutes Example 18-6 clear counters Command Cougars# clear counters Clear "show interface" counters on all interfaces [confirm]yes Cougars# 00:17:24: %CLEAR-5-COUNTERS: Clear counter on all interfaces by console Cougars# chpt_18.fm Page 801 Tuesday, May 27, 2003 2:19 PM 802 Chapter 18: Basic Router Troubleshooting The output from the show cdp neighbors command displays information about directly connected neighbors, as demonstrated in Example 18-7. This information is useful for debugging connectivity issues. If a cabling problem is suspected, enable the interfaces with the no shutdown command and then execute the show cdp neighbor detail command, as shown in Example 18-8, before any other con- figuration. The command displays specific device detail such as the active interfaces, the port ID, and the device. If the physical layer is functioning properly, all other directly connected Cisco devices should be displayed. If a known device does not show up, there is probably a Layer 1 issue. One area of concern with CDP is security. The amount of information that CDP pro- vides is so extensive that it can be a potential security risk. For security reasons, CDP should be configured only on links between Cisco devices and should be disabled on user ports or links that are not managed locally. Example 18-7 show cdp neighbors Command Output routerA# show cdp neighbors Capability Codes: R – Router, T – Trans Bridge, B – Source Route Bridge, S – Switch, H – Host, I – IGMP Device ID Local Interface Holdtime Capability Platform Port ID routerB Eth 0 151 R 2501 Eth 0 routerB Ser 0 165 R 2501 Ser 0 Example 18-8 show cdp neighbors detail Command routerA# show cdp neighbors detail Device ID: routerB Entry address(es): IP address: 198.92.68.18 Platform: 2501, Capabilities: Router Interface: Ethernet0, Port ID (outgoing port): Ethernet0 Holdtime: 143 sec chpt_18.fm Page 802 Tuesday, May 27, 2003 2:19 PM Troubleshooting Routing Issues Using show ip route and show ip protocol 803 Troubleshooting Routing Issues Using show ip route and show ip protocol The show ip protocol and show ip route commands display information about routing protocols and the routing table. The output from these commands can be used to ver- ify the routing protocol configuration. The show ip route command is perhaps the single most important command for trouble- shooting routing issues. This command displays the contents of the IP routing table. The output in Example 18-9 shows the entries for all known networks and subnet- works and how the information was obtained. If there is a problem reaching a host in a particular network, the output of the show ip route command can be used to verify that the router has a route to that network. If the output of the show ip route command does not show the expected learned routes or shows no learned routes, the problem is possibly that routing information is not being exchanged. In this case, use show ip protocols commands, as shown in Example 18-10, on the router to check for a routing protocol configuration error. Example 18-9 show ip route Command Output Cougars> show ip route Codes: C – connected, S – static, R – RIP, M – mobile, B – BGP D – EIGRP, EX – EIGRP external, O – OSPF, IA – OSPF inter area E1 – OSPF external type 1, E2 – OSPF external type 2, E – EGP i – IS-IS, L1 – IS-IS level 1, L2 – IS-IS level 2 * - candidate default Gateway of last resort is not set 144.253.0.0 is subnetted (mask is 255.255.255.0), 1 subnets C 144.253.100.0 is directly connected. Ethernet1 R 153.50.0.0 [120/1] via 183.8.128.12, 00:00:09, Ethernet0 183.8.0.0 is subnetted (mask is 255.255.255.128), 4 subnets R 183.8.0.128 [120/1] via 183.8.128.130.00, 00:00:17, Serial0 [120/1] via 183.8.64.130, 00:00:17, Serial1 C 183.8.128.0 is directly connected, Ethernet0 C 183.8.64.128 is directly connected, Serial1 C 183.8.128.128 is directly connected, Ethernet0 chpt_18.fm Page 803 Tuesday, May 27, 2003 2:19 PM 804 Chapter 18: Basic Router Troubleshooting The show ip protocols command displays values about IP routing protocol informa- tion on the entire router. This command can be used to confirm which protocols are configured, which networks are being advertised, which interfaces are sending updates, and the sources of routing updates. The show ip protocols output also displays the routing parameters about timers, filters, and other information related to the routing protocol. When multiple routing protocols are configured, the information about each protocol is listed in a separate section. The show ip protocols command output can be used to diagnose many routing issues, such as identifying a router that is suspected of delivering bad router information. The command output also can be used to confirm that the expected protocols, advertised networks, and routing neighbors are present. As with any troubleshooting process, identifying the problem is difficult, if not impossible, if there is no documentation indi- cating the expected results. Example 18-10 show ip protocols Command Output Router> show ip protocol Routing Protocol is rip Sending updates every 30 seconds, next due in 13 seconds Invalid after 180 seconds, hold down 180, flushed after 240 Outgoing update filter list for all interface is not set Incoming update filter list for all interface is not set Redistributing: rip Routing for Networks: 183.8.0.0 144.253.0.0 Routing Information Sources: Gateway Distance Last Update 183.8.128.12 120 0:00:14 183.8.64.130 120 0:00:19 183.8.128.130 120 0:00:03 Distance: (default is 120) Lab Activity Using show ip route to Examine Routing Tables In this lab, you configure RIP and IGRP on routers and then examine the impact on the routing table of multiple routing protocols using the show ip route command. chpt_18.fm Page 804 Tuesday, May 27, 2003 2:19 PM Troubleshooting Router Connections Using the show controllers serial Command 805 Troubleshooting Router Connections Using the show controllers serial Command Router configuration and troubleshooting frequently are done remotely. Therefore, it is not possible to physically inspect the router connections. The show controllers serial command, shown in Example 18-11, is used to determine the type of cable connected without inspecting the cables. Lab Activity Troubleshooting Routing Issues with show ip route and show ip protocol In this lab, you use the show ip route and show ip protocol commands to diagnose a routing configuration problem. Lab Activity Gateway of Last Resort (Default Gateway) In this lab, you configure RIP routing and add default routes (gateways) to the routers. You remove RIP and the default routes, then configure IGRP routing and add default routes (gateways) to the routers again. Lab Activity Last Route Update In this lab, you gather information about routing updates and routing proto- cols to determine the most recent routing table updates. Example 18-11 show controllers serial Command Output Cougars# show controllers serial 0/0 QUICC Serial unit 0 idb at 0x20A31A8, driver data structure at 0x20A4C60 SCC Registers: General [GSMR]=0x2:0x00000030, Protocol-specific [PSMR]=0x0 Events [SCCE]=0x0000, Mask [SCCM]=0x001F, Status [SCCS]=0x0006 Transmit on Demand [TODR]=0x0, Data Sync [DSR]=0x7E7E Interrupt Registers: … output omitted … DTE V.35 serial cable attached. output omitted chpt_18.fm Page 805 Tuesday, May 27, 2003 2:19 PM 806 Chapter 18: Basic Router Troubleshooting Being able to determine the type of cable that the controller detects is useful for finding a serial interface with no cable, the wrong type of cable, or a defective cable. The show controllers serial command queries the integrated circuit, or chip, that con- trols the serial interfaces and displays information about the physical interface. This output varies among controller chips. Even within a router type, different controller chips can be used. Regardless of the controller type, the show controllers serial command produces a tre- mendous amount of output. Other than the cable type, most of this output is internal technical detail regarding the controller chip status. Without specific knowledge of the integrated circuit, this information is not useful. Introduction to debug The debug commands assist in the isolation of protocol and configuration problems. The debug command is used to display dynamic data and events. Because the show commands display only static information, they provide a historical picture of the router operation. The debug command output provides more insight into the current events of the router. These events include traffic on an interface, error messages generated by nodes on the network, protocol-specific diagnostic packets, and other useful trouble- shooting data. Example 18-12 shows sample output from the debug ip rip command. Example 18-12 debug ip rip Command Router# debug ip rip RIP Protocol debugging is on Router# RIP: received update from 183.8.128.130 on Serial0 183.8.0.128 in 1 hops 183.8.64.128 in 1 hops 0.0.0.0 in 16 hops (inaccessible) RIP: received update from 183.8.64.140 on Seria11 183.8.0.128 in 1 hops 183.9.128.128 in 1 hops 0.0.0.0 in 16 hops (inaccessible) RIP: received update from 183.8.128.130 on Seria10 183.8.0.128 in 1 hops 183.8.64.128 in 1 hops chpt_18.fm Page 806 Tuesday, May 27, 2003 2:19 PM Introduction to debug 807 The highlighted material in Example 18-12 shows that RIP debugging has been turned on and shows which routes, networks, and interfaces are accessible and inaccessible. The dynamic output of the debug command comes at a performance cost, which pro- duces high processor overhead that can disrupt normal router operation. For this reason, you should use it conservatively. Use debug commands to examine specific types of traffic or concerns after problems have been narrowed to a few possible causes. In other words, debug commands should be used to isolate specific problems, not to monitor normal network operation. By default, the router sends the debug output and system messages to the console. If a Telnet session is being used to examine the router, the debug output and system mes- sages can be redirected to the remote terminal. This is done through the Telnet session by issuing the terminal monitor command. Use extra caution when selecting the debug commands from a Telnet session. No command should be selected that will cause the debug output to create additional traffic. If this occurs, the Telnet session rapidly saturates the link with traffic, or the router exhausts one or more resources. A good rule to fol- low to prevent this production of traffic is to never debug any activity on the port where the session is established. The output of the different debug commands varies. Some frequently generate many lines, while others output a line or two every few minutes. Example 18-13 shows the debug ip packet detail command. 0.0.0.0 in 16 hops (inaccessible) RIP: sending update to 255.255.255.255 via Ethernet0 (183.8.128.2) subnet 183.8.0.128, metric 2 subnet 183.8.64.128, metric 1 subnet 183.8.128.128, metric 1 default 0.0.0.0, metric 16 network 144.253.0.0, metric 1 RIP: sending update to 255.255.255.255 via Ethernet1 (144.253.100.202) default 0.0.0.0, metric 16 network 153.50.0.0, metric 2 network 183.8.0.0, metric 1 Example 18-12 debug ip rip Command (Continued) NOTE The debug all com- mand, in particular, should be used spar- ingly because it can disrupt router operations. chpt_18.fm Page 807 Tuesday, May 27, 2003 2:19 PM 808 Chapter 18: Basic Router Troubleshooting The highlighted material in the previous example indicates what networks are involved and shows access being denied from packets into or out of 210.107.197.105. Another Cisco IOS Software service that enhances the usefulness of the debug output is the timestamps command. This command puts a timestamp on a debug message. This information provides the time when the debug event occurred and the duration of time between events. The service timestamps debug uptime command can provide the hour:minute:second of the output, determine the amount of time since the router was last powered up, or identify when a reload command was executed. Example 18-13 debug ip packet detail Command Output Router# debug ip packet detail 10w6d: TCP src=1075, dst=80, seq=785595392, ack=3448593899, win=64240 ACK 10w6d: IP: s=192.168.120.145 (Ethernet0/0), d=192.168.119.9 (Ethernet0 /0), g=192.168.119.9, len 60, forward 10w6d: TCP src=1075, dst=80, seq=785595392, ack=3448599739, win=64240 ACK 10w6d: IP: s=192.168.120.145 (Ethernet0 /0), d=192.168.119.9 (Ethernet0 /0), g=192.168.119.9, len 60, forward 10w6d: TCP src=80, dst=1075, seq=3448603559, ack=785595392, win=8446 ACK PSH 10w6d: IP: s=192.168.120.145 (Ethernet0), d=192.168.119.9 (Ethernet0 /0), g=192.168.119.9, len 60, forward 10w6d: TCP src=1075, dst=80, seq=785595392, ack=3448604710, win=64240 ACK 10w6d: IP: s=10.1.1.81 (Serial0 /0), d=224.0.0.10, len 64, rcvd 2, proto=88 10w6d: IP: s=210.107.197.105 (Serial0 /0), d=192.168.119.255, len 1028, access denied 10w6d: ICMP type=8, code=0 10w6d: IP: s=10.1.1.82 (local), d=224.0.0.10 (Serial0 /0), len 22, sending broad/multicast, proto=88 10w6d: IP: s=0.0.0.0 (Ethernet0 /0), d=255.255.255.255, len 590, rcvd 2 10w6d: UDP src=68, dst=67 10w6d: IP: s=192.168.120.50 (Ethernet0 /0), d=192.168.120.255 (Ethernet0 /0), len 243, rcvd 3 Router# undebug all All possible debugging has been turned off GAD# IP: s=192.168.120.145 IP: s=10.1.1.81 IP: s=210.107.197.105 access denied : s=10.1.1.82 sending broad/multicast chpt_18.fm Page 808 Tuesday, May 27, 2003 2:19 PM . from 18 3.8 . 12 8 .13 0 on Serial0 18 3.8.0 . 12 8 in 1 hops 18 3.8.64 . 12 8 in 1 hops 0.0.0.0 in 16 hops (inaccessible) RIP: received update from 18 3.8.64 .14 0 on Seria 11 183.8.0 . 12 8 in 1 hops 18 3.9 . 12 8 . 12 8. to 25 5 .25 5 .25 5 .25 5 via Ethernet0 (18 3.8 . 12 8 .2) subnet 18 3.8.0 . 12 8, metric 2 subnet 18 3.8.64 . 12 8, metric 1 subnet 18 3.8 . 12 8 . 12 8, metric 1 default 0.0.0.0, metric 16 network 14 4 .25 3.0.0, metric 1 RIP:. d =22 4.0.0 .10 , len 64, rcvd 2, proto=88 10 w6d: IP: s= 21 0 .10 7 .19 7 .10 5 (Serial0 /0), d =19 2 .16 8 .11 9 .25 5, len 10 28 , access denied 10 w6d: ICMP type=8, code=0 10 w6d: IP: s =10 .1. 1. 82 (local), d =22 4.0.0 .10