Introduction to Network Testing 789 Figure 18-3 shows one approach to troubleshooting network problems. Figure 18-3 OSI Layer Troubleshooting With a structured approach, members of the network know what each member has completed to solve a problem. If a variety of ideas are used without any organization, the problem solving becomes chaotic. Without a structured approach, very few prob- lems are solved. chpt_18.fm Page 789 Tuesday, May 27, 2003 2:19 PM 790 Chapter 18: Basic Router Troubleshooting The flowcharts in Figure 18-2 and Figure 18-3 are not the only methods of trouble- shooting; however, the orderly process is very important to ensure that a network runs smoothly and efficiently. Testing by OSI Layers Testing should begin with Layer 1 of the OSI model and work to Layer 7, if necessary. Layer 1 (physical layer) errors can include the following: ■ Broken cables ■ Disconnected cables ■ Cables connected to the wrong ports ■ Intermittent cable connection ■ Wrong cables used for the task at hand (must use rollovers, crossover cables, and straight-through cables correctly) ■ Transceiver problems ■ Data communications equipment (DCE) cable problems ■ Data terminal equipment (DTE) cable problems ■ Devices turned off After exhausting all Layer 1 problems as possibilities for a network problem, the next step is to investigate Layer 2 problem possibilities. Layer 2 errors can include the following: ■ Improperly configured serial interfaces ■ Improperly configured Ethernet interfaces ■ Improper encapsulation set (High-Level Data Link Control [HDLC] is default for serial interfaces) ■ Improper clock rate settings on serial interfaces ■ Network interface card (NIC) problems After exhausting all Layer 2 problems as possibilities for a network problem, the next step is to investigate Layer 3 problem possibilities. Layer 3 errors can include the following: ■ Routing protocol not enabled ■ Wrong routing protocol enabled ■ Routing protocol incorrectly configured ■ Incorrect IP addresses chpt_18.fm Page 790 Tuesday, May 27, 2003 2:19 PM Introduction to Network Testing 791 ■ Incorrect subnet masks ■ Incorrect default gateway If errors or problems related to connectivity appear on the network, testing through the layers using connectivity tools, such as ping or telnet, should be the first step in determining the point in the network where the packet gets dropped. The ping com- mand can be used at Layer 3 to test connectivity. At Layer 7, the telnet command can be used to verify the application layer software between source and destination stations. Both of these commands are discussed in detail later in the chapter. Troubleshooting Layer 1 Using Indicators Indicators are useful tools for troubleshooting. Most interfaces or NICs have indicator lights that show whether there is a valid connection. This light usually is referred to as the link light. The interface also might have lights to indicate whether traffic is being sent (TX) or received (RX). If the interface has indicator lights that do not show a valid connection, power off the device and reset the interface card. The NIC functions at both Layers 1 and 2. Many indicators are considered Layer 1 problems in a network, including the following: ■ Broken cables ■ Disconnected cables ■ Cables connected to the wrong ports ■ Intermittent cable connection ■ Wrong cables used for the task at hand ■ Transceiver problems ■ Devices turned off A faulty or incorrect cable could result in a link light indicating a bad connection or no link. Check to make sure that all cables are connected to the appropriate ports. Make sure that all cross-connects are patched properly to the correct location using the appropri- ate cable and method. Verify that all switch or hub ports are set in the correct VLAN or collision domain, and have appropriate options set for Spanning Tree and other considerations. Verify that the proper cable is being used. If a direct connection exists between two end systems—for example, between a PC and a router or between two switches—a special crossover cable might be required. Verify that the cable from the source interface chpt_18.fm Page 791 Tuesday, May 27, 2003 2:19 PM 792 Chapter 18: Basic Router Troubleshooting is connected properly and is in good condition. If it does not seem to be a good con- nection, reset the cable and ensure that the connection is secure. Try replacing the cable with a known working cable. If the cable connects to a wall jack, use a cable tester to ensure that the jack is wired properly. Also check any transceiver in use to ensure that it is the correct type and is properly connected and configured. If replacing the cable does not resolve the problem, try replacing the transceiver, if one is being used. Before you run diagnostics or attempt complex troubleshooting, always check to make sure that the device is powered on. Some problems result from the simplest of errors. Troubleshooting Layer 2 Using the show interface Command The show interface command is perhaps the single most important tool to discover Layer 1 and Layer 2 problems with the router. The first output parameter, or line, refers to the physical layer. The second parameter, or protocol, indicates whether Cisco IOS Software processes that control the line protocol consider the interface usable. This is determined by whether keepalives successfully are received. If the interface misses three consecutive keepalives, the line protocol is marked as down. If the line is down, the protocol also must be down. This is because there would be no usable media for the protocol. This occurs when the interface is down because of a hardware issue or if it is “administratively down” because of a configuration issue. If the interface is up and the line protocol is down, a Layer 2 problem exists. The following list contains some possible causes: ■ No keepalives ■ No clock rate ■ Mismatch in encapsulation type These problems can be determined when the show interface command has been entered and a line appears stating, “interface is up and the line protocol is down.” Using the clockrate command or the encapsulation command, verify that that there is no mismatch between interfaces. The show interfaces serial command should be used after configuring a serial interface, to verify the changes and ensure that the interface is operational. chpt_18.fm Page 792 Tuesday, May 27, 2003 2:19 PM Introduction to Network Testing 793 Troubleshooting Layer 3 Using ping The ping utility is used to test network connectivity. To aid in diagnosing basic net- work connectivity, many network protocols support an echo protocol, which is used to test whether protocol packets are being routed. The ping command sends a packet to the destination host and then waits for a reply packet from that host. Results from this echo protocol can help evaluate the path-to-host reliability, delays over the path, and whether the host can be reached or is functioning. The ping output displays the mini- mum, average, and maximum times that it takes for a ping packet to find a specified system and return. ping uses the Internet Control Message Protocol (ICMP) to verify the hardware connection and the logical address of the network layer. This is a basic testing mechanism. For the network in Figure 18-4, the ping target 172.16.1.5 responded successfully to all five datagrams sent, as shown in Example 18-1. Figure 18-4 Network for Testing with ping The exclamation points (!) indicate each successful echo. If any periods (.) instead of exclamations are received on a display, the application on the router timed out waiting for a given packet echo from the ping target. The ping command can be used to diag- nose basic network connectivity. Example 18-1 ping Results for Target 172.16.1.5 Router> ping 172.16.1.5 Type escape sequence to abort Sending 5, 100 byte ICMP Echos to 172.16.1.5, timeout is 2 seconds: !!!!! Success rate is 100 percent, round-trip min/avg/max – 1/3/4 ms Router> chpt_18.fm Page 793 Tuesday, May 27, 2003 2:19 PM 794 Chapter 18: Basic Router Troubleshooting ping tests network connections by sending ICMP echo requests to a target host and waiting for replies. When testing a connection, ping tracks the number of packets sent, the number of replies received, the percentage of packets lost, and the amount of time required for the packets to reach the destination and for replies to be received. This information enables users to verify whether their workstations can communicate with other hosts, and they can determine whether any information was lost. The ping command can be initiated from both user EXEC mode and privileged EXEC mode. The ping command can be used to confirm basic network connectivity on AppleTalk, ISO Connectionless Network Service (CLNS), IP, Novell, Apollo, VINES, DECnet, or XNS networks. ICMP uses messages to accomplish various tasks. Table 18-1 shows a list of ICMP message types. Table 18-1 CMP Message Types Message Purpose Destination unreachable This tells the source host that there is a prob- lem delivering a packet. Time exceeded The time that it takes a packet to be delivered has been too long, and the packet has been discarded. Source quench The source is sending data faster than it can be forwarded. This message requests that the sender slow down. Redirect The router sending this message has received some packet for which another router would have had a better route. The message tells the sender to use the better route. Echo This is used by the ping command to verify connectivity. Parameter problem This is used to identify a parameter that is incorrect. Timestamp This is used to measure round-trip time to particular hosts. chpt_18.fm Page 794 Tuesday, May 27, 2003 2:19 PM Introduction to Network Testing 795 The extended command mode of the ping command permits users to specify the sup- ported IP header options. This allows the router to perform a more extensive range of test options. To enter ping extended command mode, type the ping command with no options in privileged mode and press Enter. You are prompted for some basic options as well as the Extended Commands option. Enter yes at the Extended commands prompt to specify various extended ping options. An extended ping works the same way as a regular ping, but it supports the manipulation of some different parameters, such as packet size and frequency. It is a good idea to use the ping command when the network is functioning properly, to see how the command works under normal conditions and so that you have something to compare against when troubleshooting. Troubleshooting Layer 3 Issues Using traceroute The traceroute command is used to discover the routes that packets take when traveling to their destinations. traceroute also can be used to test the network layer, or Layer 3, on a hop-by-hop basis and to provide performance benchmarks. The output of the traceroute command generates a list of hops that successfully were reached, as shown in Example 18-2. If the data successfully reaches the intended desti- nation, then the output indicates every router that the datagram passes through. This output can be captured and used for future troubleshooting of the internetwork. Address mask request/reply This is used to inquire about and learn the correct subnet mask to be used. Router advertisement and selection This is used to allow hosts to dynamically learn the IP addresses of the routers attached to the subnet. Example 18-2 traceroute Command Output Cougars> traceroute 168.71.8.2 traceroute to pc-b.cisco.com (168.71.8.2), 30 hops max, 40 byte packets 1 routerb (168.71.6.3) 3 ms 3 ms 3 ms 2 * * * continues Table 18-1 CMP Message Types (Continued) Message Purpose 3 ms 3 ms 3 ms chpt_18.fm Page 795 Tuesday, May 27, 2003 2:19 PM 796 Chapter 18: Basic Router Troubleshooting traceroute output also indicates the specific hop at which the failure is occurring. For each router in the path, a line of output is generated on the terminal indicating the IP address of the interface that the data entered. If an asterisk (*) appears, the packet failed. You can isolate the problem area by obtaining the last good hop from the trace- route output and comparing it to a diagram of the internetwork. traceroute also provides information indicating the relative performance of links. The round-trip time (RTT) is the time required to send an echo packet and get a response, as shown in Example 18-2. In this example, each of the three packets sent has an RTT of 3 milliseconds (ms). This is useful for an approximate idea of the delay on the link. These figures are not precise enough to be used for an accurate performance evaluation; however, this output can be captured and used for future performance troubleshooting of the internetwork. The device receiving the traceroute also must know how to send the reply back to the source of the traceroute. For the traceroute or ping data to successfully make the round trips between routers, there must be known routes in both directions. A failed response is not always an indication of a problem because ICMP messages could be rate-limited or filtered at the host site. This is especially true across the Internet. traceroute sends out a sequence of User Datagram Protocol (UDP) datagrams from the router to an invalid port address on the remote host. For the first sequence of three datagrams sent, a Time-To-Live (TTL) field value is set to 1. The TTL value of 1 causes the datagram to time out at the first router in the path. This router then responds with an ICMP time exceeded message (TEM) indicating that the datagram has expired. Three more UDP messages now are sent, each with the TTL value set to 2. This causes the second router to return ICMP TEMs. This process continues until the packets reach the other destination. Because these datagrams are trying to access an invalid port at the destination host, ICMP port unreachable messages are returned instead of ICMP TEMs. This indicates an unreachable port and signals the traceroute program, terminating the process. 3 * * * 4 * * * 5 * * * Cougars> Example 18-2 traceroute Command Output (Continued) chpt_18.fm Page 796 Tuesday, May 27, 2003 2:19 PM Introduction to Network Testing 797 Troubleshooting Layer 7 Using Telnet Telnet is a virtual terminal protocol that is part of the TCP/IP protocol suite. Telnet allows the verification of the application layer software between source and destination stations. This is the most complete test mechanism available. Telnet allows connections to be made to remote devices, to gather information and run commands and applications. Telnet is the most complete testing tool because it uses IP at the network layer and TCP at the transport layer to create a session with a remote host. If Telnet is successful, IP connectivity must be good. The telnet command provides a virtual terminal so that administrators can use Telnet operations to connect with other network devices (such as routers and switches) run- ning TCP/IP. If a router can be remotely accessed using Telnet, at least one TCP/IP application can reach the remote router. A successful Telnet connection indicates that the upper-layer application and the services of lower layers are functioning properly. Figure 18-5 illustrates a Telnet connection. Figure 18-5 Using Telnet If an administrator can Telnet to one router but not to another router, the Telnet fail- ure likely is caused by specific addressing, naming, or access permission problems. These problems can exist on the administrator’s router or on the router that failed as a Telnet target. If the Telnet to a particular server fails from one host, try connecting from a router and several other devices. Also try using ping as a more basic test. If you can ping the host, but cannot achieve a login prompt when trying to Telnet, check the following: ■ Can a reverse DNS lookup on the client’s address be found? Many Telnet servers do not allow connections from IP addresses that have no DNS entry. This is a common problem for DHCP-assigned addresses in which the administrator has not added DNS entries for the DHCP pools. chpt_18.fm Page 797 Tuesday, May 27, 2003 2:19 PM 798 Chapter 18: Basic Router Troubleshooting ■ A Telnet application might not be capable of negotiating the appropriate options and, therefore, will not connect. On a Cisco router, this negotiation process can be viewed using debug telnet. Look for error messages or invalid IP or DNS address, which would indicate this problem. ■ It is possible that Telnet is disabled or has been moved to a port other than 23 on the destination server. Recall that port 23 is the default port for Telnet. Troubleshooting Router Issues Using the show interface and show interfaces Commands Cisco IOS Software contains numerous commands for troubleshooting. Among the more widely used are the show commands. Every aspect of the router can be viewed with one or more of the show commands. The show interface command is used to check the status and statistics of the interfaces. Variations of this command can be used to check the status of the different types of interfaces. To view the status of the FastEthernet interfaces, use show interfaces fastethernet. This command also can be used to view the status of one particular interface. To view the status of the Serial 0/0 interface, use show interface serial0/0. To view the status of FastEthernet 0/0, use show interface fa0/0. The show interfaces command displays the status of two important portions of the interfaces. These can be related to the Layer 1 and the Layer 2 functions: ■ Physical (hardware) portion—The hardware includes cables, connectors, and interfaces showing the condition of the physical connection between the devices. ■ Logical (software) portion—The software status shows the state of messages such as keepalives, control information, and user information that are passed between adjacent devices. This relates to the condition of a data link layer protocol passed between two connected neighboring router interfaces. These important elements of the show interface serial command output are displayed as the line and data-link protocol status shown in Example 18-3. Lab Activity Troubleshooting Using ping and telnet In this lab, you use knowledge of OSI Layers 1, 2, and 3 to diagnose network configuration errors while using the ping and telnet utilities. chpt_18.fm Page 798 Tuesday, May 27, 2003 2:19 PM . Echos to 17 2 .16 .1. 5, timeout is 2 seconds: !!!!! Success rate is 10 0 percent, round-trip min/avg/max – 1/ 3/4 ms Router> chpt _18 .fm Page 793 Tuesday, May 27 , 20 03 2 :19 PM 794 Chapter 18 : Basic. subnet. Example 18 -2 traceroute Command Output Cougars> traceroute 16 8. 71. 8 .2 traceroute to pc-b.cisco.com (16 8. 71. 8 .2) , 30 hops max, 40 byte packets 1 routerb (16 8. 71. 6.3) 3 ms 3 ms 3 ms 2 * * * continues Table. command can be used to diag- nose basic network connectivity. Example 18 -1 ping Results for Target 17 2 .16 .1. 5 Router> ping 17 2 .16 .1. 5 Type escape sequence to abort Sending 5, 10 0