Tcpdump output 20:16:37.411690 OSI, IS-IS, length: 1492 L1 Lan IIH, hlen: 27, v: 1, pdu-v: 1, sys-id-len: 6 (0), max-area: 3 (0) source-id: 1921.6800.1008, holding time: 120s, Flags: [L1, L2] lan-id: 1921.6800.1008.02, Priority: 64, PDU length: 1492 IS Neighbor(s) TLV #6, length: 6 SNPA: 0090.692b.0e52 Protocols supported TLV #129, length: 1 NLPID(s): IPv4 (0xcc) IPv4 Interface address(es) TLV #132, length: 4 IPv4 interface address: 193.83.223.236 Area address(es) TLV #1, length: 4 Area address (length: 3): 49.0001 Restart Signaling TLV #211, length: 3 Flags [none], Remaining holding time 0s Padding TLV #8, length: 255 Padding TLV #8, length: 255 Padding TLV #8, length: 255 Padding TLV #8, length: 255 Padding TLV #8, length: 255 Padding TLV #8, length: 150 If a router exchanges these bloated Hello PDUs in both directions then it can be sure that the underlying media sufficiently supports the maximum packet sizes necessary for IS-IS. IOS and JUNOS do have different styles of how and when they do implement adja- cency checks. IOS pads each and every Hello that it transmits on the wire. On large WAN Hub Routers that terminate a lot of circuits – for example on a Router running Frame relay or ATM circuits – periodic emission of large packets can be a burden to the control plane processor. If you know that your underlying link supports at least 1492 bytes sized packets then you can turn off the artificial bloating of Hello PDUs using the no hello padding router configuration command. MTU Check 117 TLV Type TLV Length Padding Data 8 Bytes 1 1 1–255 FIGURE 5.5. The Padding TLV #8 is used to bloat IIHs up to at least 1492 bytes IOS configuration The no hello padding command turns off MTU check against the underlying media. ! router isis no hello padding [… ] ! JUNOS encompasses a technique called smart padding, where the router transmits padded Hellos only at the beginning of the Adjacency Bring up. After both ends of a router have completed the handshake procedure JUNOS automatically omits the Padding TLVs in the Hello message. That behaviour is a nice compromise between strict MTU checking and making sure that the IS-IS router does not consume excess bandwidth in tight WAN environments. The brief Tcpdump output shows the JUNOS specific vari- ation in packet sizes during an IS-IS Adjacency bring up. Tcpdump output 20:16:37.411690 OSI, IS-IS, L1 Lan IIH, src-id 1921.6800.1002, lan-id 1921.6800.1002.02, prio 64, length 1492 20:16:37.412312 OSI, IS-IS, L2 Lan IIH, src-id 1921.6800.1002, lan-id 1921.6800.1002.02, prio 90, length 1492 20:16:37.414060 OSI, IS-IS, L1 Lan IIH, src-id 1921.6800.1003, lan-id 1921.6800.1003.02, prio 70, length 1492 20:16:37.414466 OSI, IS-IS, L2 Lan IIH, src-id 1921.6800.1003, lan-id 1921.6800.1003.02, prio 64, length 1492 20:16:37.418232 OSI, IS-IS, L1 Lan IIH, src-id 1921.6800.1002, lan-id 1921.6800.1003.02, prio 64, length 65 20:16:37.418742 OSI, IS-IS, L2 Lan IIH, src-id 1921.6800.1002, lan-id 1921.6800.1002.02, prio 90, length 65 20:16:37.420914 OSI, IS-IS, L1 Lan IIH, src-id 1921.6800.1003, lan-id 1921.6800.1003.02, prio 70, length 90 20:16:37.421055 OSI, IS-IS, L2 Lan IIH, src-id 1921.6800.1003, lan-id 1921.6800.1002.02, prio 64, length 90 20:16:37.423429 OSI, IS-IS, L1 Lan IIH, src-id 1921.6800.1002, lan-id 1921.6800.1003.02, prio 64, length 65 20:16:37.423909 OSI, IS-IS, L2 Lan IIH, src-id 1921.6800.1002, lan-id 1921.6800.1002.02, prio 90, length 65 The next few sections show how the IS-IS Protocol verifies two-way connectivity over a link. From now on, the term handshaking is used as a replacement for “verifying two- way connectivity”. That is really all that handshaking means. 118 5. Neighbour Discovery and Handshaking 5.3 Handshaking In the IS-IS specification there are two general ways of handshaking: • 2-way handshake • 3-way handshake Figure 5.6 illustrates what occurs during a 2-way handshake. IS-IS is started on Router A. A Hello message is sent to Router B. As soon as Router B responds with a Hello Message of its own, Router A will declare the Adjacency with Router B up. The impor- tant aspect here is that Router A does not know if the Hello message from Router B is in response to the Hello message that Router A sent or if it is just any Hello message that Router B has generated (perhaps Router A’s Hello message has been lost on the link). There is no state that is kept. That insight is significant later when we explore a failure conditions resulting from a pure 2-way handshake check. Of course the same procedure is executed from Router B’s perspective as well. The Router B perspective is not shown in Figure 5.6, because the picture would have been too crowded and harder to under- stand. But Router B of course also sends a Hello message and as soon as Router B receives any Hello message from Router A, Router B will declare the adjacency up. Only two messages are necessary in the 2-way handshake. The 3-way handshake works differently. Handshaking 119 Router A Router B t t Router B Adjacency UP IIH Router A misc. TLVs IIH Router B misc. TLVs IS-IS enabled on the circuit Router A Adjacency UP F IGURE 5.6. For 2-way handshakes only two messages are required to declare a circuit up Figure 5.7 shows a 3-way handshake transition. Router A first sends the Hello mes- sage out, just as before. Next, Router B responds with a Hello message. Router A will know that this Hello was not sent by accident (in the 2-way case Router A never really knows) because the Hello message from Router B carries an indication that this Hello has been sent in response to Router A’s original Hello. This is done by mentioning Router A explicitly in the message body, by means of a special TLV. Later, in the finite state machine section such an event is described as Seenself. Router B receives Router A’s Hello message and now realizes that it has been seen by the neighbour (Router A) and declares the adjacency up. Router A now responds by sending a third Hello message back to Router B confirming that it has also seen Router B’s Hello message, which causes Router B to declare the adjacency (from its perspective) now up as well. The 3-way handshake is a stateful transition and much more robust than the simple 2-way version, but does require an extra message. IS-IS uses different message elements and handshaking methods depending on whether it is performing the handshaking on LAN or on point-to-point circuits. The following section shows where and in which environment the different handshaking methods are used, and what TLVs are encoded in the Hello messages to convey neigh- bour adjacency state in IS-IS. 5.3.1 The 3-way Handshake on LAN Circuits On LANs, IS-IS uses a 3-way handshake. Figure 5.8 shows the state changes on the LAN from Router A’s perspective. Please note that for better visibility again, only the state 120 5. Neighbour Discovery and Handshaking t t Router B Adjacency UP IIH Router A misc. TLVs IIH Router B “I have seen Router A” & IS-IS enabled on the circuit IIH Router A “I have seen Router B” & Router A Adjacency UP Router A Router B misc. TLVs misc. TLVs FIGURE 5.7. The 3-way handshake is a stateful transition t t Router A MAC 0090.69aa.aaaa Router B MAC 0000.0cbb.bbbb Router C MAC 0090.69cc.cccc t Router C Adjacency UP IS-IS enabled on the circuit Router A Adjacency UP IIH Router C I’ve Seen MAC 0090.69aa.aaaa misc. TLVs IIH Router A misc. TLVs IIH Router B I’ve Seen MAC 0090.69aa.aaaa misc. TLVs Router B Adjacency UP IIH Router A I’ve Seen MAC 0000.0cbb.bbbb MAC 0090.69cc.cccc misc. TLVs Router A Adjacency UP F IGURE 5.8. On LANs the routers need to send a list of visible neighbours to complete the 3-w ay handshake 121 transactions for Router A are shown in the figure. First, Router A sends a Hello onto the LAN. Routers B and C, which both get the LAN-based message of course, respond to Router A’s Hello by sending a Hello that lists Router A’s source MAC address from Router A’s original Hello message encoded in a dedicated TLV. The structure of the TLV will be discussed shortly. Router A receives these Hellos from Routers B and C and realizes “Hey, they both got my Hello message! Otherwise, my MAC address would not be listed in their Hello.” Thus, Router A declares the adjacencies to Router B and C up. To complete the 3-way handshake, Router A notifies Routers B and C that Router A has seen the recent Hello from both of them by listing Router B and C’s MAC address in one of its own TLVs. Once Routers B and C receive the Hello from Router A, the 3-way handshake is com- pleted (due to Seenself ) and the adjacency to Router A is declared up by both Router B and C. The TLV that conveys the MAC addresses is called the “IS Neighbor TLV #6”. The structure and encoding rules for this are discussed in the following section. 5.3.1.1 IS Neighbour TLV #6 Figure 5.9 shows the structure of the TLV that provides the “Hello, I have seen you” function in order to complete the 3-way handshake. The TLV code point allocated to the IS neighbour’s TLV is #6. The structure is actually very simple. It is essentially an array of SNPAs. SNPA is an abbreviation for Sub-Network Point of Attachment. On broadcast LANs a SNPA is the ISO term for a standard, 48-bit IEEE MAC address. The 48-bits equals six bytes, so the maximum length of this TLV is always a multiple of six. If it is not, then the TLV is malformed. On the network analyzer’s output, the list of MAC addresses is listed under the IS Neighbour stanza. The number of MAC addresses (4 entries) matches the TLV length of 4 bytes (4 ϫ 6 ϭ 24). Tcpdump output The IS Neighbour TLV #6 contains a list of MAC addresses of the routers that are visible from the sending router’s perspective: 122 5. Neighbour Discovery and Handshaking TLV Type TLV Length IS Neighbor MAC Address (SNPA) 6 Bytes 1 1 6 N * 6 IS Neighbor MAC Address (SNPA) 6 FIGURE 5.9. The IS Neighbour TLV #6 conveys the neighbour state for the 3-way handshaking procedure 09:38:23.996041 OSI, IS-IS, length: 74 L1 Lan IIH, hlen: 27, v: 1, pdu-v: 1, sys-id-len: 6 (0), max-area: 3 (0) source-id: 1921.6800.1012, holding time: 27s, Flags: [L1, L2] lan-id: 1921.6800.1012.02, Priority: 64, PDU length: 75 IS Neighbor(s) TLV #6, length: 24 SNPA: 0090.69b2.71ca SNPA: 0090.69b2.41cc SNPA: 0000.0c54.fadd SNPA: 0000.0c11.cc1e Protocols supported TLV #129, length: 2 NLPID(s): IPv4, IPv6 IPv4 Interface address(es) TLV #132, length: 4 IPv4 interface address: 172.16.33.1 Area address(es) TLV #1, length: 4 Area address (length: 3): 49.0001 Restart Signaling TLV #211, length: 3 Flags: [none], Remaining holding time 0s On LAN circuits there is only a single handshaking method available: the 3-way handshake using the IS-Neighbour TLV #6. On point-to-point circuits there is an imple- mentation choice between 2-way and 3-way handshakes. The next section shows how handshaking on point-to-point circuits works, what flaws have been revealed in the ori- ginal specifications, and how the handshake methods finally evolved. 5.3.2 The 2-way Handshake on Point-to-point Circuits The original ISO 10589 specification proposed just a 2-way handshake on point-to-point circuits. Through implementation and deployment experience, several scenarios are known today where the use of 2-way handshakes causes IS-IS to get blind spotted and in the worst case, to completely black hole traffic. Most of these failure scenarios are related to routers connected by unidirectional links, which is quite frequently the result of a failure to network equipment. In networking environments, unidirectional links can occur quite easily. Typically, a fibre path between a pair of routers is composed of two fibres: one for transmitting and one for receiving. If one of the two fibres breaks, the routers are reduced to one-way connectivity. In most cases this is not a big problem if there is just a simple fibre run between a pair of routers and the transmit fibre on one side breaks. The receiver on the other end of a fibre link will detect a loss of signal and the entire circuit is declared down. The trouble really starts if there is an active network element between the two routers, such as a LAN switch, so that the light is not missing on one side and the circuit always stays up. Figure 5.10 shows Router A and Router B connected through an ATM switch. Please note that this problem is not just specific to the ATM technology. The ATM switch here serves just as an example: it could be any Layer 2 technology like an active Ethernet device or a Frame Relay switch and so on. In total, there are four fibres in this small network. There is a pair of fibres between Router A and the ATM switch, and another pair of fibres between Router B and the ATM switch. Now imagine that the transmit link between Router B and the ATM switch Handshaking 123 breaks. Router A is still receiving a signal from the ATM switch, because the local link is still fine. But because “the light does not go out” at the Router A end, both sides (Router A and the ATM switch) think that everything is fine and the link is up and running to Router B. Please note that in practice there are Layer-2 management protocols like LMI, PPP LCP or ATM OAM cells that would help to detect that there is an end-to-end connectivity prob- lem. However, these protocols take time to detect error conditions and in the meantime IS-IS could have announced bogus information and flood it through the network. This example of the conditions that result in unidirectional links will be the basis for most of the issues with the 2-way handshake in IS-IS. The next two sections describe very common failure scenarios that all start with one-way connectivity as the root cause of the problem. 5.3.2.1 Failure Scenario 1: SONET/SDH APS In most carrier environments, an underlying SONET/SDH network is used to provision broadband links between routers. SONET/SDH networks are complex networks on their own and offer a variety of functions at the OSI-RM Layer 1, the Physical Layer. One of these functions is Automatic Protection Switching (APS), where “extra” bandwidth and ports in the network are provisioned to support redundancy of the SONET/SDH circuits. There are rumblings in the networking industry that this additional layer of network- ing intelligence will be made obsolete in the near future and that IP routers will soon be connected just by raw Dense Wavelength Division Multiplexing (DWDM) pipes. This might come true for very high speed (OC-48/STM-16 and beyond) links in the core, but at the edges of the network and in regional access networks, SONET/SDH networks will be present for a long time to come. And DWDM has been stalled somewhat by expensive equipment, so a discussion of SONET/SDH APS and IS-IS is still important and will be so for the foreseeable future. In any case, DWDM core or not, look at the edge of the network and assume the net- work uses transport capacity from a regional city or metropolitan area carrier. Typically the customer has the choice of an unprotected circuit or a protected circuit. In the pro- tected circuit, the regional carrier pre-provisions bandwidth and ports in order to recover from failed or broken equipment in any part of the network. Assume this is the protected flavour of the circuit, which is always a good idea if the budget allows. What follows 124 5. Neighbour Discovery and Handshaking RxTx Rx Tx My Adjacency to Router B is ok Router BRouter A ATM Switch F IGURE 5.10. Active elements between routers do not propagate downstream loss of signal errors does not require any detailed familiarity with SONET/SDH. All terms and equipment roles are fully explained as needed. Figure 5.11 shows a failure scenario where Router A and Router B are connected by a SONET/SDH pipe. Router A is located at the spoke site and Router B is located at the central hub site. Additionally, a second redundant SONET/SDH port has been pre- provisioned in case a link to one of the routers or even the router itself at the central site fails. The SONET/SDH Add–Drop Multiplexer (ADM), the network element that links the routers at both customer sites, needs to make sure its ports are still up. In SONET/SDH networks, the ends of a SONET/SDH link (in this case, the routers) can send heartbeat signalling messages in the overhead bytes of the SONET/SDH transmission frame header for redundancy purposes. Routers A and B send heartbeat signals in order to inform the ADMs that everything is okay. If the ADM does not receive a heartbeat signal from the routers for a period of 50 milliseconds (ms), then the ADM will automatically switch over to the backup circuit (Router C). Note that both Router B and Router C listen on the wire for APS signalling messages because the ADM connects both routers, receive fibres. However, Router C’s transmit fibre is not ordinarily active (it is not needed). This fibre only gets activated in failure mode when Router B or one of its links goes down. Realize that this is a purposeful, one- way connection for SONET/SDH APS. It is exactly this one-way connection that will cause trouble in IS-IS environments. Consider the following scenario: 1. Router A sends a Hello message 2. Both Routers B and C receive the Hello message 3. Router B responds with a Hello message and declares the adjacency to Router A up 4. Router C also responds with a Hello message. But the Hello response does not get through to the spoke site (no active transmit). However, Router C thinks it has suc- cessfully delivered the Hello and declares the adjacency up. So Router A knows it has an adjacency with Router B and vice versa, which is fine. The problem is that Router C also thinks it has an adjacency with Router A and therefore will forward traffic down the “broken” (inactive) link, which is only to be used for APS purposes. This is a serious issue because the traffic from Router C to Router A will get black holed because the transmit fibre is not connected all the way through the ADM. Handshaking 125 Central "Hub" site Network cloud Spoke site Network cloud SONET/SDH Add drop multiplexer (ADM) Rx Tx Rx Rx Tx 1 2 3 2 4 3 Tx Router B Router C Router A FIGURE 5.11. The protected SONET circuit is creating a unidirectional link in the backup case The whole point here is that a backup link at the Physical Layer looks like it can be used by Layer 3 (IP and IS-IS), but this is not the case. This is just a consequence of the use of the 2-way handshake on point-to-point circuits. Scenarios like this, where traffic gets black holed, are very difficult to troubleshoot. Most Network Operation Centre (NOC) teams are fooled by the fact that the router adjacency is up and their thinking is that the circuit must be delivering injected traffic. Trusting the 2-way handshake in this case leads to a serious impairment of the network. 5.3.2.2 Failure Scenario 2: Parallel Links The previous failure scenario does not do any damage, because all the IS-IS routers in the network would soon realize during the SPF calculation that Router C believes it has an adjacency with Router A, but Router A does not report an adjacency to Router C. The SPF algorithm, which is used to calculate paths through the network, has an additional stability rule built in. If two routers do not indicate to each other that they have an adja- cency, then the SPF algorithm disregards the adjacency between the two routers, which means that no transit traffic is sent over the unidirectional link. However, based on the previous failure condition, it is relatively easy to construct a four router scenario (two routers on each side of the link) where both sides report a stale adjacency that ultimately passes the 2-way check during the SPF calculation. This example simply uses a three router scenario for a clearer explanation of the underlying problem. So far, we have not mentioned the details of the SPF calculation, but there will be much more about that topic in Chapter 10 “SPF and Router Calculation”. This section shows one other example of failure. In this example even the SPF-2-way check will be spoofed, which serves as a last resort protection from black holing traffic. Consider the scenario in Figure 5.12. Here there are two routers interconnected by two circuits composed of two fibres in each direction. Now, assume there are two fibre breaks. The transmit fibre from Router A to Router B on circuit #1 has failed and, in addi- tion, the transmit fibre from Router B to Router A on circuit #2 is broken. Here is the sequence of events that happens: 1. Router A sends a Hello message on circuit #2 2. Router B responds to the Hello message on circuit #2 and declares the adjacency up 3. Router B sends a Hello message on circuit #1 4. Router A responds to the Hello message on circuit #1 and declares the adjacency up 5. Both Routers A and B tell other routers in the network that they can see each other, when in fact they can’t because of the fibre failures mentioned earlier. This failure sce- nario passes even the check during the SPF calculation. This makes both Router A and B attract transit traffic which will be black holed by both sides. So 2-way handshaking on point-to-point links in IS-IS suffers from robustness prob- lems in practice. Therefore the basic IS-IS protocol needs to be extended so that the more reliable 3-way handshakes are made on point-to-point circuits. Using the error-prone 2-way handshake procedure results in the set of problems generated by unidirectional links due to APS or multiple fibre breaks. The 3-way handshake on point-to-point circuits is discussed in the following section. 126 5. Neighbour Discovery and Handshaking . breaks. The receiver on the other end of a fibre link will detect a loss of signal and the entire circuit is declared down. The trouble really starts if there is an active network element between the. other routers in the network that they can see each other, when in fact they can’t because of the fibre failures mentioned earlier. This failure sce- nario passes even the check during the SPF calculation of the TLV that provides the “Hello, I have seen you” function in order to complete the 3-way handshake. The TLV code point allocated to the IS neighbour’s TLV is #6. The structure is actually