You can examine the contents of the Attribute block by issuing a show isis database. This command is supported on both IOS and JUNOS routers, as shown in the following: IOS command output Amsterdam# show isis database [… ] IS-IS Level-1 Link State Database: LSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OL LSPs and Revision Control 157 172.16.1/24 Pennsauken LSP New-York, Seq 0x2fc Lifetime 1200 s, OL Link Up to Pennsauken Link Up to Wash D.C. Pennsauken Frankfurt London Washington NewYork Paris FIGURE 6.8. The overloaded routers’ local prefixes are still reachable while transit-traffic is kept away New-York.00-00 0x00002fac 0xC24F 60128 1/0/0 [… ] IS-IS Level-2 Link State Database: LSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OL LINX-gw.00-00 0x00000128 0xB8EF 36163 0/0/1 LINX-gw.01-00 0x00000128 0x455A 42001 0/0/0 VIX-gw.00-00 0x00000123 0xEFC1 5023 0/0/0 [… ] In JUNOS, you can get a quick overview of the status of some of the bits by issuing the show isis database command. If you want to see the Attribute block for one LSP specifically, then you can request a specific LSP’s extensive output by issuing a show isis <LSP> extensive. There are various levels of output for the show isis database command. You can see the default and how the extensive modifier changes the output in the following: JUNOS command output hannes@New-York> show isis database IS-IS level 1 link-state database: LSP ID Sequence Checksum Lifetime Attributes New-York.00-00 0x2fac 0xc24f 62063 L1 Attached [… ] 4 LSPs IS-IS level 2 link-state database: LSP ID Sequence Checksum Lifetime Attributes LINX-gw.00-00 0x128 0xb8ef 36063 L1 L2 Overload LINX-gw.01-00 0x128 0x455a 41901 L1 L2 VIX-gw.00-00 0x123 0xefc1 4922 L1 L2 [… ] 12 LSPs hannes@New-York> show isis database LINX-gw extensive LINX-gw.00-00 Sequence: 0x128, Checksum: 0xb8ef, Lifetime: 36063 secs IS neighbor: London.01 Metric: 2000 IP prefix: 2.168.9.225/32 Metric: 0 Internal IP prefix: 172.16.6.0/28 Metric: 2000 Internal [… ] Packet: LSP id: Vienna-ts1.00-00, Length: 98 bytes, Lifetime: 36063 secs Checksum: 0xb8ef, Sequence: 0x128, Attributes: 0x7 <L1 L2 Overload> NLPID: 0x83, Fixed length: 27 bytes, Version: 1, Sysid length: 0 bytes Packet type: 20, Packet version: 1, Max area: 0 158 6. Generating, Flooding and Ageing LSPs The use of the Overload Bit is one of the more interesting stories of IS-IS. The reason why and when this bit is set has changed dramatically over the years. At first, the use was straightforward: there were many router memory constraints. Memory in the late 1980s compared to today was expensive, and the chips had little capacity. In the late 1980s, most routers were running with 512 KB of memory. So memory was definitely a con- straint, and the Overload Bit was used as intended. But what about today? Memory isn’t really an issue for IS-IS now. All modern Internet core routers have at least 256 MB of memory, and in many cases, they have even more memory. For example, the Juniper Networks T640 router has a route-processor that has a massive 2 GB for storing routing information. And IS-IS is not a particularly memory- intensive protocol. Massive amounts of memory are typically needed for storing BGP routes and paths. Even in the largest networks in the world, IS-IS does not consume more than 1.5 to 2.0 MB of memory. To give an idea how big these modern IS-IS networks are, think of an L1L2 router that is on the IS-IS Level 2 backbone with 1200 or more other routers. In addition, consider 200 or more routers at IS-IS Level 1. Thus, the link-state database is quite large. True enough, and yet it still does not consume more than between 0.1% and 1% of a modern route processor’s memory. So what is it that drives the setting of the Overload Bit today? The answer to this is addressed in the next section. 6.3.4.3 Applications of the Overload Bit In an Internet core router there are always two routing protocols running. One is for gather- ing topological knowledge and one carries a bulk amount of reachability information. The Interior Gateway Protocol (IGP) discovers the topological knowledge of the internal core network of an ISP. IS-IS is a member of the large family of IGPs. For the reachabil- ity information about bulk routes (Internet routes and customer routes), there is the Border Gateway Protocol (BGP), the interdomain routing protocol of the Internet. Each protocol does what it can do best. IS-IS quickly discovers, and then re-routes the internal network. Unfortunately, IS-IS is not very good in transporting a bulk amount of routing information across the internal network. IS-IS is in good company in being unsuited for this task with the rest of the IGPs (RIP, OSPF, EIGRP etc.). All of the IGPs suffer from the same protocol defect, in that all of the IGPs do not have any flow-control mechanisms built into the protocol. This is the reason that they fail when a large number of routes have to be carried across the internal network. As long as the IGP routers are loaded moderately, there is not much of a problem. However, as with human beings, everything is different under stress. Once the network is exposed to protocol-related stress (a large amount of re-routing, heavy LSP processing, many flap- ping links, and so on), plus a large amount of BGP reachability information, the network starts to churn. Churning IGPs was the typical reason in the 1990s when people like the NOC-team managers and the Chief Technology Officer got paged out of bed in the middle of the night. These lessons were learned and the implementations of the routing protocols did get better. There has also been more experience gained in network design. The pragmatic design rule today is that there must not be any other IP reachability information in the LSPs and Revision Control 159 IGP other than /30, /31 (point-to-point link IP sub-nets) and /32 routes (loopback addresses). No customer routes, no server farm routes, nothing except the IP sub-nets needed to run the internal infrastructure is put into the IGP’s database. Common practice is that all of the IP reachability information is injected into BGP, which, because BGP runs on top of TCP, is very good in terms of flow-control. So if a BGP peer wants to throttle down a neighbouring speaker that is too fast, the router simply delays the TCP acknowledgements. Delayed TCP ACKs emulate a small bandwidth link. The fast speaker will back off and reduce the pacing of the route transmission. But if BGP is so good, then why do we need IGPs like IS-IS? To answer this, you have to understand that there is some mutual dependency between the IGP and BGP. First of all, BGP runs on top of TCP, and in order for TCP to work, you need valid internal routes to get your BGP session up and running. Furthermore BGP needs some information about how far a BGP speaker is away to determine the best route – it is the IGP that sup- plies that information. On the downside BGP converges (produces consistent informa- tion in all BGP routers) very slowly particularly because it is not very good in detecting that a neighbour is down as it has very slow paced keep-alive timers. Once the BGP neighbour is determined to be down in the worst case, it can take up to 2 minutes for a BGP router to declare a neighbour down. IS-IS is much quicker (orders of 10–30 sec- onds) to detect absent peers. It is the slowness of BGP, more precisely the slowness of iBGP (BGP distributing information to the internal network), that mandates the use the Overload Bit today. Consider the scenario shown in Figure 6.9. It shows a transit provider providing ser- vice to the customer ASs 2 and 3. The iBGP connections as they should be in the converged state are represented in the diagram as dotted curves. However, the internal BGP sessions to Core Router #2 are down because we had to reboot Core Router #2 (the reason for this reboot is unimportant, but happens occasionally). The router reboots, starts its routing processes (IS-IS and BGP), and tries to get the iBGP sessions up, but it can’t! This is because IS-IS has not yet learned about the internal topology to acquire the necessary routing information in order to get the internal BGP sessions, which rely on TCP, up. IS-IS starts to discover its directly connected neighbours (Core Router #1, #3, Border Router A, B) and starts to synchronize its link-state databases. Database synchro- nization is discussed in more detail in Chapter 8. After database synchronization, the IS-IS routing process schedules an SPF calcula- tion and feeds the routing information resulting from that calculation into its local routing table. This is the beginning of the black hole state where packets flowing into a router have no place to go. Border Routers A and B immediately send Core Router #2 traffic targeted to local AS 2 and 3. The problem is that Core Router #2 does not yet have the transit BGP routes to know where to forward that traffic, as the iBGP sessions are not established yet. After a while, the sessions to the iBGP speakers are established (this can last up to several minutes) and Core Router #2 will have built up accurate forwarding states to pass on the traffic and so as not to black hole the packets any longer. What can be done in IS-IS to help avoid entering a black-hole scenario? Consider what would happen if Core Router #2 sets the Overload Bit when sending its first LSP. During the subsequent SPF calculation, the Border Routers A and B will detect the Overload Bit 160 6. Generating, Flooding and Ageing LSPs during processing Core Router #2’s LSP and will therefore eliminate Core Router #2 as a possible way to send transit traffic. The transit traffic will be routed around Core Router #2 over Core Routers #1 and #3. The good news is that Core Router #2’s local sub-nets will still be reachable even if the router sets the Overload Bit, so the iBGP sessions to all the internal routers can be established successfully during the overload condition. Most IS-IS implementations, including IOS and JUNOS software, support static set- ting of the Overload Bit. The following configuration statements show how to set the Overload Bit statically. The set-overload-bit command sets the Overload Bit for both levels of the IS-IS instance, as shown in the following: IOS configuration The IOS set-overload-bit command sets the Overload Bit persistent static and does not remove it after some time. Amsterdam# show running-config [… ] router isis set-overload-bit [… ] LSPs and Revision Control 161 FIGURE 6.9. IS-IS and BGP routing is mutually dependent on each other; if both do not converge at the same time, traffic is black holed 140K Internet Routes Core Router #1 Core Router #3 Border Router B Border Router A Core Router #2 iBGP eBGP 140K Internet Routes JUNOS software configuration In the JUNOS software, the configurations statement is very similar to the IOS one. In the JUNOS software the Overload Bit can be set statically by adding the overload configura- tion directive under the protocols isis configuration hierarchy. hannes@New-York> show configuration [… ] protocols { isis { overload; interface lo0.0; interface so-0/0/0; } } [… ] It is important to stress the static nature of the configuration directives. If you set the Overload Bit by using any of the two commands described, your router will not carry any transit traffic with this configuration. There are places where this can be helpful. One appropriate application for the static setting for the Overload Bit is dedicated devices such as BGP route reflectors, which are intentionally not meant to carry any transit traf- fic. Thus, do not start to panic if you see the Overload Bit set in your network – it is most likely set by the BGP route reflectors, which you do not want to carry any transit traffic. The static setting of the Overload Bit is also useful when doing hardware/software main- tenance work on a router. Set the Overload Bit to get rid of all the transit traffic for the time being. Finish the maintenance work and clear the Overload Bit to carry on forwarding transit traffic. Manual clearing of the Overload Bit is not always possible; what is needed is an automated way of clearing the Overload Bit after some amount of time. There are two strategies for the timed clearing of the Overload Bit: • Unconditional. Clear the Overload Bit after some (configurable) amount of time • Conditional. Clear the Overload Bit once all of the iBGP sessions are in the Up state The Unconditional option just requires that you know your network and how long it takes to collect BGP routes and populate the router with the correct forwarding state. The network administrator has to estimate a realistic value here. The Conditional option sim- ply waits until all the internal BGP connections are in the Up state. Architecturally, this approach lacks some robustness in practice: just ask yourself what will happen if one or more of the iBGP sessions do not come up. The Overload Bit is never cleared at all. It is relatively easy to come up with scenarios where the Overload Bit will never clear if one of the iBGP peers is under constant up-down-up flux. So most networks do not use the conditional approach and use an unconditional fixed time value of 300 seconds. This five-minute value is a good balance allowing time to bring up even large internal iBGP meshes, while still relatively quick to clear the Overload Bit. This dynamic setting and clearance of the Overload Bit is supported in both IOS and JUNOS software. IOS supports clearance of the Overload Bit according to both strat- egies (conditional wait-for-iBGP and the unconditional timer) during router startup. 162 6. Generating, Flooding and Ageing LSPs The Overload Bit can be dynamically set and cleared during startup using the set- overload-bit on-startup [<timeout> | wait-for-bgp] configuration command in router-configuration mode, as shown in the following: IOS configuration Amsterdam# show running-config [… ] router isis set-overload-bit on-startup 300 [… ] London# show running-config [… ] router isis set-overload-bit on-startup wait-for-bgp [… ] Juniper Networks engineers were a bit cautious about the dependency on both another protocol and on the all-or-nothing approach of waiting for all iBGP connections to come up. JUNOS, therefore, just supports unconditional clearance of the Overload Bit (Timer method). JUNOS software configuration In the JUNOS software, arbitrary timers between 60 and 1800 seconds can be specified that the router will wait until clearing the Overload Bit. Set the overload timeout <timeout> configuration directive under the protocols isis configuration hierarchy, as shown in the following: hannes@New-York> show configuration [… ] protocols { isis { overload timeout 300; interface lo0.0; interface so-0/0/0; } } [… ] Setting the Overload Bit during startup and dynamically clearing it with a timer is a useful tool to avoid black hole scenarios during transient network conditions. It is highly recommended that you configure the Overload timer on all the transit core routers. A lot has been said about the individual bits in the LSP header and the scenarios in which they are used. Next, the way that LSPs are distributed through the network will be examined in more detail. The mechanism used in IS-IS to distribute LSPs is called flooding. LSPs and Revision Control 163 6.4 Flooding Once a router has acquired its directly connected neighbours and built up adjacencies, it will tell other routers about its local adjacencies. To get the information across the whole network as quickly as possible, this information is flooded. But just what does flooding mean? First of all, the flooding algorithm is slightly different depending on if a router originates the information or if the router just receives the information and relays it further. Figure 6.10 and Figure 6.11 show the differences between the two approaches. In Figure 6.11, you can see that the router is originating an LSP. The router then sim- ply sends this LSP out on all interfaces that have an adjacency in the Up state. If a router is receiving an LSP that the router has not originated, as shown in Figure 6.11, the router will simply send the LSP out on all interfaces except the one where the router got the LSP. The name flooding is derived from the fact that a received LSP will be sent out almost unconditionally on all other interfaces. The basic flooding algorithm needs to be refined a little bit. One thing to make sure of is that the flooding stops at a certain point. If flooding does not stop, we would have an 164 6. Generating, Flooding and Ageing LSPs LSP Link Up to Router B Link Up to Router C Link Up to Router F Router A, Sequence 0x9 Lifetime 1200 s Router A FIGURE 6.10. Self-originated LSPs are flooded on links with an adjacency in the Up state endless, raging “LSP storm” circulating through the network. For example, consider a ring-like topology, as shown in Figure 6.12. Each router in the sample topology would just send the LSP farther on all interfaces, except on the one where the router received the LSP. So what controls the flooding and helps to prevent LSP distribution from exploding? Once again, it is the Sequence Number that controls the flooding. So let’s refine the flood- ing algorithm to use the Sequence Number to prevent endless flooding. First, a router must verify if a received LSP is newer (has a higher Sequence Number) than the one installed in the local database. If it is newer, then install the LSP in the data- base and send it out on all interfaces where there is an adjacency in the Up state. If the Sequence Number is less than or equal to the LSP in the database, then the router should simply discard the LSP. Going back to our example in Figure 6.12, where the update was circling endlessly, the difference can be seen. Once the update process is completed (after Step 6), then the flooding stops at Frankfurt because this router already knows about the LSP with Sequence Number 0x12. 6.4.1 Is Flooding Harmful? The refined basic flooding algorithm is a quite robust information distribution scheme. It is robust because the algorithm relies very little on other routers and thereby makes sure Flooding 165 LSP Link Up to Router A Link Up to Router J Link Up to Router K Router F, Sequence 0x4 Lifetime 1199 s Router A FIGURE 6.11. Relayed LSPs are flooded on links with an adjacency in the Up state, except on the one that originated the link that the message gets to the very last corner in an autonomous system. However, there is also a dark side to this robust scheme. This scheme is very inefficient in a densely meshed environment. Consider Figure 6.13. Here, four ATM switches interconnect a col- lection of routers. The physical connections are the black lines. The gray lines represent the connections at the logical level. Each gray line is a virtual circuit (VC) at the ATM level. This kind of networking setup was popular in the mid-1990s for various reasons, such as ATM interface speed, the absent traffic engineering capabilities of IP routers, and the limited processing power of software-based routers (ATM switches were hardware- based). You can learn more about these historical networking setups and why they are deployed in Chapter 14. For now, it is enough to note that ATM switches once connected IP routers, and sometimes they still do. 166 6. Generating, Flooding and Ageing LSPs LSP Link Up to Router C Link Up to Router B Router Frankfurt, Sequence 0x12 Lifetime 1200s LSP Link Up to Router C Link Up to Router B Router Frankfurt, Sequence 0x12 Lifetime 1200 s LSP Link Up to Router C Link Up to Router B Router Frankfurt, Sequence 0x12 Lifetime 1200s LSP Link Up to Router C Link Up to Router B Router Frankfurt, Sequence 0x12 Lifetime 1200 s LSP Link Up to Router C Link Up to Router B Router Frankfurt, Sequence 0x12 Lifetime 1200s LSP Link Up to Router C Link Up to Router B Router Frankfurt, Sequence 0x12 Lifetime 1200 s LSP Link Up to Router C Link Up to Router B Router Frankfurt, Sequence 0x12 Lifetime 1200s LSP Link Up to Router C Link Up to Router B Router Frankfurt, Sequence 0x12 Lifetime 1200 s LSP Link Up to Router C Link Up to Router B Router Frankfurt, Sequence 0x12 Lifetime 1200s LSP Link Up to Router C Link Up to Router B Router Frankfurt, Sequence 0x12 Lifetime 1200 s LSP Link Up to Router C Link Up to Router B Router Frankfurt, Sequence 0x12 Lifetime 1200s LSP Link Up to Router C Link Up to Router B Router Frankfurt, Sequence 0x12 Lifetime 1200 s 6 5 4 3 2 1 Pennsauken Frankfurt London Washington New York Paris FIGURE 6.12. The Sequence Number causes the flooding to stop . when people like the NOC-team managers and the Chief Technology Officer got paged out of bed in the middle of the night. These lessons were learned and the implementations of the routing protocols. synchronization, the IS-IS routing process schedules an SPF calcula- tion and feeds the routing information resulting from that calculation into its local routing table. This is the beginning of the black. Number) than the one installed in the local database. If it is newer, then install the LSP in the data- base and send it out on all interfaces where there is an adjacency in the Up state. If the Sequence