The IS-IS configuration looks alright – all interfaces are referenced. At the top there is a pointer to an export policy which we will examine closer. JUNOS configuration On first sight the static-to-isis policy looks good, however once you check the inden- tation of the terms and accept statements you will find out that the policy does not do what the network operator wanted it to do. hannes@Munich> show configuration policy-options [… ] policy-statement static-to-isis { term reject_management { from { route-filter 10.0.0.0/8 orlonger; } then reject; } term static { from protocol static; } then accept; } At first sight this policy looks good. However, once we start to compare the indenta- tion of the then part we realize that the term static does not have a valid then state- ment. Due to a misconfiguration, it got inserted at the wrong level in the policy. What the standalone then accept term does is accept every unicast route in the inet.0 routing tables and mark it for export into the IS-IS link-state database. Because there is no from statement at the same indentation level as the final then accept statement, we have an unconditional export of the entire Internet routing table into IS-IS. (The final “then” logic is executed when no terms match the routes. The logic is here “Is the route 10/8 or longer?” No, that’s a private address. “Is the route static?” No, it’s an Internet route. “Okay, then unconditionally accept the route into IS-IS.”) The distributed storage space that each node may allocate is 1492(–27) * 256 ϭ 375 Kbytes. How many IPv4 prefixes do fit in those 375 Kbytes? Figure 12.11 in Chapter 12 “IP Reachability Information” illustrates the structure and storage requirements of the Extended IP Reachability TLV #135. Worst case, the TLV consumes 9 bytes and best case 5 bytes due to variable prefix length packing. For the average Internet route we can assume a prefix length between /16 and /24 and safely assume a total storage requirement of 8 bytes per prefix. In a single TLV, on average, 31 TLVs fit, which requires 31 * 8 + 2 (TLV Overhead) ϭ 250 bytes to store. An LSP fragment is at maximum 1492 bytes in size. For TLV information there is 1492 – Header size (ϭ27) ϭ 1465 space. That means in total we can store 31 * 5 + 26 ϭ 181 routes per fragment. Inside 256 fragments we can store around 46 K routes, which is too little to hold the entire Internet routing table. As soon as the routers hit that limit, it pulls the “emergency brake” and sets the overload bit. 472 15. Troubleshooting Finally, it cleans up the mess by purging the previously generated LSPs off the distrib- uted link-state database. And that’s what the router was showing us. In order to fix the problem, the then accept statement is moved into the term static. JUNOS configuration hannes@Munich> show configuration [… ] policy-statement static-to-isis { term reject_management { from { route-filter 10.0.0.0/8 orlonger; } then reject; } term static { from protocol static; then accept; } } After committing the change, you will still see all those stale fragments in the data- base. They will be kept in the database until the garbage collection timer times out. Using default values, after a period of 20 minutes they are removed automatically. JUNOS command output After the router has changed, the broken routing policy the Overload Bit is automatically cleared. hannes@Munich> show isis database IS-IS level 2 link-state database: LSP ID Sequence Checksum Lifetime Attributes Munich.00-00 0x1c2 0x2d3b 1192 L1 L2 Pennsauken.00-00 0xc77 0xec5e 711 L1 L2 Frankfurt.00-00 0x198 0xdd86 933 L1 L2 14 LSPs [… ] The database looks normal again, and the Overload Bit has automatically been cleared. Because that problem was encountered many times in the field, Juniper Networks finally introduced a prefix-export limiter that optionally controls the export behaviour and suspends route export if a predefined threshold is reached. Case Studies 473 474 15. Troubleshooting JUNOS configuration The prefix-export-limit knob protects the rest of the network from a malicious policy by applying a threshold filter for exported routed. hannes@Munich> show configuration [… ] protocols { isis { export static-to-isis; level 2 { wide-metrics-only; prefix-export-limit 2500; } } } The amount of prefixes heavily depends on the size of your network. Good design advice is to set it to double the total number of IS-IS Level 1 and Level-2 routers in your network – The minimum number of routes should be 1000 and the maximum number of routes about 10,000. Then you have some growth for even larger numbers of routes that need to get leaked from Level 1 to Level 2. 15.4 Summary Most IS-IS problems can be resolved quickly if you stick to a troubleshooting plan and check from Layer-1 of the OSI Reference Model right up to the Application Layer. In IS-IS, the Application Layer represents the link-state database that holds the network’s link state PDUs. The network engineer needs to develop an understanding of what func- tions each layer is performing and what tools he has available to gather information. After information gathering, the collected data needs to be analyzed and interpreted, which requires knowledge of the show commands and debug outputs. For detecting mis- configuration on a router, the network engineer needs to understand where the IS-IS rele- vant data in the configuration are stored. The majority of IS-IS problems are related to adjacency formation. The network engineer needs to get familiar with all sorts of debug output for IOS and JUNOS. Just looking at the IS-IS specific configuration is often not enough to resolve a problem. We have demonstrated in the Internet route export case study that understanding of route export and policy processing is paramount for resolving complex problems. 16 Network Design For a long time, link-state protocols were believed not to scale. However, today there are operational networks with more than 1200 routers in a single level. Still, networks that run link-state protocols need to be carefully designed and a lot of factors need to be considered to get to such a scale. By ignoring certain reasonable constraints, you can easily break a network in certain scenarios. In this chapter you will learn about the critical IS-IS network design factors, all forms of router stress, including flooding stress, SPF stress and forward- ing state change stress, as well as what things to consider to build robust, fast-converging networks. 16.1 Topology and Reachability Information In service provider networks there are always at least two protocols in use. The first is an IGP (which could be OSPF or IS-IS), and the other is BGP. One of the first questions asked by networking novices is why do we need both? It turns out that all IGPs (IS-IS, OSPF, EIGRP) lack one fundamental thing, which is flow-control. For IGPs, there is no way to tell an adjacent router that their updates have overwhelmed the receiver and the sender should throttle down. The only way to deal with the situation is to throw away the updates and wait for re-transmission. However, that is still a dangerous game, as it may offload stress at the expense of the sending router, which needs to queue retransmissions and therefore consumes CPU and memory. Careful protocol heuristics need to be imple- mented to make sure that both the sending and receiving router do not take themselves out of service. Dave Katz, a software engineer with Juniper Networks, who can be blamed for writing the majority of IGP implementations on the Internet (his own self- definition) puts the complexity around finding the right heuristics in a single quote: Link State Protocols are hard! (Dave Katz) What network engineers at service providers have been doing is to apply a divide and conquer strategy and separating topology from reachability information. Topology infor- mation contains the skeleton of the network – it is a graph that describes how the routing- nodes are connected to each other. It does not contain any information about customer networks and server networks, or so on. Ideally, it does not even contain information about the directly connected sub-nets. Figure 16.1 shows that the only information that the routers advertise is their loopback IP address, which is necessary to bring up an iBGP full- mesh distribution network which handles bulk transport of the routing information. 475 476 16. Network Design When you run IS-IS over a link you typically advertise your local IP sub-net in your IS-IS LSPs. There is even the notion that local IP sub-nets should not be announced by IS-IS, but rather by BGP. Historically there has not been an option to preclude certain IP sub-nets from being announced. However, recent routing software allows you to change BGP BGP BGP BGP BGP BGP Washington IS-IS IS-IS IS-IS IS-IS IS-IS IS-IS 192.168.1.11/32 172.16.33.0/24 Pennsauken 192.168.1.18/32 172.16.33.0/24 New York 192.168.1.12/32 172.16.33.0/24 London 192.168.1.17/32 172.16.33.0/24 192.168.1.13/32 172.16.33.0/24 Frankfurt 192.168.1.14/32 172.16.33.0/24 Paris FIGURE 16.1. The minimal routing information that IS-IS needs to provide is the /32 of the Loopback IP address for bringing up the iBGP mesh. All customer routes are packed on BGP that behaviour. In IOS, there is a single knob that changes the advertising behaviour of directly connects sub-nets. Once you configure the passive-only knob, the routing software walks down the list of configured interfaces and looks for interfaces that are marked as passive. Recall that passive means that you include that interface’s sub-net in your routing update, but you do not try to establish a neighbour relationship or an adjacency over that interface. The loopback interface is by default passive and so if you configure the passive-only option, only the loopback IP address of the router is advertised in its LSP. IOS configuration In IOS controlling whether directly connected route get advertised is provided using the passive-only knob. New-York# show running-config [… ] router isis advertise passive-only ! [… ] In JUNOS there is no specific knob to control advertising behaviour. In JUNOS you write a policy for achieving that task. Later you call that policy as export policy in the protocols isis {} branch. JUNOS configuration In JUNOS you need to write an explicit policy that rejects all routes beside sub-nets on the lo0.0 interface. hannes@Frankfurt# show [… ] protocols { isis { export lo0-only; [… ] } } policy-options { policy-statement lo0-only { term lo0 { from interface lo0.0; then accept; } term final { then reject; } [… ] Topology and Reachability Information 477 The nice thing about the JUNOS policy is that you may explicitly control the level to suppress direct routes by introduction of a to {} statement. The following example shows how to restrict to the loopback0 interface related routes inside Level 2 LSPs only. policy-options { policy-statement lo0-only { term lo0 { from interface lo0.0; to { protocol isis; level 2; } then accept; } term final { then reject } } } [… ] } BGP has perfected flow-control capabilities because it runs on top of the Transmission Control Protocol (TCP). Flow control at the TCP level is built into the protocol: as soon as a receiver cannot keep up processing inbound routing updates, it can easily slow down transmission of acknowledgements or even drop the inbound update and indirectly indi- cate that the sender should back off and send information at a lower speed. Originally BGP was intended to process a certain maximum of routes. Yakov Rekhter, an Internet architect with Juniper Networks relates: Kirk Lougheed (Cisco Systems) and myself’s goal was to build a routing protocol able to convey 1000 routes and not fall into pieces – If you consider the total routes being today in the Internet we pushed the envelope a bit (Yakov Rekhter) Based on BGP’s superb scaling capabilities, the idea here is to “borrow” the existing BGP distribution mesh being used for transport of Internet routes for internal routes as well. The conclusion as to why you always need two protocols is therefore: IS-IS scales too poorly for conveying a bulk amount of routes, however, it can quickly discover a topology and provide routing connectivity between router loopback IP addresses. BGP heavily depends on these IGP-supplied routes to bring up the iBGP. Second, BGP is really in the dark when it comes to ascertaining the distance between a pair of routers. Internal BGP sessions are not “targeted” and therefore need an IGP to resolve routes and to give BGP speakers directions. In order to come up with a design recommendation, let’s first evaluate the forms of stress that routers are exposed to and develop a set of critical design factors based on those insights. From there we will set up some rules to follow when designing an IS-IS network. 478 16. Network Design 16.2 Router Stress Generally routing software can exhaust resources in three possible areas: 1. Bandwidth 2. CPU 3. Memory The next three sections investigate IS-IS implementations to see if they suffer from any limitations in those three areas. The first area is bandwidth – in IS-IS, the main band- width consumer is related to the flooding of LSPs. 16.2.1 Flooding Unlike link-local packets like Hellos (IIH) or Synchronization packets (SNP), transmit- ting link-state PDUs (LSPs) has a network-wide bandwidth usage impact. Once a router floods LSPs, it is using bandwidth equal to the number of links in a given topology times the size of the LSP. Worst case, it can be that network-wide transmission of an LSP comes at a cost of using the number of all links times the size of a LSP squared. The big gap between the best and the worst case (recall the best case is linear behaviour and the worst case is N^2 behaviour) is solely explainable by the way the topology is meshed. Consider Figure 16.2, where in a strict ring topology of six routers there is no duplicate Router Stress 479 Paris Ring Topology Full-mesh Topology Pennsauken New York Washington Frankfurt London Pennsauken New York Washington Paris Frankfurt London FIGURE 16.2. In a dense-meshed environment there are lots of duplicate LSPs to process transmission of an LSP. As soon as a link breaks, the LSP travels round until every node gets a copy. Note that for greater visibility the propagation of only one LSP is shown. Of course, in real networks both ends of the link that breaks would originate a new LSP. As soon as you add links to the topology, the more redundant the transmission of LSPs gets. In the ring-topology each router sees the LSP one time. The worst case is a full-mesh of all routers, where a single router failure triggers (N – 1) LSPs being flooded over (N – 2) links (ϭ O(N 2 )) through the network. The big problem in a dense- or full-mesh environment is that nodes that already got a copy of LSPs receive many redundant duplicates with the same information. An additional source of flooding stress comes from turning on the TE extensions. Once you turn on features like Traffic Engineering, DiffServ Traffic Engineering or Auto Bandwidth, then the TEDs throughout the network topology need to be updated through the use of the IS-IS flooding sub-system. That means that every router in the network sees (and needs to see) accurate TE information. However, if the TE implementation permits changes to flooding timers, then let having very conservative timers guide your design. TE extensions are a major source of LSP updates and there should be an effort to reduce these to the minimum possible. It is recommended that you consider the topology to evaluate the stress resulting from receipt of duplicate LSPs. Densely meshed environments scale poorly in flooding environ- ments. Try to avoid full-mesh or near-full mesh topologies. Sometimes a lot of extra redundancy does not turn into more resiliency. 16.2.2 SPF Stress Link-state routing protocols were once believed to be CPU intense algorithms that exhausted an embedded system’s sparse resources. Because of that belief, both link-state IGPs (OSPF, IS-IS) have provisions to split the size of the link-state domains to smaller units. In OSPF multiple areas, and in IS-IS two levels, are an attempt to spare the control plane CPU when doing the SPF run. A lot has changed in the last decade. CPUs became (in line with Moore’s Law) faster by a factor of 8000; Trunk bandwidth grew from T1 speeds to OC-192c/STM-64. The only thing that has not changed at all is the paranoid thinking that SPF may exhaust the CPU resources of a router. The fact is, the demand that SPF puts on router resources has been outpaced by the processing power of modern CPUs. Table 16.1 shows how SPF execution fares on modern route processors like the Cisco Systems GRP or a Juniper Networks RE 3.0. The CPU requirements of an SPF operation are well understood and well documented by computer scientists. The fundamental relationship is O(N * log(N )), which describes a curve where the CPU requirements grow a little more than linearly, with N being the num- ber of total routers in the network. In practice it is a little more than just log N due to the 2-way check that is needed to verify that a node is connected on both ends and not a dead end. The results from the simulation in Table 16.1 are impressive. It means that processing a grid of 2000 routers, which are in total connected by 5000 links, has a typical execu- tion runtime of only 100–245 milliseconds. If you consider this table then it is obvious that raw SPF execution time is not a problem for large IS-IS networks. So what is it then? 480 16. Network Design Why are we all so scared of routers running excessive number of SPF runs back to back? What is it besides the SPF calculation itself that scares network operators so much? 16.2.3 Forwarding State Change Stress The purpose of the SPF calculation is to find out the shortest path to every edge of the network. However, just the insight that there are better paths available is not enough. There are no good things, unless you do them! (Erich Kästner) The router has to pass on the new proximity results to a subsystem called the resolver, which is used to map third party next-hops to forwarding next-hops. Consider Figure 16.3, if the link between Washington and New York breaks, the SPF calculation will be finished in a matter of microseconds. Each IS-IS speaker is also a BGP speaker and car- ries several thousand active BGP routes. If the IS-IS topology changes, then the BGP routes that depend on IS-IS need to get changed as well. The resolver needs now to back- track through all the BGP routes and verify that the BGP next-hop is affected by a change in the core topology. As you can imagine, walking down a table of several hundreds of thousands of BGP route-entries is a resource intensive task. In our example, there are tons of forwarding state changes to do: all Washington and New York routes need to be changed in a very short time. After the BGP dependencies have been worked out, this may generate changes in the BGP topology as well: recall that the IGP distance is part of the BGP route selection process. But that is only half of the story, as those things still occur on the control plane. Router Stress 481 TABLE 16.1. Modern route processors can calculate topologies for thousands of nodes and links sub second. SPF runtime (ms) Juniper Networks Cisco Systems Routers Links Routing Engine 3.0 GRP 12000 100 250 1,92 4,80 200 500 4,97 12,42 400 1000 12,49 31,22 600 1500 21,18 52,94 800 2000 30,67 76,67 1000 2500 40,78 101,94 1500 3750 68,11 170,27 2000 5000 97,68 244,21 2500 6250 128,98 322,45 3000 7500 161,69 404,22 4000 10000 230,53 576,33 5000 12500 303,09 757,72 6000 15000 378,67 946,67 7000 17500 456,82 1142,04 8000 20000 537,19 1342,98 9000 22500 619,55 1548,86 10000 25000 703,67 1759,18 . unicast route in the inet.0 routing tables and mark it for export into the IS-IS link-state database. Because there is no from statement at the same indentation level as the final then accept statement,. have an unconditional export of the entire Internet routing table into IS-IS. (The final “then” logic is executed when no terms match the routes. The logic is here “Is the route 10/8 or longer?” No,. purging the previously generated LSPs off the distrib- uted link-state database. And that’s what the router was showing us. In order to fix the problem, the then accept statement is moved into the