troubleshooting much easier. You are doing operations and support people a big favour if you avoid fancy and complicated System-ID schemes. 16.3.7 Align Throttling Timers Based on Global Network Delay In most IS-IS implementations there are many timers that the network operator can adjust. In order to build a network that converges in the sub-second range, you often need to tweak those timers. The first thought may be the faster the better, however, that’s not always the case. The typical throttles that are on by default are LSP origination and SPF delay timers. Both JUNOS and IOS have a similar strategy to apply these throttles. Both implementations in common behave fast (almost no delay) for the first events in a series. However, the more quickly changes come, the more restrictive, and hence slower, the sys- tem behaves. This is achieved as a step function in JUNOS (the first three events are han- dled the fast way, and then the system immediately backs off to slow behaviour) and in IOS the router gets slower using an exponential curve. However, after three or four events, the system fully backs off to the slower behaviour. The art of good network design is to find a healthy compromise so that the majority (95 per cent) of network events falls under the fast window and you can take full advantage of the open throttles. Consider Figure 16.7. When parts of a network fail, then there is always more than one LSP in flight. Once the link between Washington and New York breaks, both routers have to update their LSPs. Ideally both LSPs arrive at all the routers at the same point in time. Now you need to find a compromise between reasonably fast behaviour and waiting long enough that you can catch and process an SPF run for all the LSPs belonging to a particular network fault condition. How long is long enough? If you take a closer look at how an incident is processed, then the dominating element after mutual detection of a link-break is the propagation of the LSP. LSP propagation with reasonably fast circuits (greater then OC-3/STM-1 speeds) is mostly a function of the light-speed in a fibre plus the LSP processing delay of routers. A good estimation for the average global flooding delay is the worst case delay for network control traffic, plus the average hop count, multi- plied by 10 ms (average LSP processing delay). Speed of light in fibres is about 200.000 km/s ϭ 200 km/ms Example 1: Continental network (diameter 4.000 km) spanning over average 6 routers. 4.000 km/200 km/ms ϩ 6 * 10 ms ϭ 20 ms ϩ 6 * 10 ms ϭ 80 ms Example 2: Intercontinental network (diameter 30.000 km) spanning over an average of 8 routers. 30.000 km/200 km/ms ϩ 8 * 10 ms ϭ 150 ms ϩ 8 * 10 ms ϭ 230 ms It is recommended to change the LSP origination timer to a value that safely catches most link flaps. It should not be set too aggressively. A good recommendation here is 150 ms, which gives the router enough time to fully build an LSP no matter how large it may be. It is recommended that you change the SPF delay timer based on the calculation for the average processing and LSP propagation delay. A recommendation for delaying the SPF calculation is between 50 ms and 250 ms depending on network size. 492 16. Network Design Design Recommendations 493 New York LSP Washington LSP Pennsauken Frankfurt London Washington New York Paris FIGURE 16.7. For true fast-convergence routing the throttling behaviour should match the LSP propagation properties of the network 16.3.8 Single Level Where You Can – Multi-level Where You Must IS-IS has many useful scaling tools built in. One of them is the multi-level orientation that allows having hundreds of routing domains composed of hundreds of routers and routes between them. An important question to ask yourself is how much scalability do you really need? A couple of years ago, everybody in the communication industry was 494 16. Network Design crazy over scaling, and even pushing the envelope for scalability was not enough for some network service providers. Fortunately today some normality (some say sanity) has come back to the industry, and most people now realize their true scaling needs and the associated time-span until a next generation of network and network routers are needed. According to our experience, for the majority of networks, there is a simple design rule, which is to start with a single level of IS-IS. Unless your topological domain (IS-IS level) grows to a total number of beyond 800–1500 routers, there is no need to split things up. Do not be misled and think that splitting up a single level network into a multi-level net- work must automatically improve the scaling properties in all dimensions of that net- work. There are some examples where the additional complexity of processing two topological domains may result in worse resource demands. A good case study is shown in Figure 16.8. In a given multi-level network, in order to still get optimal BGP routing, you need to turn on route leaking and propagate all /32 loopback routes. An individual /32 prefix originating in Area 49.0100 Level 1 is leaked up to the Level 2 Area 49.0001 and, due to route leaking, it will be propagated down to Area 49.0200 Level 1. If you now sup- pose (for example) that the San Francisco router goes down, then the timing properties of all the individual operations are: 1. Flood inside Level 1 Area 49.0100 (20ms) 2. Atlanta: SPF delay, local SPF calculation, leak to Level 2, LSP build delay (100 ms ϩ 10 ms ϩ 100 ms) 3. Flood inside Level 2 Area 49.0001 (20 ms) 4. Amsterdam: SPF delay, local SPF calculation, leak to Level 1, LSP build delay (100 ms ϩ 10 ms ϩ 100 ms) 5. Flood inside Level 1 Area 49.0200 (20 ms) 6. Stockholm: SPF delay, local SPF calculation (10 ms ϩ 10 ms) If you sum up all of these operations then the total Level-1-to-Level-1 convergence takes 500 ms compared to the single level flooding delay, plus SPF delay and SPF calcu- lation (130 ms). Although the network should behave better, it actually got worse by a factor of 4 in terms of convergence. Typically, you start a green-field design with a single Level-2 IS-IS network. You may ask why to start with Level 2 first and not with Level 1, and extend it later to a Level-2 network? Figure 16.9 shows the two transitioning examples. On the top, the migration from a Level-2 to a multi-level network works without any disruption at all. Only two level settings need to be changed. On the bottom the conversion from a Level-1 to Level- 2 is troublesome, because Level-1 adjacencies require that the Area address matches on both ends, and in order to create several Level-1 areas you need to renumber the areas and touch every router. This means that for a short time you must disrupt your service. In the top example we assume that the network administrator has allocated the appropriate distinct Area-IDs so that they can change the level anytime. In Level 2 routing, different Area-IDs are tolerated – the Area-ID does not need to match. A little piece of tolerance is generally a good recipe for easy transitioning, and here it helps to be able to renumber the Area-IDs without any disruption. That is good news, as you do not need to worry about addressing from the start. You can at a later stage change the entire IS-IS Area addressing and add anything at any time. 495 Frankfurt London Washington New York Paris Miami San Jose Vienna Munich Amsterdam Atlanta San Fran Area 49.0100 Area 49.0001 Level 2-only Pennsauken Stockholm Area 49.0200 F IGURE 16.8. In a two level IS-IS network propagation through all levels does slo w down convergence 2 2 2 2 1 1 1 1 1 2 2 1 1 2 2 1 Amsterdam London Atlanta New York Amsterdam London Atlanta New York Amsterdam London Atlanta New York Amsterdam London Atlanta New York 49.0100 Pennsauken 49.0001 Level 2-only 49.0200 Multi Level 49.0100 Pennsauken 49.0001 49.0200 Level 1-only 49.0001 Pennsauken 49.0100 Pennsauken 49.0001 49.0200 Multi Level F IGURE 16.9. Level 2-only to multi-level transition is easier than Level 1-only to multilevel , as it does not require renumbering Area-IDs 496 497 In the case where you have a massive number of routers, or certain smaller routers with limited CPU and memory resources, you may decide to run a multi-level network. Best current practice is to have a point of presence (POP) where all the internal routers, reflectors, access servers and servers are routed using Level-1 routing. One or two L1L2 routers provide core access as illustrated in Figure 16.10. Note the link between the two core routers – both a Level 1 and a Level 2 adjacency is established between the core routers. This is an example where one physical link can be shared between two IS-IS topologies. The purpose of the link is for backup in case there is a topology break either in the core or in the edge. A POP-centric design has the nice advantage that there is a clean separation between core and edge, which is also typically run by different teams at a service provider. So there are almost no incidents where the edge-router department at a service provider modifies core-router related configuration and vice versa (some call this the missing cross talk problem). At large service providers it often turns out that technology has to follow organizational requirements. The POP centric design perfectly reflects that idea. 16.3.9 Do Not Rely on Default Routes Default routes are a nice tool for scaling the network, particularly for routers that do not have the CPU and memory resources to load and hold full BGP Internet routing tables. POP design 49.0100 49.0900 2 2 12 1 11 11 1 1 1 L2 L1L2 L1 L2 L1L2 L1 L1 L1 FIGURE 16.10. Common multi-level designs have two L1L2 routers and intra-POP routing is provided using L1 routing Design Recommendations 497 498 However, in an Internet environment, any form of default routing disturbs more than it does good. By introducing a default route, you are also keeping out more specific entries that may contain valuable information and harm network properties like convergence. Consider Figure 16.8, where San Francisco and Stockholm are BGP speakers in differ- ent areas. In order to calculate a meaningful metric between the two BGP speakers the L1L2 (Atlanta, Amsterdam) routers leak all /32 addresses down in the Level 1. Next, suppose that the San Francisco router becomes unreachable through a topology break in the core. The Atlanta router builds a new Level 2 LSP, which does not contain San Francisco’s loopback IP address. Amsterdam quickly notices that San Francisco speaker is unreachable within the Level 2 domain and does not leak the prefix further down in Level 1. The Stockholm Level 1 router now tries to resolve the BGP routes through any other IGP route, which is in this case the default route. The default route is locally gen- erated by all L1 IS-IS routers because the L1L2 routers have the Attached (ATT) Bit set as part of their Level 1 LSP. BGP thinks that the other end is still reachable and does not realize (due to the default route) that the San Francisco router is not available anymore. If there is a direct iBGP connection between San Francisco and Stockholm, then as a last resort the BGP session will time out. However, if the San Francisco prefixes are learned via a BGP Route Reflector infrastructure, then there is no way for Stockholm to realize that a remote BGP speaker is down. Most router implementations have a configuration option that ignores the ATT Bit in Level 1 LSPs and hence suppresses generation of a local default route. Going back to Figure 16.8, that means that all Level 1 BGP speakers are configured not to listen to ATT Bits. Then the resolver immediately knows that there are no alternate paths available, and the BGP routes are immediately withdrawn and alternate paths become active. Try to design your core network so that it does not rely on a default route in any sin- gle place. Try to avoid any form of explicit 0/0 route embedded in IP Reachability TLVs and actively filter those routes on any L1L2 choke point. Also try to control all implicit default routes that may be locally generated through the ATT Bit. If there are no default routes in the core network, then your end-customers will greatly benefit from the improved network convergence. 16.3.10 Use Wide-metrics Only Both IS-IS implementations discussed in this book ship with a compatibility mode where both old-style and wide-style metrics are advertised in a router’s LSP. That dual support for both old-style TLV #2, #128, #130 and new-style TLV #22, #135 comes at some cost. The drawback is that the metrics of the old-style topology and the new-style topology have to stay congruent in order to produce non-divergent topologies. The clipping of metrics makes the new-style metrics, although they have a broader metric width, behave like the old-style 6-bit metrics and does not carry any advantage despite being compat- ible at all costs. The default settings of JUNOS and IOS are either small-metrics and compatibility mode. It is recommended to turn off generation of any form of old-style TLV. All recent routing software has support for wide metrics, and there are no disad- vantages to suppressing any old-style TLVs. Most real-world IS-IS router configurations turn off small metric generation and it is just a matter of time before router vendors pick 498 16. Network Design Design Recommendations 499 up the deployed reality and it becomes the default behaviour in JUNOS and IOS. Often downwards compatibility is preferred when changing defaults, and generally software engineers are reluctant to change behaviour. However, downward compatibility at all costs is not always the smartest way to evolve a protocol. 16.3.11 Make Use of the Overload Bit The Overload (OL) Bit was intended to be an indicator that a router ran out of resources. Today it is used for administration purposes in order to avoid persistent transit traffic or to avoid the delay of transit traffic until the BGP mesh has been brought up. It is a con- venient tool to avoid traffic flowing over certain nodes. The SPF calculation tries to find the shortest path between a pair of nodes in the IS-IS topology, however, sometimes the shortest path through a node may not be desired at all, particularly when it has to be routed over small-bandwidth routers serving (for example) as a route reflector. Consider Figure 16.11, where the core link between Router Munich and Router Frankfurt fail and the traffic can be re-routed over the dual homed route reflector. However, the route reflec- tor is just attached with OC-3/STM-1 link speed, and if inadvertently abused as a transit router, may fully congest those links. Good connectivity of route reflectors is a desired property and multi-homing is often a way of achieving this. If you statically set the Overload Bit, then all routers consider your route reflectors as non-viable transit paths for passing traffic. Any Edge Node, such as data centre access routers, may be candidates for setting the Overload Bit statically. 16.3.12 Turn on HMAC-MD5 Authentication IS-IS is believed to be an inherently secure protocol because it natively runs on OSI Reference Model Layer 2 and hence an attacker cannot inject any malicious packets from remote locations, as can be done with the IP routing protocols. But there is no such Route reflector 30% load oc-48 1% load oc-3 1% load oc-3 Frankfurt Munich RR Overload FIGURE 16.11. Static setting of the Overload Bit saves the route reflector from forwarding transit traffic 500 16. Network Design 500 thing as too much security. Part of the problem with securing IS-IS well was that until recently Cisco Systems did not ship HMAC-MD5 authentication support with their soft- ware, and simple text authentication does not yield true protection. Despite that lack, many service providers decided not to turn on authentication. But do not be misled – the security of your IS-IS network should not be regarded lightly. There are dedicated toolkits like Nemesis http://www.packetfactory.net/projects/nemesis/, which can generate all sorts of routing protocol packets, including IS-IS. There are forms of attacks known where a hacker having access to your flooding domain via some Ethernet segment can easily inject bogus LSPs and take all the routers in your network out of service. A dread- ful form of attack called the end-of-sequence-space attack is illustrated in Figure 16.12. Every 10 seconds the DIS on that LAN transmits a full CSNP which contains all LSP- IDs of the network. The attacker needs to wiretap those CSNPs and extract MAC addresses and source IDs of IS-IS routers in a LAN in order to spoof some LSPs. Next, the attacker forges so-called purge LSPs which are typically used to revoke an issued LSP before it expires (this is sometimes called premature ageing). Tcpdump output 06:56:44.928898 OSI, IS-IS, length: 54 L2 LSP, hlen: 27, v: 1, pdu-v: 1, sys-id-len: 6 (0), max-area: 3 (0) lsp-id: 1921.6800.1018.00-00, seq: 0xffffffff, lifetime: 0s chksum: 0x0000, PDU length: 27, Flags: [ L1L2 IS ] 49.0900 lsp-id: 1921.6800.1001.00-00 seq: 0x00000562, life:5017s lsp-id: 1921.6800.1003.00-00 seq: 0x0000073a, life:3110s lsp-id: 1921.6800.1006.00-00 seq: 0x00000d20, life:5219s lsp-id: 1921.6800.1034.00-00 seq: 0x000006f4, life:48962s CSNP source 1921.6800.1018.00 MAC 0090:69ab:7860 LSP lsp-id 1921.6800.1001.00-00 MAC 0090:69ab:7860 Milan 1 Milan 2 Attackers Host FIGURE 16.12. In an unauthenticated IS-IS environment it is very easy to take down an entire network using an end-of-sequence-space attack The LSP in question does not contain any TLV at all. Note the sequence number 0xffffffff – this is the highest sequence number possible. When the packet is flooded through the Level 2 domain, it is ultimately received by the router with System-ID 1921.6800.1018. This router realizes that it was not the originator of that packet. Whenever a router sees non-self originated LSP carrying their very own System-ID number, they try to increment the sequence number and re-issue an accurate version of their LSP. However, that is not possible any more because the sequence space is exhausted. ISO 10589 requires that a router that reaches end-of-sequence space goes dormant for a max-LSP-age period. Most routers are configured with a max-LSP-age of 65,535 seconds, which is 18.2 hours. Can you imagine the turmoil if all your routers go dormant for 18.2 hours and your iBGP distribution mesh falls to pieces? And it does not take much in the way of resources to achieve that – just an open Ethernet segment and a possibility to inject spoofed packets. Realize the small effort involved: no brute force Denial of Service attack is needed to take down an entire service provider’s network, just a few hundred spoofed LSPs that can easily be generated with common security tools. Security through obscurity does not generally work well. Running an IS-IS network without proper authentication using cryptographically strong algorithms like HMAC- MD5 is grossly negligent. Given the potential threat of an attack to the IGP, every net- working architect should be concerned and it should be mandatory to turn on IS-IS strong authentication. 16.3.13 Turn on Graceful Restart/Non-stop Forwarding Once the IGP fails, a network is literally breaking into pieces. If the IGP goes down, the BGP mesh will fail, and subsequently routing and forwarding is severely disrupted. Modern routing software is written where some components share a common fate. If, for instance, there is a bug in the resolver, then the entire routing process will crash. After a couple of seconds a watchdog component may find out that the process has been falling on its face and restart the faulty routing sub-system. The software will behave like on a cold start and send out IS-IS Hellos with no neighbour-related state mentioned in the adjacency-related TLVs (#6, #240), which will cause neighbouring routers to drop those adjacencies. Graceful restart/non-stop forwarding helps to protect the IGP subsystem in case of a failure. After being restarted, the router will request a graceful restart by setting the Restart Request Bit in TLV #211, which will cause neighbouring routers that support graceful restart/non-stop forwarding not to take down adjacencies for a grace-period (typ- ically 180 seconds). Graceful restart/non-stop forwarding is all about tolerance of nodes that have failed previously. It is recommended to turn graceful restart/nonstop forwarding on wherever possible. You gain more availability of your network in case of local software failure and the problem does not spread network-wide anymore. 16.4 Conclusion The biggest improvement for getting to a scalable IS-IS network is separating IP reach- ability information from IS reachability information and do not let IS-IS carry IP routes, Conclusion 501 . topologies. The purpose of the link is for backup in case there is a topology break either in the core or in the edge. A POP-centric design has the nice advantage that there is a clean separation. does not leak the prefix further down in Level 1. The Stockholm Level 1 router now tries to resolve the BGP routes through any other IGP route, which is in this case the default route. The default. all L1 IS-IS routers because the L1L2 routers have the Attached (ATT) Bit set as part of their Level 1 LSP. BGP thinks that the other end is still reachable and does not realize (due to the default