40 K prefixes, the total convergence time until the last prefix is updated takes about 40 seconds (if not minutes) depending on the router hardware. Updating forwarding tables under load is hard and quite expensive to achieve. 10.5.2 Hierarchical Forwarding Table The alternate solution does not map the prefixes directly to the forwarding next-hops. There is instead the notion of an indirect next-hop. An indirect Next-hop is the originator of a prefix that is not directly connected to the router. It is a different way of modelling the dependency from BGP routes to IS-IS routes at a forwarding table level. Consider the sample network again, where there are six BGP speakers. Therefore the number of indirect next-hops equals six. Figure 10.21 shows how the forwarding table is implemented. The 120,000 prefixes do not point to the forwarding next-hop directly anymore, but rather to their corresponding indirect next-hops. The indirect next-hops finally point to the forward- ing next-hop. This is the key point: 120.000 interfaces point to six indirect next-hops, which point to only three physical interfaces. What is gained from that, except another memory lookup in the forwarding path to find out the indirect next-hop? Actually, a lot is gained once it comes down to changing the fowarding state. Look at the previous example where the link between Washington and Frankfurt fails and assume the forwarding table is structured as in Figure 10.21. After LSP propagation, a full SPF run, and BGP recursion some BGP speakers (most notably Washington) are found to have a different path. Previously the link outage traffic to Washington has been routed straight, but now it is re-routed via Paris. The good news is that now at least 40.000 pointers that point from the prefixes to the forwarding next-hops do not have to be changed. The only thing that the router needs to do is flip one pointer! The hierarchical scheme does trade some forwarding lookup latency (less than 1 microsecond) against maintainability and convergence of the forwarding tables. However, given that there are multiservice networks deployed today transporting all kinds of traffic, convergence considerations are always important and need to be considered. There is one case when the abstraction of indirect next-hops to next-hops does not help, which is the case when an indirect next-hop (a BGP speaker) goes down. Then BGP needs to find out during the recursion run which BGP routes have to change their next- 278 10. SPF and Route Calculation Forwarding engine 81.21.0.0/20 192.168.0.8 so-7/3/0.0 100s of BGP Next-hops100000s of Prefixes 10s of forwarding Next-hops FIGURE 10.21. In a hierarchical forwarding table a prefix points first to an indirect next-hop, which maps finally to a forwarding next-hop hop information and reassign these prefixes to other indirect next-hops. Ultimately this will result in a massive amount of forwarding state changes at the forwarding plane. The speed of the raw forwarding table changes at about 5000–6000 prefixes per second. A common design practice is (if possible) to balance the number of active BGP paths evenly across a network by incorporating a good and solid peering mesh with other ASs. Reducing the number of active routes per-AS makes sure that if an indirect next-hop (BGP speaker) goes down, the re-routing is done within several seconds. 10.6 Conclusion The CPU impact of the SPF algorithm really has been “demonized” in the past. Fears that an SPF calculation may lock up a router entirely are not justified today. Modern control plane processors have sufficient CPU power to recalculate even large topologies in the sub-second range, so the bare SPF algorithm itself is not the problem anymore. The associ- ated BGP recursion check can be completed in sub-second range as well. What persists as a challenge is the final stage of the post-processing cycle of the SPF results. Forwarding table maintenance in modern routers can fully engage the CPUs for several seconds, much more than SPF ever could do. This challenge can be tackled from two sides: on the vendor side, by applying clever implementation techniques like indirect Next-hops in the forwarding path and, on the service provider side, by doing a proper network design. With proper balancing of active BGP paths evenly across the network, a network carrying even 120,000 routes converges in a both fast and stable manner. Conclusion 279 11 TLVs and Sub-TLVs Charles Darwin’s classic theory of evolution and its basic mechanism of natural selection through mutation can be re-applied to technology and the Internet. As the environment changes, entities within the environment must change as well or become extinct. Indeed, the relationship between the evolution of living creatures and the evolution of Internet technology is impressive. Each age of technology builds upon the discoveries of the pre- vious age resulting in a constant change, which also helps humankind to adapt to or cope with their technological environment. Experience from the last decade of networking has shown that Darwin’s Law is just as present and active in networking as anywhere else in life. When Darwin is translated to the routing protocols environment this means that not the strongest, most lightweight or most optimal routing protocol, algorithm and imple- mentation at a fixed, given time will prevail in the end. The routing protocol that wins the technology prize for survival of the fittest will be the one that adapts best to its changing Internet environment and can add features mandated by passing time. Based on the his- torical development of the OSPF routing protocol, it will be shown why Darwin’s Law is valid in the technology environment. 11.1 Taxonomy for Extensibility First of all, the term extensibility as used in this chapter will be defined in more detail. In any discussion of extensibility, there are three things that need to be considered: • How do current software maturation models work? • What are the ramifications if a routing protocol is barely or not extensible at all? • What does it mean when a routing protocol is called extensible? Software quality assurance is still a young topic in computer science. This is especially true for the additional complexity of distributed systems, where one failing node may impact others, no really clueful model has been found. Best common practice to ensure software quality is hardening by deploying. This essentially means that if software is deployed frequently, all the inherent bugs will finally go away after a maturation cycle. In the next paragraph, a typical software maturation cycle is described. 11.1.1 Current Software Maturation Models The most common software maturation model applicable in the last decade is briefly described in Figure 11.1 and explained as follows: 1. Develop the base (alpha) code. This is the first engineering cut at the program and will not be released publicly. This software is tested at the router vendor’s internal test bed. 281 Once bugs are discovered they will be fixed. Typically the alpha code testing efforts are accomplished in a few days or at most a couple of weeks. 2. Bring the code into a closed environment and test functionality. This is often called the “beta” program. The beta code is usually released to friendly customers. The goal of the beta program is to get input from the various customers’ testing labs. Each of these testing labs will focus their testing efforts of the feature or feature of most interest to the particular customer. The customer will test the beta to see if the features of interest are correctly supported, and working properly. 3. Gather problem and bug reports, then apply fixes. This is the feedback phase of the beta program. Typically lots of bugs are filed and fixed in this phase. Generally, the more bugs found at this stage, the more stable the release will be later. The customers will get the fixes to test against their respective network environment before the code is released. 4. Release the code to deploy it on a larger scale. Although closed environment testing is fine, and it is always a good idea to do some base-line testing, there is no better test than to expose the code to the real world. Here for the first time a routing protocol leaves the safety of the pool and is hit by the crashing waves of the Internet. Now there are things like constant BGP updates, flapping links causing IGPs to recalculate their SPF graph every couple of seconds, and many other events all putting stress on both router hardware and software. This is the phase where new, undocumented problems are found, and often actions that are nowhere mentioned in any specification. These new and unexpected issues sometimes lead to entire software projects being dismissed because of the problems that no one was aware of during the writing of the internal design specifications. 5. Gather problem and bug reports, apply fixes. Not all problems are discovered on Day One of an ongoing deployment – Murphy’s Law is also, and especially, valid in the 282 11. TLVs and Sub-TLVs 1. Develop base (alpha) code 2. Bring it into a closed environment to test functionality (beta program) 3. Gather problem and bug reports, apply fixes 4. Release the code, deploy it on a larger scale 5. Gather problem and bug reports, apply fixes 6. Repeat steps 4–5 several times Maturity loop 1 2 3 4 5 6 FIGURE 11.1. The maturity cycle of IP routing protocol software networking environment. Typically, more problems are discovered when the last few routers in a customer’s network are updated with the new software. If there is improp- erly functioning software in the customer’s live network, then it is the router vendor’s responsibility to provide fast, “hot” fixes for the problems being discovered. This is a scary phase for the ISP, because sometimes the hot fix has not been subjected to the full internal test bed, mostly because of urgency to bring the fix into the field. A full regression test of routing software can last from a few days to many weeks, depending on the complexity of the overall software architecture. 6. Repeat steps 4 and 5 (deploy and fix) several times. This is probably the most important phase of all. Because this software development activity concerns the Internet, an ever expanding and growing thing, there is no such thing as software that is working and complete at the same time. It might be that during the software life cycle the Internet as a whole or the ISP’s network is hitting a new scaling barrier for the routing software. This is the development phase that one large router vendor refers to as the “Internet classroom” where both ISPs and router vendors have to learn every day about the changing environment. But this results in a feedback loop of constant improvement through extensibility. The full deployment and fixing steps in the process and the repeated iteration of these key deploy and fix steps are increasingly important for the overall quality and maturity of the routing protocol software. Even if this software maturation model sounds expensive at first for the customers, who are asked to deploy software that is admittedly incomplete, there is really no other way to complete the process. Some customer agents buying soft- ware and hardware from the router vendors might ask if the software should or could be tested better and more rigorously before going into general release for customer use. This is a valid point. But experience has shown that even the most rigorous of testing environ- ments cannot detect all bugs or lack of needed functions in a routing protocol. Now, for the first public release after the beta-program, enough experience and testing should have been done to avoid a large percentage of “severe” bugs, which are bugs that cause the failure of the routing protocol and/or router. But less severe bugs, or feature shortcomings in the routing protocol, often only reveal themselves during extensive and large-scale operations in the Internet environment. What the deploy-and-fix steps really boil down to is the repeated application of the fol- lowing related steps: • Deploying routing software • Getting further experience (sometimes based on a slightly changed environment), and • Improving routing software These steps are very important for the maturity – and continued survival – of the proto- cols. Ideally, routing protocols should be almost like good wine – the older, the better. 11.1.2 Ramifications of Non-extensible Routing Protocols The allowance for constant improvement of the routing protocol through added features and bug fixes forms the essence of the extensible routing protocol. How do non-extensible Taxonomy for Extensibility 283 routing protocols negatively impact the maturity cycle of routing protocol software? A routing protocol becomes stable after spending enough cycles in the maturity (deploy- and-fix) loop. So what happens if this maturity cycle is disrupted by (for example) the need to add new features to the protocol? It is clear that there will always be enhancements, new capabilities and features requested by customers dictating innovation by voting with their dollars. No router vendor (or any type of company, for that matter) can withstand customer- originated pressure and refuse to add needed and desirable functionality to an already stable protocol. Ironically, one of the toughest challenges for the router vendors is to strike a bal- ance between the customers desiring rock-solid, stable routing protocols and the customers at the edge of the technology pushing for innovation. So the question turns into “How do I introduce new functionality without harming the existing code base?” It is important to real- ize what the last part of this question states – due to the prevailing software development model, vendors do not want to disrupt the maturity cycle by creating something radically different, and so incompatible, with the existing routing protocol. It turns out that this desired property (extensible, but not harmful) is solely dependent on the routing protocol’s architecture. This architecture determines how easy it is (or is not) for the developer to incor- porate new features into the routing protocol. First of all it is hard to extend a protocol whose architecture was never prepared for extension. The ramification of this uphill battle will be additional demand for time and resources for bringing the protocol to a mature state, which may delay new enhancements to your network. Competitively speaking, it may be that a competitor is already provisioning services while other vendors are still testing in the labs to verify the protocol and the accompanying new features are prime-time ready. 11.1.3 What Does it Mean When a Routing Protocol Is Called Extensible? How can anyone tell if a routing protocol is really extensible or not? There are two places to look at in order to determine if a routing protocol is friendly to the developer (that is, ready for extensions). The two places are: • Hellos and Capability Announcement messages • NLRI (Network Layer Reachability Information) messages 11.1.3.1 Hellos and Capability Announcement Messages Hellos are the packets that are regularly exchanged between routers to determine if there is basic connectivity between the routers. Since these types of packets are common and exchanged whenever adjacencies are established between routers, Hello packets are also a good place to indicate new capabilities in the routing protocol. These capabilities might include, but are not limited to, things such as: • Support for additional networking protocols beside IPv4 (for example, IPv6, CNLP, IPX, and so on). • Changes and modifications to the protocol interaction (for example, the basic hand- shake procedures for the protocol). • Changes and modifications to the topological understanding of the routing protocol (for example, support for multiple topologies and link types, more than one area, and so on). 284 11. TLVs and Sub-TLVs • Changes and modifications to the basic timers such as establishing a different baseline for the timers. Think of an extension to the routing protocol that allows for sub-second convergence and requires other routers to re-interpret their own timers, sometimes in violation of the original specification. For instance, a router wanting to indicate that the dead-timer value of 300 should not be interpreted in units of seconds (300 seconds), but should be now interpreted as a 300-millisecond dead-timer. • Changes to the authentication scheme used on the link (for example, public-key authen- tication used instead of a simple password). • Non-stop forwarding (also known as graceful restart) behaviour that allows neigh- bouring node to still forward traffic through a router with a failing control plane, as described in draft-ietf-isis-restart-05. 11.1.3.2 NLRIs NLRI (Network Layer Reachability Information) is a term borrowed from BGP and used with IGPs like ISIS and OSPF. But the idea of an NLRI can be used for virtually any net- working and routing protocol that has to pass on OSI-RM Layer 3 (L3) network prefixes (routes). The NLRI is basically a per-Layer3 protocol envelope for passing on reachability information (“if you have any packets for the IP addresses that follow, you can send them here. I know how to reach them.”). Keeping the eventual need to transition to IPv6 in mind, any protocol should support multiprotocol extensions to a degree that one can convey both IPv4 and IPv6 NLRIs concurrently. The concept of a per-protocol container is needed because, virtually every network layer protocol has different L3 address formats and rules that routers follow to parse them. For instance, the bit string 192.168.1/24 could and does mean something totally different in IPv4 than in IPv6. So each prefix has to be packaged in a dedi- cated envelope telling the other routers what network layer protocol the prefix is targeted for. It would be nice if Hellos and NLRIs formed two completely independent mechanisms of extensibility. Unfortunately, Hellos and NLRIs are related to each other. For example, if a node sends IPv6 reachability information inside an IPv6 NLRI to another router, the receiving router has to have a way of finding out ahead of time which of its neighbours can forward IPv6 in order to route the traffic closer to the destination. So a single look to see if a routing protocol supports a different NLRI type is only half the story. This makes it clear that a truly extensible routing protocol must easily accommodate both new cap- abilities in its Hello Messages as well as support for different NLRIs during route prefix update exchanges to facilitate extensions. Two routing protocols often cited as models of extensibility and for their ease of adding new features while remaining stable are OSPF and IS-IS. The rest of this chapter is an analysis of how OSPF and then IS-IS achieve their extensibility. 11.2 Analysis of OSPF Extensibility The analysis in this section is based on RFC 2328 “OSPF Version 2” Appendix A “OSPF Data Formats”, and also RFC 2370 The OSPF Opaque LSA Option. What this section basically does is try to pose some hypothetical questions of the routing protocol. For instance, we could ask, “in what parts or fields of the routing protocol could Analysis of OSPF Extensibility 285 OSPF extensions be placed?” Now, no one really would do this in real life. This procedure is just an exercise for the sake of finding out if a protocol is truly extensible or not. If there is no place to put extensions, then the routing protocol is just not extensible, no matter what the standard might say in this regard. Figure 11.2 shows a generic OSPF header and an OSPF Hello in detail. One feature of the OSPF routing protocol is immediately obvious just by glancing at the figure. The OSPF routing protocol has a frame structure aligned to 32-bit (4 bytes) boundaries. Today, 32-bit alignment is merely a legacy from the times when CPU processing power was small and realigning a byte stream would have had a serious CPU impact. This optimiza- tion effort is considered to be a non-issue in modern networks. For CPU architectures that store 32-Bit integers in non-network order format (like for example the Intel x86 Architecture) this optimization is even more irrelevant because for network order to host order conversion the stream is read in 8-bit quantities anyway). The generic OSPF header as well as the OSPF Hello header has some occurrences of IPv4 addresses without explicit fields establishing that these addresses are in the IPv4 format. This is a bad thing, because it means that OSPF cannot send Hellos for any other address family such as IPv6. How would the parser in the receiving router know what’s inside a field like the Designated Router field without implicitly expecting an IPv4 address? What if inside the address field is an IPv6 address, or even something else? Simply put, current OSPF extension mech- anisms cannot be extended in that way to indicate multiple address families in the header. 286 11. TLVs and Sub-TLVs Neighbour List Version Router ID Area Hello Interval 2 Bytes 2 4 4 2 2 8 4 2 Checksum Authentication Data Network Mask Options 1 4 Priority Dead Interval Packet Length Type Authentication Type 1 1 1 1 Designated Router Backup Designated Router 4 4 N * 4 Bit 0 1 2 3 4 5 6 7 Option Bit Name Reserved Opaque Demand Circuit External Attributes NSSA support Multicast Support External Type of Service Support Options FIGURE 11.2. The OSPF Hello packet has no provisions for optional fields and the 8-bit options field is almost fully populated But what about using fields after the Hello message? There is plenty of room in an IP packet (65,515 bytes for the IPMAX size field in the IP packet header. A good idea, but it won’t work. The last field in the Hello message is the Neighbour List. The Neighbour List is an implementation of a 3-way handshake protocol. Consider when Router A and Router B first communicate. In the Hellos exchange, the Neighbour List basically lists routers, including Router A, by IP address in Router B’s Hellos in order to indicate that there is bi-directional connectivity between Router A and Router B. However, the Neighbour List lacks a Length indicator field. How does the receiver then know where the “Neighbour List” ends? Recall that the software parsing the Hello knows the length of the entire OSPF packet. Knowing this, the router simply subtracts the OSPF generic header plus the OSPF Hello, which are in total 44 bytes. The result divided by four (each entry in the Neighbour-ID list is a 32-bit IPv4 address) and so reveals the number of neighbours that this specific router has “seen”. From a purely bit-field perspective the OSPF Hello is not extensible at all. This is because neither re-interpretation of existing fields nor attaching new fields at the end of the frame works to extend OSPF to other address families without breaking the base OSPF proto- col. If such a multi-family feature were added to OSPF, the result would be incompatible with older versions of OSPF. Right from it first version, OSPF included an 8-bit field called the “Options Field”, which can be seen in Figure 11.2. Throughout the last decade, the Options field has been utilized for several extensions to OSPF. These extensions include: • O – The router supports the Opaque LSA Types 9,10,11 which are mainly used for Traffic Engineering applications • DC – Demand circuits (circuits that are not up all the time, such as dial-ups) • EA – External LSA (deprecated, this option must no longer be used) • N/P – NSSA support (allows external routes in a stub area) • MC – Multicast OSPF (using OSPF to distribute multicast routing information) • E – Indicating ASBRs (routers redistributing routes from another routing source such as RIP) • T – TOS (type of service) routing support As shown in Figure 11.2, the Options field has only three unused (Reserved) bits left. So there is not-much room to extend the protocol in a backward-compatible way. The option negotiating mechanism in OSPF is solved quite nicely because a router receiving Hellos and not having certain capabilities replies in its own Hellos with the Options field bits that are not supported cleared. In the common OSPF frame there are fields indicating the Version and the Packet- Type. In Figure 11.2, the Packet-Type for the Hello is set to 1 (Hello Packet-Type) and the Version field is set to 2 indicating OSPF protocol version 2. Mention has just been made of the OSPF Packet-Type field. A value of 1 indicates an OSPF Hello packet. In the base specification OSPF supports five different packet types: 1. Hello 2. Database description 3. Link-state request Analysis of OSPF Extensibility 287 4. Link-state update 5. Link-state acknowledgment Database description, link-state update, and link-state acknowledgment are purely for synchronizing the link-state databases on demand. The element carrying network layer reachability information, the main source of detailed link state information, is the link- state update packet. If OSPF wants to announce IP prefixes and their related metrics, the vehicle to get this information across to other OSPF routers is the link-state update packets (LSAs). The next step in exploring the extensibility of OSPF is to make sure the link-state update packet can be extended. Figure 11.3 shows that there is a field called Link-State Type. Simply adding new Link-State Types can extend OSPF. It is not necessary to go deeper into what link-state types exist today and what the various packet formats look like to understand that this is how OSPF can be extended for new features and functions. This information is available from a number of sources. After all, this is a book about IS-IS and not a book about OSPF. 288 11. TLVs and Sub-TLVs Version Router ID Area LSA age 2 Bytes 2 4 4 2 2 8 4 2 Checksum Authentication Data LSA Count Options 1 4 LSA Type Link State ID Packet Length Type Authentication Type 1 1 1 4 Advertising Router Sequence Number 4 4 Checksum Length 2 2 FIGURE 11.3. The LSA type field would be a possibility for extending OSPF . testing, there is no better test than to expose the code to the real world. Here for the first time a routing protocol leaves the safety of the pool and is hit by the crashing waves of the Internet and complete at the same time. It might be that during the software life cycle the Internet as a whole or the ISP’s network is hitting a new scaling barrier for the routing software. This is the. However, the Neighbour List lacks a Length indicator field. How does the receiver then know where the “Neighbour List” ends? Recall that the software parsing the Hello knows the length of the entire OSPF