690 DEPLOYMENT CONSIDERATIONS Table 13.1 Traffic matrix for the long-haul mesh network case study. The fiber topology is shown in Figure 13.8(a). The traffic is shown in terms of the number of 10 Gb/s wavelengths between pairs of nodes in the upper-right triangle of this matrix. Node Node Destination Node Number Total Number Name 12345 6 78 91011 1213 1415 161718 19 Traffic 1 Seattle 2 San Francisco 3 Los Angeles 4 Salt Lake City 5 E1 Paso 6 Denver 7 Houston 8 Dallas 9 Kansas City 10 Chicago 11 Nashville 12 Atlanta 13 Tampa 14 Miami 15 Charlotte 16 Philadelphia 17 New York 18 Boston 19 Cleveland 0222212331 3 3 2 1 3 3 2 3 3 41 0032333321 1 1 1 1 1 2 3 2 1 35 0001123113 1 2 1 3 1 1 3 1 3 33 0000211321 2 3 1 2 1 2 2 3 1 32 0000012322 3 2 1 1 3 2 1 2 1 34 0000002232 1 3 2 2 3 1 1 2 2 34 0000000132 2 3 3 3 3 2 3 2 1 41 0000000012 1 3 1 2 1 3 1 1 1 33 0000000001 2 3 3 1 1 2 3 1 1 35 0000000000 1 3 2 2 3 3 3 3 1 36 0000000000 0 2 3 3 1 2 2 3 1 34 0000000000 0 0 2 1 2 3 1 1 2 40 0000000000 0 0 0 3 2 1 2 2 3 35 0000000000 0 0 0 0 2 1 2 2 1 33 0000000000 0 0 0 0 0 1 1 1 2 32 0000000000 0 0 0 0 0 0 2 1 1 33 0000000000 0 0 0 0 0 0 0 2 3 37 00001300000 0 0 0 0 0 0 0 0 2 34 0000000000 0 0 0 0 0 0 0 0 0 30 as ultra-long-haul (ULH) systems. We also look at the benefits of different types of protection architectures. The network of Figure 13.8(a) has 19 nodes and 28 links interconnecting the nodes. Table 13.1 shows the assumed traffic matrix between the various nodes in terms of 10 Gb/s channels. The total end-to-end traffic amounts to 3.31 Tb/s and represents a fairly realistic network in the 2002-2003 time frame. The first step in the design process is to route the end-to-end traffic and determine the amount of working and protection capacity required. Sophisticated algorithms are used to perform this function in practice, but we use fairly simple algorithms for this study. For 1 + 1 protection, we have to calculate a pair of working and protection paths which are node disjoint, that is, do not have any intermediate nodes (and links) in common. This ensures that the protection path will be available in case a node or link along the working path fails. We choose the working path as the shortest-length path between the end nodes. To calculate the protection path for a given pair of 13.2 Designing the Transmission Layer 691 end nodes, we delete the intermediate nodes in the working path between those two nodes, and calculate the shortest-length path in the resulting topology. For shared mesh protection, we use the same working and protection paths as in the 1 + 1 protection case. However, we do not need to allocate protection capacity for each path separately. Instead we provide only as much protect capacity as is needed to reroute the working paths affected by a single link failure. To do this, we calculate the protection capacity required on the links for every possible link failure and take the maximum over all possible link failures. Table 13.2 shows the assumed link distances and the number of 10 Gb/s wave- lengths required on each link as a result of the routing and capacity allocation discussed above. Even though the end-to-end traffic requirement between any pair of nodes is no more than 30 Gb/s (three 10 Gb/s wavelengths), there are several links that carry more than 100 wavelengths (or equivalently over i Tb/s of capacity). For example, the Denver-Kansas City link carries 77 working wavelengths and 78 pro- tection wavelengths (in the case of 1 + 1 protection), or 41 protection wavelengths (in the case of shared mesh protection). In many of these links, we will end up using multiple WDM systems in parallel to meet the capacity demand. We assume each of the 19 nodes has one or more electrical core crossconnects. The crossconnects terminate all the traffic at the node, including both traffic pass- ing through the node as well as traffic being added/dropped at the node. Thus, there is no optical passthrough at the nodes. Table 13.3 shows the number of crossconnect ports required for the 1 + 1 and shared mesh protection cases. Each node requires a few hundred such ports. For 1 + 1 protection, the largest node is Nashville, which has 566 ports and handles 5.66 Tb/s of traffic. For shared mesh pro- tection, the largest node is Kansas City, which has 413 ports and handles 4.13 Tb/s of traffic. The next step in the design is to cost out the network, based on the type and quantity of equipment deployed at all the sites. Table 13.4 shows the capabilities and costs of the LH and ULH systems assumed for this study, as well as the crossconnects. Table 13.5 shows the quantity of different types of LH and ULH equipment and crossconnects required to support the link distances and capacities shown in Table 13.2. Figure 13.12 shows the corresponding network costs in graphical form and illustrates how the network cost varies with the different options as well as the cost breakdown among the various components. Observe that both ULH and mesh protection provide cost savings. Also, with this model, the amplifier cost is relatively small compared to the cost of transponders/regenerators and crossconnects. Note that we have assumed the use of crossconnects for both the 1 + 1 case and the shared mesh case. Crossconnects are essential in the shared mesh scenario, as they are the ones that provide this capability. However, 1 + 1 protection can be implemented directly by the transponders, and we do not need crossconnects for this purpose. 692 DEPLOYMENT CONSIDERATIONS Table 13.2 Link distances in the network topology of Figure 13.8(a). Also shown are the number of wavelengths required on each link to support the working traffic and the protection traffic for the cases of 1 + 1 and shared mesh protection, assuming the traffic matrix of Table 13.1. Link Length Working Protection Protection (km) Capacity Capacity Capacity 1 + 1 Shared Mesh Seattle-San Francisco 1235 4 43 33 San Francisco-Los Angeles 616 35 12 10 Seattle-Salt Lake City 1251 25 35 16 Los Angeles-Salt Lake City 1073 44 37 22 Seattle-Chicago 3146 12 39 39 Salt Lake City-Denver 693 91 28 14 Los Angeles-E1 Paso 1294 14 54 52 E1 Paso-Denver 1011 21 77 37 Denver-Chicago 1657 45 27 18 Denver-Kansas City 999 77 78 41 E1 Paso-Houston 1213 19 71 35 Houston-Dallas 409 44 46 30 Dallas-Kansas City 805 42 50 15 Dallas-Atlanta 1305 19 43 26 Kansas City-Nashville 873 68 102 80 Kansas City-Chicago 750 26 61 29 Nashville-Atlanta 388 63 47 14 Atlanta-Tampa 742 48 30 14 Tampa-Miami 370 33 45 25 Miami-Charlotte 1183 14 64 48 Nashville-Charlotte 598 30 108 47 Charlotte-Philadelphia 814 18 74 47 Nashville-Cleveland 816 27 87 35 Boston-Cleveland 1020 26 11 8 New York-Cleveland 751 50 59 22 Philadelphia-New York 165 35 57 25 New York-Boston 343 8 29 26 Chicago-Cleveland 546 89 51 42 At the intermediate nodes, passthrough connections can be patched through using manual patch panels. However, if full flexibility is desired in provisioning end-to-end connections, then crossconnects will be needed in both cases. The outcome of the study depends critically on the relative cost and capabilities of different types of equipment, and the routing algorithm used. For instance, we have assumed that there is a small premium in cost for ULH amplifiers and transponders relative to their LH counterparts, and a small decrease in number of wavelengths 13.2 Designing the Transmission Layer 693 Table 13.3 Number of crossconnect ports required at each of the 19 nodes in the case of 1+1 and shared mesh protection. In 1 + 1 protection, each add/drop wavelength consumes three crosscon- nect ports, one for the local add/drop, one on the working path, and one on the protection path. The passthrough traffic consists of both working and protection traffic not terminating at the local node. In the shared mesh case, each add/drop wavelength consumes one port for the local add/drop and one additional port for the working path. The passthrough ports include ports to carry all the working traffic passing through the node, as well as all the ports reserved for shared protection. 1 + 1 Protection Shared Mesh Protection Node Add/drop Passthrough Total Passthrough Total )~ )~ Ports )~ Ports Seattle 41 76 199 88 170 San Francisco 35 24 129 47 117 Los Angeles 33 130 229 144 210 Salt Lake City 32 196 292 180 244 E1 Paso 34 188 290 144 212 Denver 34 376 478 310 378 Houston 41 98 221 87 169 Dallas 33 178 277 143 209 Kansas City 35 434 539 343 413 Chicago 36 278 386 264 336 Nashville 34 464 566 330 398 Atlanta 40 170 290 144 224 Tampa 35 86 191 85 155 Miami 33 90 189 87 153 Charlotte 32 244 340 172 236 Philadelphia 33 118 217 92 158 New York 37 164 275 129 203 Boston 34 6 108 34 102 C|eveland 30 340 430 269 329 per system. If the relative cost changes, the study conclusions can change quite substantially. Figure 13.13 plots the relative cost of LH and ULH options as a function of the relative cost of transponders (and regenerators) and amplifiers. We have only touched some of the issues affecting network design. A number of additional factors need to be taken into account while designing a more realistic network: 9 We can use LH systems on shorter links and ULH systems on longer links to optimize the cost further. 9 Many ULH systems include optical add/drop capability to pass through signals at intermediate nodes in the optical domain, rather than requiring all wavelengths to 694 DEPLOYMENT CONSIDERATIONS Table 13.4 Characteristics of the equipment used in the backbone network study. All costs are in thousands of U.S. dollars. The ULH amplifier and transponder costs are somewhat higher compared to their LH counterparts, and the ULH system has fewer wavelengths than the LH system. For terminals (including transponders), regenerators, and crossconnects, there is a common equipment cost, and in addition a cost per port equipped. For example, an LH terminal equipped with 10 transponders would cost $800,000, and a crossconnect equipped with two ports would cost $380,000. LH System Number of wavelengths per system Spans between regeneration Terminal common equipment cost 10 Gb/s transponder cost Regenerator common equipment cost 10 Gb/s regenerator cost Amplifier cost 80 6 x 80 km (640 km total) $200 $60 $200 $100 $200 ULH System Number of wavelengths per system Spans between regeneration Terminal common equipment cost 10 Gb/s transponder cost Regenerator common equipment cost 10 Gb/s regenerator cost Amplifier cost 60 25 x 80 km (2000 km total) $200 $7s $200 $125 $240 Crossconnect Number of 10 Gb/s ports Common equipment cost Cost per 10 Gb/s port 128 $300 $40 be terminated. This capability can be used to reduce the nodal costs by eliminating some of the transponders required to terminate the passthrough traffic. In this case, we also have to deal with the routing and wavelength assignment problem discussed in Chapter 8, as signals being passed through optically cannot be converted to other wavelengths. ,, Using more sophisticated routing and capacity allocation algorithms will bring the cost down for both 1 + 1 and shared mesh protection. 13.2 Designing the Transmission Layer 695 Table 13.5 Number of amplifiers, transponders, regenerators, and crossconnects required for LH and ULH systems to realize the capacities and link distances shown in Table 13.2, for both 1 + 1 and shared mesh protection. Part Quantity 1 + 1 Shared Mesh LH ULH LH ULH Amplifiers Transponders Terminal common equipment Regenerators Regenerator common equipment Crossconnect ports Crossconnect common equipment 364 487 275 413 4984 4984 3754 3754 75 96 59 77 1866 51 1432 51 33 1 24 1 5646 5646 4416 4416 55 55 43 43 Figure 13.12 Breakdown of network costs for LH and ULH systems with 1 -t- 1 and shared mesh protection. 696 DEPLOYMENT CONSIDERATIONS 1.3 1.2 r~ O o 1.1 1 "~ 0.9 0.8 0.7 ULH ampli ULH amplifier cost = LH am~ ~~ ULH amplifier cost=LH amplifier cost Baseline costs used in the text u u u i u n u 1 1.1 1.2 1.3 1.4 1.5 1.6 1.7 Relative ULH transponder and regenerator cost Figure 13.13 Sensitivity of study results to the relative cost of ULH and LH transpon- ders (and regenerators) and amplifiers. The x axis indicates the ULH transponder and regenerator cost relative to the LH transponder and regenerator cost. The y axis indicates the relative network cost for ULH and LH systems assuming 1 + 1 protection. 9 We have decoupled the network costing from the routing and capacity allocation. However, further cost optimization is possible by considering the two parts together. For example, in the LH case, we might choose slightly longer paths if it means using fewer regenerators on some of the links in the path. 9 We have not taken into account the cost of blocking when considering cross- connects. Observe that many nodes require more than one crossconnect, given our assumption of a 1.28 Tb/s crossconnect. In this analysis, we have simply used as many crossconnects as needed to obtain the desired port counts, without considering the cost of scaling the crossconnect or the cost of blocking. 9 We have implicitly assumed that there is no protection between the client equip- ment (for example, routers) and the optical layer equipment (such as crosscon- nects). In practice, we'll need to have some protection here as well and factor its cost into account. 9 Traffic demands are at 10 Gb/s. We haven't dealt with aggregating and grooming lower-speed demands. 13.2 Designing the Transmission Layer 697 13.2.7 Long-Haul Undersea Networks The economics of long-haul undersea links is similar to that of the long-haul terres- trial links, but with a few subtle differences. First, there are several types of undersea links commonly deployed. One type spans several thousands of kilometers across the Atlantic or Pacific oceans to interconnect North America with Europe or Asia, as shown in Figure 13.14. Another type tends to be relatively shorter haul (a few hun- dred kilometers), interconnecting countries either in a festoon type of arrangement or by direct links across short stretches of water. The term festoon means a string suspended in a loop between two points. In this context, it refers to an undersea cable used to connect two locations that are not separated by a body of water, usu- ally neighboring countries. A trunk-and-branch configuration is also popular, where an undersea trunk cable serves several countries. Each country is connected to the trunk cable by a branching cable, with passive optical components used to perform the branching at the branching units. If a branch cable is cut, access to a particular country is lost, but other countries continue to communicate via the trunk cable. WDM is widely deployed in all these types of links. The long-haul undersea systems tend to operate at the leading edge of technol- ogy and have to overcome significant impairments to attain the distances involved. The links use the dispersion management technique described in Section 5.8.6 by having alternating spans with positive and negative dispersion fiber to realize a total chromatic dispersion of zero but at the same time have finite chromatic dispersion at all points along the link. The shorter-distance undersea links also stretch design objectives but in a different way. The main objective with these links is to eliminate any undersea amplifiers or repeater stations, due to their relatively higher cost of installation and maintenance. As a result, these systems use relatively high-power transmitters. The trunk-and-branch configuration is also evolving. The early branching units contained passive splitters and combiners, but optical add/drop multiplexers are now being used to selectively drop and add specific wavelengths at different locations. Undersea systems are designed to provide very high levels of reliability and availability due to the high cost of servicing or replacing failed parts of the network. Optical amplifiers with redundant pumping arrangements have proven to be highly reliable devices, and their failure rates are much lower than those of electronic regenerators. Likewise, optical add/drop multiplexers using passive WDM devices have been qualified for use in undersea branching configurations. Undersea networks are very expensive to build, and the capacity on these net- works is shared among a number of users. WDM allows traffic from different users to be segregated by carrying them on different channelsma useful feature. 698 DEPLOYMENT CONSIDERATIONS Figure 13.14 Different types of undersea networks, showing a couple of ultra-long-haul trans- Atlantic links, shorter-haul direct repeaterless links, a trunk-and-branch configuration, and a festoon. One key difference between undersea links and terrestrial links is that, in most cases, undersea links are deployed from scratch with new fibers rather than over existing fiber plant. It is rare to upgrade an existing long-haul amplified undersea link, as the cost of laying a new link is not significantly higher than the cost of upgrading an existing link. This provides more flexibility in design choices. 13.2.8 Metro Networks The metro network can be broken up into two parts. The first part is the metro access network and extends from the carrier's central office to the carrier's customer locations, serving to collect traffic from them into the carrier's network. The second part of this network is the metro interoffice network~the part of the network 13.2 Designing the Transmission Layer 699 between carrier central offices. The access network today typically consists of rings a few kilometers to a few tens of kilometers in diameter, and traffic is primarily hubbed into the central office. The interoffice network tends to be several to a few tens of kilometers between sites, and traffic tends to be more distributed. Because of the shorter spans involved, the case for WDM links is less compelling in metro networks. The other alternatives, namely, using multiple fibers or using higher-speed TDM, are quite viable in many situations. Despite this, however, there hasn't been widespread deployment of OC-192 in the metro network. One reason is that OC-192 interfaces have only recently appeared on metro systems. Another reason is that carriers in this part of the network are interested in delivering low-speed services at DS1 (1.5 Mb/s) or DS3(45 Mb/s) rates and OC-192 equipment is only now becoming a cost-effective alternative for this application. On the other hand, reasons other than pure capacity growth are driving the deployment of WDM in these networks. Metro carriers need to provide a variety of different types of connections to their customers. The service mix includes leased private line services; statistical multiplexing types of services such as frame relay, ATM, and IP; Gigabit Ethernet; ESCON; and Fibre Channel. In many cases this service mix is supported by having a set of overlay networks, each dedicated to supporting a different service. These overlay networks are ideally realized using a single infrastructure. Due to its transparent nature, a WDM network provides a better infrastructure than most others, such as SONET/SDH, for this purpose. Another factor is that the traffic distribution changes much more rapidly in metro networks than in long-haul networks. This drives the need to be able to rearrange network capacity quickly and efficiently as needed. Reconfigurable WDM networks allow capacity to be provided as needed in an efficient manner. A big driver for WDM deployment in metro networks has been the need for large enterprises to interconnect their data centers. These data centers are separated by several kilometers to a few tens of kilometers. All transactions are mirrored at both sites. This allows the enterprise to recover quickly from a disaster when one of the centers fails. There may be other reasons for this as well, such as lower real estate costs at one location than at the other. Peripheral equipment such as disk farms can be placed at the cheaper site. The bandwidth requirement for such applications is large. The large mainframes at these data centers need to be interconnected by several hundred channels, each at up to 1 Gb/s. For example, IBM mainframes communicate using hundreds of ESCON channels, discussed in Chapter 6, running at 200 Mb/s each, or Fibre Channel at 1 Gb/s, as discussed in Chapter 6. Typically, these data centers tend to be located in dense metropolitan areas where most of the installed fiber is already in use. Moreover, these networks use a large variety of protocols and bit rates. These two factors make WDM an attractive option for these types of networks. . Nashville-Atlanta 388 63 47 14 Atlanta-Tampa 742 48 30 14 Tampa-Miami 370 33 45 25 Miami-Charlotte 1183 14 64 48 Nashville-Charlotte 598 30 108 47 Charlotte-Philadelphia 814 18 74 47 Nashville-Cleveland. 78 41 E1 Paso-Houston 1213 19 71 35 Houston-Dallas 409 44 46 30 Dallas-Kansas City 805 42 50 15 Dallas-Atlanta 1305 19 43 26 Kansas City-Nashville 873 68 102 80 Kansas City-Chicago 750 26. Capacity Capacity Capacity 1 + 1 Shared Mesh Seattle-San Francisco 1235 4 43 33 San Francisco-Los Angeles 616 35 12 10 Seattle-Salt Lake City 1251 25 35 16 Los Angeles-Salt Lake City 1073