Corporate Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-4000 800 553-NETS (6387) Fax: 408 526-4100 Data Center High Availability Clusters Design Guide Customer Order Number: Text Part Number: OL-12518-01 THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS. THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY. The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California. NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Data Center High Availability Clusters Design Guide © 2006 Cisco Systems, Inc. All rights reserved. CCSP, CCVP, the Cisco Square Bridge logo, Follow Me Browsing, and StackWise are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn, and iQuick Study are service marks of Cisco Systems, Inc.; and Access Registrar, Aironet, BPX, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unity, Enterprise/Solver, EtherChannel, EtherFast, EtherSwitch, Fast Step, FormShare, GigaDrive, GigaStack, HomeLink, Internet Quotient, IOS, IP/TV, iQ Expertise, the iQ logo, iQ Net Readiness Scorecard, LightStream, Linksys, MeetingPlace, MGX, the Networkers logo, Networking Academy, Network Registrar, Pa cke t, PIX, Post-Routing, Pre-Routing, ProConnect, RateMUX, ScriptShare, SlideCast, SMARTnet, The Fastest Way to Increase Your Internet Quotient, and TransPath are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries. All other trademarks mentioned in this document or Website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0601R) iii Data Center High Availability Clusters Design Guide OL-12518-01 CONTENTS Document Purpose ix Intended Audience ix Document Organization ix Document Approval x CHAPTER 1 Data Center High Availability Clusters 1-1 High Availability Clusters Overview 1-1 HA Clusters Basics 1-4 HA Clusters in Server Farms 1-5 Applications 1-6 Concept of Group 1-7 LAN Communication 1-9 Virtual IP Address 1-9 Public and Private Interface 1-10 Heartbeats 1-11 Layer 2 or Layer 3 Connectivity 1-11 Disk Considerations 1-12 Shared Disk 1-13 Quorum Concept 1-13 Network Design Considerations 1-16 Routing and Switching Design 1-16 Importance of the Private Link 1-17 NIC Teaming 1-18 Storage Area Network Design 1-21 Complete Design 1-22 CHAPTER 2 Data Center Transport Technologies 2-1 Redundancy and Client Protection Technologies 2-1 Dark Fiber 2-3 Pluggable Optics Characteristics 2-3 CWDM 2-4 DWDM 2-6 Maximum Distances and B2B Considerations 2-9 CWDM versus DWDM 2-10 Contents iv Data Center High Availability Clusters Design Guide OL-12518-01 Fiber Choice 2-11 SONET/SDH 2-12 SONET/SDH Basics 2-12 SONET UPSR and BLSR 2-13 Ethernet Over SONET 2-14 Service Provider Topologies and Enterprise Connectivity 2-15 Resilient Packet Ring/Dynamic Packet Transport 2-17 Spatial Reuse Protocol 2-17 RPR and Ethernet Bridging with ML-series Cards on a SONET Network 2-18 Metro Offerings 2-18 CHAPTER 3 Geoclusters 3-1 Geoclusters Overview 3-1 Replication and Mirroring 3-3 Geocluster Functional Overview 3-5 Geographic Cluster Performance Considerations 3-7 Server Performance Considerations 3-8 Disk Performance Considerations 3-9 Transport Bandwidth Impact on the Application Performance 3-10 Distance Impact on the Application Throughput 3-12 Benefits of Cisco FC-WA 3-13 Distance Impact on the Application IOPS 3-17 Asynchronous Versus Synchronous Replication 3-19 Read/Write Ratio 3-21 Transport Topologies 3-21 Two Sites 3-21 Aggregating or Separating SAN and LAN Transport 3-21 Common Topologies 3-22 CWDM and DWDM Topologies 3-22 SONET Topologies 3-23 Multiprotocol Label Switching Topologies 3-25 Three or More Sites 3-26 Hub-and-Spoke and Ring Topologies with CWDM 3-26 Hub-and-Spoke and Ring Topologies with DWDM 3-29 Shared Ring with SRP/RPR 3-32 Virtual Private LAN Service 3-33 Geocluster Design Models 3-34 Campus Cluster 3-34 Metro Cluster 3-37 Contents v Data Center High Availability Clusters Design Guide OL-12518-01 Regional Cluster 3-39 Continental Cluster 3-40 Storage Design Considerations 3-43 Manual Disk Failover and Failback 3-43 Software-Assisted Disk Failover 3-47 Network Design Considerations 3-50 LAN Extension and Redundancy 3-50 EtherChannels and Spanning Tree 3-51 Public and Private Links 3-52 Routing Design 3-52 Local Area Mobility 3-55 CHAPTER 4 FCIP over IP/MPLS Core 4-1 Overview 4-1 Typical Customer Requirements 4-2 Compression 4-3 Compression Support in Cisco MDS 4-3 Security 4-5 Cisco Encryption Solutions 4-6 Write Acceleration 4-7 Using FCIP Tape Acceleration 4-7 FCIP 4-8 TCP Operations 4-8 TCP Parameters 4-8 Customer Premises Equipment (CPE)—Cisco 9216/9216i and Cisco 7200 4-10 Cisco 9216 4-10 Cisco MDS 9216i 4-11 Cisco 7200 4-12 CPE Selection—Choosing between the 9216i and 7200 4-12 QoS Requirements in FCIP 4-13 Applications 4-14 Synchronous Replication 4-14 Asynchronous Replication 4-14 Service Offerings over FCIP 4-15 Service Offering Scenario A—Disaster Recovery 4-15 Service Offering Scenario B—Connecting Multiple Sites 4-16 Service Offering Scenario C—Host-based Mirroring 4-17 MPLS VPN Core 4-18 Using VRF VPNs 4-19 Contents vi Data Center High Availability Clusters Design Guide OL-12518-01 Testing Scenarios and Results 4-20 Test Objectives 4-20 Lab Setup and Topology 4-20 VPN VRF—Specific Configurations 4-21 MP BGP Configuration—PE1 4-21 Gigabit Ethernet Interface Configuration—PE1 4-22 VRF Configuration—PE1 4-22 MP BGP Configuration—PE2 4-22 Gigabit Ethernet Interface Configuration—PE2 4-22 VRF Configuration—PE2 4-23 Scenario 1—MDS 9216i Connection to GSR MPLS Core 4-23 Configuring TCP Parameters on CPE (Cisco MDS 9216) 4-24 Configuring the MTU 4-24 Scenario 2—Latency Across the GSR MPLS Core 4-25 Scenario 3—Cisco MDS 9216i Connection to Cisco 7500 (PE)/GSR (P) 4-26 Scenario 4—Impact of Failover in the Core 4-27 Scenario 5—Impact of Core Performance 4-27 Scenario 6—Impact of Compression on CPE (Cisco 9216i) Performance 4-28 Application Requirements 4-29 Remote Tape-Backup Applications 4-30 Conclusion 4-30 CHAPTER 5 Extended Ethernet Segments over the WAN/MAN using EoMPLS 5-1 Introduction 5-1 Hardware Requirements 5-1 Enterprise Infrastructure 5-2 EoMPLS Designs for Data Center Interconnectivity 5-3 EoMPLS Termination Options 5-4 MPLS Technology Overview 5-8 EoMPLS Design and Configuration 5-11 EoMPLS Overview 5-11 EoMPLS—MTU Computation 5-15 Core MTU 5-15 Edge MTU 5-17 EoMPLS Configuration 5-18 Using Core IGP 5-18 Set MPLS Globally 5-19 Enable MPLS on Core Links 5-19 Verify MPLS Connectivity 5-19 Contents vii Data Center High Availability Clusters Design Guide OL-12518-01 Create EoMPLS Pseudowires 5-20 Verify EoMPLS Pseudowires 5-20 Optimize MPLS Convergence 5-20 Backoff Algorithm 5-21 Carrier Delay 5-21 BFD (Bi-Directional Failure Detection) 5-22 Improving Convergence Using Fast Reroute 5-24 High Availability for Extended Layer 2 Networks 5-27 EoMPLS Port-based Xconnect Redundancy with Multiple Spanning Tree Domains 5-28 IST Everywhere 5-28 Interaction between IST and MST Regions 5-29 Configuration 5-32 EoMPLS Port-based Xconnect Redundancy with EtherChannels 5-33 Remote Failure Detection 5-34 EoMPLS Port-based Xconnect Redundancy with Spanning Tree 5-36 CHAPTER 6 Metro Ethernet Services 6-1 Metro Ethernet Service Framework 6-1 MEF Services 6-2 Metro Ethernet Services 6-2 EVC Service Attributes 6-3 ME EVC Service Attributes 6-7 UNI Service Attributes 6-8 Relationship between Service Multiplexing, Bundling, and All-to-One Bundling 6-11 ME UNI Service Attributes 6-13 Ethernet Relay Service 6-14 Ethernet Wire Service 6-15 Ethernet Private Line 6-16 Ethernet Multipoint Service 6-17 ME EMS Enhancement 6-17 Ethernet Relay Multipoint Service 6-18 APPENDIX A Configurations for Layer 2 Extension with EoMPLS A-1 Configurations A-6 Enabling MPLS A-6 Port-based Xconnect A-6 Configuring the Loopback Interface A-6 Configuring OSPF A-7 Configuring ISIS A-7 Contents viii Data Center High Availability Clusters Design Guide OL-12518-01 Aggregation Switch Right (Catalyst 6000 Series Switch-Sup720-B)—Data Center 1 A-8 Enabling MPLS A-8 Port-based Xconnect A-8 Configuring the Loopback Interface A-8 Configuring VLAN 2 A-8 Configuring Interface fa5/1 (Connected to a Remote Catalyst 6000 Series Switch) A-8 Configuring OSPF A-9 Configuring ISIS A-9 Aggregation Switch Left (Catalyst 6000 Series Switch-Sup720-B)— Data Center 2 A-9 Enabling MPLS A-9 Port-based Xconnect A-9 Configuring the Loopback Interface A-10 Configuring OSPF A-10 Configuring ISIS A-11 Aggregation Switch Right (Catalyst 6000 Series Switch-Sup720-B)— Data Center 2 A-11 Enabling MPLS A-11 Port-based Xconnect A-11 Configuring the Loopback Interface A-11 Configuring VLAN 2 A-12 Configuring Interface G5/1 (Connected to Remote Catalyst 6000 Series Switch) A-12 Configuring OSPF A-12 Configuring ISIS A-12 MTU Considerations A-13 Spanning Tree Configuration A-13 MST Configuration A-14 Failover Test Results A-19 Data Center 1 (Catalyst 6000 Series Switch—DC1-Left) A-19 Data Center 1 (Catalyst 6000 Series Switch—DC1-Right) A-20 Data Center 2 (Catalyst 6000 Series Switch—DC2-Left) A-20 Data Center 2 (Catalyst 6000 Series Switch—DC2-Right) A-20 G LOSSARY ix Data Center High Availability Clusters Design Guide OL-12518-01 Preface Document Purpose Data Center High Availability Clusters Design Guide describes how to design and deploy high availability (HA) clusters to provide uninterrupted access to data, even if a server loses network or storage connectivity, or fails completely, or if the application running on the server fails. Intended Audience This guide is intended for system engineers who support enterprise customers that are responsible for designing, planning, managing, and implementing local and distributed data center IP infrastructures. Document Organization This guide contains the chapters in the following table. Section Description Chapter 1, “Data Center High Availability Clusters.” Provides high-level overview of the use of HA clusters, including design basics and network design recommendations for local clusters. Chapter 2, “Data Center Transport Technologies.” Describes the transport options for interconnecting the data centers. Chapter 3, “Geoclusters.” Describes the use and design of geoclusters in the context of business continuance as a technology to lower the recovery time objective. Chapter 4, “FCIP over IP/MPLS Core.” Describes the transport of Fibre Channel over IP (FCIP) over IP/Multiprotocol Label Switching (MPLS) networks and addresses the network requirements from a service provider (SP) perspective. Chapter 5, “Extended Ethernet Segments over the WAN/MAN using EoMPLS.” Describes the various options available to extend a Layer 2 network using Ethernet over Multiprotocol Label Switching (EoMPLS) on the Cisco Sup720-3B. Chapter 6, “Metro Ethernet Services.” Describes the functional characteristics of Metro Ethernet services. x Data Center High Availability Clusters Design Guide OL-12518-01 Preface Document Organization Appendix A “Configurations for Layer 2 Extension with EoMPLS.” Describes the lab and test setups. Glossary. Provides a glossary of terms. [...]... OpenVMS High Availability Cluster Service—This clustering solution was originally developed for VAX systems, and now runs on HP Alpha and HP Integrity servers This is an OS-level clustering that offers an SSI For more information, see the following URL: http://h71000.www7.hp.com/ Data Center High Availability Clusters Design Guide 1-2 OL-12518-01 Chapter 1 Data Center High Availability Clusters High Availability. .. the clustered application “shared folder” called “test” Data Center High Availability Clusters Design Guide 1-4 OL-12518-01 Chapter 1 Data Center High Availability Clusters HA Clusters Basics Figure 1-3 Virtual Address Configuration with MSCS HA Clusters in Server Farms Figure 1-4 shows where HA clusters are typically deployed in a server farm Databases are typically clustered to appear as a single... is used at the very bottom of the processing tiers to protect application data Data Center High Availability Clusters Design Guide OL-12518-01 1-5 Chapter 1 Data Center High Availability Clusters HA Clusters Basics Figure 1-4 HA Clusters Use in Typical Server Farms Default GW Web servers Email servers Application servers 154272 Database Servers Applications An application running on a server that has... Complete Design Figure 1-23 shows the end-to-end design with a typical data center network Each clustered server is dual-homed to the LAN and to the SAN NIC teaming is configured for the public interface; with this design, it might be using the ALB mode (also called TLB depending on the NIC vendor) to take Data Center High Availability Clusters Design Guide 1-22 OL-12518-01 Chapter 1 Data Center High Availability. .. heartbeats heartbeats node2 154277 private LAN Data Center High Availability Clusters Design Guide 1-10 OL-12518-01 Chapter 1 Data Center High Availability Clusters HA Clusters Basics Heartbeats From a network design point of view, the type of heartbeats used by the application often decide whether the connectivity between the servers can be routed For local clusters, it is almost always assumed that the... which node owned the disk group for the application Data Center High Availability Clusters Design Guide 1-16 OL-12518-01 Chapter 1 Data Center High Availability Clusters HA Clusters Basics switch1 Typical Local Cluster Configuration switch2 public switch1 public private node1 switch2 node2 private node1 node2 quorum quorum Application data Application data 154284 Figure 1-16 Importance of the Private... result is that both nodes in the cluster go offline Data Center High Availability Clusters Design Guide OL-12518-01 1-17 Chapter 1 Data Center High Availability Clusters HA Clusters Basics Figure 1-17 Cluster Configuration with a Promiscuous Port—No Private Link switch1 switch2 heartbeat node1 node2 Application data 154285 quorum For this reason, Cisco highly recommends using at least two NICs; one for... identify the same disk that appears on two different HBAs Data Center High Availability Clusters Design Guide OL-12518-01 1-21 Chapter 1 Data Center High Availability Clusters HA Clusters Basics Figure 1-22 SAN Configuration User Applications SCSI Port/Generic Driver User Applications File System/Database Filesyste User Applications File System/Database Volume Mgr Virtualization Replacement Disk Driver... owns the disk that stores the application data When a failover happens, node2 mounts the disk and starts the application by using the API provided by the Application DLL Figure 1-7 Failover of Group node1 node2 Application data 154275 quorum Data Center High Availability Clusters Design Guide 1-8 OL-12518-01 Chapter 1 Data Center High Availability Clusters HA Clusters Basics The failover can also be... the picture) continues on the private links connected to the remaining access switch Figure 1-24 (a) shows the design with a loop-free access Data Center High Availability Clusters Design Guide OL-12518-01 1-23 Chapter 1 Data Center High Availability Clusters HA Clusters Basics Figure 1-24 Design with a Loop-free Access (a) and an Important Failure Scenario (b) (a) agg1 (b) agg2 agg1 acc2 acc1 acc1 . CHAPTER 1-1 Data Center High Availability Clusters Design Guide OL-12518-01 1 Data Center High Availability Clusters High Availability Clusters Overview Clusters. address 1-2 Data Center High Availability Clusters Design Guide OL-12518-01 Chapter 1 Data Center High Availability Clusters High Availability Clusters Overview