Corporate Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-4000 800 553-NETS (6387) Fax: 408 526-4100 Data Center: Infrastructure Architecture SRND Solutions Reference Network Design March, 2004 Customer Order Number: 956513 THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS. THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY. The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California. NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Data Cemter Networking: Enterprise Distributed Data Centers Copyright © 2004, Cisco Systems, Inc. All rights reserved. CCIP, CCSP, the Cisco Arrow logo, the Cisco Powered Network mark, Cisco Unity, Follow Me Browsing, FormShare, and StackWise are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn, and iQuick Study are service marks of Cisco Systems, Inc.; and Aironet, ASIST, BPX, Catalyst, CCDA, CCDP, CCIE, CCNA, CCNP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, the Cisco IOS logo, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Empowering the Internet Generation, Enterprise/Solver, EtherChannel, EtherSwitch, Fast Step, GigaStack, Internet Quotient, IOS, IP/TV, iQ Expertise, the iQ logo, iQ Net Readiness Scorecard, LightStream, MGX, MICA, the Networkers logo, Networking Academy, Network Registrar, Packet, PIX, Post-Routing, Pre-Routing, RateMUX, Registrar, ScriptShare, SlideCast, SMARTnet, StrataView Plus, Stratm, SwitchProbe, TeleRouter, The Fastest Way to Increase Your Internet Quotient, TransPath, and VCO are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and certain other countries. All other trademarks mentioned in this document or Web site are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0304R) iii Data Center: Infrastructure Architecture SRND 956513 CONTENTS Preface vii Document Purpose vii Document Organization viii Obtaining Documentation viii World Wide Web viii Documentation CD-ROM viii Ordering Documentation ix Documentation Feedback ix Obtaining Technical Assistance ix Cisco.com ix Technical Assistance Center x Cisco TAC Web Site x Cisco TAC Escalation Center xi CHAPTER 1 Data Center Infrastructure Architecture 1-1 Data Center Architecture 1-1 Hardware and Software Recommendations 1-3 Aggregation Switches 1-3 Service Appliances 1-5 Service Modules 1-5 Access Switches 1-6 Software Recommendations 1-8 Data Center Multi-Layer Design 1-9 Core Layer 1-9 Aggregation and Access Layer 1-10 Service Switches 1-10 Server Farm Availability 1-11 Load-Balanced Servers 1-12 Data Center Protocols and Features 1-15 Layer 2 Protocols 1-15 Layer 3 Protocols 1-16 Security in the Data Center 1-18 Scaling Bandwidth 1-18 Network Management 1-19 Contents iv Data Center: Infrastructure Architecture SRND 956513 1-20 CHAPTER 2 Data Center Infrastructure Design 2-1 Routing Between the Data Center and the Core 2-1 Layer 3 Data Center Design 2-1 Using OSPF 2-3 Using EIGRP 2-7 Designing Layer 3 Security 2-8 Switching Architecture for the Server Farm 2-9 Using Redundant Supervisors 2-9 Layer 2 Data Center Design 2-10 Using Three-Tier and Two-Tier Network Designs 2-10 Layer 2 and Layer 3 Access Design 2-11 Using VLANs to Segregate Server Farms 2-12 VLAN Scalability 2-13 Using Virtual Trunking Protocol 2-14 Choosing a Spanning-Tree Algorithm 2-14 Using Loopguard and UDLD 2-15 Using PortFast and TrunkFast 2-17 Using a Loop-Free Topology 2-18 Designing Layer 2 Security 2-19 Assigning the Default Gateway in the Data Center 2-21 Using Gateway Redundancy Protocols 2-22 Tuning the ARP Table 2-23 CHAPTER 3 HA Connectivity for Servers and Mainframes: NIC Teaming and OSA/OSPF Design 3-1 Overview 3-1 Ensuring Server Farm and Mainframe Availability 3-2 Load Balanced Servers 3-4 NIC Teaming 3-4 Mainframe Sysplex 3-6 NIC Teaming Architecture Details 3-7 Hardware and Software 3-8 Deployment Modes 3-8 Fault Tolerance Modes 3-8 Load Balancing Modes 3-12 Link Aggregation Modes 3-13 Layer 3 Multihoming 3-14 Interoperability with Security 3-16 Contents v Data Center: Infrastructure Architecture SRND 956513 Intrusion Detection 3-17 Port Security 3-17 Private VLANs 3-19 Mainframe OSA and OSPF Architecture Details 3-20 Overview 3-20 Attachment Options 3-21 IP Addressing 3-22 OSPF Routing on a Mainframe 3-23 Sysplex 3-24 Configuration Details 3-26 Speed and Duplex Settings 3-27 Layer 2 Implementation 3-27 Spanning Tree 3-27 PortFast and BPDU Guard 3-28 Port Security 3-29 Server Port Configuration 3-29 CHAPTER 4 Data Center Infrastructure Configuration 4-1 Configuring Network Management 4-1 Username and Passwords 4-1 VTY Access 4-2 SNMP 4-3 Logging 4-3 VLAN Configuration 4-3 Spanning Tree Configuration 4-6 Rapid PVST+ 4-6 MST 4-7 Protection From Loops 4-7 VLAN Interfaces and HSRP 4-8 Switch-To-Switch Connections Configuration 4-9 Channel Configuration 4-9 Trunk Configuration 4-10 Server Port Configuration 4-12 Speed and Duplex Settings 4-12 PortFast and BPDU Guard 4-13 Port Security 4-13 Configuration Example 4-14 Sample Configurations 4-14 Aggregation1 4-14 Contents vi Data Center: Infrastructure Architecture SRND 956513 Aggregation2 4-18 Access 4-21 G LOSSARY I NDEX vii Data Center: Infrastructure Architecture SRND 956513 Preface This publication provides solution guidelines for enterprises implementing Data Centers with Cisco devices. The intended audiences for this design guide include network architects, network managers, and others concerned with the implementation of secure Data Center solutions, including: • Cisco sales and support engineers • Cisco partners • Cisco customers Document Purpose The convergence of voice and video in today’s enterprise networks has placed additional requirements on the infrastructure of enterprise data centers, which must provide the following services: • Hosting enterprise-wide servers • Supporting critical application services • Supporting traditional data services • 24X7 availability These requirements are based on the applications supported rather than the size of the data center. The process of selecting the proper data center hardware and software versions that meet the necessary Layer 2, Layer 3, QoS, and Multicast requirements can be a daunting task. This solutions reference network design (SRND) provides design and implementation guidelines for building a redundant, scalable enterprise data center. These guidelines cover the following areas: • Data center infrastructure and server farm design • Server farm design including high availability • Designing data centers for mainframe connectivity • Enhancing server-to-server communication viii Data Center: Infrastructure Architecture SRND 956513 Preface Document Organization Document Organization This document consists of the following chapters: Obtaining Documentation The following sections explain how to obtain documentation from Cisco Systems. World Wide Web You can access the most current Cisco documentation on the World Wide Web at the following URL: http://www.cisco.com Translated documentation is available at the following URL: http://www.cisco.com/public/countries_languages.shtml Documentation CD-ROM Cisco documentation and additional literature are available in a Cisco Documentation CD-ROM package, which is shipped with your product. The Documentation CD-ROM is updated monthly and may be more current than printed documentation. The CD-ROM package is available as a single unit or through an annual subscription. Chapter Description Chapter 1, “Data Center Infrastructure Architecture” Provides background information, including hardware recommendations for designing a data center infrastructure that is secure, scalable, and resilient. Chapter 2, “Data Center Infrastructure Design” Describes design issu, including routing between the data center and the core, switching within the server farm Chapter 3, “HA Connectivity for Servers and Mainframes: NIC Teaming and OSA/OSPF Design” Describes how to include server connectivity with NIC teaming and mainframe connectivity in your data center infrastructure architcture. Chapter 4, “Data Center Infrastructure Configuration” Provides configuration procedures and sample listings for implementing the recommended infrastructure architecture. ix Data Center: Infrastructure Architecture SRND 956513 Preface Obtaining Technical Assistance Ordering Documentation Cisco documentation is available in the following ways: • Registered Cisco Direct Customers can order Cisco product documentation from the Networking Products MarketPlace: http://www.cisco.com/cgi-bin/order/order_root.pl • Registered Cisco.com users can order the Documentation CD-ROM through the online Subscription Store: http://www.cisco.com/go/subscription • Nonregistered Cisco.com users can order documentation through a local account representative by calling Cisco corporate headquarters (California, USA) at 408 526-7208 or, elsewhere in North America, by calling 800 553-NETS (6387). Documentation Feedback If you are reading Cisco product documentation on Cisco.com, you can submit technical comments electronically. Click Leave Feedback at the bottom of the Cisco Documentation home page. After you complete the form, print it out and fax it to Cisco at 408 527-0730. You can e-mail your comments to bug-doc@cisco.com. To submit your comments by mail, use the response card behind the front cover of your document, or write to the following address: Cisco Systems Attn: Document Resource Connection 170 West Tasman Drive San Jose, CA 95134-9883 We appreciate your comments. Obtaining Technical Assistance Cisco provides Cisco.com as a starting point for all technical assistance. Customers and partners can obtain documentation, troubleshooting tips, and sample configurations from online tools by using the Cisco Technical Assistance Center (TAC) Web Site. Cisco.com registered users have complete access to the technical support resources on the Cisco TAC Web Site. Cisco.com Cisco.com is the foundation of a suite of interactive, networked services that provides immediate, open access to Cisco information, networking solutions, services, programs, and resources at any time, from anywhere in the world. Cisco.com is a highly integrated Internet application and a powerful, easy-to-use tool that provides a broad range of features and services to help you to • Streamline business processes and improve productivity • Resolve technical issues with online support x Data Center: Infrastructure Architecture SRND 956513 Preface Obtaining Technical Assistance • Download and test software packages • Order Cisco learning materials and merchandise • Register for online skill assessment, training, and certification programs You can self-register on Cisco.com to obtain customized information and service. To access Cisco.com, go to the following URL: http://www.cisco.com Technical Assistance Center The Cisco TAC is available to all customers who need technical assistance with a Cisco product, technology, or solution. Two types of support are available through the Cisco TAC: the Cisco TAC Web Site and the Cisco TAC Escalation Center. Inquiries to Cisco TAC are categorized according to the urgency of the issue: • Priority level 4 (P4)—You need information or assistance concerning Cisco product capabilities, product installation, or basic product configuration. • Priority level 3 (P3)—Your network performance is degraded. Network functionality is noticeably impaired, but most business operations continue. • Priority level 2 (P2)—Your production network is severely degraded, affecting significant aspects of business operations. No workaround is available. • Priority level 1 (P1)—Your production network is down, and a critical impact to business operations will occur if service is not restored quickly. No workaround is available. Which Cisco TAC resource you choose is based on the priority of the problem and the conditions of service contracts, when applicable. Cisco TAC Web Site The Cisco TAC Web Site allows you to resolve P3 and P4 issues yourself, saving both cost and time. The site provides around-the-clock access to online tools, knowledge bases, and software. To access the Cisco TAC Web Site, go to the following URL: http://www.cisco.com/tac All customers, partners, and resellers who have a valid Cisco services contract have complete access to the technical support resources on the Cisco TAC Web Site. The Cisco TAC Web Site requires a Cisco.com login ID and password. If you have a valid service contract but do not have a login ID or password, go to the following URL to register: http://www.cisco.com/register/ If you cannot resolve your technical issues by using the Cisco TAC Web Site, and you are a Cisco.com registered user, you can open a case online by using the TAC Case Open tool at the following URL: http://www.cisco.com/tac/caseopen If you have Internet access, it is recommended that you open P3 and P4 cases through the Cisco TAC Web Site. [...]... serial number Data Center: Infrastructure Architecture SRND 956513 xi Preface Obtaining Technical Assistance Data Center: Infrastructure Architecture SRND xii 956513 C H A P T E R 1 Data Center Infrastructure Architecture This chapter provides background information for designing a secure, scalable, and resilient data center infrastructure It includes the following sections: • Data Center Architecture. .. in the data center architecture described in this design guide Data Center: Infrastructure Architecture SRND 956513 1-17 Chapter 1 Data Center Infrastructure Architecture Scaling Bandwidth Security in the Data Center Describing the details of security in the data center is beyond the scope of this document, but it is important to be aware of it when building the infrastructure Security in the data center... with explicitly defined primary and backup traffic paths Data Center: Infrastructure Architecture SRND 956513 1-1 Chapter 1 Data Center Infrastructure Architecture Data Center Architecture • Security—Prevent flooding, avoid exchanging protocol information with rogue devices, and prevent unauthorized access to network devices The data center infrastructure must provide port density and Layer 2 and Layer... between the data center and the core typically is performed on the MSFC The Layer 3 portion of the data center design changes slightly depending on it is an Internet data center or an intranet data center Figure 2-1 shows the physical topology of an intranet data center on the left, and the logical topology on the right Data Center: Infrastructure Architecture SRND 956513 2-1 Chapter 2 Data Center Infrastructure. .. view of the intranet data center shows that the data center is one spoke of a hub-and-spoke topology As such, there is very little dynamic routing You only need to inject a default route into the data center and advertise the data center subnets to the core Data Center: Infrastructure Architecture SRND 2-2 956513 Chapter 2 Data Center Infrastructure Design Routing Between the Data Center and the Core... specific network performance requirements and traffic patterns Data Center: Infrastructure Architecture SRND 1-14 956513 Chapter 1 Data Center Infrastructure Architecture Data Center Protocols and Features Data Center Protocols and Features This section provides background information about protocols and features that are helpful when designing a data center network for high availability, security and scalability... Ethernet data center infrastructure Data Center: Infrastructure Architecture SRND 956513 1-3 Chapter 1 Data Center Infrastructure Architecture Hardware and Software Recommendations The Catalyst 6500 is available in several form factors: • 6503: 3 slots 3 RUs • 6506: 6 slots 12 RUs • 7606: 6 slots 7 RUs • 6509: 9 slots 15 RUs • 6513: 13 slots, 19 RUs The 6509 and 6513 are typically deployed in the data. .. improvements in the features that are supported on the access switch platforms Data Center: Infrastructure Architecture SRND 1-8 956513 Chapter 1 Data Center Infrastructure Architecture Data Center Multi-Layer Design described in this design document, it isn't possible to give a recommendation on the software release you should deploy in your data center The choice of the software release depends on the hardware... PVST+ is ensured by building the Data Center: Infrastructure Architecture SRND 956513 1-15 Chapter 1 Data Center Infrastructure Architecture Data Center Protocols and Features “Common Spanning-Tree” (CST) by using VLAN 1 Cisco switches build a CST with IEEE 802.1d switches, and the BPDUs for all the VLANs other than VLAN 1 are tunneled through the 802.1d region Cisco data centers feature a fully-switched... between the firewalls and the MSFC, but this is subject to slower convergence in case of firewall Data Center: Infrastructure Architecture SRND 1-16 956513 Chapter 1 Data Center Infrastructure Architecture Data Center Protocols and Features failures Delays are caused by the process of neighbor establishment, data base exchange, running the SPF algorithm and installing the Layer 3 forwarding table in the . vi Data Center: Infrastructure Architecture SRND 956513 Aggregation2 4-18 Access 4-21 G LOSSARY I NDEX vii Data Center: Infrastructure Architecture SRND. xii Data Center: Infrastructure Architecture SRND 956513 Preface Obtaining Technical Assistance CHAPTER 1-1 Data Center: Infrastructure Architecture SRND