Corporate Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-4000 800 553-NETS (6387) Fax: 408 526-4100 Data Center: SAN Extension for Business Continuance Solutions Reference Network Design Version 1.2 THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS. THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY. The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California. NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Data Center: SAN Extension for Business Continuance Copyright © 2004, Cisco Systems, Inc. All rights reserved. CCIP, CCSP, the Cisco Arrow logo, the Cisco Powered Network mark, Cisco Unity, Follow Me Browsing, FormShare, and StackWise are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn, and iQuick Study are service marks of Cisco Systems, Inc.; and Aironet, ASIST, BPX, Catalyst, CCDA, CCDP, CCIE, CCNA, CCNP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, the Cisco IOS logo, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Empowering the Internet Generation, Enterprise/Solver, EtherChannel, EtherSwitch, Fast Step, GigaStack, Internet Quotient, IOS, IP/TV, iQ Expertise, the iQ logo, iQ Net Readiness Scorecard, LightStream, MGX, MICA, the Networkers logo, Networking Academy, Network Registrar, Packet, PIX, Post-Routing, Pre-Routing, RateMUX, Registrar, ScriptShare, SlideCast, SMARTnet, StrataView Plus, Stratm, SwitchProbe, TeleRouter, The Fastest Way to Increase Your Internet Quotient, TransPath, and VCO are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and certain other countries. All other trademarks mentioned in this document or Web site are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0304R) iii Data Center: SAN Extension for Business Continuance Version 1.2 CONTENTS Preface vii Document Organization vii Document Conventions viii CHAPTER 1 SAN Extension for Business Continuance: Overview 1-1 Overview 1-1 Replication and Mirroring 1-3 Synchronous Replication 1-4 Network Latency 1-5 Asynchronous Replication 1-6 SAN Extension Transport Overview 1-7 Dark Fiber 1-8 CWDM 1-8 DWDM (Dense Wave Division Multiplexing) 1-9 SONET/SDH 1-9 FCIP 1-9 Flow Control over Long Distance 1-11 SAN Extension Solutions 1-12 Metro Optical SAN Extension with the ONS15530/ONS15540 and MDS 9000 1-12 Optical Protection Options 1-12 Scaling the Infrastructure 1-12 Extending the Distance 1-12 Campus SAN Extension using CWDM and MDS 9000 1-13 Protection 1-14 Scaling 1-14 Distance Capabilities 1-15 Metro/Regional – ONS15454 1-15 Synchronous Replication using FCIP on the MDS 9000 IP Services Module 1-16 MDS 9000 FCIP for Synchronous Data Replication 1-17 MDS 9000 FCIP for Asynchronous Data Replication 1-17 FCIP for SAN Extension with PA-FC-1G Port Adapter for 7200/7400 1-18 FCIP Compression and Encryption Solutions 1-19 Encryption for Data Privacy 1-19 Compression for Increased Throughput Over WAN links 1-20 Contents iv Data Center: SAN Extension for Business Continuance Version 1.2 Summary 1-22 CHAPTER 2 Designing FCIP SAN Extension for Cisco SAN Environments 2-1 Overview 2-1 Why Use FCIP? 2-1 Solution Topology 2-2 Required Components 2-2 Using Compression 2-3 Compression and Available Bandwidth 2-4 Compression Modes and Rates 2-5 Using Write Acceleration 2-7 Write Acceleration Overview 2-7 Write Exchanges with More Than Two Round Trips 2-9 Expected Performance Gains When Using Write Acceleration 2-12 When to Use Write Acceleration 2-14 Considerations and Guidelines for Using Write Acceleration 2-16 CHAPTER 3 Implementing FCIP SAN Extension in Cisco SAN Environments 3-1 Implementation Details 3-1 FCIP Configuration Overview 3-1 Configuring the Gigabit Ethernet Interface, FCIP Profile, and FCIP Interface 3-2 Configuring the Gigabit Ethernet Interface 3-5 Setting the MTU Size 3-6 Automatic Path MTU Discovery 3-6 Manually Determining Path MTU Using Extended Ping 3-7 Configuring the FCIP Profile 3-8 FCIP Interface Configuration 3-9 Fibre Channel Configuration Considerations 3-10 Fibre Channel Receive Buffer Configuration Guidelines 3-10 Monitoring for Frame Expiry 3-13 Using Shared FCIP Links 3-13 Impact of WAN Packet Loss on FCIP Throughput 3-14 FCIP Tunnel Latency 3-14 Configuring Compression 3-15 Recommended Topologies 3-15 Single FCIP Link 3-16 Switch-A Configuration 3-16 Switch-B Configuration 3-17 Single FCIP Link with Multiple VSANs 3-18 Contents v Data Center: SAN Extension for Business Continuance Version 1.2 Switch-A Configuration 3-19 Switch-B Configuration 3-19 High Availability Replication Design Using Multiple FCIP Links and Port Channels 3-20 High Availability Protection Levels 3-21 Port Channel Load Balancing 3-21 FCIP Link and Profile Configuration 3-22 Rate Limiting FCIP Links and Port Channel Members 3-22 Switch-A Configuration 3-23 Switch-B Configuration 3-24 Switch-C Configuration 3-25 Switch-D Configuration 3-26 CHAPTER 4 FCIP SAN Extension In Legacy Environments 4-1 Overview 4-1 Performance and Configuration Considerations 4-1 Cisco IOS Requirements 4-2 7200 VXR Series Router PCI Bus Considerations 4-2 7200 VXR Series Router CPU Performance Considerations 4-4 7200 VXR Series Router Gigabit Ethernet Considerations 4-5 Software Configuration 4-5 TCP Maximum Window Size (MWS) 4-7 Background Traffic 4-9 Interface MTU Size 4-9 Fibre Channel Configuration 4-10 Fibre Channel Timers 4-10 BB Credits 4-10 Fibre Channel Frame Size 4-11 Fibre Channel Switch Interoperability 4-12 Solution Topologies 4-12 CHAPTER 5 SAN Extension with Compression and Encryption in WAN Environments 5-1 FCIP Compression and Encryption Overview 5-1 Compression Overview 5-2 Selecting a Compression Solution 5-3 Determining the Compression Ratio 5-5 Encryption Overview 5-6 IPSec Network Security 5-7 IPSec Encapsulating Security Protocol (ESP) 5-7 IPSec Authentication Header (AH) 5-7 Contents vi Data Center: SAN Extension for Business Continuance Version 1.2 IPSec Modes of Operation 5-7 DES/3DES Encryption 5-9 AES Encryption 5-9 Cisco Encryption Solutions 5-9 Combined Compression and Encryption FCIP Storage Extension Solutions 5-11 Compression and Encryption Design Considerations 5-14 IPS-8 Compression Performance Options 5-14 Configuring SA-VAM, SA-VAM2, IPSec VPN Services Module Compression and Encryption 5-15 Configuring IPS-8 FCIP and Compression 5-19 Compression and Encryption Configuration Examples 5-21 SA-VAM / PA-FC-1G 5-21 SA-VAM2 / IPS-8 5-23 SA-VAM / SA-VAM2 High Availability 5-25 IPS-8 SANOS v1.3 Compression 5-27 IPS-8 High Availability 5-28 IPS-8 / IPSec VPN Services Module 5-31 IPSec VPN Services Module High Availability Options 5-34 Conclusion 5-35 vii Data Center: SAN Extension for Business Continuance Version 1.2 Preface The purpose of this document is to provide product knowledge and design guidance about how to extend the capacity of Storage Area Networks (SAN) using Cisco products. This document is intended for network design architects and support engineers who are responsible for planning, designing, and implementing storage area networks (SAN) for business continuance and disaster recovery. Document Organization This document contains the following chapters: Chapter or Appendix Description Chapter 1, “SAN Extension for Business Continuance: Overview” Provides an overview of SAN extension for business continuance. Chapter 2, “Designing FCIP SAN Extension for Cisco SAN Environments” Provides background information on the FCIP protocol and describes performance considerations involved when designing a specific implementation. Chapter 3, “Implementing FCIP SAN Extension in Cisco SAN Environments” Describes design scenarios for implementing FCIP SAN Extension using the and examines the tuning and configuration instructions required for optimum operation. Chapter 4, “FCIP SAN Extension In Legacy Environments” Provides product knowledge and design guidance about how to extend the capacity of a Storage Area Networks (SAN) using the Cisco PA-FC-1G PAM. Chapter 5, “SAN Extension with Compression and Encryption in WAN Environments” Provides product knowledge and design guidance for compressing and encrypting Fibre Channel over IP (FCIP) traffic using the following Cisco products: • Cisco SA-VAM and SA-VAM2 service modules for the 7200 VXR Series Routers • IPS-8 IP Storage Services Module for the MDS 9216 and 9500 Fibre Channel switches and directors • IPSec VPN Services Module for the Catalyst 6500 switch and 7600 router Appendix A, “Glossary” A glossary of terms often used when describing encryption and compression technology. viii Data Center: SAN Extension for Business Continuance Version 1.2 Preface Document Conventions Document Conventions This guide uses the following conventions to convey instructions and information: Table 1 Document Conventions Convention Description boldface font Commands and keywords. italic font Variables for which you supply values. [ ] Keywords or arguments that appear within square brackets are optional. {x | y | z} A choice of required keywords appears in braces separated by vertical bars. You must select one. screen font Examples of information displayed on the screen. boldface screen font Examples of information you must enter. < > Nonprinting characters, for example passwords, appear in angle brackets. [ ] Default responses to system prompts appear in square brackets. CHAPTER 1-1 Data Center: SAN Extension for Business Continuance Version 1.2 1 SAN Extension for Business Continuance: Overview This chapter provides product knowledge and design guidance about how to extend the capacity of Storage Area Networks (SAN) using Cisco products. It includes the following topics: • Overview, page 1-1 • Replication and Mirroring, page 1-3 • SAN Extension Transport Overview, page 1-7 • SAN Extension Solutions, page 1-12 • Summary, page 1-22 Overview Storage Area Network (SAN) Extension is one of the enabling technologies for enterprise business continuance. An extended SAN increases the geographic distance allowed for SAN storage operations, in particular for data replication and copy operations. By replicating or copying data to an alternate site, an enterprise can protect its data in the event of disaster at the primary site. You should consider the following three factors when determining your business continuance requirements: • Recovery time objective (RTO) • Recovery point objective (RPO) • Disaster radius The RTO and RPO provide a measurable target for business continuance and disaster recovery purposes. It also presents a firm basis from which to design the underlying SAN extension network. RTO refers to how long it takes for the enterprise systems to resume operation after a disaster. RTO is the longest time that your organization can tolerate. This is shown below in Figure 1-1. 1-2 Data Center: SAN Extension for Business Continuance Version 1.2 Chapter 1 SAN Extension for Business Continuance: Overview Overview Figure 1-1 Recovery Time and Recovery Time Objective The technologies and processes used in a network largely determine how well an organization achieves its RTO. For example, an extended cluster offers better recovery capabilities than manual migration or tape-based restoration. RPO refers to how current or fresh the data is after a disaster. This is shown below in Figure 1-2. RPO is the maximum data loss that your organization can tolerate after a disaster. Figure 1-2 Recovery Point and Recovery Point Objective As with the RTO, the systems, processes and technologies employed will determine if you achieve your RPO. Additionally, the distance between the data centers, and how well your applications tolerate network latency, determine whether zero RPO is possible. The following recovery technologies support increasingly lower RPO at increasingly higher cost: • Tape backup and restore • Periodic replication and backups • Asynchronous replication • Synchronous replication. The disaster radius refers to how extensive the disaster is. Earthquakes, floods, fires, hurricanes, cyclones and attacks will all have varying probabilities and disaster radii according to the geographic region in which they occur. The key issue is that the backup site must not be within the radius of possible threats. For example, an earthquake could destroy both primary and secondary data centers if they were both separated geographically but connected by a major fault line. Many enterprises adopt a multi-hop strategy to minimize their exposure to this situation—two data centers located within metro distances, and a third disaster recovery site located out of the region. This is shown below in Figure 1-3. 104725 Disaster occurs Systems Inoperative Systems Recovered and Operational time t 1 Recovery time time t 2 time 104726 Disaster occurs Period over which data is "lost" Last point at which data is in a valid state time t 0 Recovery point time t 1 time [...]... information, with special consideration of the trade-offs between application performance, geographical separation, and RPO/RTO, will provide a solid basis for determining the appropriate replication and SAN Extension solution for your enterprise Data Center: SAN Extension for Business Continuance 1-22 Version 1.2 Chapter 1 SAN Extension for Business Continuance: Overview Summary Data Center: SAN Extension. .. Summary Data Center: SAN Extension for Business Continuance Version 1.2 1-23 Chapter 1 SAN Extension for Business Continuance: Overview Summary Data Center: SAN Extension for Business Continuance 1-24 Version 1.2 C H A P T E R 2 Designing FCIP SAN Extension for Cisco SAN Environments This chapter provides background information on the FCIP protocol and describes performance considerations involved when... bandwidth utilization Data Center: SAN Extension for Business Continuance 1-6 Version 1.2 Chapter 1 SAN Extension for Business Continuance: Overview SAN Extension Transport Overview Figure 1-8 Asynchronous Replication –Ordered Writing Local Data Center Host Remote Data Center SCSI Write Transfer Ready Disk I/O Service Time FCP Data (2kB frames) SCSI Write SCSI Status=good Transfer Ready FCP Data (2kB frames)... appropriate SFPs for distance and fiber type MDS 9000 at each end with CWDM SFPs CWDM OADMs and Muxes ONS15530 ONS15540 ONS15454 MSTP ONS15454 MSPP (SONET/SDH) MDS 9000 IP Services Module PA-FC-1G for 7200 and 7400 SN5428-2 Data Center: SAN Extension for Business Continuance 1-10 Version 1.2 Chapter 1 SAN Extension for Business Continuance: Overview SAN Extension Transport Overview Table 1-1 SAN Extension. .. application performance (providing adequate network bandwidth is available for ongoing replication) Data Center: SAN Extension for Business Continuance Version 1.2 1-17 Chapter 1 SAN Extension for Business Continuance: Overview SAN Extension Solutions A number of WAN deployment alternatives are available depending upon user requirements Some of these are shown in the Figure 1-14 below In this dual data center... IP Services Module at a central data center, shown in Figure 1-17 Figure 1-15 Simple FCIP SAN Extension Solution Over IP WAN Using PA-FC-1G Port Adapters in 7200 or 7400 routers IP Network FCIP Cisco 7200 or 7400 with PA-FC-1G 104739 FCIP Data Center: SAN Extension for Business Continuance 1-18 Version 1.2 Chapter 1 SAN Extension for Business Continuance: Overview SAN Extension Solutions The SN5428-2... others applicable only to asynchronous replication A SAN Extension transport overview is shown below in Figure 1-10 Data Center: SAN Extension for Business Continuance Version 1.2 1-7 Chapter 1 SAN Extension for Business Continuance: Overview SAN Extension Transport Overview Figure 1-9 Replication Methods with Optical and IP Transports Increasing Distance Data Center Campus Metro Regional National Dark... into one 32 Gbps logical ISL VSAN trunking permits transparent sharing of any or all of these port channels between isolated SAN fabrics Data Center: SAN Extension for Business Continuance 1-14 Version 1.2 Chapter 1 SAN Extension for Business Continuance: Overview SAN Extension Solutions Distance Capabilities Unlike DWDM, CWDM lambdas or channels cannot be amplified, except for the single lambda around... vendor implementations may vary Data Center: SAN Extension for Business Continuance 1-4 Version 1.2 Chapter 1 SAN Extension for Business Continuance: Overview Replication and Mirroring Figure 1-6 Synchronous Replication – Protocol Detail Local Data Center Host Remote Data Center SCSI Write Transfer Ready FCP Data (2kB frames) SCSI Write Disk I/O Service Time Transfer Ready FCP Data (2kB frames) t Response... appropriate for encryption over DS3 or E3 links When employed on the 7200, this hardware 3DES encryption port adapter can be used in conjunction with the PA-FC-1G port adapter for a single-chassis FCIP solution with encryption and compression See Figure18— below Data Center: SAN Extension for Business Continuance Version 1.2 1-19 Chapter 1 SAN Extension for Business Continuance: Overview SAN Extension . utilization. 1-7 Data Center: SAN Extension for Business Continuance Version 1.2 Chapter 1 SAN Extension for Business Continuance: Overview SAN Extension Transport. P Sync 1-9 Data Center: SAN Extension for Business Continuance Version 1.2 Chapter 1 SAN Extension for Business Continuance: Overview SAN Extension Transport