NCAR design proposal

Cisco Systems Advanced Services National Center for Atmospheric Research Campus Design Review And Best Practices Recommendation Version 1.2 Corporate Headquarters Cisco Systems, Inc 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-4000 800 553-NETS (6387) Fax: 408 526-4100 THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY The following information is for FCC compliance of Class A devices: This equipment has been tested and found to comply with the limits for a Class A digital device, pursuant to part 15 of the FCC rules These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment This equipment generates, uses, and can radiate radio-frequency energy and, if not installed and used in accordance with the instruction manual, may cause harmful interference to radio communications Operation of this equipment in a residential area is likely to cause harmful interference, in which case users will be required to correct the interference at their own expense The following information is for FCC compliance of Class B devices: The equipment described in this manual generates and may radiate radio-frequency energy If it is not installed in accordance with Cisco’s installation instructions, it may cause interference with radio and television reception This equipment has been tested and found to comply with the limits for a Class B digital device in accordance with the specifications in part 15 of the FCC rules These specifications are designed to provide reasonable protection against such interference in a residential installation However, there is no guarantee that interference will not occur in a particular installation You can determine whether your equipment is causing interference by turning it off If the interference stops, it was probably caused by the Cisco equipment or one of its peripheral devices If the equipment causes interference to radio or television reception, try to correct the interference by using one or more of the following measures: Turn the television or radio antenna until the interference stops Move the equipment to one side or the other of the television or radio Move the equipment farther away from the television or radio Plug the equipment into an outlet that is on a different circuit from the television or radio (That is, make certain the equipment and the television or radio are on circuits controlled by different circuit breakers or fuses.) Modifications to this product not authorized by Cisco Systems, Inc could void the FCC approval and negate your authority to operate the product The following third-party software may be included with your product and will be subject to the software license agreement: CiscoWorks software and documentation are based in part on HP OpenView under license from the Hewlett-Packard Company HP OpenView is a trademark of the Hewlett-Packard Company Copyright  1992, 1993 Hewlett-Packard Company The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public domain version of the UNIX operating system All rights reserved Copyright  1981, Regents of the University of California Network Time Protocol (NTP) Copyright  1992, David L Mills The University of Delaware makes no representations about the suitability of this software for any purpose Point-to-Point Protocol Copyright  1989, Carnegie-Mellon University All rights reserved The name of the University may not be used to endorse or promote products derived from this software without specific prior written permission The Cisco implementation of TN3270 is an adaptation of the TN3270, curses, and termcap programs developed by the University of California, Berkeley (UCB) as part of the UCB’s public domain version of the UNIX operating system All rights reserved Copyright  1981-1988, Regents of the University of California Cisco incorporates Fastmac and TrueView software and the RingRunner chip in some Token Ring products Fastmac software is licensed to Cisco by Madge Networks Limited, and the RingRunner chip is licensed to Cisco by Madge NV Fastmac, RingRunner, and TrueView are trademarks and in some jurisdictions registered trademarks of Madge Networks Limited Copyright  1995, Madge Networks Limited All rights reserved Xremote is a trademark of Network Computing Devices, Inc Copyright  1989, Network Computing Devices, Inc., Mountain View, California NCD makes no representations about the suitability of this software for any purpose The X Window System is a trademark of the X Consortium, Cambridge, Massachusetts All rights reserved NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH ALL FAULTS CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PRACTICAL PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES AccessPath, AtmDirector, Browse with Me, CCDE, CCIP, CCSI, CD-PAC, CiscoLink, the Cisco NetWorks logo, the Cisco Powered Network logo, Cisco Systems Networking Academy, Fast Step, Follow Me Browsing, FormShare, FrameShare, GigaStack, IGX, Internet Quotient, IP/VC, iQ Breakthrough, iQ Expertise, iQ FastTrack, the iQ logo, iQ Net Readiness Scorecard, MGX, the Networkers logo, Packet, RateMUX, ScriptBuilder, ScriptShare, SlideCast, SMARTnet, TransPath, Unity, Voice LAN, Wavelength Router, and WebViewer are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn, Discover All That’s Possible, and Empowering the Internet Generation, are service marks of Cisco Systems, Inc.; and Aironet, ASIST, BPX, Catalyst, CCDA, CCDP, CCIE, CCNA, CCNP, Cisco, the Cisco Certified Internetwork Expert Logo, Cisco IOS, the Cisco IOS logo, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Enterprise/Solver, EtherChannel, EtherSwitch, FastHub, FastSwitch, IOS, IP/TV, LightStream, MICA, Network Registrar, PIX, Post-Routing, Pre-Routing, Registrar, StrataView Plus, Stratm, SwitchProbe, TeleRouter, and VCO are registered trademarks of Cisco Systems, Inc and/or its affiliates in the U.S and certain other countries All other trademarks mentioned in this document or Web site are the property of their respective owners The use of the word partner does not imply a partnership relationship between Cisco and any other company (0105R) INTELLECTUAL PROPERTY RIGHTS: THIS DOCUMENT CONTAINS VALUABLE TRADE SECRETS AND CONFIDENTIAL INFORMATION OF CISCO SYSTEMS, INC AND IT’S SUPPLIERS, AND SHALL NOT BE DISCLOSED TO ANY PERSON, ORGANIZATION, OR ENTITY UNLESS SUCH DISCLOSURE IS SUBJECT TO THE PROVISIONS OF A WRITTEN NON-DISCLOSURE AND PROPRIETARY RIGHTS AGREEMENT OR INTELLECTUAL PROPERTY LICENSE AGREEMENT APPROVED BY CISCO SYSTEMS, INC THE DISTRIBUTION OF THIS DOCUMENT DOES NOT GRANT ANY LICENSE IN OR RIGHTS, IN WHOLE OR IN PART, TO THE CONTENT, THE PRODUCT(S), TECHNOLOGY OF INTELLECTUAL PROPERTY DESCRIBED HEREIN Design Review Template Copyright  2001-2, Cisco Systems, Inc All rights reserved COMMERCIAL IN CONFIDENCE A PRINTED COPY OF THIS DOCUMENT IS CONSIDERED UNCONTROLLED Contents Contents Document Control History Review About The Document Document Purpose Business Profile Current Topology Current Design Overview Overview of Recommendations Executive Summary Short Term Design Enhancements Foothills-CenterGreen: The Radio Link Mixed Core (L2 + L3) Hierarchical (MultiLayer) Network Design: An Overview Overview 11 11 Long Term: AS-Proposed Design 14 Configuration Best Practices 16 VLAN 16 VLAN Trunking Protocol (VTP) 17 Trunking Mode 17 Spanning Tree Protocol (STP) 18 Unidirectional Link Detection (UDLD) 20 EtherChannel (PAgP) NCAR Design Review and Recommendations v1.2 21 Document Control Author: Hazim Dahir Advanced Services – Central Engineering History Table Revision History Version No Issue Date Status Reason for Change 1.0 20-Feb-2003 Draft, Released First release 1.1 11-Apr-2003 1.2 24-Apr-2003 Introduced Mixed Core Final Updated Diagrams and added Configuration observations Review Table Revision Review Reviewer’s Name Version No Tsegerada Beyen and Niels Brunsgaard 1.0 Date Advanced Services Network Engineering Team 1.1 Apr-17-2003 NCAR Change Forecast: High This document will be kept under revision control A printed copy of this document is considered uncontrolled NCAR Design Review and Recommendations v1.2 About The Document Document Purpose This network design review document is intended to provide an overall assessment of the design aspects of the network and select operational functions The comments presented in this document are a result of information learned about the network from customer-documentation as well as the weekly discussions This assessment is part of the Performance Engineering and Optimization services provided by the Central Engineering Team This service will give a best practice assessment of the network as a complete system It uses data collected about individual devices or interfaces to generate an assessment of “Campus Best Practices” This assessment would consider network Availability, Scalability, Convergence, Modularity, Hierarchical Design and other network stability aspects Business Profile Understanding the business goals of a company or institution is very important when analyzing a network design The goal of a good network design is to empower users in meeting company objectives The network should provide an acceptable level of performance and reliability while not wasting capital and other resources in the process of over-engineering the network Nor should a network be under-engineered such that it fails to meet the service levels necessary to meet the business objectives Many design decisions are a result of thoughtful risk/benefit analysis The National Center for Atmospheric Research, NCAR, was established in 1960 to serve as a focus for research on atmospheric and related science problems and is recognized for its scientific contributions to our understanding of the earth system, including climate change, changes in atmospheric composition, Earth-Sun interactions, weather formation and forecasting, and the impacts of all of these components on human societies With two major sites in Boulder, I.M Pei's Mesa Laboratory and a newer Foothills Laboratory, NCAR's research is conducted in several principal disciplinary areas: atmospheric chemistry; mesoscale and microscale meteorology; solar and solar-terrestrial physics; and climate and the linking of climate with other environmental systems Focused contributions are also made to national scientific initiatives There are multi-disciplinary and cross-disciplinary efforts aimed at the development of a coupled climate system model which will simulate the complex interrelations between climate, weather, the sun, and the biosphere and oceans Research on the societal interactions with atmospheric processes is an integral part of NCAR's program NCAR Design Review and Recommendations v1.2 About The Document Current Topology UNAVCO M e sa Lab Pe arl Stre e t POTS ml-mr-c2-gs ps-3018-c1-ts ml-062-c1-gs uv-18-c1-es ps-3018-c1-es ps-1027a-c1es ps-2008-c1-es ml-mr-c2-ts Ml-371-c1gs Foothills Lab f l2-2076-c1-e s netscm netscm2 ml-50-c1-gs f l1-2152-c1-gsf l3-3087-c1-gs f l3-3068-c1-gs f l4-1012-c1-gs netscm3 ml-16c-c1-gs f l1-2037a-c1-gs f l2-3095-c1-as ml-mr-c2-es f l2-3095-c1-ts f l2-3095-c1-gs gin ml-mr-c3-gs f l2-2104-c1-gs f l2-2069-c1-gs f l4-2060-c1-gs ml-47-c1-gs ml-mr-c1-g s ml-mr-c1-ts Ce nte r Gre e n cg1-2010-c3-e s cg1-2036-c1-gs cg2-mr-c1-ts ml-mr-c1-as ml-243b-c1-gs voip1r-n2 cg-voipr ml-y2k-c1-gs ml-y2k-c1-as cg2-mr-c1-gs cg1-3010-c1-es vbnsr cg1-2010-c4-es cg1-2010-c1-escg1-3010-c2-es M arshall jef -126-c1-as marshallr-n406 mar-26a-c1-es f b-12-c1-es jef -126-c1-es NCAR Network Diagram A TM links Gigabit Ethernet links f b-12-c2-es jef -126-c1-ts Fle ischman Building cg1-2010-c2-es cg1-3036-c1-gs Je ffCo L3 sw itch L2 Sw itch A TM Sw it ch Terminal Server Current Design Overview The current design spreads over three major campuses The largest and most populated are Mesa and Foothills Center Green utilizes an L2 switch as an aggregation point The existing design facilitates for several VLANs to span multiple switches as well as multiple sites Although this is not recommended, at NCAR this does not present any immediate problems or issues The single homing of switches to the perspective core switch and the absence of a Spanning Tree loop at the core provide for a stable environment At the current time, NCAR is satisfied with the current “availability” model and hope to improve it in the future For example, the Mesa site, acts as a transport site for all traffic exchanged between CenterGreen NCAR Design Review and Recommendations v1.0 About The Document and Foothills A total failure of the “ml-mr-c1-gs” switch will isolate all three major sites Relying on internal redundancy (Dual-Supervisor and HA feature) helps reduce the chance of that type of failure NCAR Design Review and Recommendations v1.0 Overview of Recommendations Executive Summary The advent of high-speed L3 switches has moved modern Enterprise/Campus Network Design away from the flat L2 vlan-based model Cisco’s current Campus Reference Design Model, commonly known as the Multilayer Model, features high-powered L3 switches placed in key areas of the Enterprise Network This document concentrates on key design concepts required for mission critical networks The most important ones are: - Hierarchical Network Model: Characterization of traffic flow - Modularity: Network made up of distinct network blocks - Scalability: Allow network to grow without major changes or redesign - High Availability: Internal, External, and path redundancy - Predictability: Traffic Flows, delays, bounds, fail-over paths are predictable - Simplicity: Satisfy network requirements with the least amount of effort or Hardware The following sections attempt to describe two design improvement approaches: Short Term Design Enhancement Long Term AS Proposed Design NCAR Design Review and Recommendations v1.2 Short Term Design Enhancements Foothills-CenterGreen: The Radio Link NCAR is testing a Radio Link (TeraBeam) for possible deployment into the production environment to connect Foothills with CenterGreen If we allow the Radio Link to act a trunk, then we are creating an environment that is STP dependent for convergence That in return will force one of the Core links to be in the “Blocking” state Reliability and Utilization common sense force us to “block” the Radio Link All Links (including the Radio Link) are better utilized in an L3 Core With the three major sites participating, this would be a full mesh Mesa will no longer be the only link between Foothills and CenterGreen Mixed Core (L2 + L3) The presence of several campus-wide VLANs requires Trunking (ISL or dot1Q) in the Core Those VLANs would be handled by two trunks connecting the three sites This is exactly how all traffic is handled today Other VLANs that are unique to an L2 switch or to one of the sites will be cleared from the L2 trunk and can be routed via a separate L3 connection This is best described by the following diagram By adding a routing engine to the CenterGreen switch, three unique VLANs can be configured to represent the L3 core The other L2 will only carry traffic for the VLANs requiring campus-wide configuration (all other VLANs must be cleared from the trunk) An important decision needs to be taken here regarding the Active gateway(s) for the ‘L2” VLANs Any two routers in any of the three sites can handle that requirement We can also consider M-HSRP and have one router active for half the VLANs and another router active for the other half (Point of Discussion: This document will updated accordingly) NCAR Design Review and Recommendations v1.2 Error! Reference source not found Mixed Core Design - The L3 Core consists of the three independent point-to-point VLANs X, Y, and Z - The L2 trunk illustrated by the blue lines will carry VLANs that need site-wide accessibility - VLANs X, Y, and Z to be cleared from the dot1Q trunk - All Site-specific VLANs to be cleared from the dot1Q trunks NCAR Design Review and Recommendations v1.0 10 Hierarchical (MultiLayer) Network Design: An Overview Overview The hierarchical three tiered campus design has become the preferred architecture for most networks The three tiered architecture is comprised of an access layer that directly connect network users by means of switches, normally placed in wiring closets positioned throughout the campus Access layer switches are also connected to a distribution layer The distribution layer sites will have a number of access switches connected to it The number of access switches connected to the distribution layer is often determined by geographic proximity, such as all the access switches in a building homing into one distribution site for that building The distribution sites normally consist of switches with layer and layer functions The distribution layer switches are usually deployed in pairs for system redundancy The distribution layer switches are then connected to a core layer switch The core switches are also usually deployed in pairs for redundancy and may support layer as well as layer functions An overall campus design such as this might have a numerical profile of 8000 users connected to 40 access switches that are then connected to distribution sites that are finally connected to core site Access Distribution Backbone Core Building block additions Data Center WAN Internet Remote Access NCAR Design Review and Recommendations v1.2 11 Hierarchical (MultiLayer) Network Design A hierarchical network design distributes networking function at each layer through a layered organization Modular designs are made out of building blocks Modules and can be added or removed without redesigning the network A modular design is also easier to grow and troubleshoot Cisco’s multilayer design is an example of modular hierarchical design model The key elements of the structured hierarchy are the Core, Distribution and Access layer in a network The Access Layer This layer is made up of pure L2 (layer 2) switches each with dual uplinks to two L3 distribution switches There are basically two types of access vlans that can be implemented in this area and most often we see a mix of both The Access Layer model use one of two methods to provide redundancy and/or load balancing: The HSRP Model: It does not utilise STP and has faster convergence in the face of failures There is no regular routing protocol running over the uplinks, only hsrp (passive vlan interfaces) Load-balancing is achieved by having half of the vlans use one distribution switch as their default gateway and the other half use the other distribution switch It should be pointed out that the standard access model does not call for STP to be turned off on the switches STP should be enabled but no L2 loops should physically exist This is a precaution to avoid the network to collapse in case of wrong cabling or when channeling feature misbehaves One common perception is that a unique vlan must be configured on each access switch, this is not true as it is allowed and sometimes advantageous to configure several vlans per access switch; the only thing that is not allowed is to configure one same vlan on more than one access switch The Spanning Tree Model: In the early days of the VLAN technology it was a common design to group clients and their corresponding servers within the same vlan to fully take advantage of the performance of L2 switching As a result, workgroup servers were usually attached at the distribution layer where all vlans would meet Each of these vlans would also span the whole network thereby potentially creating extended STP topologies These designs were heavily based on the vlan concept accompanied with its highly publicized flexibility (mobility) and therefore had quite a bit of a success Although, building-wide vlans are discouraged in the multilayer model it has to be admitted that real networks often make heavy use of their conveniences and cannot work around them easily Hence, another access model is presented with a very simple STP topology such that it can be tolerated in the multilayer model In the STP/VLAN-based access model, we restrict the STP topology into a triangle Primary and secondary roots are always located at the distribution layer so that each vlan blocks once on each access switch Many materials explain how to design, implement and troubleshoot STP optimally for this situation In this case, you want to have at least two vlans per access switch in order to use the bandwidth on all links including blocked uplinks On one access switch, each vlan should block a different uplink – this can easily be achieved by using a different primary root at the distribution layer for each vlan Helpful hints and features to deploy in the access layer: • • • • • • Use UDLD and loop-guard to prevent the adverse effects of STP failures Keep the topology as simple as possible Triangular shaped topologies are very well suited to implement the STP’s enhancements (x-fast) Use multiple vlans and PVST+ to loadshare traffic across blocked ports Location of blocked ports must be very easily predicted (a consequence of simple triangular topologies) The distance to the root must be kept to a minimum Carefully choose the root location The Distribution Layer NCAR Design Review and Recommendations v1.0 12 Hierarchical (MultiLayer) Network Design This layer interfaces the Access layer with the Core Layer; it is a major aggregation point and plays a very important role The distribution layer terminates all access vlans from the layer perspective High performing Catalysts 6k are typically used to route the vlans efficiently at wire speed towards the backbone The switches have routing intelligence to participate in any routing protocol with core routers as well as providing HSRP functionality to load share traffic coming from the access layer The Distribution switches not inject routing updates into the access layer in normal circumstances Although it consists only of two L3 switches, the way we implement and interconnect them to other layers is not trivial Helpful hints and features to deploy in the access layer: • • Aggregates the Access Layer (wiring closet) switches Provides uplink connectivity to the Core LayerReduces the need of high density peering with a Core-only Layer • Redundancy (Availability), Load Balancing and Capacity Planning (Provisioning) are the most important aspects o Use Layer Switching in this Layer o Use HSRP and HSRP-Tracking for first-hop redundancy The Core Layer The backbone of the network Aggregates the Distribution Layer switches and plays the primary role of connecting other network building blocks It should provide redundancy, rapid convergence using high-speed connections and high-speed Layer switching The Server Farm: A server farm is implemented as a high-capacity building block attached to the campus backbone, and is treated as its own distribution block Server farm is an aggregation point for much of the traffic from the whole campus As such, it makes sense to design the server farm with less over subscription of switch and trunk resources than the normal building block For example, if the other campus building blocks connect to the backbone with Fast Ethernet, then the server farm should probably attach to the backbone with Gigabit Ethernet How we handle Campus or Enterprise-Wide VLANs? These are definitely incompatible with the Multilayer model, which is a strictly routed environment There are generally two possible ways to force these protocols on the model, either we build Campus-Wide vlans with complex Spanning Tree topologies spanning the whole network or we need to bridge these protocols on the routers Both alternatives undermine the robustness of the network NCAR Design Review and Recommendations v1.0 13 Long Term: AS-Proposed Design This section will detail a design proposal from the AS team The following design is hierarchical, redundant, and modular If fully implemented, the design presents a solution where each of the campuses is totally independent from the other from a traffic and fail-over point of views Mesa/Marshall Fleischman Jeffco Server Farm POTS/WAN 10/100/1000 L3 GEC MESA Si Si Foot Hills Center Green Gigabit Core Si Si Si Si FootHills/Pearl Street/UNAVCO Semi-Fully Meshed Core design A link or Switch failure at any of the buildings will not affect any other building The design is also optimised for IP Telephony and Multicast and any QoS strategy needed for any technology or application The above diagram provides an idea of the final design; a phased migration is provided later The proposed design offers the following advantages and common features: At the Access Layer: - Switches would have unique VLANs (one or several that exist only on that switch) This in return eliminates dependency on Spanning Tree Protocol for convergence NCAR Design Review and Recommendations v1.2 14 Hierarchical (MultiLayer) Network Design - User ports to be configured with PortFast (See best practices section below) - ISL or dot1Q trunks to the Distribution layer Allow only Access Layer VLANs on the trunk - The ability to load-balance traffic over the Access-Distribution trunks (This is done using HSRP) If VLANs span multiple switches or buildings, then we can achieve load balancing using STP by alternating “Root” configuration between Distribution Switches - If STP is the choice for redundancy, the UplinkFast feature is utilized to offer almost instantaneous fail-over of uplinks At the Distribution Layer: - Two Distribution switches per building to provide maximum redundancy and performance The two distribution switches are also directly connected to the Core using L3 connections - Terminate and limit the size of broadcast domains With unique VLANs configured at the Access switches, the link between the two Distribution in every building can be an L3 connections - A failure in one of the closets will only be limited to that closet - Configure HSRP with alternating priority levels between the two Distribution-MSFCs to provide redundancy as well as load balancing of the Uplinks - Easy to manage and troubleshoot - Root bridges are local and known No Root is across the WAN links At the Core Layer: - Layer Core - Highly redundant - Simple Design - Point-to-Point L3 connections NCAR Design Review and Recommendations v1.0 15 Configuration Best Practices VLAN VLAN1 has a special significance in Catalyst networks, and can be the cause of some confusion The Catalyst Supervisor always uses the Default VLAN, 1, to tag a number of control and management protocols when trunking, such as CDP, VTP and PAgP! All ports including the internal sc0 interface are configured to be members of VLAN1 at initial boot up All trunks carry VLAN1 by default, and in older Catalyst code, (prior to CatOS 5.3), it was not possible to block user data in VLAN1 Two other definitions are needed to help clarify some well-used terms in Catalyst networking: • The Management VLAN is where sc0 resides, and can be changed • The Native VLAN is defined as the VLAN a port will return to when not trunking, and is the untagged VLAN on an 802.1Q trunk There are several good reasons to tune a network and alter the behavior of ports in VLAN1: • When the diameter of VLAN1, like any other VLAN, gets large enough to be a risk to stability, particularly from an STP perspective, it needs to be pruned back • Control plane data on VLAN1 should be kept separate from user data in other VLANs, and ideally the management VLAN, to simplify troubleshooting and maximise available CPU • Layer-2 loops in VLAN1 must be avoided when designing Multilayer Campus networks without STP, yet trunking is still required to the access-layer multiple there are multiple VLANs and IP subnets To this, manually clear VLAN1 from trunk ports Note: • CDP, VTP and PAgP updates are always forwarded with a VLAN1 tag even if VLAN1 has been cleared on trunks and is not the native VLAN One implication is that when trunking to a router, an interface must be defined in VLAN1 to detect CDP • DTP updates are forwarded with VLAN1 tags on ISL trunks, but use the native VLAN on 802.1Q • 802.1Q IEEE BPDUs are forwarded untagged on the ‘Common Spanning Tree’ VLAN1 (like ISL) for interoperability with other vendors, unless VLAN1 has been cleared from the trunk Cisco PerVLAN Spanning tree (PVST+) BPDUs are sent and tagged for all other VLANs Analysis NCAR Design Review and Recommendations v1.2 16 Hierarchical (MultiLayer) Network Design VLAN Trunking Protocol (VTP) Before you create VLANs you must decide what VTP mode to use in your network With VTP you can make VLAN configuration changes centrally on one or more switches and have those changes automatically communicated to all the other switches in the network VTP is a Layer messaging protocol that maintains VLAN configuration consistency by managing the addition, deletion, and renaming of VLANs on a network-wide basis VTP minimizes mis-configurations and configuration inconsistencies that can result in a number of problems such as duplicate VLAN names, incorrect VLAN-type specifications, and security violations The VLAN database built is stored in NVRAM, separately the configuration AS recommends transparent mode As the majority of customers not create or delete VLANs frequently, when a new VLAN is needed it is not much effort to update all switches in a domain, usually numbering 20 or less • This practice encourages good change control • Limits the risk of a user error, such as deleting a VLAN, impacting the entire domain • Eliminated the risk of any VTP bug affecting the entire network • There is no risk from a new switch being introduced into the network with a higher VTP revision number and over-writing the entire domain's VLAN configuration There is a positive and negative side to VTP being able to make changes very easily on a network - most enterprises prefer a cautious approach • STP per VLAN and unnecessary flooding should be limited by explicit configuration (i.e pruning) of what VLANs are propagated on what trunks A per switch VLAN configuration also encourages this practice • The extended VLAN range in CatOS 6.x, numbers1025-4094, can only be configured in this way Trunking Mode Purpose: DTP is the second generation of DISL (Dynamic ISL) and exists to ensure that the different parameters involved in sending ISL or 802.1Q frames, like the configured encapsulation type, native VLAN, hardware capability, etc are agreed by the Catalysts at either end of a trunk This also helps protect against non-trunk ports flooding tagged frames, a potentially serious security risk, by ensuring ports and their neighbors are either in a safe trunking or non-trunking state Operational Overview: DTP is a layer-2 protocol that negotiates configuration parameters between a switch port and it's neighbor It uses another well-known multicast MAC address of 01-00-0c-cc-cc-cc and a SNAP protocol type of 0x2004 Here is a summary of the configuration modes: Note: ISL and 802.1Q encapsulation type can be set or negotiated - ISL will be preferred over dot1Q, but is recommended to be set • DTP assumes point-to-point connection, and Cisco devices will support 802.1Q trunk ports that are only point-to-point • During DTP negotiation, the ports will not participate in STP Only after the port type becomes one of the three types (Access, ISL or 802.1Q), the port will be added to STP (If PAgP is running that is the next process to run prior to the port participating in STP) • VLAN will usually be there on the trunk port If the port is trunking in ISL mode, DTP packets are sent out on VLAN 1, otherwise (for 802.1Q trunking or non-trunking ports) on the Native VLAN NCAR Design Review and Recommendations v1.0 17 Hierarchical (MultiLayer) Network Design • In desirable mode DTP packets transfer the VTP domain name, which must match for a negotiated trunk to come up, plus trunk configuration and admin status • Messages are sent every 1s during negotiation, and every 30s after that • Be careful that it is understood modes (on, nonegotiate, off) explicitly specify in which state the port will end up A bad configuration can lead to a dangerous inconsistent state where one side is trunking and the other is not A port in on, auto, or desirable sends DTP frames periodically If a port in auto or desirable mode doesn't see a DTP packet in 5min it will be set to non-trunk AS recommends an explicit trunk configuration of ‘Desirable-Desirable’ In this mode, network operators can trust syslog and command line status messages that a port is up and trunking, unlike the mode ‘on’ that can make a port appear up even though the neighbor is mis-configured With the consistency of using the same configuration at both ends of the trunk, the ambiguous and default setting ‘auto’ can be avoided set trunk desirable Importantly, also set 'trunk off' on all non-trunk ports This helps eliminate wasted negotiation time when bringing host ports up (in conjunction with ‘port channel mode off’ and PortFast enabled - set port host) Set trunk off all By default, Trunking ports will propagate information about all VLANs, but AS recommends limiting that to the VLANs defined on the wiring closet switches This practice has many advantages, most importantly, is the ability to isolate issues, broadcasts, and loops to one wiring closet instead of the entire network Spanning Tree Protocol (STP) Spanning tree maintains a loop free layer two environment in redundant switched and bridged networks Without STP, frames would loop and/or multiply indefinitely causing a network meltdown as all devices in the broadcast domain are interrupted by high traffic As an initial observation, all Catalyst switches have STP enabled by default AS recommends leaving the feature enabled for these reasons: • When there is a loop (induced by mis-patching/bad cable, etc.), STP will prevent detrimental effects to the network, potentially made much worse with multicast and broadcast data • Protection against channel breaking down • Most of networks are configured with STP and hence gets maximum exposure More exposure generally equates to more stable code • Protection against dual attached NICs misbehaving (or routing enabled on servers) • Many protocols are closely related to STP in the code (such as PAgP, IGMP snooping, trunking etc.) running without it may lead to undesirable results During a reported network disruption, TAC and DE will most likely suggest that non-usage of STP will be at the centre of the fault if at all conceivable NCAR Design Review and Recommendations v1.0 18 Hierarchical (MultiLayer) Network Design set spantree enable all ! default PortFast PortFast is used to bypass normal spanning-tree operation on access-ports to speed up connectivity between end-stations and the services they need to connect to after link initialization AS recommend STP PortFast be set on for all enabled host ports Must be diligent to catch all ports!!! AS recommends using the “host” macro to configure PortFast on Access ports: Set port host ! macro for the following commands set spantree portfast enable set trunk off set channel mode off Note that PortFast doesn’t mean that we not run spanning-tree at all on those ports: BPDUs are still sent, received and processed It must also be understood that PortFast will not work on trunks since these ports are typically not access-ports This may cause confusion on access-ports running ISL or dot1q connected to multi-homed trunking capable end-stations PortFast BPDU-Guard provides a method for preventing loops by moving a non-trunking port into an ErrDisable state when a BPDU is received on that port Under “normal” conditions we should never receive a BPDU packet on an access-port configured for PortFast, so if we for some reason should see a BPDU coming in, it indicates an invalid hence dangerous configuration and the action we take upon this is to shut down the access-port When the BPDU Guard feature is enabled on the switch spanning tree shuts down PortFast-configured interfaces that receive BPDUs instead of putting them into the spanning-tree blocking state set spantree portfast bpdu-guard enable UplinkFast UplinkFast is a solution for Access Layer switches to move it’s blocking link almost instantaneously to forwarding if there is a direct link failure to the root switch Access Layer switches using UplinkFast forward their CAM table over the new root port, preventing unknown unicasts to flood UplinkFast feature uses uplink group, which consists of the root port and all the ports that provide an alternate connection to the root bridge (blocking redundant links) If the root port fails (primary uplink failure), a port from the uplink group is selected to immediately replace it Uplinkfast feature is only useful when there is a redundant blocking link Uplinkfast and the following Cross stack uplinkfast feature are not necessary if a design with no layer loops is implemented NCAR Design Review and Recommendations v1.0 19 Hierarchical (MultiLayer) Network Design Spanning Tree Root (Primary) Trunk Distribution Access Spanning Tree Root (Secondary) Gi0/2 Gi0/1 CatOS> (enable) set spantree uplinkfast enable BackboneFast In the above figure if the link between the two distribution layer switches failed then it’s considered an indirect root link failure After an indirect failure, the secondary root bridge will start transmitting configuration BPDUs claiming itself to be the root Access switch upon receiving the BPDU will realize that the BPDU is inferior compared to what it has on its root port The access switch will then transmit a Root Link Query message on its root port to determine if the original root is still alive Upon receiving a positive response from the root that it is still alive, Access switch will transmit a BPDU to the secondary root bridge so that it can calculate the path to root Access switch will also transition the blocking port to listening and learning state then forwarding Without backbonefast the Access switch won’t have reacted to the inferior BPDU until max-age (20 seconds by default) on the port expired Therefore backbonefast cuts the convergence time for indirect failure by 20 seconds The access port still goes through listening and learning states, which takes 30 seconds in total CatOS> (enable) set spantree backbonefast enable Unidirectional Link Detection (UDLD) The Uni-Directional Link Detection (UDLD) feature is intended to resolve some of the following issues: - Monitor physical cabling configurations - shutting down any mis-wired ports as ‘ErrDisabled’ - Protect against Uni-directional links, on Fibre-optical and copper Ethernet interfaces When a unidirectional link is detected -due to media or port/interface malfunction- the affected port is shutdown as ‘ErrDisabled’ and a corresponding syslog message generated - To be clear: it is the top switch in these diagrams that see a unidirectional link and will place the port in Errdisable Spanning tree, with its steady state unidirectional BPDU flow, was an acute sufferer from the above failures It is easy to see how a port may suddenly be unable to transmit BPDUs, causing an STP state change from 'blocking' to 'forwarding' on the neighbor, yet a loop still existing as the port is still able to receive For maximum protection against symptoms resulting from uni-directional links, AS recommends enabling UDLD on point-to-point FE/GE links between “aggressive-UDLD capable” Cisco switches – where the message interval is configurable / set to 15 seconds default NCAR Design Review and Recommendations v1.0 20 Hierarchical (MultiLayer) Network Design By default UDLD is disabled globally, and enabled in readiness on fiber ports As UDLD is an infrastructure protocol needed between switches only, it is disabled by default on copper ports as these tend to be used for host access Set udld enable ! once globally enabled, all FE and GE fibre ports have UDLD enabled by default Set udld enable ! for additional specific ports and copper media if needed Set udld aggressive-mode enable ! all point to point links Enabling UDLD ‘aggressive-mode’ provides additional benefits: This feature provides enhanced protection against dangerous unidirectional link conditions in the following situations, and includes attempts to re-establish a connection with the neighbor upon failure detection: - One side of a link has a port stuck (either Tx and Rx) - One side of a link remains up while the other side of the link has gone down This reduces the reliance on Layer FEFI mechanisms - After eight failed retries, the port is transitioned to an ErrDisable state and a syslog message logged - In these cases, UDLD aggressive mode will ErrDisable both of the ports on the link and stops the black-holing of traffic Aggressive mode UDLD also allows the possibility to manually configure the UDLD probe/echo message interval between values ranging from 7-90 seconds, the default interval being 15 seconds EtherChannel (PAgP) EtherChannel technologies allow the inverse multiplexing of multiple (up to eight on Cat6k) channels into a single logical link, and although each platform differs from the next in implementation, it is important to understand the common requirement - An algorithm to statistically multiplex frames over multiple channels - Creation of a logical port so that a single instance of STP can be run An algorithm to statistically multiplex frames over multiple channels - A channel management protocol PAgP is a management protocol that will check for parameter consistency at either end of the link, and assist the channel in adapting to link failure or addition - PAgP requires that all ports in the channel belong to the same VLAN or are configured as trunk ports (Because dynamic VLANs can force the change of a port into a different VLAN, they are not included in EtherChannel participation.) - When a bundle already exists and a VLAN of a port is modified, all ports in the bundle are modified to match that VLAN - PAgP does not group ports that operate at different speeds or port duplex If speed and duplex are changed when a bundle exists, PAgP changes the port speed and duplex for all ports in the bundle Configuration Options: EtherChannels can be configured in different modes, summarized in the table below: Mode On Configurable Options: PAgP not in operation The port is channeling regardless of how the NCAR Design Review and Recommendations v1.0 21 Hierarchical (MultiLayer) Network Design Off Auto (Default) Desirable Non-silent (Default on 5k fiber FE and GE ports) Silent (Default on all 6k and 4k ports, and 5k copper ports) peer port is configured If the peer port mode is on, a channel is formed The port is not channeling regardless of how the peer port is configured Aggregation is under control of the PAgP protocol Places a port into a passive negotiating state and no PAgP packets are sent on the interface until at least one PAgP packet is received that indicates that the sender is operating in desirable mode Aggregation is under control of the PAgP protocol Places a port into an active negotiating state, in which the port initiates negotiations with other ports by sending PAgP packets A channel is formed with another port group in either desirable or auto mode An auto or desirable mode keyword If no data packets are received on the interface, then the interface is never attached to an agport and cannot be used for data This bi-directionality check was provided for specific Cat5k hardware (EBC and SAINT4/5 ASICS), as some link failures result in the channel being broken apart and is to be avoided More flexible bundling and improved bi-directionality checks are present by default in the 4k and 6k hardware See section on auto-negotiation F2H1A0C4 An auto or desirable mode keyword If no data packets are received on the interface, then after a 15s timeout period, the interface is attached by itself to an agport and can thus be used for data transmission Silent mode also allows for channel operation when the partner can be an analyzer or server that never sends PAgP AS recommends enabling PAgP on all switch-to-switch channel connections (ie avoid the mode ‘on’) The preferred way is to set ‘desirable mode’ at both ends of a link The additional recommendation is to allow the ‘silent/non-silent’ keyword to default, i.e silent on 6ks and 4ks, non-silent on 5k fiber ports As discussed in previous sections, explicitly configuring channeling off on all other port is helpful for rapid data forwarding as ports come up Waiting up to 15s for PAgP to timeout on a port that will not be used for channeling should be avoided, especially as the port is then handed over to STP which itself can take 30s to allow data forwarding, 45s in total set port channel mode desirable set port channel mode off! where ports not in use, part of the ! set port host macro NCAR Design Review and Recommendations v1.0 22 Corporate Headquarters Cisco Systems, Inc 170 West Tasman Drive San Jose, CA 95134-1706 USA www.cisco.com Tel: 408 526-4000 800 553-NETS (6387) Fax: 408 526-4100 European Headquarters Cisco Systems Europe 11 Rue Camille Desmoulins 92782 Issy-Les-Moulineaux Cedex France www-europe.cisco.com Tel: 33 58 04 60 00 Fax: 33 58 04 61 00 Americas Headquarters Cisco Systems, Inc 170 West Tasman Drive San Jose, CA 95134-1706 USA www.cisco.com Tel: 408 526-7660 Fax: 408 527-0883 Asia Pacific Headquarters Cisco Systems Australia, Pty., Ltd Level 9, 80 Pacific Highway P.O Box 469 North Sydney NSW 2060 Australia www.cisco.com Tel: +61 8448 7100 Fax: +61 9957 4350 Cisco Systems has more than 200 offices in the following countries and regions Addresses, phone numbers, and fax numbers are listed on the Cisco Web site at www.cisco.com/go/offices Argentina • Australia • Austria • Belgium • Brazil • Bulgaria • Canada • Chile • China • Colombia • Costa Rica • Croatia • Czech Republic Denmark • Dubai, UAE Finland • France • Germany • Greece • Hong Kong SAR • Hungary • India • Indonesia • Ireland • Israel • Italy • Japan • Korea • Luxembourg • Malaysia • Mexico The Netherlands • New Zealand • Norway • Peru • Philippines • Poland • Portugal • Puerto Rico • Romania • Russia • Saudi Arabia • Singapore • Slovakia • Slovenia South Africa • Spain • Sweden • Switzerland • Taiwan • Thailand • Turkey • Ukraine • United Kingdom • United States • Venezuela • Vietnam • Zimbabwe ... attempt to describe two design improvement approaches: Short Term Design Enhancement Long Term AS Proposed Design NCAR Design Review and Recommendations v1.2 Short Term Design Enhancements Foothills-CenterGreen:... robustness of the network NCAR Design Review and Recommendations v1.0 13 Long Term: AS-Proposed Design This section will detail a design proposal from the AS team The following design is hierarchical,... additions Data Center WAN Internet Remote Access NCAR Design Review and Recommendations v1.2 11 Hierarchical (MultiLayer) Network Design A hierarchical network design distributes networking function at