STF 5 monitoring managing the datacenter

Copyright © 2006 EMC Corporation Do not Copy - All Rights Reserved Section - Monitoring and Managing the Data Center Introduction © 2006 EMC Corporation All rights reserved Welcome to Section of Storage Technology Foundations – Monitoring and Managing the Data Center Copyright © 2006 EMC Corporation All rights reserved These materials may not be copied without EMC's written consent EMC believes the information in this publication is accurate as of its publication date The information is subject to change without notice THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE Use, copying, and distribution of any EMC software described in this publication requires an applicable software license EMC2, EMC, Navisphere, CLARiiON, and Symmetrix are registered trademarks and EMC Enterprise Storage, The Enterprise Storage Company, The EMC Effect, Connectrix, EDM, SDMS, SRDF, Timefinder, PowerPath, InfoMover, FarPoint, EMC Enterprise Storage Network, EMC Enterprise Storage Specialist, EMC Storage Logix, Universal Data Tone, E-Infostructure, Access Logix, Celerra, SnapView, and MirrorView are trademarks of EMC Corporation All other trademarks used herein are the property of their respective owners Monitoring and Managing the Data Center - Copyright © 2006 EMC Corporation Do not Copy - All Rights Reserved Section Objectives Upon completion of this section, you will be able to: y Describe areas of the data center to monitor y Discuss considerations for monitoring the data center y Describe techniques for managing the data center © 2006 EMC Corporation All rights reserved Monitoring and Managing the Data Center - The objectives for this section are shown here Please take a moment to read them Monitoring and Managing the Data Center - Copyright © 2006 EMC Corporation Do not Copy - All Rights Reserved In This Section This section contains the following modules: y Monitoring in the Data Center y Managing in the Data Center © 2006 EMC Corporation All rights reserved Monitoring and Managing the Data Center - This section contains modules, monitoring in the data center and managing in the data center Monitoring and Managing the Data Center - Copyright © 2006 EMC Corporation Do not Copy - All Rights Reserved Monitoring in the Data Center After completing this module, you will be able to: y Discuss data center areas to monitor y List metrics to monitor for different data center components y Describe the benefits of continuous monitoring y Describe the challenges in implementing a unified and centralized monitoring solution in heterogeneous environments y Describe industry standards for data center monitoring © 2006 EMC Corporation All rights reserved Monitoring and Managing the Data Center - In this module, you will learn about different aspects of monitoring data center components, including the benefits of pro-active monitoring and the challenges of managing a heterogeneous environment (multiple hardware/software from various vendors) Monitoring and Managing the Data Center - Copyright © 2006 EMC Corporation Do not Copy - All Rights Reserved Monitoring Data Center Components Client HBA Port HBA Keep Alive Port IP IP Network SAN Storage Arrays Health Capacity Performance Cluster Hosts/Servers with Applications © 2006 EMC Corporation All rights reserved Security Monitoring and Managing the Data Center - The Business Continuity Overview module discussed the importance of resolving all single points of failure when designing data centers Having designed a resilient data center, the next step is to ensure that all components that make up the data center are functioning properly and are available on a 24x7 basis The way to achieve this is by monitoring the data center on a continual basis System Monitoring is essential to ensure that the underlying IT infrastructure business critical applications are operational and optimized The main objective is to ensure that the various hosts, network systems and storage are running smoothly and to know how loaded each system and component is and how effectively it is being utilized The major components within the data center that should be monitored include: y Servers, databases and applications y Network ((SAN) and IP Networks (switches, routers, bridges)) y Storage Arrays Each of these components should be monitored for health, capacity, performance, and security Monitoring and Managing the Data Center - Copyright © 2006 EMC Corporation Do not Copy - All Rights Reserved Why Monitor Data Centers y Availability – Continuous monitoring ensures availability – Warnings and errors are fixed proactively y Scalability – Monitoring allows for capacity planning/trend analysis which in turn helps to scale the data center as the business grows y Alerting – Administrators can be informed of failures and potential failures – Corrective action can be taken to ensure availability and scalability © 2006 EMC Corporation All rights reserved Monitoring and Managing the Data Center - Continuous monitoring of health, capacity, performance and security of all data center components is critical to ensure data availability and scalability For example, information about component failures can be sent to appropriate personnel for corrective actions Ongoing trends show that the data storage environment continues to grow at a rapid pace According to the International Data Corporation (IDC), external storage-system capacity growth will increase at a compound annual growth rate (CAGR) of approximately 50% through 2007 This represents a doubling of the current capacity every years or so Automated monitoring and alerting solutions are becoming increasingly important Monitoring the data center closely and effectively optimizes data center operations and avoids downtime Monitoring and Managing the Data Center - Copyright © 2006 EMC Corporation Do not Copy - All Rights Reserved Monitoring Health y Why monitor health of different components – Failure of any hardware/software component can lead to outage of a number of different components ¾ Example: A failed HBA could cause degraded access to a number of data devices in a multi-path environment or to loss of data access in a single path environment y Monitoring health is fundamental and is easily understood and interpreted – At the very least health metrics should be monitored – Typically health issues would need to be addressed on a high priority © 2006 EMC Corporation All rights reserved Monitoring and Managing the Data Center - Health deals with the status/availability of a particular hardware component or a software process (i.e., status of SAN device or port, database instance up/down, HBA status, disk/drive failure, etc.) If a component has failed, it could lead to down time unless redundancy exists Monitoring the health of data center components is very important and is easy to understand and interpret (i.e., a component is either available or it has failed) Monitoring for capacity, performance, and security depend on the health and availability of different components Monitoring and Managing the Data Center - Copyright © 2006 EMC Corporation Do not Copy - All Rights Reserved Monitoring Capacity y Why monitor capacity – Lack of proper capacity planning can lead to data un-availability and the ability to scale – Trend reports can be created from all the capacity data ¾ Enterprise is well informed of how IT resources are utilized y Capacity monitoring prevents outages before they can occur – More preventive and predictive in nature than health metrics ¾ Based on reports one knows that 90% of a file system is full and that the file system is filling up at a particular rate ¾ 95% of all the ports have been utilized in a particular SAN fabric, a new switch should added if more arrays/servers are to be added to the same fabric © 2006 EMC Corporation All rights reserved Monitoring and Managing the Data Center - From a monitoring perspective, capacity deals with the amount of resources available Examples: y Available free/used space on a file system or a database table space y Amount of space left in a RAID Group y Amount of disk space available on storage arrays y Amount of file system or mailbox quota allocated to users y Number of available ports in a switch (e.g., 52 out of 64 ports in use, leaving 12 free ports for expansion) Monitoring and Managing the Data Center - Copyright © 2006 EMC Corporation Do not Copy - All Rights Reserved Monitoring Performance y Why monitor Performance metrics – Want all data center components to work efficiently/optimally – See if components are pushing performance limits or if they are being under utilized – Can be used to identify performance bottlenecks y Performance Monitoring/Analysis can be extremely complicated – Dozens of inter-related metrics depending on the component in question – Most complicated of the various aspects of monitoring Monitoring and Managing the Data Center - © 2006 EMC Corporation All rights reserved Performance monitoring measures the efficiency of operation of different data center components Examples: y Number of I/Os thorough a front-end port of a storage array y Number of I/Os to disks in a storage array y Response time of an application y Bandwidth utilization y Server CPU utilization Monitoring and Managing the Data Center - Copyright © 2006 EMC Corporation Do not Copy - All Rights Reserved Monitoring Security y Why monitor security – Prevent and track unauthorized access ¾ Accidental or malicious y Enforcing security and monitoring for security breaches is a top priority for all businesses © 2006 EMC Corporation All rights reserved Monitoring and Managing the Data Center - 10 Security prevents and tracks unauthorized access Examples of security monitoring are: y Login failures y Unauthorized storage array configuration/re-configuration y Monitoring physical access (via badge readers, biometric scans, video cameras, etc.) y Unauthorized Zoning and LUN masking in SAN environments or changes to existing zones Monitoring and Managing the Data Center - 10 Copyright © 2006 EMC Corporation Do not Copy - All Rights Reserved Scenario – Running out of File System Space Solutions y Offload non-critical data File System – Delete non-essential data – Move older/seldom used data to other media ¾ ILM/HSM strategy ¾ Easy retrieval if needed y Extend File System Warning: FS is 66% Full – Operating System and Logical Volume Manager dependent – Management tasks seen in Scenario will apply here as well Critical: FS is 80% Full © 2006 EMC Corporation All rights reserved Monitoring and Managing the Data Center - 52 In this scenario, we will explore data center management tasks that you would possibly have to execute to prevent a file system from getting 100% full When a file system is running out of space, either: y Actively perform tasks which off load data from the existing file system (keep file system the same size) − Delete unwanted files − Offload files that have not been accessed for a long time to tape or to some other media from which it can be easily retrieved if necessary y Extend the file system to make it bigger − Considerations for extending file systems ¾ Dynamic extension of file systems is dependent on the specific operating system or logical volume manager (LVM) in use − The possible tasks to extend file systems is discussed in more detain in the next slide In reality, a good data center administrator should constantly monitor file systems and offload non-critical data and also be ready to extend the file system, if necessary Monitoring and Managing the Data Center - 52 Copyright © 2006 EMC Corporation Do not Copy - All Rights Reserved Scenario – Running out of File System Space, continued Correlate File System with Volume Group or Disk Group Done No Is there free space available in the VG? Yes No Execute Command to extend File System Is the File System being replicated? Yes Does the server have additional devices available? Yes Execute Command to extend VG No Does the Array have configured LUNs that can be allocated? Yes Perform tasks to ensure that the larger File System and Volume Group are replicated correctly Allocate LUNs to server No Does the array have unconfigured capacity? No © 2006 EMC Corporation All rights reserved Yes Configure new LUNs Identify/Procure another array Monitoring and Managing the Data Center - 53 The steps/considerations prior to the extension of a file system have been illustrated in the flow chart The goal is to increase the size of the file system to avoid application outage Other considerations revolve around local/remote replication/protection employed for the application For instance, if the application is protected via remote/local replication and a new device is added to the Volume Group, ensure that this new device is replicated as well The steps include: y Correlate the file system to the logical volume and volume group if an LVM is in use y If there is enough space in the volume group – extend the file system y If the volume group does not have space – does the server have access to other devices which can be use to extend the volume group – extend the volume group – extend the file system y If the server does not have access to additional devices – allocate additional devices to the server – many or all of the steps discussed in scenario will have to be used to this (configure new LUNs on array, LUN mask, reconfigure server to recognize new devices – extend volume group – extend file system) Monitoring and Managing the Data Center - 53 Copyright © 2006 EMC Corporation Do not Copy - All Rights Reserved Scenario – Chargeback Report Storage Arrays VG Production (Green) LV FS VG Remote Replica (Red) DB App LV FS Port DB AppVG SW1 LV Port FS Port DB App SW2 Hosts/Servers with Applications © 2006 EMC Corporation All rights reserved Port Local Replica (Blue) Monitoring and Managing the Data Center - 54 Scenario 3: In this scenario, we will explore the various data center tasks that will be necessary to create a specific report A number of servers (50 – only shown in picture) with HBAs each and are connected to a Storage Array via two switches SW1 and SW2 Each server has independent paths (2 HBAs) to the storage array via switch SW1 and switch SW2 Applications are running on each of the servers, array replication technology is used to create local and remote replicas The Production devices are represented by the green devices, local replica by the blue devices and the remote replicas by the red devices A report documenting the exact amount of storage used by each application (including that used for local and remote replication) has to be created The amount of raw storage used must be reported as well The cost of the raw storage consumed by each application must be billed to the application owners A sample report is shown in the picture The report shows the information for two applications Application Payroll_1 has been allocated 100 GB of storage Production volumes are RAID volumes hence the raw space used by the production volumes is 200 GB Local replicas are on unprotected (no fault tolerance) volumes, hence raw space used by local replicas is 100 GB The remote replicas are on RAID5 (5 disk group) volumes, hence raw space used for remote replicas is 125 GB What are the various data center management steps to perform in order to create such a report? Monitoring and Managing the Data Center - 54 Copyright © 2006 EMC Corporation Do not Copy - All Rights Reserved Scenario – Chargeback Report – Tasks – Correlate Application Ỉ File Systems Ỉ Logical Volumes Ỉ Volume Groups Ỉ Host Physical Devices Ỉ Array Devices (Production) – Determine Array Devices used for Local Replication – Determine Array Devices used for Remote Replication – Determine storage allocated to application based on the size of the array devices Example: VG DB App Array Remote Array LV Source Vol Local Replica Vol Remote Replica Vol FS Source Vol Local Replica Vol Remote Replica Vol © 2006 EMC Corporation All rights reserved Monitoring and Managing the Data Center - 55 The first step in determining the chargeback costs associated with an application is to correlate the application with the array devices that are in use As indicated in the picture, trace the application to the file systems, logical volumes, volume groups, and eventually to the array devices Since the applications are being replicated, determine the array devices used for local replication and the array devices used for remote replication In the example shown, the application is using “Source Vol 1&2” (in Array 1) The replication devices are “Local Replica Vol 1&2” (in Array 1) and “Remote Replica Vol 1&2” (in the Remote Array) Keep in mind that this can change over time As the application grows, more file systems and devices may be used Thus, before a new report is generated, the correlation of application to the array devices should be done to ensure that the most current information is used After the array devices are identified, the amount of storage allocated to the application can be easily computed In this case “Source Vol 1&2” are each 10GB in size Thus the storage allocated to the application is 20GB (10+10) The allocated storage for replication would be 20GB for local and 20GB for remote The allocated storage is the actual storage that can be used, it does not represent the actual raw storage used by the application To determine the raw space, determine the RAID protection that is used to the various array devices Monitoring and Managing the Data Center - 55 Copyright © 2006 EMC Corporation Do not Copy - All Rights Reserved Scenario – Chargeback Report – Tasks, continued – Determine RAID type for Production/Local Replica/Remote Replica devices – Determine the total raw space allocated to application for production/local replication/remote replication – Compute the chargeback amount based of price/raw GB of storage – Repeat steps for each application and create report – Repeat the steps each time the report is to be created (weekly/monthly) Example: Source Vols = 2*10GB RAID = 2* 20GB raw = 40GB Local Replica Vols = 2*10GB = 2*10GB raw = 20GB Remote Replica Vols = 2*10 GB RAID = 2*12.5 GB raw = 25GB Total raw storage = 40+20+25 = 85GB Chargeback cost = 85*0.25/GB = 21.25 © 2006 EMC Corporation All rights reserved Monitoring and Managing the Data Center - 56 To determine the raw space, review the steps displayed on the slide using the example listed Determine RAID type for Production/Local Replica/Remote Replica devices In the example shown, production devices are 10GB RAID 1, Local replica devices are 10GB with no protection, and remote replica devices are 10GB RAID (5 disk group) devices Determine the total raw space allocated to application for production, local replication, and remote replication Based on the values from step 1, you can determine that the total raw space used by the application is 85GB (Total raw storage = 40+20+25 = 85GB) Compute the chargeback amount based on price per raw GB of storage Based on the cost per GB of storage (for the example this equals 25/GB), the chargeback cost can be computed (Chargeback cost = 85*0.25/GB = 21.25) Repeat these steps for each application and create a report Repeat the steps each time the report is to be created (weekly/monthly) The exercise would have to repeated for every single application in the enterprise in order to generate the require report These tasks can be done manually Manual creation of the report may be acceptable if only one or two applications exist The process can become extremely tedious if many applications exist The best way to create this report would be to automate these various tasks Monitoring and Managing the Data Center - 56 Copyright © 2006 EMC Corporation Do not Copy - All Rights Reserved Information Lifecycle Management y Information Management Challenges y Information Lifecycle y Information Lifecycle Management – Definition – Process – Benefits – Implementation © 2006 EMC Corporation All rights reserved Monitoring and Managing the Data Center - 57 Information Lifecycle Management (ILM) is a key approach for assuring availability, capacity, and performance Let’s look at some of the aspects of ILM Monitoring and Managing the Data Center - 57 Copyright © 2006 EMC Corporation Do not Copy - All Rights Reserved Key Challenges of Information Management CHALLENGE Scaling infrastructure within budget constraints Information growth is relentless CHALLENGE Scaling resources to manage complexity CHALLENGE Information is more strategic than ever Access, availability, and protection of critical information assets at optimal cost Information changes in value over time Ability to prioritize information management based on data value CHALLENGE Reducing risk of non-compliance CHALLENGE © 2006 EMC Corporation All rights reserved Monitoring and Managing the Data Center - 58 Companies face three key challenges related to information management: Strong growth of information: y Post-dot com rate of growth is around 50%, driven by digitization, increased use of e-mail, etc y Just planning for growth can take up to 50% of storage resources y Meeting growth needs has increased the complexity of a customer environment Information is playing a more important role in determining business success: y New business applications provide more ways to extract a competitive advantage in the marketplace, e.g., companies like Dell, WalMart, and Amazon, where, at the heart of their respective business models, is the strategic use of information Finally, information changes in value, and many times not necessarily in a linear fashion y For example, customers become inactive, reducing the need for account information; pending litigation makes certain information more valuable, etc y Understanding the value of information should be at the heart of managing information in general Monitoring and Managing the Data Center - 58 Copyright © 2006 EMC Corporation Do not Copy - All Rights Reserved The Information Lifecycle Sales Order Application Example New Order Record Order Processing Warranty Claim TIME VALUE Orders Fulfilled Protect Create © 2006 EMC Corporation All rights reserved Warranty Voided Migrate Access Dispose Archive Monitoring and Managing the Data Center - 59 Information that is stored on a computer has a different value to a company, depending on how long it is stored on the network In the above example, this sales order goes through differing value to the company from the time that it is created to the time that the warrantee is eventually voided In a typical sales example as this one, the value of information is highest when a new order is created and processed After order fulfillment, there is potentially less need to have real-time access to customer/order data, unless a warranty claim or other event triggers that need Similarly, after the product has entered EOL, or after the account is closed, there is little value in the information and it can be disposed Monitoring and Managing the Data Center - 59 Copyright © 2006 EMC Corporation Do not Copy - All Rights Reserved Information Lifecycle Management Definition © 2006 EMC Corporation All rights reserved Monitoring and Managing the Data Center - 60 Information Lifecycle Management is a strategy, not a product or service in itself; further, this strategy is proactive and dynamic in helping plan for IT growth as it relates to business needs, and reflects the value of information in a company A successful information lifecycle management strategy must be: y Business-centric by tying closely with key processes, applications, and initiatives of the business y Centrally managed, providing an integrated view into all information assets of the business, both structured and unstructured y Policy-based, anchored in enterprise-wide information management policies that span all processes, applications, and resources y Heterogeneous, encompassing all types of platforms and operating systems y Aligned with the value of data, matching storage resources to the value of the data to the business at any given point in time Monitoring and Managing the Data Center - 60 Copyright © 2006 EMC Corporation Do not Copy - All Rights Reserved Information Lifecycle Management Process Policy-based Alignment of Storage Infrastructure with Data Value AUTOMATED Classify data / applications based on business rules Implement policies with information management tools Manage storage environment Tier storage resources to align with data classes FLEXIBLE Storage infrastructure that is Application and Lifecycle Aware © 2006 EMC Corporation All rights reserved Monitoring and Managing the Data Center - 61 The process of implementing the ongoing modification of an Information Lifecycle Management strategy consists of four activities: y Classify data and applications on the basis of business rules and policies to enable differentiated treatment of information y Implement policies with information management tools—from creation to disposal of data y Manage the environment with integrated tools that interface with multi-vendor platforms, and reduce operational complexity y Tier storage resources to align with data classes - storing information in the right type of infrastructure based on its current value Monitoring and Managing the Data Center - 61 Copyright © 2006 EMC Corporation Do not Copy - All Rights Reserved Information Lifecycle Management Benefits Information growth is relentless Information is more strategic than ever Information changes in value over time © 2006 EMC Corporation All rights reserved Improve utilization of assets through tiered storage platforms Simplify and automate management of information and storage infrastructure Provide more cost-effective options for access, business continuity and protection Ensure easy compliance through policy-based management Deliver maximum value at lowest TCO by aligning storage infrastructure and management with information value Monitoring and Managing the Data Center - 62 Implementing an ILM strategy delivers key benefits that directly address the challenges of information management y Improved utilization, by the use of tiered platforms, and increased visibility into all enterprise information y Simplified management by integration of process steps and interfaces to individual tools in place today, and by increased automation y A wider range of options backup, protection, and recovery to balance the need for continuity with the cost of losing specific information y Painless compliance by having better control upfront in knowing what data needs to be protected and for how long y Lower TCO while meeting required service levels through aligning the infrastructure and management costs with information value so resources are not wasted or complexity introduced by managing low-value data at the cost of high-value data Monitoring and Managing the Data Center - 62 Copyright © 2006 EMC Corporation Do not Copy - All Rights Reserved Path to Enterprise – Wide ILM App App App Data Data Data App Automated Networked Storage Step Networked Tiered Storage y y y Enable networked storage Automate environment Classify applications / data App App Data Data App ILM for Specific Applications Step Application-specific ILM y y Define business policies for various information types Deploy ILM components into principal applications App App Cross-Application ILM Step Enterprise-wide ILM y y y Implement ILM across applications Policy-based automation Full visibility into all information Lower cost through increased automation © 2006 EMC Corporation All rights reserved Monitoring and Managing the Data Center - 63 Implementing ILM enterprise wide will take time, and no one believes it can be done instantaneously A three step roadmap to enterprise-wide ILM is illustrated y Step and are tuned to products and solutions available today, with the goal to be “ILMenabled” across a few enterprise-critical applications In step 1, the goal is to get the environment to an automated networked storage environment This is the basis for any policy-based information management The value of tiered storage platforms can be exploited manually In fact, many enterprises are already in this state y Step takes ILM to the next level with detailed application/data classification and linkage to business policies While done in a manual way, the resultant policies can be automatically executed with tools for one or more applications, resulting in better management and optimal allocation of storage resources y Step of the vision is to automate more of the “front-end” or classification and policy management activities so as to scale to a wider set of enterprise applications It is consistent with the need for more automation and greater simplicity of operations Monitoring and Managing the Data Center - 63 Copyright © 2006 EMC Corporation Do not Copy - All Rights Reserved Module Summary Key points covered in this module: y Individual component tasks that would have to be performed in order to achieve overall data center management objectives were illustrated – Allocation of storage to a new application server – Running out of file system space – Creating a chargeback report y Concept of Information Lifecycle Management © 2006 EMC Corporation All rights reserved Monitoring and Managing the Data Center - 64 These are the key points covered in this module Please take a moment to review them Monitoring and Managing the Data Center - 64 Copyright © 2006 EMC Corporation Do not Copy - All Rights Reserved Section Summary Key points covered in this section: y Areas of the data center to monitor y Considerations for monitoring the data center y Techniques for managing the data center © 2006 EMC Corporation All rights reserved Monitoring and Managing the Data Center - 65 This completes Section – Monitoring and Managing the Data Center Please take a moment to review the key points covered in this section Monitoring and Managing the Data Center - 65 Copyright © 2006 EMC Corporation Do not Copy - All Rights Reserved Course Summary Key points covered in this course: y Storage concepts and architecture y Evolution of storage and storage environments y Logical and physical components of storage systems y Storage technologies and solutions y Core data center infrastructure elements and activities for monitoring and managing the data center y Options for Business Continuity © 2006 EMC Corporation All rights reserved Monitoring and Managing the Data Center - 66 This completes the Storage Technology Foundations training Please take a moment to review the key points covered in this course Monitoring and Managing the Data Center - 66 ... shown using the green and red lines in the line graph If the actual utilization prior to deploying the new server is the green line, then there is room to add the new server Otherwise, the deployment... modules, monitoring in the data center and managing in the data center Monitoring and Managing the Data Center - Copyright © 2006 EMC Corporation Do not Copy - All Rights Reserved Monitoring in the. .. Array provide the required storage to the new server? Monitoring and Managing the Data Center - 25 This example illustrates the importance of monitoring the capacity of arrays A number of servers