©2009, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc (www.ashrae.org) For personal use only Additional reproduction, distribution, or transmission in either print or digital form is not permitted without ASHRAE's prior written permission ©2009, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc (www.ashrae.org) For personal use only Additional reproduction, distribution, or transmission in either print or digital form is not permitted without ASHRAE's prior written permission ’ Any updates/errata to this publication will be posted on the ASHRAE Web site at www.ashrae.org/publicationupdates ©2009, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc (www.ashrae.org) For personal use only Additional reproduction, distribution, or transmission in either print or digital form is not permitted without ASHRAE's prior written permission ©2009, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc (www.ashrae.org) For personal use only Additional reproduction, distribution, or transmission in either print or digital form is not permitted without ASHRAE's prior written permission ISBN 978-1-933742-73-1 ©2009 American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc 1791 Tullie Circle, NE Atlanta, GA 30329 www.ashrae.org All rights reserved Printed in the United States of America Cover design by Joe Lombardo, DLB Associates ASHRAE has compiled this publication with care, but ASHRAE has not investigated, and ASHRAE expressly disclaims any duty to investigate, any product, service, process, procedure, design, or the like that may be described herein The appearance of any technical data or editorial material in this publication does not constitute endorsement, warranty, or guaranty by ASHRAE of any product, service, process, procedure, design, or the like ASHRAE does not warrant that the information in the publication is free of errors, and ASHRAE does not necessarily agree with any statement or opinion in this publication The entire risk of the use of any information in this publication is assumed by the user No part of this book may be reproduced without permission in writing from ASHRAE, except by a reviewer who may quote brief passages or reproduce illustrations in a review with appropriate credit; nor may any part of this book be reproduced, stored in a retrieval system, or transmitted in any way or by any means—electronic, photocopying, recording, or other—without permission in writing from ASHRAE Requests for permission should be submitted at www.ashrae.org/permissions ©2009, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc (www.ashrae.org) For personal use only Additional reproduction, distribution, or transmission in either print or digital form is not permitted without ASHRAE's prior written permission Acknowledgments ix PART BASICS CHAPTER INTRODUCTION 1.1 Objectives for this Book 11 1.2 How to Use this Book 12 CHAPTER HOW, WHAT, & WHERE TO MEASURE 15 2.1 Overview 15 2.2 Quantifying Energy Efficiency Metrics 17 CHAPTER MEASUREMENT DEVICES 21 3.1 Overview 21 3.2 Sensor Accuracy 23 3.3 Temperature 24 3.4 Pressure 29 3.5 Flow—Liquid 31 3.6 Flow—Gas 40 3.7 Current 44 3.8 Voltage 49 3.9 Power 53 CHAPTER MEASUREMENT COLLECTION SYSTEMS—ARCHITECTURE & SOFTWARE 59 4.1 Overview 59 4.2 Business Questions 60 4.3 Scalable Hardware/Software Architecture 63 4.4 Measurement Levels 64 v ©2009, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc (www.ashrae.org) For personal use only Additional reproduction, distribution, or transmission in either print or digital form is not permitted without ASHRAE's prior written permission Real-Time Energy Consumption Measurements in Data Centers PART COOLING SYSTEMS—AIR MEASUREMENTS CHAPTER AIR HANDLERS 79 5.1 Overview 79 5.2 Measurement Levels 81 CHAPTER COMPUTER ROOM UNITS 85 6.1 Overview 85 6.2 Measurement Levels 94 PART COOLING SYSTEMS —HYDRONIC MEASUREMENTS CHAPTER PUMPS 101 7.1 Overview 101 7.2 Measurement Levels—Electrical 101 7.3 Measurement Levels—Fluid 105 CHAPTER COOLING TOWERS 113 8.1 Overview 113 8.2 Measurement Levels 116 CHAPTER CHILLERS 125 9.1 Overview 125 9.2 Measurement Levels 129 CHAPTER 10 HEAT EXCHANGERS 141 10.1 Overview 141 10.2 Measurement Levels 143 PART POWER SYSTEMS MEASUREMENTS CHAPTER 11 INTRODUCTION TO CRITICAL POWER DISTRIBUTION 149 11.1 Overview 149 11.2 Critical Power versus Essential Power 150 vi ©2009, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc (www.ashrae.org) For personal use only Additional reproduction, distribution, or transmission in either print or digital form is not permitted without ASHRAE's prior written permission Contents CHAPTER 12 UPSTREAM CRITICAL POWER DISTRIBUTION 155 12.1 Overview 155 12.2 Service Entrance Equipment 155 12.3 Automatic Transfer Switch (ATS) 159 12.4 Primary Electrical Distribution Switchgear 161 CHAPTER 13 UNINTERRUPTIBLE POWER SUPPLY (UPS) 165 13.1 Overview 165 13.2 UPS Metering, Power Module Level 168 13.3 UPS Metering, System Level 169 CHAPTER 14 COMPUTER ROOM TRANSFORMER & POWER DISTRIBUTION UNIT (PDU) 173 14.1 Overview 173 14.2 Stand-Alone Transformers 173 14.3 Computer Room Power Distribution Units (PDU) 175 14.4 Rack-Mounted Power Distribution Unit (RPDU) 179 PART IT SYSTEMS MEASUREMENTS CHAPTER 15 COMPUTE & STORAGE SYSTEMS 185 15.1 Overview 185 15.2 Measurement Levels 190 CHAPTER 16 NETWORKING SYSTEMS 201 16.1 Overview 201 16.2 Measurement Levels 206 APPENDIX A PUMPS 213 A.1 Power and Efficiency 213 A.2 Real-Time Power Measurements 216 vii ©2009, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc (www.ashrae.org) For personal use only Additional reproduction, distribution, or transmission in either print or digital form is not permitted without ASHRAE's prior written permission Real-Time Energy Consumption Measurements in Data Centers APPENDIX B CHILLERS 223 B.1 Variables Affecting RLA and Power Rating 223 B.2 Integrated or Non-Standard Part Load Value 224 APPENDIX C MIXED-USE FACILITIES 227 C.1 Real-Time Cooling Tower Power Consumption 227 C.2 Real-Time Chiller Power Consumption 229 APPENDIX D UNINTERRUPTIBLE POWER SUPPLY (UPS) 233 D.1 Technology 234 D.2 Redundancy and Availability 236 D.3 Rules of Thumb for Minimum Practical Level of UPS Instrumentation 241 D.4 Sample Case Study: A Partial PUE and DCiE Determination for the Critical Power Path within the Data Center 243 APPENDIX E ONSITE POWER GENERATION AND CCHP IN DATA CENTER APPLICATIONS 249 E.1 Overview 249 E.2 CCHP 251 E.3 Measurement Levels 258 E.4 Example Calculations for a CCHP Installation 263 APPENDIX F ABBREVIATIONS AND GLOSSARY 267 REFERENCES 283 viii ©2009, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc (www.ashrae.org) For personal use only Additional reproduction, distribution, or transmission in either print or digital form is not permitted without ASHRAE's prior written permission ASHRAE TC 9.9 and The Green Grid would like to thank the following individuals for their substantial contributions to the book: Lead editor / author – Tahir Cader, HP (formerly SprayCool) Co-editor after First Draft – Don Beaty, DLB Associates Chapter – Tahir Cadera,b, HP (lead) Chapter – Tahir Cadera,b, HP (lead); Mike Mangan, DLB Associates; Jeff Jaworksi, DLB Associates Chapter – John Beana,b, APC/Schneider; Randall Woffordb, Dell; Ross Ignalla, Dranetz-BMI; Michael Kennedy, DLB Associates Chapter – Ken Uhlmanb, Eaton (lead); Harry Rogersb, Microsoft Chapter – Robert Wasilewski, DLB Associates (lead) Chapter – Jeff Trowera, DataAire (lead); Cliff Federspiel, Federspiel Controls Chapter – John Beana,b, APC/Schneider (lead) Chapter – Daryn Clinea, Evapco (lead) Chapter – Jonathan Spreemana, Trane (lead); Tahir Cadera,b, HP Chapter 10 – Robert Wasilewski, DLB Associates Chapters 11, 12, 13, 14 – Steve McCluera,b, APC/Schneider Electric (lead); Bill Campbellb, Emerson Network Power; John Messerb, Emerson Network Power Chapter 15 – Mike Pattersona,b, Intel (lead); Bob MacArthurb, EMC Chapter 16 – Kevin Engelberta,b, Cisco (lead) Chapter 17 – Kevin Wymana, Carrier Corporation (lead); Greg Palmerb, HP (formerly UTC Power) (lead); Appendix A – John Beana,b, APC/Schneider (lead) Appendix B – Jonathan Spreemana, Trane (lead) Appendix C – Tahir Cadera,b, HP (lead); Jonathan Spreemana, Trane ix ©2009, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc (www.ashrae.org) For personal use only Additional reproduction, distribution, or transmission in either print or digital form is not permitted without ASHRAE's prior written permission Real-Time Energy Consumption Measurements in Data Centers GLOSSARY ACPI: cabinet: crest factor: a waveform measurement calculated by peak amplitude divided by the RMS 272 ©2009, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc (www.ashrae.org) For personal use only Additional reproduction, distribution, or transmission in either print or digital form is not permitted without ASHRAE's prior written permission Appendix F—Abbreviations and Glossary circuit setter: a valve that is carefully calibrated to provide flow balancing and flow metering cylindrical unloading: data center: the ratio of the total amount of power used by a data center facility to the power delivered to IT equipment, as a percent DCMI: 273 ©2009, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc (www.ashrae.org) For personal use only Additional reproduction, distribution, or transmission in either print or digital form is not permitted without ASHRAE's prior written permission Real-Time Energy Consumption Measurements in Data Centers of total source energy by total UPS energy; source energy differs from site energy in the fact that it accounts for the types of fuels that are consumed to provide energy to the facility flicker: a rapid fluctuation in voltage Measurements are defined by IEC 61000-4-15, Testing and Measurement Techniques – Flickermeter – Functional and design specifications, Ediction 1.1, 2003-03 harmonics: electrical voltages and currents resulting from certain loads that are the source of many power quality issues 274 ©2009, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc (www.ashrae.org) For personal use only Additional reproduction, distribution, or transmission in either print or digital form is not permitted without ASHRAE's prior written permission Appendix F—Abbreviations and Glossary a simple and inexpensive means of addressing lower than designed or expected capacity by diverting hot discharge gas to falsely load an otherwise lightly loaded compressor 275 ©2009, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc (www.ashrae.org) For personal use only Additional reproduction, distribution, or transmission in either print or digital form is not permitted without ASHRAE's prior written permission Real-Time Energy Consumption Measurements in Data Centers IPMI: PMBus: Power Usage Effectiveness (PUE): the ratio of power delivered to IT equipment to the total amount of power used by a data center facility precision: the quality of being reproducible rack 276 ©2009, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc (www.ashrae.org) For personal use only Additional reproduction, distribution, or transmission in either print or digital form is not permitted without ASHRAE's prior written permission Appendix F—Abbreviations and Glossary rack-mounted equipment ratchet charge: a utility rate provision under which the demand charge for each month (or other period) is based on the highest measured demand (or its percentage) over the previous year (or other period) reactive power: portion of apparent power that does no work It is measured commercially by kilvars Reactive power must be supplied to most types of magnetic equipment, such as motors It is supplied by generators or by electrostatic equipment, such as capacitors 277 ©2009, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc (www.ashrae.org) For personal use only Additional reproduction, distribution, or transmission in either print or digital form is not permitted without ASHRAE's prior written permission Real-Time Energy Consumption Measurements in Data Centers 278 ©2009, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc (www.ashrae.org) For personal use only Additional reproduction, distribution, or transmission in either print or digital form is not permitted without ASHRAE's prior written permission Appendix F—Abbreviations and Glossary SMASH: SMBus: SNMP: spark spread: the difference between the market price of electricity or natural gas and its production costs To calculate the spread, the heat rate of a generating unit or power system is multiplied by the cost of energy measured in dollars per BTUs (British thermal units) standard pressure: reference condition for the pressure (assumed sea level) is defined by the ICAO (International Civil Aeronautics Organization) as 101.325kPa, which is exactly 1013.24 millibars In I-P units the value is approximately 14.696 psi, or 29.921 inches of mercury at 32 °F standard temperature: reference condition for the rating of equipment, usually 68 °F (20 °C) 279 ©2009, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc (www.ashrae.org) For personal use only Additional reproduction, distribution, or transmission in either print or digital form is not permitted without ASHRAE's prior written permission Real-Time Energy Consumption Measurements in Data Centers ventilation 280 ©2009, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc (www.ashrae.org) For personal use only Additional reproduction, distribution, or transmission in either print or digital form is not permitted without ASHRAE's prior written permission Appendix F—Abbreviations and Glossary 281 ©2009, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc (www.ashrae.org) For personal use only Additional reproduction, distribution, or transmission in either print or digital form is not permitted without ASHRAE's prior written permission ©2009, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc (www.ashrae.org) For personal use only Additional reproduction, distribution, or transmission in either print or digital form is not permitted without ASHRAE's prior written permission AHRI 2003 AHRI Standard 550/590-2003, Standard for Water Chilling Packages Using the Vapor Compression Cycle http://www.ahrinet.org/ARI/util/showdoc.aspx?doc=1472 ANSI / ASHRAE 2007 ANSI / ASHRAE 127-2007, Method of Testing for Rating Computer and Data Processing Room Unitary AirConditioners Atlanta, GA: American Society of Heating, Refrigerating, and Air-Conditioning Engineers, Inc ANSI / ASHRAE 2008 ANSI / ASHRAE 135-2008, BACnet – A Data Communication Protocol for Building Automation and Control Networks Atlanta, GA: American Society of Heating, Refrigerating, and Air-Conditioning Engineers, Inc ASHRAE 2002 Guideline 14-2002 – Measurement of Energy and Demand Savings Atlanta, GA: American Society of Heating, Refrigerating, and Air-Conditioning Engineers, Inc ASHRAE 2006 Liquid Cooling Guidelines for Datacom Equipment Centers Atlanta, GA: American Society of Heating, Refrigerating, and Air-Conditioning Engineers, Inc ASHRAE 2006 ASHRAE Green Guide: The Design Construction and Operation of Sustainable Buildings, Second Edition Atlanta, GA: American Society of Heating, Refrigerating, and Air-Conditioning Engineers, Inc ASHRAE 2006 2006 ASHRAE Handbook – Refrigeration Atlanta, GA: American Society of Heating, Refrigerating, and Air-Conditioning Engineers, Inc ASHRAE 2008 Thermal Guidelines for Data Processing Environments – Second Edition Atlanta, GA: American Society of Heating, Refrigerating, and Air-Conditioning Engineers, Inc ASHRAE 2008 Evaporative Cooling Choices to Maximize Waterside Economizer Use in Datacom Installations ASHRAE Transactions 2008, Vol 114, Part Paper NY-08-006 ASHRAE 2008 2008 ASHRAE Handbook – HVAC Systems and Equipment Atlanta, GA: American Society of Heating, Refrigerating, and Air-Conditioning Engineers, Inc ©2009, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc (www.ashrae.org) For personal use only Additional reproduction, distribution, or transmission in either print or digital form is not permitted without ASHRAE's prior written permission Real-Time Energy Consumption Measurements in Data Centers ASHRAE 2008 ASHRAE Guideline 22-2008, Instrumentation for Monitoring Central Chilled-Water Plant Efficiency Atlanta, GA: American Society of Heating, Refrigerating, and Air-Conditioning Engineers, Inc ASHRAE 2009 Best Practices for Datacom Facility Energy Efficiency Atlanta, GA: American Society of Heating, Refrigerating, and AirConditioning Engineers, Inc ASHRAE 2009 Fundamentals Handbook Atlanta, GA: American Society of Heating, Refrigerating, and Air-Conditioning Engineers, Inc ASHRAE 1991 ASHRAE Terminology of Heating, Ventilation, Air Conditioning, & Refrigeration - Second Edition Atlanta, GA: American Society of Heating, Refrigerating, and Air-Conditioning Engineers, Inc Cisco 2009 “Managing EnergyWise Using CiscoWorks LAN Management Solution.” Cisco White Paper C11-558123-00 Sep 2009 Clark, Don 2005 How Sensor and Gauge Accuracy Impact Chiller Efficiency Do You Really Know How Your Chillers Are Running http://www.efftec.com/theimpactofsensorcalibration.html DMTF 2008 Web Services for Management (WS-Management) Specification v1.0.0 Distributed Management Task Force, Inc http://www.dmtf.org/standards/published_documents/DSP0226 _1.0.0.pdf EPA 2007 Report to Congress on Server and Data Center Energy Efficiency Public Law 109-431 EPA 2008 Energy Star Program Requirements for Computers, Version 5.0 www.energystar.gov/ia/partners/products_specs/program_reqs/Comp uter_Spec_Final.pdf Efficiency Valuation Organization 2007 EVO 10000-1.2007: International Performance Measurement and Verification Protocol, Concepts and Options for Determining Energy and Water Savings Volume http://www.evo-world.org/ Figliola, R S., and Beasley, D E., 1991 Theory and Design for Mechanical Measurements John Wiley & Sons IEEE 2003 802.3af: Amendment: Data Terminal Equipment (DTE) Power via Media Dependent Interface (MDI) 284 ©2009, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc (www.ashrae.org) For personal use only Additional reproduction, distribution, or transmission in either print or digital form is not permitted without ASHRAE's prior written permission References Intel 2004 Intelligent Platform Management Interface IPMI v2.0 Specifications Document Revision 1.0 http://download.intel.com/design/servers/ipmi/IMPIv2_0rev1_0.pdf Lippis, Nicholas John III 2009 “Controlling Corporate Energy Consumption via the Enterprise Network: A New Approach to Achieving Energy Efficiency by Leveraging Network Infrastructure.” Lippis Report Jan 2009 Philips Semiconductors 2000 I2C Bus Specification Version 2.1 SPEC, 2008 SPECpower_ssj2008, SPEC – Power and Performance TGG 2007 Qualitative Analysis of Power Distribution Configurations for Data Centers The Green Grid TGG 2008 A Framework for Data Center Energy Productivity The Green Grid TGG 2008 Green Grid Data Center Power Efficiency Metrics: PUE And DCiE The Green Grid TGG 2009 Proper Sizing of IT Power and Cooling Loads The Green Grid TGG 2009 Usage and Public Reporting Guidelines for The Green Grid’s Infrastructure Metrics PUE/DCiE The Green Grid United States Department of Energy (DOE) Federal Energy Management Program (FEMP) 2008 M & V Guidelines: Measurement and Verification for Federal Energy Project, Version 3.0 http://www.eere.energy.gov/femp 285 ©2009, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc (www.ashrae.org) For personal use only Additional reproduction, distribution, or transmission in either print or digital form is not permitted without ASHRAE's prior written permission ... permission Real- Time Energy Consumption Measurements in Data Centers Figure 1.4 - Key metering locations in a data center 10 ©2009, American Society of Heating, Refrigerating and Air-Conditioning Engineers,... permission Real- Time Energy Consumption Measurements in Data Centers the most extensive, with the more difficult to instrument parts of the facility remaining uninstrumented Limited modification to infrastructure... written permission Introduction real- time energy consumption data using energy efficiency and productivity metrics An example of a real- time energy efficiency metric is the real- time version of