Real time energy consumption measurements in data centers

Figure 1.1 - Projected data center energy use scenarios EPA, 2007 The alarming trend of escalating electricity consumption in US data centers has spurred the ICT industry to aggressively

Any updates/errata to this publication will be posted on the ASHRAE Web site at www.ashrae.org/publicationupdates

ISBN 978-1-933742-73-1

©2009 American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc

1791 Tullie Circle, NE Atlanta, GA 30329 www.ashrae.org

All rights reserved

Printed in the United States of America Cover design by Joe Lombardo, DLB Associates

ASHRAE has compiled this publication with care, but ASHRAE has not investigated, and ASHRAE expressly disclaims any duty to investigate, any product, service, process, procedure, design, or the like that may be described herein The appearance of any technical data or editorial material in this publication does not constitute endorsement, warranty, or guaranty by ASHRAE of any product, service, process, procedure, design, or the like ASHRAE does not warrant that the information in the publication is free of errors, and ASHRAE does not necessarily agree with any statement or opinion in this publication The entire risk of the use of any information in this publication is assumed

by the user

No part of this book may be reproduced without permission in writing from ASHRAE, except by a reviewer who may quote brief passages or reproduce illustrations in a review with appropriate credit; nor may any part of this book be reproduced, stored in a retrieval system, or transmitted in any way or by any means—electronic, photocopying, recording,

or other—without permission in writing from ASHRAE Requests for permission should

be submitted at www.ashrae.org/permissions

Acknowledgments ix

PART 1 BASICS CHAPTER 1 INTRODUCTION 3

1.1 Objectives for this Book 11

1.2 How to Use this Book 12

CHAPTER 2 HOW, WHAT, & WHERE TO MEASURE 15

2.1 Overview 15

2.2 Quantifying Energy Efficiency Metrics 17

CHAPTER 3 MEASUREMENT DEVICES 21

3.1 Overview 21

3.2 Sensor Accuracy 23

3.3 Temperature 24

3.4 Pressure 29

3.5 Flow—Liquid 31

3.6 Flow—Gas 40

3.7 Current 44

3.8 Voltage 49

3.9 Power 53

CHAPTER 4 MEASUREMENT COLLECTION SYSTEMS—ARCHITECTURE & SOFTWARE 59

4.1 Overview 59

4.2 Business Questions 60

4.3 Scalable Hardware/Software Architecture 63

4.4 Measurement Levels 64

PART 2 COOLING SYSTEMS—AIR MEASUREMENTS

CHAPTER 5 AIR HANDLERS 79

5.1 Overview 79

5.2 Measurement Levels 81

CHAPTER 6 COMPUTER ROOM UNITS 85

6.1 Overview 85

6.2 Measurement Levels 94

PART 3 COOLING SYSTEMS —HYDRONIC MEASUREMENTS CHAPTER 7 PUMPS 101

7.1 Overview 101

7.2 Measurement Levels—Electrical 101

7.3 Measurement Levels—Fluid 105

CHAPTER 8 COOLING TOWERS 113

8.1 Overview 113

8.2 Measurement Levels 116

CHAPTER 9 CHILLERS 125

9.1 Overview 125

9.2 Measurement Levels 129

CHAPTER 10 HEAT EXCHANGERS 141

10.1 Overview 141

10.2 Measurement Levels 143

PART 4 POWER SYSTEMS MEASUREMENTS CHAPTER 11 INTRODUCTION TO CRITICAL POWER DISTRIBUTION 149

11.1 Overview 149

11.2 Critical Power versus Essential Power 150

CHAPTER 12 UPSTREAM CRITICAL

POWER DISTRIBUTION 155

12.1 Overview 155

12.2 Service Entrance Equipment 155

12.3 Automatic Transfer Switch (ATS) 159

12.4 Primary Electrical Distribution Switchgear 161

CHAPTER 13 UNINTERRUPTIBLE POWER SUPPLY (UPS) 165

13.1 Overview 165

13.2 UPS Metering, Power Module Level 168

13.3 UPS Metering, System Level 169

CHAPTER 14 COMPUTER ROOM TRANSFORMER & POWER DISTRIBUTION UNIT (PDU) 173

14.1 Overview 173

14.2 Stand-Alone Transformers 173

14.3 Computer Room Power Distribution Units (PDU) 175

14.4 Rack-Mounted Power Distribution Unit (RPDU) 179

PART 5 IT SYSTEMS MEASUREMENTS CHAPTER 15 COMPUTE & STORAGE SYSTEMS 185

15.1 Overview 185

15.2 Measurement Levels 190

CHAPTER 16 NETWORKING SYSTEMS 201

16.1 Overview 201

16.2 Measurement Levels 206

APPENDIX A PUMPS 213

A.1 Power and Efficiency 213

A.2 Real-Time Power Measurements 216

APPENDIX B CHILLERS 223

B.1 Variables Affecting RLA and Power Rating 223

B.2 Integrated or Non-Standard Part Load Value 224

APPENDIX C MIXED-USE FACILITIES 227

C.1 Real-Time Cooling Tower Power Consumption 227

C.2 Real-Time Chiller Power Consumption 229

APPENDIX D UNINTERRUPTIBLE POWER SUPPLY (UPS) 233

D.1 Technology 234

D.2 Redundancy and Availability 236

D.3 Rules of Thumb for Minimum Practical Level of UPS Instrumentation 241

D.4 Sample Case Study: A Partial PUE and DCiE Determination for the Critical Power Path within the Data Center 243

APPENDIX E ONSITE POWER GENERATION AND CCHP IN DATA CENTER APPLICATIONS 249

E.1 Overview 249

E.2 CCHP 251

E.3 Measurement Levels 258

E.4 Example Calculations for a CCHP Installation 263

APPENDIX F ABBREVIATIONS AND GLOSSARY 267

REFERENCES 283

ASHRAE TC 9.9 and The Green Grid would like to thank the following individuals for their substantial contributions to the book:

Lead editor / author – Tahir Cader, HP (formerly SprayCool) Co-editor after First Draft – Don Beaty, DLB Associates

Chapter 1 – Tahir Cadera,b, HP (lead)

Chapter 2 – Tahir Cadera,b, HP (lead); Mike Mangan, DLB

Associates; Jeff Jaworksi, DLB Associates

Chapter 3 – John Beana,b, APC/Schneider; Randall Woffordb, Dell; Ross Ignalla, Dranetz-BMI; Michael Kennedy, DLB Associates Chapter 4 – Ken Uhlmanb, Eaton (lead); Harry Rogersb, Microsoft Chapter 5 – Robert Wasilewski, DLB Associates (lead)

Chapter 6 – Jeff Trowera, DataAire (lead); Cliff Federspiel,

Federspiel Controls

Chapter 7 – John Beana,b, APC/Schneider (lead)

Chapter 8 – Daryn Clinea, Evapco (lead)

Chapter 9 – Jonathan Spreemana, Trane (lead); Tahir Cadera,b, HP Chapter 10 – Robert Wasilewski, DLB Associates

Chapters 11, 12, 13, 14 – Steve McCluera,b, APC/Schneider

Electric (lead); Bill Campbellb, Emerson Network Power; John Messerb, Emerson Network Power

Chapter 15 – Mike Pattersona,b, Intel (lead); Bob MacArthurb, EMC

Chapter 16 – Kevin Engelberta,b, Cisco (lead)

Chapter 17 – Kevin Wymana, Carrier Corporation (lead); Greg Palmerb, HP (formerly UTC Power) (lead);

Appendix A – John Beana,b, APC/Schneider (lead)

Appendix B – Jonathan Spreemana, Trane (lead)

Appendix C – Tahir Cadera,b, HP (lead); Jonathan Spreemana, Trane

Appendix D – Steve McCluer , APC/Schneider (lead); Bill

Campbella, Emerson Network Power; John Messerb, Emerson Network Power

Appendix E – Kevin Wymana, Carrier Corporation; Greg Palmera,b,

HP (formerly UTC Power) (lead);

The following individuals also provided significant feedback and guidance in the writing of this book: Roger Schmidt, IBM; Don Beaty, DLB Associates (major commenter on 1st edition)

Production of final book including creation of most graphics – Jeff Jaworski, DLB Associates; Mike Mangan, DLB Associates

Book cover design – Joe Lombardo, DLB Associates

a Member ASHRAE TC 9.9

b Member The Green Grid

Over the last several years, energy consumption by data centers in the US as well as worldwide has become a topic of intense discussion within the Information and Communication Technologies (ICT) world There are numerous publications presenting statistics on the impact of data center power consumption on the supply of electricity One of the more comprehensive studies was that requested by the US Congress in Public Law 109-431, in which the EPA was mandated to quantify the

electricity usage by US data centers, resulting in Report to Congress on

Server and Data Center Energy Efficiency Public Law 109-431, 2007

The key finding of this study is that in 2006, US data centers consumed 1.5% of all electricity used in the US and that according to historical trends, this consumption would rise to 2.9% by 2011 The 1.5% electricity usage included servers and the infrastructure to support servers, but did not include network or storage equipment A graph of the findings is shown in Figure 1.1

Figure 1.1 - Projected data center energy use scenarios (EPA, 2007)

The alarming trend of escalating electricity consumption in US data centers has spurred the ICT industry to aggressively increase energy efficiency in order to dramatically reduce power consumption in data centers Together, the DOE and The Green Grid have stated that a goal for 2011 is to achieve a reduction of energy to 100B kWh / year instead

of the current projection of 120B kWh / year for 2011 One of the key ways in which the industry can achieve the state-of-the-art curve is via real-time energy efficiency, which is achievable only through the use of

real-time energy consumption data using energy efficiency and productivity metrics An example of a real-time energy efficiency metric

is the real-time version of the Power Utilization Effectiveness (PUE)

metric as proposed by The Green Grid (Green Grid Data Center Power

Efficiency Metrics: PUE And DCiE, 2008) This metric is defined and

discussed further in Chapter 2 The focus of this book is real-time energy consumption measurements, with the resulting data to be used in all the relevant energy efficiency and productivity metrics

Real-time energy consumption measurements are only possible if all key subsystems are appropriately instrumented and properly communicating through use of data center level software Existing data centers have varying levels of instrumentation, ranging from very poor to excellent For this book, three approaches to instrumentation and measurement for any given subsystem will be followed The following loose guidelines are provided:

 Minimum Practical Measurement

 Best Practical Measurement

 State-of-the-Art Measurement

When deciding what level of measurement to target, a data center owner / operator needs to keep in mind key items such as capital cost, data accuracy and resolution, and end-use of the data These factors will

be dealt with in further detail in subsequent chapters The following guidelines, summarized in Table 1.1, are suggested:

 Minimum Practical Measurement – This will require some level of human activity to perform periodic measurements This approach will require zero to limited infrastructure upgrades, and zero to limited investment in instrumentation This approach may rely more heavily on staff (most likely existing) to manually record data, and will also rely on manufacturers’ equipment data

 Best Practical Measurement – This will require a lower level of human activity than the minimum case in order to manually record data For this case, it is anticipated that data will be logged in real-time with extensive trending possible The instrumentation used may not necessarily be of the highest accuracy nor will it likely be

the most extensive, with the more difficult to instrument parts of the facility remaining uninstrumented Limited modification to infrastructure should be expected, and some tasks may be beyond the competency of the existing staff Less reliance on manufacturers’ data is expected

 State-of-the-Art Measurement – This will not require human activity

to gather and record data Data will be collected by automated systems in real-time and will support extensive trending and analysis The instrumentation will be of accuracy suitable for revenue grade There will likely be a requirement to upgrade the existing infrastructure, and it is very likely some level of contractor or consultant support will be needed for the implementation

Mixed-use facilities offer the greatest challenge in which to quantify real time energy consumption Figures 1.2 through 1.4 show a generic layout in a mixed-use facility These figures are schematic in nature and are not intended to be fully representative of all possible configurations Figure 1.2 shows a schematic representation of the electrical distribution system in a mixed-use data center, while Figure 1.3 shows the mechanical layout of the same data center type

Figure 1.2 - Schematic representation

of the electrical system in a

mixed-use facility

Figure 1.3 - Schematic representation

of the mechanical layout of a data center housed in a mixed-use facility

Figure 1.4 is a simple graphic representing the key metering locations in a typical data center The meters acquire power consumption data from all the electrical and mechanical subsystems shown in Figures 1.2 and 1.3, respectively Each metering location is associated with its own hardware and software protocols, and in many cases hardware and software is provided by multiple vendors These systems generally do not communicate with each other, which creates a significant issue in progressing toward the display of real-time energy and productivity metrics The subsequent chapters will discuss these issues in further detail

Figure 1.4 - Key metering locations in

a data center

1.1 OBJECTIVES FOR THIS BOOK

The following are key objectives for the book:

 Provide an overview of the state of real-time energy consumption measurements in the data center The book will cover both legacy as well as state-of-the-art data centers

 Discuss the minimum and best practical levels of measurement, as well as state-of-the-art measurement for real-time energy consumption measurements (see Chapter 1 – Introduction)

 Provide a detailed discussion of how the measured real-time data will be used, and in particular how this information will be turned into knowledge that can lead to actionable items This will cover the latest industry data center productivity and energy efficiency metrics from organizations such as The Green Grid and ASHRAE TC9.9 Emphasis will also be placed on quantifying the data center’s power consumption for a data center housed in a mixed-use facility

The idea behind the state-of-the-art measurement is that the industry will eventually arrive at the ―plug and play‖ data center Such a data center will rely on the widespread availability of network-enabled equipment For example, at some point in the future, a data center owner / operator can expect to ―plug in‖ a key subsystem such as a pump and have the data center’s operating system recognize the pump in real-time This will be followed shortly thereafter by real-time reporting of energy consumption measurements, and in turn real-time data center productivity and energy efficiency metrics

This book will focus on monitoring and control for optimization of data center energy efficiency There are, in fact, other benefits that may arise from real-time monitoring and control One key benefit includes predicting the health of the infrastructure by tracking performance trends Additionally, while the book presents several examples using the Power Usage Effectiveness (PUE) metric from The Green Grid, use of any specific metric such as the DOE’s Energy Usage Effectiveness (EUE), or The Green Grid’s Data Center Energy Productivity (DCeP) is left entirely up to the data center owner / operator

1.2 HOW TO USE THIS BOOK

While it is recommended that this book be read in its entirety, it is possible to benefit from reading only parts of the book For the benefit

of the reader, the book has been divided into five parts, each containing chapters dedicated to key components or subsystems The five parts included are:

Part 1 Basics

Part 2 Cooling Systems – Air Measurements

Part 3 Cooling Systems – Hydronic Measurements

Part 4 Power Systems Measurements

Part 5 IT Systems Measurements

Part 1 will provide an overview of the book including measurement devices and software Chapter 1 sets the stage for the book Chapter 2 will focus on How, What & Where To Measure Chapter 3, the Measurement Devices chapter, will provide an overview of the various sensor types available The chapter will provide an overview of sensors for all electrical (e.g., voltage, current, etc.) and mechanical (e.g., pressure, temperature, flow, etc.) systems in the data center Chapter 4, the Measurements Collection Systems chapter, will cover the business objectives that will guide a data center owner / operator to a given level

of instrumentation (i.e., minimum practical, best practical, or the-art level of measurement) This chapter will also provide an overview of the various standards and protocols to facilitate communication with IT equipment and facilities equipment The objective of such protocols is to acquire the real-time power consumption data and make it readily available to the data center owner / operator Finally, this chapter will provide some discussion of how the acquired and reduced data can be turned into knowledge and subsequent actionable items that affect the business

state-of-Parts 2 and 3 will provide an overview of the various cooling systems and subsystem types (e.g., chillers) that are deployed today Each chapter will focus on the single most widely deployed subsystem type and provide a more detailed discussion of the three levels of instrumentation The reader will be shown a high level discussion (not a detailed description) of how to, at each level of instrumentation, use

measured data and manufacturer’s data to quantify the power consumption of the specific subsystem type Special attention is given, where appropriate, to show the reader how to quantify that part of the subsystem’s power consumption that is attributable to a data center housed in a mixed-use facility For example, in mixed-use facilities, the cooling towers, chillers, and pumps typically support all parts of the facility, including the data center

Part 4 will focus on the power delivery path from the point of entry into the facility, to the point of delivery to the IT equipment Specific attention is paid to Uninterruptible Power Supplies (UPS) and transformers As with the other chapters, an overview is provided with emphasis on the most widely deployed UPSs and transformers

Part 5 will provide a description of the servers, storage, and networking equipment deployed in data centers The three levels of instrumentation will be discussed, and the reader will be shown how to roll the total IT equipment power consumption into a single power consumption number for later use by the data center owner / operator There are also Appendices meant to provide additional information

or detail for different subsystems or components Appendix A provides additional information for calculating real-time pump efficiency Appendix B describes additional methods for quantifying chiller efficiency Appendix C focuses on a specific example for calculations within a mixed-use facility Appendix D provides additional information

on Uninterruptible Power Supply efficiency measurements Some of-thumb calculations are also provided to enable the reader to perform calculations on power conversion losses Appendix E, Onsite Combined Cooling, Heat, and Power (CCHP), describes in some detail the specifics

rule-of CCHP and waste heat recovery An important feature rule-of the chapter is the description of how to accommodate CCHP within the calculation of energy efficiency metrics such as PUE for data centers Appendix F lists the nomenclature in the book

A references section is located at the end of the book

2.1 OVERVIEW

Understanding the overall goals for measurements is as important as implementing the measurement system and obtaining measurements within the data center While accuracy of the measurement devices can

be critical, benefits can be realized through simply obtaining useful data Depending on what, where, and how measurements are taken, varying levels of accuracy of the devices may be implemented

Potential uses for measured data can include understanding energy usage as a whole, trending over time, understanding the instantaneous power consumption of key pieces of equipment, billing, or calculating energy efficiency using one of the metrics described in this book Depending on the purpose, different factors may be paramount for the collected data Assuming that a fixed budget exists for obtaining a set of measurements or calculating a metric, the owner / operator may need to balance measurement accuracy with frequency, sensor quantity and location These decisions ultimately need to be made in accordance with understanding how, what, and where to measure

Common to all systems is the opportunity to measure the real-time power consumption for each subsystem The real-time energy consumption can be measured directly via current and voltage measurements, or accurately via the measurement of power For purely electrical equipment such as UPSs and transformers, the only choice of accounting for power consumption (losses in the case of this type of equipment) is via direct measurements of current and voltage or power For mechanical subsystems such as pumps, compressors, and blowers, estimated power consumption can be indirectly calculated via

the measurement of temperature, flow rate, and pressure drop Power consumption can also be obtained through direct measurements on each subsystem The measured data can then be used in conjunction with manufacturers’ performance data in order to determine efficiency It is instructive for the user to consult ASHRAE Guideline 22-2008 for some guidance with respect to instrumentation of the chilled-water plant (includes cooling towers, condenser water pumps, chillers, and chilled water pumps) It is important to note, however, that the level of instrumentation required for isolating the real-time power consumption

of a data center housed in a mixed-use facility is higher than that shown

in ASHRAE Guideline 22-2008

For facilities using air side economizers it is important to understand the condition of the air external to the data center to properly react to changing external conditions Particulate and gaseous contamination will not be discussed in this book, however, more information can be found in ASHRAE’s ―Particulate and Gaseous Contamination in Datacom Environments‖

Measurements can be taken either manually or automatically The specific device installed will dictate which option can be used Generally, manual readings will be the minimum practical measurement, while automatic readings will occur in the best practical, and state-of-the art data center Automated readings can be stored electronically and trended over time Trending can also be achieved with manual readings, but will take more time and effort to produce

The decision whether to use minimum practical, best practical, or state-of-the-art measurements is ultimately a function of the facility and its stakeholders This book will introduce a multitude of measurement devices, locations, and techniques to understand the energy consumption

of common components of a data center Obtaining the most useful data should always dictate the techniques employed

2.2 QUANTIFYING ENERGY EFFICIENCY METRICS

The Green Grid recently proposed Power Utilization Effectiveness

(PUE) as an energy efficiency metric (Data Center infrastructure

Efficiency (DCiE) is the reciprocal of PUE) This metric highlights the

amount of power that is consumed in total by the data center, including

IT loads, and the amount for IT and physical infrastructure to support the

IT Figure 2.1 shows a simple schematic detailing the key data center

subsystems that are accounted for in PUE or DCiE

Figure 2.1 - Data center energy efficiency metrics

In keeping with the discussion of each of the data center subsystems

covered in the remainder of the book, Power Utilization Effectiveness

(PUE) is defined as:

IT

facP P PUE (2.1) [Equation 2.1]

where P fac is total power consumed by the facility and P IT is total

power consumed by the IT equipment

The IT equipment consists of the servers, network gear, storage

equipment etc At the facility level, the key power consuming

subsystems include the chillers, IT equipment, Computer Room Air

Conditioners / Handlers (CRACs / CRAHs), cooling towers, pumps,

UPSs, etc Taking these into account, Equation 2-1 can be re-written as:

n stor net serv

n p ct crac c stor net serv

P P P P

P P P P P P P P

where P serv is total power consumed by the servers, P net is total power

consumed by the network equipment, P stor is total power consumed by

the storage equipment, P c is total power consumed by the chiller(s), P crac

is total power consumed by the Computer Room Air Conditioner(s)

(CRACs), P ct is total power consumed by the cooling tower(s), P p is total

power consumed by the pump(s), and P n is total power consumed by the

nth subsystem

As previously mentioned, the individual chapters on subsystems will

describe how to measure the real-time power consumption for each given

subsystem For example, ―Chapter 9 Chillers‖ describes the real-time

measurement of P c for the three levels of instrumentation described in

―Chapter 1 Introduction‖ In addition, examples are given in the

Appendices for each subsystem describing how to quantify what

percentage of the full facility power, P fac, the data center is responsible

for in a mixed-use facility

A simple example showing the calculation of DCiE and PUE is

illustrative of this point Assume that a data center’s total facility power

consumption is 2.2 MW, while the IT equipment power consumption is 1

MW Under this scenario, the DCiE is calculated as:

PUE = (P fac /P IT)

= (2200 kW/1000 kW) (2.3)

= 2.2

[Equation 2.3]

DCiE = (P IT /P fac) · 100%

= (1000 kW/2200 kW) · 100% (2.4) = 45%

The Green Grid is currently gathering data center performance information and intends to publish a white paper, in the near future, that will put the above-calculated values of PUE and DCiE in perspective In order to improve the efficiency of the data center, strategies should be developed that address the efficiency and utilization of both the IT and facilities systems Depending on which strategies are implemented, the numerator or denominator could change, thereby impacting these metrics The data center operator should be aware that technologies like virtualization can lower total IT power, effectively increasing PUE, while efficiency has gone up It is recommended that the user follow The Green Grid’s ―Usage and Public Reporting Guidelines for The Green Grid’s Infrastructure Metrics PUE/DCiE‖ for PUE of DCiE

While the usage of PUE and DCiE has been highlighted here, the objective is not to focus on any metric in particular To re-iterate, a key objective for the book is to educate the data center owner / operator on how to acquire the real-time power consumption measurement data This data can subsequently be used in the determination of any energy efficiency, data center productivity, or other metric of choice

3.1 OVERVIEW

In the data center, sensors are used to measure key variables such as temperature, flow rate, current, voltage, pressure, humidity, etc Meters generally form part of a monitoring system that has its information gathered by the appropriate software to display an aggregate view of the information from a device (e.g., a data center subsystem), a facility, or an enterprise Clearly, the various types of software (see Chapter 4) need the capability to communicate with the various sensor / meter combinations

Sensors and meters come in a wide variety of configurations, accuracies, and connectivity styles This chapter is intended to provide the reader with an overview of the various sensor and meter styles, their basic modes of operations, their general accuracy levels, and their general applicability to the various subsystems in the data center

Table 3.1 lists the key data center subsystems covered in this book, each subsystem’s major components, and the key variables to be measured for each subsystem The table shows that there are some commonalities between subsystems and components For example, pumps are deployed in cooling towers for spraying water, for moving condenser and chilled water, and for moving condensate in CRACs The condenser and chilled water pumps will be large pumps (>20 hp [15 kW]), the cooling tower pumps will be mid-size (>5 hp [4 kW]), and the condensate pumps will be the smallest pumps (<1 hp [0.75 kW]) The real-time power consumption for each of these pump types can be directly measured via current and voltage, or with a power meter The power consumption for the pumps can be indirectly measured using flow rate and pressure drop, but this approach is not likely to be effective for the small volumes of water moved by the condensate pumps Chapter 7 - Pumps provides a detailed discussion with respect to the measurement of the real-time power consumption of pumps

With respect to the instrumentation, the key specifications such as span (instrument range), resolution, and accuracy will change as a function of the equipment that is being measured For example, it is not recommended to use a power meter capable of measuring over a range of

0 to 1,000 kW on a pump that consumes a maximum of 20 kW Additionally, an important part of sensor accuracy is system calibration and understanding the potential drift inherent in the type of sensor selected Calibration should always be performed with the system installed compared with from the factory System calibration can be impacted by software filtering, slope of the sensor reading and software scaling

This chapter will discuss the various sensor types to be used to measure the key variables, and will provide some high level guidance with respect to specifying the most accurate instrumentation within the bounds of the minimum practical, best practical, and state-of-the-art measurements This guidance will be provided for all the data center subsystems and associated components listed in Table 3.1

This chapter will provide details on sensors and in some cases, sensor / meter combinations for the measurement of the following key variables:

 Power (separate from combination of current and voltage)

Sensor selection is dependent upon the quality (accuracy, precision, drift, rate of response), quantity, installation restrictions, method of measurement required, signal output requirements (or signal conditioning), measurement range, turndown ratio, the capabilities of the intended data recording devices, and the resources available to purchase and / or support them

The impact of inaccurate data can have a dramatic effect on energy consumption For example, a 1 °F (0.56 °C) decrease in chilled water temperature caused by an inaccurate high reading can create a 2 - 4 percent increase in energy usage to maintain that unnecessary low temperature Not knowing the real temperatures can cost a fortune in wasted energy, not to mention wear and tear on the chiller components

by running outside of intended parameters The four main contributors

to bad data are inaccurate temperature sensors, pressure sensors, turbulent flow, and human error

The following example of how inaccurate data can affect energy efficiency:

Example

Assume a chiller has a design specification of 600 tons (2.11 MW), drawing 500 full load amps at 460 V with a power factor of 0.9, producing a 10 °F (5.6 °C) ∆T at a flow rate of 1,440 gpm (327 m3/hr) and a design efficiency of 0.598 kW/ton, where ―kW‖ refers to the

electrical input to the chiller and ―ton‖ refers to the cooling output of the chiller (1 ton = 12,000 Btu/h [3.52 kW]) If this chiller’s evaporator water temperature sensor is reading low by 1 °F (0.56 °C) and the evaporator water out temperature sensor is reading high by 1 °F (0.56 °C), a combined error of 2 °F (1.1 °C) or an 8 °F (4.4 °C) ∆T at full load exists When the efficiency is calculated, it equals 0.747 kW/ton Dividing 0.598 by 0.747 gives 80 percent actual efficiency A 2 °F (1.1 °C) error can make it appear that the chiller is 20 percent inefficient

The inaccuracies shown in the above example can alter scheduling of maintenance, produce inaccurate cost analysis, and skew the plant load profile by 20 percent, making decisions concerning chiller sizing very difficult Operating at 80 percent of full load rated efficiency, a 600 ton (2.11 MW) chiller running at 50 percent load, 24 hours / 365 days, at

$0.06 / kWh would indicate a $24,912 loss This emphasizes the reason that sensors and gauges should be accurately calibrated to their specified accuracies

There are many different types of temperature sensors, each based on different technologies This section will not discuss every type, but instead the ones most common to data center systems The main sensor types focused on are thermocouples, thermistors, and RTDs (resistance temperature detectors) The reader should be aware that there are lead length limitations for each of these sensor types

Table 3.2 gives a brief overview of the sensors to be discussed in the

subsequent subsections, based on ASHRAE Handbook – Fundamentals,

2009

Source: ASHRAE Handbook – Fundamentals

3.3.1 Thermocouples

Figure 3.1 shows a typical thermocouple probe A thermocouple consists of two electrical conductors that are made of dissimilar metals and have at least one electrical connection (see for example Figliola and Beasley (1991), p 282) The two conductors are formed into a junction via soldering or twisting to make good electrical contact The junction of the two dissimilar metals, typically called the hot junction, produces a small voltage signal in proportion to the temperature of the junction One junction (the hot junction) is typically encased in a sensor probe at the point of measurement, while the other junction (the cold junction) is connected to the measuring instrument Thermocouples are among the easiest temperature sensors to use and have the advantage of being self-powered, relatively low cost, stable and durable

Figure 3.1 - Photograph of a thermocouple probe

Thermocouples are also rugged and very reliable An example of a typical use of thermocouples is shown in Figure 3.2 For the chiller plant (shown with condenser and chilled water pumps), thermocouples are installed in wells in the water piping These should be installed close to the upstream (inlet) of the temperature-changing device, such as chillers and cooling towers, but as far downstream (outlet) as practical to ensure that there is no temperature stratification in the outlet flow where the measurement is taken

Figure 3.2 - Schematic representation

of the use of thermocouples in a chilled water plant

3.3.2 Thermistors

Thermistors are special solid-state transducers that behave like temperature-sensitive electrical resistors that exhibit a change in electrical resistance with a change in temperature (see for example Figliola and Beasley (1991), p 275) Thermistors are special solid-state transducers that behave like temperature-sensitive electrical resistors that exhibit a change in electrical resistance with a change in temperature Figure 3.3 shows three thermistor probes A small and measured direct current is forced through the thermistor, the voltage drop is measured, and the resistance determined A functional relationship between resistance and temperature is used to determine the temperature Compared to other temperature sensors, thermistors are very accurate and precise over relatively small temperature range An advantage of a thermistor is the ability to retain specified characteristics after being subjected to designated environmental or electrical test conditions Thermistors can provide high sensitivity and accuracy, but can be more costly

Figure 3.3 - Photograph of thermistor probes

3.3.3 Resistance Temperature Detectors

(RTDs)

Resistance Temperature Detectors (RTDs) are mature and well understood temperature sensors that have been in use for many years Full technical descriptions for RTDs are readily available in any

engineering text (see for example Figliola and Beasley (1991), p 267) The following text has been excerpted from the website www.temperatures.com/rtds.html:

―Resistance Temperature Detectors (RTDs) are wire wound and thin film devices that measure temperature by making use of the physical principle of the positive temperature coefficient of electrical resistance of metals (see Figure 3.4 for selected RTDs) The hotter they become, the larger or higher the value of their electrical resistance They are among the most precise temperature sensors available with resolution and measurement uncertainties of ±0.2 °F (±0.1 °C) or better possible in special designs The most commonly used type of RTD element is platinum; these models are often referred to as Platinum Resistance Thermometer (PRTs) Additionally, platinum elements are popular because they can be used over a wide range of temperatures and feature a quick response time Platinum's coefficient of resistance is nearly linear

By using a platinum element, resolutions of ±0.2 °F (±0.1 °C) or better are possible.‖

Figure 3.4 - Photographs of RTDs

―The advantages of RTD’s include: stable output over a long period

of time, ease of recalibration, and highly accurate readings over relatively narrow temperature spans When compared to thermocouples their disadvantages are smaller overall temperature range and a greater initial cost ―

There are numerous types of pressure sensing methods utilizing different electrical signals and various means of local indication Local indicating sensors will have some form of attached display that allows observation of the reading, typically in engineering units Local indicating sensors may also include a provision for an electrical signal output for remote monitoring For simplicity only the more common subset of what is available is discussed The scope of information provided is not intended to eliminate the use of other styles of devices but rather to maintain focus on the more likely scenarios In all cases, it

is necessary to select wetted sensor materials that are compatible with fluids to be measured, that the pressure range is appropriate for the particular application, and that the electrical output signal is compatible with the analog inputs of the particular BMS (Building Management System) or Supervisory Control and Data Acquisition (SCADA) system used

It is important to understand that not all pressure sensors are suitable for use with liquids and that they may only be intended for applications using air or compressed gasses Another important consideration when selecting a pressure instrument is to understand whether it measures absolute or gauge pressure Gauge pressure is referenced to atmospheric pressure and is appropriate for most non-laboratory measurements of fluids within a piping system Gauge pressure can either indicate positive pressure above atmospheric conditions or vacuum level below atmospheric conditions Compound gauges are capable of measuring both positive pressures and vacuum

Table 3.3 gives a brief overview of the sensors to be discussed in the

subsequent subsections, based on ASHRAE Handbook – Fundamentals,

2009