How to Measure Anything Finding the Value of “Intangibles in Business” Third Edition DOUGLAS W HUBBARD Cover design: Wiley Cover image: © iStockphoto.com (clockwise from the top); © graphxarts, © elly99, © derrrek, © procurator, © Olena_T, © miru5 Copyright © 2014 by Douglas W Hubbard All rights reserved Published by John Wiley & Sons, Inc., Hoboken, New Jersey First edition published by John Wiley & Sons, Inc., in 2007 Second edition published by John Wiley & Sons, Inc., in 2010 Published simultaneously in Canada No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8600, or on the Web at www.copyright.com Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/permissions Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose No warranty may be created or extended by sales representatives or written sales materials The advice and strategies contained herein may not be suitable for your situation Y ou should consult with a professional where appropriate Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at (800) 762-2974, outside the United States at (317) 572-3993, or fax (317) 5724002 Wiley publishes in a variety of print and electronic formats and by print-on-demand Some material included with standard print versions of this book may not be included in e-books or in print-on-demand If this book refers to media such as a CD or DVD that is not included in the version you purchased, you may download this material at http://booksupport.wiley.com For more information about Wiley products, visit www.wiley.com Library of Congress Cataloging-in-Publication Data Hubbard, Douglas W., 1962– How to measure anything : finding the value of intangibles in business / Douglas W Hubbard.—Third edition pages cm Includes bibliographical references and index ISBN 978-1-118-53927-9 (cloth); ISBN 978-1-118-83644-6 (ebk); ISBN 978-1-118-83649-1 (ebk) Intangible property—Valuation I Title HF5681.I55H83 2014 657'.7—dc23 2013044540 I dedicate this book to the people who are my inspirations for so many things: to my wife, Janet, and to our children, Evan, Madeleine, and Steven, who show every potential for being Renaissance people I also would like to dedicate this book to the military men and women of the United States, so many of whom I know personally I've been out of the Army National Guard for many years, but I hope my efforts at improving battlefield logistics for the U.S Marines by using better measurements have improved their effectiveness and safety CONTENTS Preface to the Third Edition Acknowledgments About the Author PART I: The Measurement Solution Exists CHAPTER 1: The Challenge of Intangibles The Alleged Intangibles Yes, I Mean Anything The Proposal: It’s about Decisions A “Power Tools” Approach to Measurement A Guide to the Rest of the Book CHAPTER 2: An Intuitive Measurement Habit: Eratosthenes, Enrico, and Emily How an Ancient Greek Measured the Size of Earth Estimating: Be Like Fermi Experiments: Not Just for Adults Notes on What to Learn from Eratosthenes, Enrico, and Emily Notes CHAPTER 3: The Illusion of Intangibles: Why Immeasurables Aren’t The Concept of Measurement The Object of Measurement The Methods of Measurement Economic Objections to Measurement The Broader Objection to the Usefulness of “Statistics” Ethical Objections to Measurement Reversing Old Assumptions Notes Note PART II: Before You Measure CHAPTER 4: Clarifying the Measurement Problem Toward a Universal Approach to Measurement The Unexpected Challenge of Defining a Decision If You Understand It, You Can Model It Getting the Language Right: What “Uncertainty” and “Risk” Really Mean An Example of a Clarified Decision Notes Notes CHAPTER 5: Calibrated Estimates: How Much Do You Know Now? Calibration Exercise Calibration Trick: Bet Money (or Even Just Pretend To) Further Improvements on Calibration Conceptual Obstacles to Calibration The Effects of Calibration Training Notes Notes CHAPTER 6: Quantifying Risk through Modeling How Not to Quantify Risk Real Risk Analysis: The Monte Carlo An Example of the Monte Carlo Method and Risk Tools and Other Resources for Monte Carlo Simulations The Risk Paradox and the Need for Better Risk Analysis Notes CHAPTER 7: Quantifying the Value of Information The Chance of Being Wrong and the Cost of Being Wrong: Expected Opportunity Loss The Value of Information for Ranges Beyond Yes/No: Decisions on a Continuum The Imperfect World: The Value of Partial Uncertainty Reduction The Epiphany Equation: How the Value of Information Changes Everything Summarizing Uncertainty, Risk, and Information Value: The PreMeasurements Notes PART III: Measurement Methods CHAPTER 8: The Transition: From What to Measure to How to Measure Tools of Observation: Introduction to the Instrument of Measurement Decomposition Secondary Research: Assuming You Weren’t the First to Measure It The Basic Methods of Observation: If One Doesn’t Work, Try the Next Measure Just Enough Consider the Error Choose and Design the Instrument Note CHAPTER 9: Sampling Reality: How Observing Some Things Tells Us about All Things Building an Intuition for Random Sampling: The Jelly Bean Example A Little about Little Samples: A Beer Brewer’s Approach Are Small Samples Really “Statistically Significant”? When Outliers Matter Most The Easiest Sample Statistic Ever A Biased Sample of Sampling Methods Notes Notes CHAPTER 10: Bayes: Adding to What You Know Now The Basics and Bayes Using Your Natural Bayesian Instinct Heterogeneous Benchmarking: A “Brand Damage” Application Bayesian Inversion for Ranges: An Overview The Lessons of Bayes Notes PART IV: Beyond the Basics CHAPTER 11: Preference and Attitudes: The Softer Side of Measurement Observing Opinions, Values, and the Pursuit of Happiness A Willingness to Pay: Measuring Value via Trade-Offs Putting It All on the Line: Quantifying Risk Tolerance Quantifying Subjective Trade-Offs: Dealing with Multiple Conflicting Preferences Keeping the Big Picture in Mind: Profit Maximization versus Purely Subjective Trade-Offs Notes CHAPTER 12: The Ultimate Measurement Instrument: Human Judges Homo Absurdus: The Weird Reasons behind Our Decisions Getting Organized: A Performance Evaluation Example Surprisingly Simple Linear Models How to Standardize Any Evaluation: Rasch Models Removing Human Inconsistency: The Lens Model Panacea or Placebo?: Questionable Methods of Measurement Comparing the Methods Example: A Scientist Measures the Performance of a Decision Model Notes CHAPTER 13 : New Measurement Instruments for Management The Twenty-First-Century Tracker: Keeping Tabs with Technology Prediction Markets: A Dynamic Aggregation of Opinions Notes CHAPTER 14: A Universal Measurement Method: Applied Information Economics Bringing the Pieces Together Case: The Value of the System That Monitors Your Drinking Water Case: Forecasting Fuel for the Marine Corps Case: Measuring the Value of ACORD Standards Ideas for Getting Started: A Few Final Examples Summarizing the Philosophy Notes APPENDIX: Calibration Tests (and Their Answers) Index List of Tables Appendix Calibration Survey for Ranges: A Answers for Calibration Survey for Ranges: A Calibration Survey for Ranges: B Answers to Calibration Survey for Ranges: B Calibration Survey for Binary: A Answers for Calibration Survey for Binary: A Calibration Survey for Binary: B Answers to Calibration Survey for Binary: B List of Illustrations Chapter Exhibit 4.1 IT Security for the Department of Veterans Affairs Exhibit 4.2 Department of Veterans Affairs Estimates for the Effects of Virus Attacks Chapter Exhibit 5.1 Sample Calibration Test Exhibit 5.2 Actual versus Ideal Scores: Initial 10 Question 90% CI Test Exhibit 5.3 Spin to Win! Exhibit 5.4 Methods to Improve Your Probability Calibration Exhibit 5.5 Aggregate Group Performance Exhibit 5.6 90% Confidence Interval Test Score Distribution after Training (Final 20-Question Test)21 Exhibit 5.7 Calibration Experiment Results for 20 IT Industry Predictions in 1997 Chapter Exhibit 6.1 The Normal Distribution Exhibit 6.2 Simple Monte Carlo Layout in Excel Exhibit 6.3 Histogram Exhibit 6.4 The Binary (a.k.a Bernoulli) Distribution Exhibit 6.5 The Uniform Distribution Exhibit 6.6 Optional: Additional Monte Carlo Concepts for the More Ambitious Student Exhibit 6.7 A Few Monte Carlo Tools Chapter Exhibit 7.1 Extremely Simple Expected Opportunity Loss Example Exhibit 7.2 EOL “Slices” for Range Estimates Exhibit 7.3 Example EVPI Calculation for Segments in a Range (total number of rows in actual table would be 20) Exhibit 7.4 Example of the Relative Threshold Exhibit 7.5 Expected Opportunity Loss Factor Chart Exhibit 7.6 Loss Functions for Decisions on a Continuum Exhibit 7.7 The Value verses Cost of Partial Information Exhibit 7.8 The Effect of Time Sensitivity on EVPI and EVI Exhibit 7.9 Measurement Inversion Chapter Exhibit 9.1 Simplified t-Statistic Pick the nearest sample size (or interpolate if you prefer more precision) Exhibit 9.2 How Uncertainty Changes with Sample Size Exhibit 9.3 Varying Rates of Convergence for the Estimate of the Mean Exhibit 9.4 Mathless 90% CI for the Median of Population Exhibit 9.5 Population Proportion 90% CI for Small Samples Exhibit 9.6 Example Distributions for Estimates of Population Proportion from Small Samples Exhibit 9.7 Comparison of World War II German Mark V Tank Production Estimates Exhibit 9.8 Serial Number Sampling Exhibit 9.9 Threshold Probability Calculator Exhibit 9.10 Example for a Customer Support Training Experiment Exhibit 9.11 Probability of Correct Guesses Out of 280 Trials in Emily Rosa’s Experiment assuming a 50% chance per guess of being correct Exhibit 9.12 Examples of Correlated Data Exhibit 9.13 Promotion Period versus Ratings Points for a Cable Network Exhibit 9.14 Selected Items from Excel’s Regression Tool “Summary Output” Table Exhibit 9.15 Promotion Time versus Ratings Chart with the “Best-Fit” Regression Line Added Chapter 10 Exhibit 10.1 Selected Basic Probability Concepts Exhibit 10.2 The Bayesian Inversion Calculator Spreadsheet Exhibit 10.3 Probability That the Majority Is Green, Given the First Five Samples* Exhibit 10.4 Calibrated Subjective Probabilities versus Bayesian Exhibit 10.5 Confidence versus Information Emphasis Exhibit 10.6 Customer Retention Example Comparison of Prior Knowledge, Sampling without Prior Knowledge, and Sampling with Prior Knowledge (Bayesian Analysis) Exhibit 10.7 Summary of Results of the Three Distributions versus Thresholds Exhibit 10.8 Example Prior Distribution of Ranges (Low Resolution) Exhibit 10.9 Chance of Each Population Distribution Based on Example of Sampling ... But the title chosen, How to Measure Anything: Finding the Value of Intangibles in Business, seemed to grab the right audience and convey the point of the book without necessarily excluding... Applied Information Economics Bringing the Pieces Together Case: The Value of the System That Monitors Your Drinking Water Case: Forecasting Fuel for the Marine Corps Case: Measuring the Value of. . .How to Measure Anything Finding the Value of Intangibles in Business” Third Edition DOUGLAS W HUBBARD Cover design: Wiley Cover image: © iStockphoto.com (clockwise from the top); © graphxarts,