This edition first published 2015 © 2015 John Wiley and Sons, Ltd Registered office John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, United Kingdom For details of our global editorial offices, for customer services and for information about how to apply for permission to reuse the copyright material in this book please see our website at www.wiley.com The right of the author to be identified as the author of this work has been asserted in accordance with the Copyright, Designs and Patents Act 1988 All rights reserved No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by the UK Copyright, Designs and Patents Act 1988, without the prior permission of the publisher Wiley also publishes its books in a variety of electronic formats Some content that appears in print may not be available in electronic books Designations used by companies to distinguish their products are often claimed as trademarks All brand names and product names used in this book are trade names, service marks, trademarks or registered trademarks of their respective owners The publisher is not associated with any product or vendor mentioned in this book This publication is designed to provide accurate and authoritative information in regard to the subject matter covered It is sold on the understanding that the publisher is not engaged in rendering professional services If professional advice or other expert assistance is required, the services of a competent professional should be sought Trademarks: Wiley and the Wiley logo are trademarks or registered trademarks of John Wiley & Sons, Inc and/or its affiliates in the United States and/or other countries, and may not be used without written permission All other trademarks are the property of their respective owners John Wiley & Sons, Ltd is not associated with any product or vendor mentioned in the book A catalogue record for this book is available from the British Library ISBN 978-1-118-94371-7 (paperback); 978-1-118-94373-1 (ePDF); 978-1-11894372-4 (ePub) Set in 9.5/11.5 MinionPro-Regular by SPS/TCS Printed in U.S by Bind Rite Robbinsville This book is for those who are willing to lead, in any endeavor Most of all, this book is for Kristin, Will, Abby, and Sam And, a special thanks to my big sister — Rob Thomas To my parents Agnes and Patrick, wife Emmeline, and children Isolde, Theodore and Caspian — Patrick McSharry About the Authors Rob Thomas is Vice President of Product Development for Big Data and Information Management in the IBM Software Group With over 15 years in the technology industry, Mr Thomas has had the opportunity to consult to a variety of global businesses He has experience in business and operational strategy, high technology, acquisitions and divestitures, manufacturing operations, and product design and development Mr Thomas is currently responsible for product development and engineering for IBM’s Big Data and Information Management product line As Vice President of Business Development in IBM Software, Mr Thomas led the acquisition of Netezza and Vivisimo, both leaders in the data era Mr Thomas has extensive international experience, leading IBM’s high technology services business in Asia Pacific, while living in Tokyo, Japan Prior to that, he was a Partner in IBM’s consulting business Mr Thomas graduated from Vanderbilt University with a BA in Economics He earned his Masters in Business Administration from the University of Florida Mr Thomas publishes regularly on his blog (http://www.robdthomas.com) and has an active following on Twitter (@robdthomas) He is an avid golfer, reader, and exercise enthusiast He lives in New Canaan, Connecticut with his wife (Kristin) and three children (Will, Abby, and Sam) Most of what he has learned in his life came from his parents, his wife, and his two sisters Patrick McSharry is a Senior Research Fellow at the Smith School of Enterprise and the Environment, Faculty Member of the Oxford Man Institute of Quantitative Finance at Oxford University, Visiting Professor at the Department of Electrical and Computer Engineering, Carnegie Mellon University, Fellow of the Royal Statistical Society and Senior Member of the IEEE He takes a multidisciplinary approach to developing quantitative techniques for data science, decision-making, and risk management His research focuses on big data, forecasting, predictive analytics, machine learning, and the analysis of human behavior He has published over 90 peer-reviewed papers, participated in knowledge exchange programs and consults for national and international government agencies and the insurance, finance, energy, telecoms, environment, and healthcare sectors Patrick received a first class honours BA in Theoretical Physics and an MSc in Engineering from Trinity College Dublin and a DPhil in Mathematics from Oxford University About the Technical Editor Carin Anderson is a freelance technical editor She has edited, compiled, and written numerous grants and proposals over the last decade and a half Carin developed a mobile application company, creating multi-user gaming platforms She also cofounded an informational website targeting families with young children In her spare time, she enjoys spending time with her family and friends, running and reading PUBLISHER’S ACKNOWLEDGEMENTS Some of the people who helped bring this book to market include the following: Editorial and Production VP Consumer and Technology Publishing Director: Michelle Leete Associate Director–Book Content Management: Martin Tribe Professional Technology & Strategy Director: Barry Pruett Commissioning Editor: Ellie Scott Development Editor: Tom Dinse Copy Editor: Laura Miller Technical Editor: Carin Anderson Marketing Associate Marketing Director: Louise Breinholt Marketing Manager: Lorna Mein Marketing Executive: Polly Thomas Composition Services Proofreader: Wordsmith Editorial Indexer: Potomac Indexing, LLC Big Data Revolution Table of Contents Prologue Berkeley, 1930s Pattern Recognition Committing to One Percent The Big Data Revolution Introduction Storytelling Objective Outline Storytelling (Continued) Part I: The Revolution Starts Now: 9 Industries Transforming with Data Chapter 1: Transforming Farms with Data 3 Chapter 2: Why Doctors Will Have Math Degrees 2 Nelson Peltz Part I “The Revolution Starts Now: 9 Industries Transforming with Data” Part II “Learning from Patterns in Big Data” Part III “Leading the Revolution” California, 2013 Brief History of Farming The Data Era Potato Farming Precision Farming Capturing Farm Data Deere & Company Versus Monsanto Integrated Farming Systems Data Prevails The Climate Corporation Growsafe Systems Farm of the Future California, 2013 (Continued) United States, 2014 The history of Medical Education Scientific Method Rise of Specialists We have a Problem Collecting Data Telemedicine Innovating with Data Implications of a Data-Driven Medical World The Future of Medical School Vinod Khosla The Data Era Ben Goldacre A Typical Medical School A Medical School for the Data Era United States, 2030 Chapter 3: Revolutionizing Insurance: Why Actuaries Will Become Data Scientists 3 4 Chapter 4: Personalizing retail and fashion 2 3 Middle of Somewhere, 2012 Short History of Property & Casualty Insurance and Underwriting Actuarial Science In Insurance Pensions, Insurance, Leases Compound Interest Probability Mortality Data Modern-Day Insurance Eight Weeks to Eight Days Online Policies The Data Era Dynamic Risk Management Catastrophe Risk Open Access Modeling Opportunities Middle of Somewhere, 2012 (Continued) Karolina A Brief History of Retail Retail Eras Aristide Boucicaut The Shift The Data Era Stitch Fix Keaton Row Zara Karolina (Continued) Chapter 5: Transforming Customer relationships with data 2 3 Chapter 6: Intelligent Machines 3 Chapter 7: Government and Society 3 7 Buying a House Brief History of Customer Service Customer Service Over Time Boeing Financial Services The Data Era An Automobile Manufacturer Zendesk Buying a House (Continued) Denmark Intelligent Machines Machine Data The Data Era General Electric Drones Tesla Networks of Data Denmark (Continued) Egypt, 2011 Social Media Intelligence Snowden Effect Privacy Risk Versus Reward Observation or Surveillance Development Targets Open Data Hackathons Open Access Ensuring Personal Protection Private Clouds Sanitizing Data Evidence-Based Policy Public-Private Partnerships Impact Bonds Social Impact Bond Table of Contents Prologue Berkeley, 1930s Pattern Recognition Nelson Peltz Committing to One Percent The Big Data Revolution Introduction Storytelling Objective Outline Part I “The Revolution Starts Now: Industries Transforming with Data” Part II “Learning from Patterns in Big Data” Part III “Leading the Revolution” Storytelling (Continued) Part I: The Revolution Starts Now: 9 Industries Transforming with Data Chapter 1: Transforming Farms with Data California, 2013 Brief History of Farming The Data Era Potato Farming Precision Farming Capturing Farm Data Deere & Company Versus Monsanto Integrated Farming Systems Data Prevails The Climate Corporation Growsafe Systems Farm of the Future California, 2013 (Continued) Chapter 2: Why Doctors Will Have Math Degrees United States, 2014 The history of Medical Education Scientific Method Rise of Specialists We have a Problem Ben Goldacre Vinod Khosla The Data Era Collecting Data Telemedicine Innovating with Data Implications of a Data-Driven Medical World The Future of Medical School A Typical Medical School A Medical School for the Data Era United States, 2030 Chapter 3: Revolutionizing Insurance: Why Actuaries Will Become Data Scientists Middle of Somewhere, 2012 Short History of Property & Casualty Insurance and Underwriting Actuarial Science In Insurance Pensions, Insurance, Leases Compound Interest Probability Mortality Data Modern-Day Insurance Eight Weeks to Eight Days Online Policies The Data Era Dynamic Risk Management Catastrophe Risk Open Access Modeling Opportunities Middle of Somewhere, 2012 (Continued) Chapter 4: Personalizing retail and fashion Karolina A Brief History of Retail Retail Eras Aristide Boucicaut The Shift The Data Era Stitch Fix Keaton Row Zara Karolina (Continued) Chapter 5: Transforming Customer relationships with data Buying a House Brief History of Customer Service Customer Service Over Time Boeing Financial Services The Data Era An Automobile Manufacturer Zendesk Buying a House (Continued) Chapter 6: Intelligent Machines Denmark Intelligent Machines Machine Data The Data Era General Electric Drones Tesla Networks of Data Denmark (Continued) Chapter 7: Government and Society Egypt, 2011 Social Media Intelligence Snowden Effect Privacy Risk Versus Reward Observation or Surveillance Development Targets Open Data Hackathons Open Access Ensuring Personal Protection Private Clouds Sanitizing Data Evidence-Based Policy Public-Private Partnerships Impact Bonds Social Impact Bond Development Impact Bonds The Role of Big Data Egypt, 2011 (Continued) Chapter 8: Corporate Sustainability City of London Global Megaforces Population Carbon Footprint Water Scarcity Environmental Risk BP and Exxon Mobile Early Warning Systems Social Media Risk and Resilience Measuring Sustainability Long-Term Decision Making Stranded Assets City of London (Continued) Chapter 9: Weather and Energy India, 2012 The Weather Forecasting the Weather When are Weather Forecasts Wrong? Chaos Ensemble Forecasts Communication Renewable Energy Solar, Hydro, and Wind Power Volatile or Intermittent Supply Energy Consumption Smart Meters Intelligent Demand-Side Management India, 2012 (Continued) Part II: Learning from Patterns in Big Data Chapter 10: Pattern Recognition Elements of Success Rhyme Pattern Recognition: A Gift or Trap? What Fish Teach us about Pattern Recognition Bayes’ Theorem Tsukiji Market Pattern Recognition Rochester Institute of Technology A Method for Recognizing Patterns Elements of Success Rhyme (Continued) Chapter 11: Why Patterns in Big Data Have Emerged Meatpacking District Business Models in the Data Era Data as a Competitive Advantage Data Improves Existing Products or Services Data as the Product Dun & Bradstreet CoStar IHS Meatpacking District (Continued) Chapter 12: Patterns in Big Data The Data Factor Summary of Big Data Patterns Redefining a Skilled Worker Creating and Utilizing New Sources of Data Building New Data Applications Transforming and Creating New Business Processes Data Collection for Competitive Advantage Exposing Opinion-Based Biases Real-Time Monitoring and Decision Making Social Networks Leveraging and Creating Data Deconstructing the Value Chain New Product Offerings Building for Customers Instead of Markets Tradeoff Between Privacy and Insight Changing the Definition of a Product Inverting the Search Paradigm for Data Discovery Data Security New Partnerships Founded on Data Shortening the Innovation Lifecycle Defining New Channels to Market New Economic Models Forecasting and Predicting Future Events Changing Incentives New Partnerships (Public/Private) Real-Time Monitoring and Decision Making (Early Warning Systems) A Framework for Big Data Patterns Part III: Leading the Revolution Chapter 13: The Data Opportunity What Oil Teaches Us About Data Bain Study Seizing the Opportunity Chapter 14: Porsche Rome Ferdinand Porsche The Birth of Porsche The Porsche Sports Car Porsche Today Rome (Continued) Chapter 15: Puma Herzogenaurach Advertising Wars Jochen Zeitz Environmental Profit and Loss Herzogenaurach (Continued) Chapter 16: Patterns in Big Data Introduction The Method Step 1: Understand Data Assets The Patterns Step 2: Explore Data Challenges Questions Hypotheses Data Models Statistical Significance Step 3: Design the Future The Patterns Step 4: Design a Data-Driven Business Model The Patterns Step 5: Transform Business Processes for the Data Era The Patterns Step 6: Design for Governance and Security The Patterns Step 7: Share Metrics and Incentives Chapter 17: Big Data Architecture Introduction Architect for the Future Lessons from Stuttgart Big Data Reference Architectures Leveraging Investments in Architecture Big Data Reference Architectures Business View Logical View Chapter 18: Business View Reference Architecture Introduction Men’s Trunk: a Retailer in the Data Era The Business View Reference Architecture Answer Fabric Data Virtualization Data Engines Management Data Governance User Interface, Applications, and Business Processes Summary Chapter 19: Logical View Reference Architecture Introduction Men’s Trunk: a Retailer in the Data Era (Continued) The Logical View Reference Architecture Data Ingest Analytics Discovery Landing Operational Warehouse Information Insight Operational Data Governance Men’s Trunk: a Retailer in the Data Era (Continued) Chapter 20: The Architecture of the Future Men’s Trunk: a Retailer in the Data Era (Continued) Men’s Trunk: Applying the Methodology Step 1: Understand Data Assets Step 2: Explore the Data Step 3: Design the Future Step 4: Design a Data-Driven Business Model Step 5: Transform Business Processes for the Data Era Step 6: Design for Governance and Security Step 7: Share Metrics and Incentives Men’s Trunk: the Business View Reference Architecture Answer Fabric Data Virtualization Data Engines Management Data Governance User Interface, Applications, and Business Processes Men’s Trunk: the Logical View Reference Architecture Approach Men’s Trunk: a Retailer in the Data Era (Continued) Epilogue The Time is Now Taking Action Fear not Usual Competitors The Future Table of Contents Begin Reading 10 11 12 13 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 69 70 71 72 73 74 75 76 77 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 131 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 ... half is working.” Transforming retail, however, is more than just using data to better target clients It’s about using data to transform the role of a retailer and truly serve a customer of one Chapter 5: Transforming Customer Relationships with Data Data will increase... Chapter 11: Why Patterns in Big Data Have Emerged Chapter 12: Patterns in Big Data 2 10 11 12 13 14 15 16 17 18 19 20 21 Meatpacking District Business Models in the Data Era Data as a Competitive Advantage Data Improves Existing Products or Services... The Patterns Step 4: Design a Data- Driven Business Model The Patterns Step 5: Transform Business Processes for the Data Era The Patterns Step 6: Design for Governance and Security The Patterns