TEAMFLY Team-Fly ® Page iii Data Mining Cookbook Modeling Data for Marketing, Risk, and Customer Relationship Management Olivia Parr Rud Page iv Publisher: Robert Ipsen Editor: Robert M. Elliott Assistant Editor: Emilie Herman Managing Editor: John Atkins Associate New Media Editor: Brian Snapp Text Design & Composition: Argosy Designations used by companies to distinguish their products are often claimed as trademarks. In all instances where John Wiley & Sons, Inc., is aware of a claim, the product names appear in initial capital or ALL CAPITAL LETTERS. Readers, however, should contact the appropriate companies for more complete information regarding trademarks and registration. Copyright © 2001 by Olivia Parr Rud. All rights reserved. Published by John Wiley & Sons, Inc. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per- copy fee to the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 750-4744. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 605 Third Avenue, New York, NY 10158-0012, (212) 850-6011, fax (212) 850-6008, E-Mail: PERMREQ @ WILEY.COM. This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold with the understanding that the publisher is not engaged in professional services. If professional advice or other expert assistance is required, the services of a competent professional person should be sought. This title is also available in print as 0-471-38564-6 For more information about Wiley product, visit our web site at www.Wiley.com. Page v What People Are Saying about Olivia Parr Rud's Data Mining Cookbook In the Data Mining Cookbook, industry expert Olivia Parr Rud has done the impossible: She has made a very complex process easy for the novice to understand. In a step-by-step process, in plain English, Olivia tells us how we can benefit from modeling, and how to go about it. It's like an advanced graduate course boiled down to a very friendly, one-on-one conversation. The industry has long needed such a useful book. Arthur Middleton Hughes Vice President for Strategic Planning, M\S Database Marketing This book provides extraordinary organization to modeling customer behavior. Olivia Parr Rud has made the subject usable, practical, and fun. . . . Data Mining Cookbook is an essential resource for companies aspiring to the best strategy for success— customer intimacy. William McKnight President, McKnight Associates, Inc . In today's digital environment, data flows at us as though through a fire hose. Olivia Parr Rud's Data Mining Cookbook satisfies the thirst for a user-friendly "cookbook" on data mining targeted at analysts and modelers responsible for serving up insightful analyses and reliable models. Data Mining Cookbook includes all the ingredients to make it a valuable resource for the neophyte as well as the experienced modeler. Data Mining Cookbook starts with the basic ingredients, like the rudiments of data analysis, to ensure that the beginner can make sound interpretations of moderate- sized data sets. She finishes up with a closer look at the more complex statistical and artificial intelligence methods (with reduced emphasis on mathematical equations and jargon, and without computational formulas), which gives the advanced modeler an edge in developing the best possible models. Bruce Ratner Founder and President, DMStat1 Page vii To Betty for her strength and drive. To Don for his intellect. Page ix CONTENTS Acknowledgments xv Foreword xvii Introduction xix About the Author xxiii About the Contributors xxv Part One: Planning the Menu 1 Chapter 1: Setting the Objective 3 Defining the Goal 4 Profile Analysis 7 Segmentation 8 Response 8 Risk 9 Activation 10 Cross-Sell and Up-Sell 10 Attrition 10 Net Present Value 11 Lifetime Value 11 Choosing the Modeling Methodology 12 Linear Regression 12 Logistic Regression 15 Neural Networks 16 Genetic Algorithms 17 Classification Trees 19 The Adaptive Company 20 Hiring and Teamwork 21 Product Focus versus Customer Focus 22 Summary 23 Chapter 2: Selecting the Data Sources 25 Types of Data 26 Sources of Data 27 Internal Sources 27 External Sources 36 Selecting Data for Modeling 36 Data for Prospecting 37 Data for Customer Models 40 Data for Risk Models 42 Constructing the Modeling Data Set 44 How big should my sample be? 44 Page x Sampling Methods 45 Developing Models from Modeled Data 47 Combining Data from Multiple Offers 47 Summary 48 Part Two: The Cooking Demonstration 49 Chapter 3: Preparing the Data for Modeling 51 Accessing the Data 51 Classifying Data 54 Reading Raw Data 55 Creating the Modeling Data Set 57 Sampling 58 Cleaning the Data 60 Continuous Variables 60 Categorical Variables 69 Summary 70 Chapter 4: Selecting and Transforming the Variables 71 Defining the Objective Function 71 Probability of Activation 72 Risk Index 73 Product Profitability 73 Marketing Expense 74 Deriving Variables 74 Summarization 74 Ratios 75 Dates 75 Variable Reduction 76 Continuous Variables 76 Categorical Variables 80 Developing Linear Predictors 85 Continuous Variables 85 Categorical Variables 95 Interactions Detection 98 Summary 99 Chapter 5: Processing and Evaluating the Model 101 Processing the Model 102 Splitting the Data 103 Method 1: One Model 108 Method 2: Two Models— Response 119 Page xi Method 2: Two Models — Activation 119 Comparing Method 1 and Method 2 121 Summary 124 Chapter 6: Validating the Model 125 Gains Tables and Charts 125 Method 1: One Model 126 Method 2: Two Models 127 Scoring Alternate Data Sets 130 Resampling 134 Jackknifing 134 Bootstrapping 138 Decile Analysis on Key Variables 146 Summary 150 Chapter 7: Implementing and Maintaining the Model 151 Scoring a New File 151 Scoring In-house 152 Outside Scoring and Auditing 155 Implementing the Model 161 Calculating the Financials 161 Determining the File Cut-off 166 Champion versus Challenger 166 The Two-Model Matrix 167 Model Tracking 170 Back-end Validation 176 Model Maintenance 177 [...]... the field At Data Miners, the analytic marketing consultancy I founded in 1997, we firmly believe that data mining projects succeed or fail on the basis of the quality of the data mining process and the suitability of the data used for mining The choice of particular data mining techniques, algorithms, and software is of far less importance It follows that the most important part of a data mining project... great models in no time MICHAEL J A BERRY FOUNDER, DATA MINERS, INC CO -AUTHOR, DATA MINING TECHNIQUES AND MASTERING DATA MINING Page xix INTRODUCTION What is data mining? Data mining is a term that covers a broad range of techniques being used in a variety of industries Due to increased competition for profits and market share in the marketing arena, data mining has become an essential practice for maintaining... on data mining, database design, predictive modeling, Web modeling and marketing strategies Data Square is a premier database marketing consulting firm offering business intelligence solutions through the use of cutting-edge analytic services, database design and management, and e-business integration As part of the total solution, Data Square offers Web-enabled data warehousing, data marting, data mining, ... of data mining began gaining popularity in the marketing arena in the late 1980s and early 1990s A few cutting edge credit card banks saw a form of data mining, known as data modeling, as a way to enhance acquisition efforts and improve risk management The high volume of activity and unprecedented growth provided a fertile ground for data modeling to flourish The successful and profitable use of data. .. familiarity with how to apply these tools in a data mining context in order to support database marketing and customer relationship management goals If you are a statistician or marketing analyst who has been called upon to implement data mining models to increase response rates, increase profitability, increase customer loyalty or reduce risk through data mining, this book will have you cooking up great... for data mining increases Researchers develop new methods, software manufacturers automate existing methods, and talented analysts continue to push the envelope with standard techniques Data mining and, more specifically, data modeling, is becoming a strategic necessity for companies to maintain profitability My desire for this book serves as a handy reference and a seasoned guide as you pursue your data. .. customer lifecycle Historically, one form of data mining was also known as ' 'data dredging." This was considered beneath the standards of a good researcher It implied that a researcher might actually search through data without any specific predetermined hypothesis Recently, however, this practice has become much more acceptable, mainly because this form of data mining has led to the discovery of valuable... CEO of MS Database Marketing, Inc., a technology-driven database marketing company focused on maximizing the value of Page xxviii their clients' databases MS delivers CRM and prospect targeting solutions that are implemented via the Web and through traditional direct marketing programs Their core competency is in delivering database solutions to marketing and sales organizations by mining data to identify... efficiency of the next campaign Each project may have different data requirements or may utilize different analytic methods, or both We begin our culinary data journey with a discussion of the building blocks necessary for effective data modeling In chapter 1, I introduce the steps for building effective data models I also provide a review of common data mining techniques used for marketing risk and customer... Finally, thanks to Brandon, Adam, Vanessa, and Dean for tolerating my unavailability for the last year Page xvii FOREWORD I am a data miner by vocation and home chef by avocation, so I was naturally intrigued when I heard about Olivia Parr Rud's Data Mining Cookbook What sort of cookbook would it be, I wondered? My own extensive and eclectic cookery collection is comprised of many different styles It includes . information about Wiley product, visit our web site at www.Wiley.com. Page v What People Are Saying about Olivia Parr Rud& apos;s Data Mining Cookbook In the Data Mining Cookbook, industry expert. today's digital environment, data flows at us as though through a fire hose. Olivia Parr Rud& apos;s Data Mining Cookbook satisfies the thirst for a user-friendly " ;cookbook& quot; on. as mushrooms or tofu. There are books devoted to the cuisine of a particular country or region; books devoted to particular cooking methods like steaming or barbecue; books that comply with