Undergraduate Topics in Computer Science Undergraduate Topics in Computer Science (UTiCS) delivers high-quality instructional content for undergraduates studying in all areas of computing and information science From core foundational and theoretical material to final-year topics and applications, UTiCS books take a fresh, concise, and modern approach and are ideal for self-study or for a one- or two-semester course The texts are all authored by established experts in their fields, reviewed by an international advisory board, and contain numerous examples and problems Many include fully worked solutions Also in this series Iain Craig Object-Oriented Programming Languages: Interpretation 978-1-84628-773-2 Hanne Riis Nielson and Flemming Nielson Semantics with Applications: An Appetizer 978-1-84628-691-9 Max Bramer Principles of Data Mining Max Bramer, BSc, PhD, CEng, FBCS, FIEE, FRSA Digital Professor of Information Technology, University of Portsmouth, UK Series editor Ian Mackie École Polytechnique, France and King’s College London, UK Advisory board Samson Abramsky, University of Oxford, UK Chris Hankin, Imperial College London, UK Dexter Kozen, Cornell University, USA Andrew Pitts, University of Cambridge, UK Hanne Riis Nielson, Technical University of Denmark, Denmark Steven Skiena, Stony Brook University, USA Iain Stewart, University of Durham, UK David Zhang, The Hong Kong Polytechnic University, Hong Kong British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library Library of Congress Control Number: 2007922358 Undergraduate Topics in Computer Science ISSN 1863-7310 ISBN-10: 1-84628-765-0 e-ISBN 10: 1-84628-766-9 ISBN-13: 978-1-84628-765-7 e-ISBN-13: 978-1-84628-766-4 Printed on acid-free paper © Springer-Verlag London Limited 2007 Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms of licences issued by the Copyright Licensing Agency Enquiries concerning reproduction outside those terms should be sent to the publishers The use of registered names, trademarks, etc in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant laws and regulations and therefore free for general use The publisher makes no representation, express or implied, with regard to the accuracy of the information contained in this book and cannot accept any legal responsibility or liability for any errors or omissions that may be made 987654321 Springer Science+Business Media springer.com Contents Introduction to Data Mining 1 Data for Data Mining 1.1 Standard Formulation 1.2 Types of Variable 1.2.1 Categorical and Continuous Attributes 1.3 Data Preparation 1.3.1 Data Cleaning 1.4 Missing Values 1.4.1 Discard Instances 1.4.2 Replace by Most Frequent/Average Value 1.5 Reducing the Number of Attributes 1.6 The UCI Repository of Datasets Chapter Summary Self-assessment Exercises for Chapter 11 11 12 14 14 15 17 17 17 18 19 20 20 Introduction to Classication: Naăve Bayes and Nearest Neighbour 2.1 What is Classification? 2.2 Naăve Bayes Classiers 2.3 Nearest Neighbour Classification 2.3.1 Distance Measures 2.3.2 Normalisation 2.3.3 Dealing with Categorical Attributes 2.4 Eager and Lazy Learning Chapter Summary 23 23 24 31 34 37 38 38 39 vi Principles of Data Mining Self-assessment Exercises for Chapter 39 Using Decision Trees for Classification 3.1 Decision Rules and Decision Trees 3.1.1 Decision Trees: The Golf Example 3.1.2 Terminology 3.1.3 The degrees Dataset 3.2 The TDIDT Algorithm 3.3 Types of Reasoning Chapter Summary Self-assessment Exercises for Chapter 41 41 42 43 44 47 49 50 50 Decision Tree Induction: Using Entropy for Attribute Selection 4.1 Attribute Selection: An Experiment 4.2 Alternative Decision Trees 4.2.1 The Football/Netball Example 4.2.2 The anonymous Dataset 4.3 Choosing Attributes to Split On: Using Entropy 4.3.1 The lens24 Dataset 4.3.2 Entropy 4.3.3 Using Entropy for Attribute Selection 4.3.4 Maximising Information Gain Chapter Summary Self-assessment Exercises for Chapter 51 51 52 53 55 56 57 59 60 62 63 63 Decision Tree Induction: Using Frequency Tables for Attribute Selection 5.1 Calculating Entropy in Practice 5.1.1 Proof of Equivalence 5.1.2 A Note on Zeros 5.2 Other Attribute Selection Criteria: Gini Index of Diversity 5.3 Inductive Bias 5.4 Using Gain Ratio for Attribute Selection 5.4.1 Properties of Split Information 5.5 Number of Rules Generated by Different Attribute Selection Criteria 5.6 Missing Branches Chapter Summary Self-assessment Exercises for Chapter 65 65 66 68 68 70 72 73 74 75 76 77 Contents vii Estimating the Predictive Accuracy of a Classifier 6.1 Introduction 6.2 Method 1: Separate Training and Test Sets 6.2.1 Standard Error 6.2.2 Repeated Train and Test 6.3 Method 2: k-fold Cross-validation 6.4 Method 3: N -fold Cross-validation 6.5 Experimental Results I 6.6 Experimental Results II: Datasets with Missing Values 6.6.1 Strategy 1: Discard Instances 6.6.2 Strategy 2: Replace by Most Frequent/Average Value 6.6.3 Missing Classifications 6.7 Confusion Matrix 6.7.1 True and False Positives Chapter Summary Self-assessment Exercises for Chapter 79 79 80 81 82 82 83 84 86 87 87 89 89 90 91 91 Continuous Attributes 93 7.1 Introduction 93 7.2 Local versus Global Discretisation 95 7.3 Adding Local Discretisation to TDIDT 96 7.3.1 Calculating the Information Gain of a Set of Pseudoattributes 97 7.3.2 Computational Efficiency 102 7.4 Using the ChiMerge Algorithm for Global Discretisation 105 7.4.1 Calculating the Expected Values and χ2 108 7.4.2 Finding the Threshold Value 113 7.4.3 Setting minIntervals and maxIntervals 113 7.4.4 The ChiMerge Algorithm: Summary 115 7.4.5 The ChiMerge Algorithm: Comments 115 7.5 Comparing Global and Local Discretisation for Tree Induction 116 Chapter Summary 118 Self-assessment Exercises for Chapter 118 Avoiding Overfitting of Decision Trees 119 8.1 Dealing with Clashes in a Training Set 120 8.1.1 Adapting TDIDT to Deal With Clashes 120 8.2 More About Overfitting Rules to Data 125 8.3 Pre-pruning Decision Trees 126 8.4 Post-pruning Decision Trees 128 Chapter Summary 133 Self-assessment Exercise for Chapter 134 viii Principles of Data Mining More About Entropy 135 9.1 Introduction 135 9.2 Coding Information Using Bits 138 9.3 Discriminating Amongst M Values (M Not a Power of 2) 140 9.4 Encoding Values That Are Not Equally Likely 141 9.5 Entropy of a Training Set 144 9.6 Information Gain Must be Positive or Zero 145 9.7 Using Information Gain for Feature Reduction for Classification Tasks 147 9.7.1 Example 1: The genetics Dataset 148 9.7.2 Example 2: The bcst96 Dataset 152 Chapter Summary 154 Self-assessment Exercises for Chapter 154 10 Inducing Modular Rules for Classification 155 10.1 Rule Post-pruning 155 10.2 Conflict Resolution 157 10.3 Problems with Decision Trees 160 10.4 The Prism Algorithm 162 10.4.1 Changes to the Basic Prism Algorithm 169 10.4.2 Comparing Prism with TDIDT 170 Chapter Summary 171 Self-assessment Exercise for Chapter 10 171 11 Measuring the Performance of a Classifier 173 11.1 True and False Positives and Negatives 174 11.2 Performance Measures 176 11.3 True and False Positive Rates versus Predictive Accuracy 179 11.4 ROC Graphs 180 11.5 ROC Curves 182 11.6 Finding the Best Classifier 183 Chapter Summary 184 Self-assessment Exercise for Chapter 11 185 12 Association Rule Mining I 187 12.1 Introduction 187 12.2 Measures of Rule Interestingness 189 12.2.1 The Piatetsky-Shapiro Criteria and the RI Measure 191 12.2.2 Rule Interestingness Measures Applied to the chess Dataset 193 Contents ix 12.2.3 Using Rule Interestingness Measures for Conflict Resolution 195 12.3 Association Rule Mining Tasks 195 12.4 Finding the Best N Rules 196 12.4.1 The J-Measure: Measuring the Information Content of a Rule 197 12.4.2 Search Strategy 198 Chapter Summary 201 Self-assessment Exercises for Chapter 12 201 13 Association Rule Mining II 203 13.1 Introduction 203 13.2 Transactions and Itemsets 204 13.3 Support for an Itemset 205 13.4 Association Rules 206 13.5 Generating Association Rules 208 13.6 Apriori 209 13.7 Generating Supported Itemsets: An Example 212 13.8 Generating Rules for a Supported Itemset 214 13.9 Rule Interestingness Measures: Lift and Leverage 216 Chapter Summary 218 Self-assessment Exercises for Chapter 13 219 14 Clustering 221 14.1 Introduction 221 14.2 k-Means Clustering 224 14.2.1 Example 225 14.2.2 Finding the Best Set of Clusters 230 14.3 Agglomerative Hierarchical Clustering 231 14.3.1 Recording the Distance Between Clusters 233 14.3.2 Terminating the Clustering Process 236 Chapter Summary 237 Self-assessment Exercises for Chapter 14 238 15 Text Mining 239 15.1 Multiple Classifications 239 15.2 Representing Text Documents for Data Mining 240 15.3 Stop Words and Stemming 242 15.4 Using Information Gain for Feature Reduction 243 15.5 Representing Text Documents: Constructing a Vector Space Model 243 15.6 Normalising the Weights 245 ... types of data Labelled and Unlabelled Data In general we have a dataset of examples (called instances), each of which comprises the values of a number of variables, which in data mining are often... algorithms of the data mining stage of knowledge discovery will be its prime concern Applications of Data Mining There is a rapidly growing body of successful applications in a wide range of areas... version remains with me Max Bramer Digital Professor of Information Technology University of Portsmouth, UK January 2007 Data for Data Mining Data for data mining comes in many forms: from computer