38mm Cover design: Gary Thompson PATTERN RECOGNITION Third Edition Third Edition Andrew R. Webb Keith D. Copsey PATTERN RECOGNITION Third Edition STATISTICAL Webb Copsey RECOGNITION STATISTICAL PATTERN Andrew R. Webb and Keith D. Copsey Mathematics and Data Analysis Consultancy, Malvern, UK Statistical pattern recognition relates to the use of statistical techniques for analysing data measurements in order to extract information and make justified decisions. It is a very active area of study and research, which has seen many advances in recent years. Applications such as data mining, web searching, multimedia data retrieval, face recognition, and cursive handwriting recognition, all require robust and efficient pattern recognition techniques. This third edition provides an introduction to statistical pattern theory and techniques, with material drawn from a wide range of fields, including the areas of engineering, statistics, computer science and the social sciences. The book has been updated to cover new methods and applications, and includes a wide range of techniques such as Bayesian methods, neural networks, support vector machines, feature selection and feature reduction techniques. Technical descriptions and motivations are provided, and the techniques are illustrated using real examples. Statistical Pattern Recognition, Third Edition: • Provides a self-contained introduction to statistical pattern recognition. • Includes new material presenting the analysis of complex networks. • Introduces readers to methods for Bayesian density estimation. • Presents descriptions of new applications in biometrics, security, finance and condition monitoring. • Provides descriptions and guidance for implementing techniques, which will be invaluable to software engineers and developers seeking to develop real applications. • Describes mathematically the range of statistical pattern recognition techniques. • Presents a variety of exercises including more extensive computer projects. The in-depth technical descriptions make this book suitable for senior undergraduate and graduate students in statistics, computer science and engineering. Statistical Pattern Recognition is also an excellent reference source for technical professionals. Chapters have been arranged to facilitate implementation of the techniques by software engineers and developers in non-statistical engineering fields. www.wiley.com/go/statistical_pattern_recognition STATISTICAL RED BOX RULES ARE FOR PROOF STAGE ONLY. DELETE BEFORE FINAL PRINTING. www.it-ebooks.info P1: OTA/XYZ P2: ABC JWST102-fm JWST102-Webb September 8, 2011 8:52 Printer Name: Yet to Come www.it-ebooks.info P1: OTA/XYZ P2: ABC JWST102-fm JWST102-Webb September 8, 2011 8:52 Printer Name: Yet to Come Statistical Pattern Recognition www.it-ebooks.info P1: OTA/XYZ P2: ABC JWST102-fm JWST102-Webb September 8, 2011 8:52 Printer Name: Yet to Come www.it-ebooks.info P1: OTA/XYZ P2: ABC JWST102-fm JWST102-Webb September 8, 2011 8:52 Printer Name: Yet to Come Statistical Pattern Recognition Third Edition Andrew R. Webb • Keith D. Copsey Mathematics and Data Analysis Consultancy, Malvern, UK A John Wiley & Sons, Ltd., Publicatio n www.it-ebooks.info P1: OTA/XYZ P2: ABC JWST102-fm JWST102-Webb September 8, 2011 8:52 Printer Name: Yet to Come This edition first published 2011 © 2011 John Wiley & Sons, Ltd Registered office John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, United Kingdom For details of our global editorial offices, for customer services and for information about how to apply for permission to reuse the copyright material in this book please see our website at www.wiley.com. The right of the author to be identified as the author of this work has been asserted in accordance with the Copyright, Designs and Patents Act 1988. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by the UK Copyright, Designs and Patents Act 1988, without the prior permission of the publisher. Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books. Designations used by companies to distinguish their products are often claimed as trademarks. All brand names and product names used in this book are trade names, service marks, trademarks or registered trademarks of their respective owners. The publisher is not associated with any product or vendor mentioned in this book. This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold on the understanding that the publisher is not engaged in rendering professional services. If professional advice or other expert assistance is required, the services of a competent professional should be sought. Library of Congress Cataloging-in-Publication Data Webb, A. R. (Andrew R.) Statistical pattern recognition / Andrew R. Webb, Keith D. Copsey. – 3rd ed. p. cm. Includes bibliographical references and index. ISBN 978-0-470-68227-2 (hardback) – ISBN 978-0-470-68228-9 (paper) 1. Pattern perception–Statistical methods. I. Copsey, Keith D. II. Title. Q327.W43 2011 006.4–dc23 2011024957 A catalogue record for this book is available from the British Library. HB ISBN: 978-0-470-68227-2 PB ISBN: 978-0-470-68228-9 ePDF ISBN: 978-1-119-95296-1 oBook ISBN: 978-1-119-95295-4 ePub ISBN: 978-1-119-96140-6 Mobi ISBN: 978-1-119-96141-3 Typeset in 10/12pt Times by Aptara Inc., New Delhi, India www.it-ebooks.info P1: OTA/XYZ P2: ABC JWST102-fm JWST102-Webb September 8, 2011 8:52 Printer Name: Yet to Come To Rosemary, Samuel, Miriam, Jacob and Ethan www.it-ebooks.info P1: OTA/XYZ P2: ABC JWST102-fm JWST102-Webb September 8, 2011 8:52 Printer Name: Yet to Come www.it-ebooks.info P1: OTA/XYZ P2: ABC JWST102-fm JWST102-Webb September 8, 2011 8:52 Printer Name: Yet to Come Contents Preface xix Notation xxiii 1 Introduction to Statistical Pattern Recognition 1 1.1 Statistical Pattern Recognition 1 1.1.1 Introduction 1 1.1.2 The Basic Model 2 1.2 Stages in a Pattern Recognition Problem 4 1.3 Issues 6 1.4 Approaches to Statistical Pattern Recognition 7 1.5 Elementary Decision Theory 8 1.5.1 Bayes’ Decision Rule for Minimum Error 8 1.5.2 Bayes’ Decision Rule for Minimum Error – Reject Option 12 1.5.3 Bayes’ Decision Rule for Minimum Risk 13 1.5.4 Bayes’ Decision Rule for Minimum Risk – Reject Option 15 1.5.5 Neyman–Pearson Decision Rule 15 1.5.6 Minimax Criterion 18 1.5.7 Discussion 19 1.6 Discriminant Functions 20 1.6.1 Introduction 20 1.6.2 Linear Discriminant Functions 21 1.6.3 Piecewise Linear Discriminant Functions 23 1.6.4 Generalised Linear Discriminant Function 24 1.6.5 Summary 26 1.7 Multiple Regression 27 1.8 Outline of Book 29 1.9 Notes and References 29 Exercises 31 2 Density Estimation – Parametric 33 2.1 Introduction 33 www.it-ebooks.info P1: OTA/XYZ P2: ABC JWST102-fm JWST102-Webb September 8, 2011 8:52 Printer Name: Yet to Come viii CONTENTS 2.2 Estimating the Parameters of the Distributions 34 2.2.1 Estimative Approach 34 2.2.2 Predictive Approach 35 2.3 The Gaussian Classifier 35 2.3.1 Specification 35 2.3.2 Derivation of the Gaussian Classifier Plug-In Estimates 37 2.3.3 Example Application Study 39 2.4 Dealing with Singularities in the Gaussian Classifier 40 2.4.1 Introduction 40 2.4.2 Na ¨ ıve Bayes 40 2.4.3 Projection onto a Subspace 41 2.4.4 Linear Discriminant Function 41 2.4.5 Regularised Discriminant Analysis 42 2.4.6 Example Application Study 44 2.4.7 Further Developments 45 2.4.8 Summary 46 2.5 Finite Mixture Models 46 2.5.1 Introduction 46 2.5.2 Mixture Models for Discrimination 48 2.5.3 Parameter Estimation for Normal Mixture Models 49 2.5.4 Normal Mixture Model Covariance Matrix Constraints 51 2.5.5 How Many Components? 52 2.5.6 Maximum Likelihood Estimation via EM 55 2.5.7 Example Application Study 60 2.5.8 Further Developments 62 2.5.9 Summary 63 2.6 Application Studies 63 2.7 Summary and Discussion 66 2.8 Recommendations 66 2.9 Notes and References 67 Exercises 67 3 Density Estimation – Bayesian 70 3.1 Introduction 70 3.1.1 Basics 72 3.1.2 Recursive Calculation 72 3.1.3 Proportionality 73 3.2 Analytic Solutions 73 3.2.1 Conjugate Priors 73 3.2.2 Estimating the Mean of a Normal Distribution with Known Variance 75 3.2.3 Estimating the Mean and the Covariance Matrix of a Multivariate Normal Distribution 79 3.2.4 Unknown Prior Class Probabilities 85 3.2.5 Summary 87 3.3 Bayesian Sampling Schemes 87 3.3.1 Introduction 87 www.it-ebooks.info [...]... the development of statistical pattern recognition methodology and its application to practical sensor data analysis problems The book is aimed at advanced undergraduate and graduate courses Some of the material has been presented as part of a graduate course on pattern recognition and at pattern recognition summer schools It is also designed for practitioners in the field of pattern recognition as well... this book is on pattern recognition procedures, providing a description of basic techniques Statistical Pattern Recognition, Third Edition Andrew R Webb and Keith D Copsey © 2011 John Wiley & Sons, Ltd Published 2011 by John Wiley & Sons, Ltd www.it-ebooks.info P1: OTA/XYZ JWST102-c01 P2: ABC JWST102-Webb 2 August 26, 2011 15:51 Printer Name: Yet to Come INTRODUCTION TO STATISTICAL PATTERN RECOGNITION. .. edition) is available on the book’s website Scope The book presents most of the popular methods of statistical pattern recognition However, many of the important developments in pattern recognition are not confined to the statistics literature and have occurred where the area overlaps with research in machine learning Therefore, where we have felt that straying beyond the traditional boundaries of statistical. .. character recognition Therefore, we see that the term pattern , in its technical meaning, does not necessarily refer to structure within images www.it-ebooks.info P1: OTA/XYZ JWST102-c01 P2: ABC JWST102-Webb August 26, 2011 15:51 Printer Name: Yet to Come STATISTICAL PATTERN RECOGNITION 3 Figure 1.1 Pattern classifier The main topic in this book may be described by a number of terms including pattern. .. indicator function, I(θ ) = 1 if θ = true else 0 www.it-ebooks.info P1: OTA/XYZ JWST102-c01 P2: ABC JWST102-Webb August 26, 2011 15:51 Printer Name: Yet to Come 1 Introduction to statistical pattern recognition Statistical pattern recognition is a term used to cover all stages of an investigation from problem formulation and data collection through to discrimination and classification, assessment of results... intensive methods and less on a statistical approach, but there is strong overlap between the research areas of statistical pattern recognition and machine learning 1.1.2 The basic model Since many of the techniques we shall describe have been developed over a range of diverse disciplines, there is naturally a variety of sometimes contradictory terminology We shall use the term pattern to denote the p-dimensional... example, ‘transactions’ between individuals – email traffic, purchases) Understanding these datasets requires additional tools in the pattern recognition toolbox Therefore, we also examine developments such as methods for analysing data that may be represented as a graph Pattern recognition as a field of study developed significantly in the 1960s It was very much an interdisciplinary subject Some people entered... introduction to statistical pattern recognition theory and techniques Most of the material presented in this book is concerned with discrimination and classification and has been drawn from a wide range of literature including that of engineering, statistics, computer science and the social sciences The aim of the book is to provide descriptions of many of the most useful of today’s pattern processing... assume that there exist C groups or classes, denoted ω1 , , ωC and associated with each pattern x is a categorical variable z that denotes the class or group membership; that is, if z = i, then the pattern belongs to ωi , i ∈ {1, , C} Examples of patterns are measurements of an acoustic waveform in a speech recognition problem; measurements on a patient made in order to identify a disease (diagnosis);... important diverse topics including model selection Book website The website www.wiley.com/go /statistical_ pattern_ recognition contains supplementary material on topics including measures of dissimilarity, estimation, linear algebra, data analysis and basic probability Acknowledgements In preparing the third edition of this book we have been helped by many people We are especially grateful to Dr Gavin . Gary Thompson PATTERN RECOGNITION Third Edition Third Edition Andrew R. Webb Keith D. Copsey PATTERN RECOGNITION Third Edition STATISTICAL Webb Copsey RECOGNITION STATISTICAL PATTERN Andrew. xix Notation xxiii 1 Introduction to Statistical Pattern Recognition 1 1.1 Statistical Pattern Recognition 1 1.1.1 Introduction 1 1.1.2 The Basic Model 2 1.2 Stages in a Pattern Recognition Problem 4 1.3. are illustrated using real examples. Statistical Pattern Recognition, Third Edition: • Provides a self-contained introduction to statistical pattern recognition. • Includes new material