Machine Learning A Constraint-Based Approach Machine Learning A Constraint-Based Approach Marco Gori Università di Siena Morgan Kaufmann is an imprint of Elsevier 50 Hampshire Street, 5th Floor, Cambridge, MA 02139, United States Copyright © 2018 Elsevier Ltd All rights reserved No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or any information storage and retrieval system, without permission in writing from the publisher Details on how to seek permission, further information about the Publisher’s permissions policies and our arrangements with organizations such as the Copyright Clearance Center and the Copyright Licensing Agency, can be found at our website: www.elsevier.com/permissions This book and the individual contributions contained in it are protected under copyright by the Publisher (other than as may be noted herein) Notices Knowledge and best practice in this field are constantly changing As new research and experience broaden our understanding, changes in research methods, professional practices, or medical treatment may become necessary Practitioners and researchers must always rely on their own experience and knowledge in evaluating and using any information, methods, compounds, or experiments described herein In using such information or methods they should be mindful of their own safety and the safety of others, including parties for whom they have a professional responsibility To the fullest extent of the law, neither the Publisher nor the authors, contributors, or editors, assume any liability for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions, or ideas contained in the material herein Library of Congress Cataloging-in-Publication Data A catalog record for this book is available from the Library of Congress British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library ISBN: 978-0-08-100659-7 For information on all Morgan Kaufmann publications visit our website at https://www.elsevier.com/books-and-journals Publisher: Katey Birtcher Acquisition Editor: Steve Merken Editorial Project Manager: Peter Jardim Production Project Manager: Punithavathy Govindaradjane Designer: Miles Hitchen Typeset by VTeX To the Memory of My Father who provided me with enough examples to appreciate the importance of hard work to achieve goals and to disclose the beauty of knowledge Contents Preface Notes on the Exercises • • • • xiii xix CHAPTER The Big Picture 1.1 Why Do Machines Need to Learn? 1.1.1 Learning Tasks J.1.2 Symbolic and SubsymboJic Representations of the 1.1.3 Biological and Artificial Neural Networks 1.1.4 Protocols of Learning Env iron ment 1.2 II 13 Principles and Practice 28 1.2.1 The Puzz ling Nature of Induction 28 1.2.2 Lea rn ing Principles 34 1.2.3 The Role of Time in Learning Processes 34 1.2.4 Foclls of Attention 35 on Experien ce 38 1.3.1 Me asuring the Sliccess of Experiments 39 40 · ].3.2 Handwritten Character Recognition 1.3.3 Setting up a Machine Learning Exper m e nt 1.3.4 Test and Experimental Remarks 42 i Learning to See Speech Understanding ].4.3 Agents Living in Their Own Environment 50 1.4.1 45 in Machine Learning 1.4.2 1.5 Scholia • 52 54 61 61 60 2.1.2 Ill-Position of Constraint·lnduced Risk Functions 2.1.3 Risk Minimization 2.1.4 The Bias-Variance Dilemma 2.2 Stati stical Learnjng 2.2.1 M_ax.imum LikeUhood ESlimalion 2.2.2 Bayesian Inference , , 69 75 83 , , 83 • 86 • 88 89 92 Bayesian Learnlng 2.2.4 Graphical Modes 2.2.5 Frequentist and Bayesian Approach Information-Based Learning A Motivating Example , 71 2.2.3 2.3 50 51 Learning Principles Environmental Constraints 2.1.1 Loss and Risk Functions 2.3 19 1.4 Chall enges 2.1 1.1.5 Constraint-Based Lea rn ing 1.3 Hands CHAPTER , • 95 95 VII � -viii Contents 2.3.2 Principle of MaximumEntropy 2.3.3 MaximumMutualInformation 2.4 LearningUnder 3.1 3.2 3.3 • • 97 theParsimony Principle 104 104 2.4.2 Minimum Description Length, , , , , , , , , , , MDLandRegulai r zation 2.4.4 Statislicallnterpretation of Regularization 113 • • Linear Threshold Machines 3.1.1 NormalEquations 3.1.2 Undetermined Problems andPseudoinversion 3.1.3 RidgeRegression 3.1.4 Primal and DualRepresentations 123 128 129 132 134 L inearMachinesWith Threshold Units 141 3.2.1 3.2.2 OptimalityforL inearlyS - eparableExamples 3.2.3 Failing Predicate-Orderand RepresentationalIssues StatisticalView toSeparate 149 142 lSI 155 BayesianDecislon and LinearDiscrimination 155 3.3.2 LogisticRegression 3.3.3 The Parsimony PrincipleMeetsthe Bayesian Decision 3.3.4 LMS in theStatisticalFramework 158 Algorithmic[ssues 156 159 162 3.4.1 GradientDescent 3.4.2 StochasticGradientDescent 3.4.3 ThePerceptron Algorithm 165 3.4.4 Complexity Tssues t69 162 164 t75 Kernel Machines 186 FeatureSpace 187 Polynomial Preproces s ing 187 4.1.2 BooleanEni r chment 188 4.1.3 InvariantFeatllreMaps 189 4.1.4 Linear-Separability 190 inHigh-Dimensional Spaces MaximumMarginProblem , " 4.2.1 4.3 I IS 122 L inearMachines 4.1.1 4.2 110 3.5 Scholi 4.1 104 2.4.3 CHAPTER 99 TheParsimony Principle 3.3 ] 3.4 2.4 2.5 Scholia CHAPTER 194 ClassificationUnderLinear·Separability 194 4.2.2 Dealil l gWith Soft·Constraints 4.2.3 Regression KernelFunctions 4.3.1 Similarity andKernelTrick 4.3.2 Characterization of Kernels 198 201 • • 207 • • 207 • • 208 � Contents 4.3.3 The Reproducing Kernel Map 212 4.3.4 Types of Kernels 14 4.4 Regularization 220 4.4.1 Regularized Risks 220 4.4.2 Regularization in RKHS 222 4.4.3 Minimization of Regularized Risks 223 4.4.4 Regularization Operators 22 4.5 Scholia 230 CHAPTER Deep Architectures 5.1 Architectural Issues 5.2 236 237 5.1.1 Digraphs and Feedforward Networks 5.1 Deep Paths 5.1.3 From Deep to Relaxation-Based Architectures 5.1.4 Classifiers, Regressors, and Auto-Encoders 238 240 24 244 Realization of Boolean Functions 247 5.2.1 Canonical Realizations by and-or Gates 247 5.2.2 Universal na nd Realization 5.2.3 Shallow vs Deep Realizations 251 5.2.4 LTU-Based Realizations and Complexity Issues 5.3 Realization of Real-Valued Functions 5.3.1 Computational Geometry-Based Realizations 5.3.2 Universal Approximation 5.3.3 Solution Space and Separation Surfaces 5.3.4 Deep Networks and Representational Issues 5.4 Convolutional Networks 251 254 26 26 268 271 276 280 4.1 Kernels, Convolutions, and Receptive Fields 280 5.4.2 Incorporating Invariance 28 5.4.3 Deep Convolutional Networks 5.5 Learning in Feedforward Networks 5.5.1 Supervised Learning 5.5.2 Backpropagation 293 298 298 298 5.5.3 Symbolic and Automatic Differentiation 306 5.5.4 Regularization Issues 308 5.6 Complexity Issues 5.6.1 On the Problem of Loeal Minima 5.6.2 Facing Saturation 5.6.3 Complexity and Numerical Issues 5.7 Scholia CHAPTER Learning ami Reasoning With Constraints 6.1 Constraint Machines 13 313 319 323 326 340 43 6.1.1 Walking Throllgh Learning and Inference 43 6.1.2 A Unified View of Constrained Environments 352 ix � - x Contents 6.2 6.3 6.1 FunctionalRepresentation ofLearningTask s 359 6.1 Reasoning With Constraints 36 LogicConstraints inthe Environment, , , FormalLogic ,\Od Complexity ofReasoning 373 2.2 EnviromnentsWithSymbols and Subsymbols 376 6.2.3 T-Norms 384 6.2.4 LukasiewiczPropositionalLogic 388 DiffusionMachines 392 6.3 DataModels 393 6.3.2 Diffusion inSpatioternporaJ Environments , , 399 6.3.3 Recurrent Neural Networks 400 6.4 Algorithmic[ssues 404 PointwiseContent-Based Constraints 64 PropositionalConstraints in theTnputSpace 408 6.4.3 Supev r isedLearningWithLinearConstraints 6.4.4 LearningUnder Dif f usion Constraints Lif eL · ongLearningAgents 405 424 6.5.1 CognitiveAction and TemporalManifolds 425 6.5.2 EnergyBalance 430 6.5.3 Focus ofAttention, Teaching, and ActiveLearning 431 65 Developmental Learning 6.6 6.4 6.5 373 , Scholia CHAPTER Epilogue CHAPTER Answers to Exercises Section J.J Sc e tion 1.2 Section 1.3 437 446 452 453 • • 45 455 _ 455 Section 2.1 Section 2.2 Section Section 3.2 Section 3.3 Section 3.4 459 465 • • • Section 5.3 Section 5.4 • • • • Section 5.5 486 487 475 479 Section 4.4 Section 5.2 472 473 Section 4.2 Section 5.1 468 471 Section 4.1 Section 4.3 433 489 490 492 494 .. .Machine Learning A Constraint-Based Approach Machine Learning A Constraint-Based Approach Marco Gori Università di Siena Morgan Kaufmann is an imprint of Elsevier 50 Hampshire Street,... nature An interesting example is the car evaluation artificial learning task proposed in the UCI Machine Learning repository https://archive.ics.uci.edu/ml/datasets/Car+Evaluation The evaluation... description is what opens the doors for a fundamental rethinking of algorithmic approaches to many problems that are naturally solved by humans A learning machine can still be given an algorithmic