the computational complexity of machine learning - michael j kearns

[...]... models and the model we consider: the explicit emphasis here on the computational e ciency of learning algorithms The model we use, sometimes known as the distribution-free model or the model of probably approximately correct learning, was introduced by L.G Valiant 93 in 1984 and has been the catalyst for a renaissance of research in formal models of machine learning known as computational learning theory... nitions We expect these consequences to take the form of learning algorithms along with proofs of 2 Introduction their correctness and performance, lower bounds and hardness results that delineate the fundamental computational and information-theoretic limitations on learning, and general principles and phenomena that underly the chosen model The notion of a mathematical study of machine learning is by... future examples While there is no ultimate defense against either of these two kinds of error, the distribution-free model allows the probabilities of their occurrence to be controlled by the parameters and respectively 2.4 Other de nitions and notation Sample complexity Let A be a learning algorithm for a representation class C Then we denote by SA  ;  the number of calls to the oracles POS and NEG... and partly out of interest in the phenomenon of learning in its own right, the goal of the research presented here is to provide some mathematical foundations for a science of e cient machine learning More precisely, we wish to de ne a formal mathematical model of machine learning that is realistic in some but inevitably not all important ways, and to analyze rigorously the consequences of our de nitions... on computational complexity Learning algorithms are required to be e cient, in the standard polynomial-time sense The question we therefore address and partially answer in these pages is: What does complexity theory have to say about machine learning from examples? As we shall see, the answer to this question has many parts We begin in Chapter 2 by giving the precise de nition of the distribution-free... We feel that the results presented here and elsewhere in computational learning theory demonstrate that a wide variety of topics in theoretical computer science and other branches of mathematics have a direct and signi cant bearing on natural problems in machine learning We hope that this line of research will continue to illuminate the phenomenon of e cient machine learning, both in the model studied... schemes see Garey and Johnson 39 , and denote by jxj and jcj the length of these encodings measured in bits or in the case of real-valued domains, some other reasonable measure of length that may depend on the model of arithmetic computation used; see Aho, Hopcroft and Ullman 3  Parameterized representation classes We will often study parameterized classes of representations Here we have a strati ed domain... some cases for computational reasons we may not wish to restrict H beyond it being polynomially evaluatable If the algorithm produces an accurate and easily evaluated hypothesis, then our learning problem is essentially solved, and the actual form of the hypothesis is of secondary concern A major theme of this book is the importance of allowing a wide choice of representations for a learning algorithm... positive-only and negative-only weak learnability Note that although the learning algorithm receives only one type of examples, the hypothesis output must 14 De nitions and Motivation for Distribution-free Learning still be accurate with respect to both the positive and negative distributions Several learning algorithms in the distribution-free model are positiveonly or negative-only The study of positive-only.. .The Computational Complexity of Machine Learning 1 Introduction Recently in computer science there has been a great deal of interest in the area of machine learning In its experimental incarnation, this eld is contained within the broader con nes of arti cial intelligence, and its attraction for researchers stems from many sources Foremost among these is the hope that an understanding of a computer's