Neural Networks and Intellect: Using Model-Based Concepts Leonid I. Perlovsky OXFORD UNIVERSITY PRESS NEURAL NETWORKS AND INTELLECT This page intentionally left blank NEURAL NETWORKS AND INTELLECT Using Model-Based Concepts Leonid I. Perlovsky New York • Oxford OXFORD UNIVERSITY PRESS 2001 Oxford University Press Oxford New York Athens Auckland Bangkok Bogot ´ a Buenos Aires Calcutta Cape Town Chennai Dar es Salaam Delhi Florence Hong Kong Istanbul Karachi Kuala Lumpur Madrid Melbourne Mexico City Mumbai Nairobi Paris S ˜ ao Paulo Shanghai Singapore Taipei Tokyo Toronto Warsaw and associated companies in Berlin Ibadan Copyright © 2001 by Oxford University Press, Inc. Published by Oxford University Press, Inc., 198 Madison Avenue, New York, New York, 10016 http://www.oup-usa.org Oxford is a registered trademark of Oxford University Press All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior permission of Oxford University Press. Library of Congress Cataloging-in-Publication Data Perlovsky, Leonid I. Neural networks and intellect : using model-based concepts / Leonid I. Perlovsky. p. cm. Includes bibliographical references and index. ISBN 0-19-511162-1 1. Neural networks (computer science) 2. Mathematical models. I. Title. QA76.87.P435 2000 006.3'2—dc21 00-026297 Printing (last digit):987654321 Printed in the United States of America on acid-free paper CONDENSED TABLE OF CONTENTS PREFACE xix PART ONE: OVERVIEW: 2300 YEARS OF PHILOSOPHY, 100 YEARS OF MATHEMATICAL LOGIC, AND 50 YEARS OF COMPUTATIONAL INTELLIGENCE 1 Introduction: Concepts of Intelligence 3 2 Mathematical Concepts of Mind 51 3 Mathematical versus Metaphysical Concepts of Mind 125 PART TWO: MODELING FIELD THEORY: NEW MATHEMATICAL THEORY OF INTELLIGENCE WITH ENGINEERING APPLICATIONS 4 Modeling Field Theory 153 5 MLANS: Maximum Likelihood Adaptive Neural System for Grouping and Recognition 206 6 Einsteinian Neural Network 263 7 Prediction, Tracking, and Dynamic Models 289 8 Quantum Modeling Field Theory (QMFT) 321 9 Fundamental Limitations on Learning 329 10 Intelligent System Organization: MFT, Genetic Algorithms, and Kant 356 PART THREE: FUTURISTIC DIRECTIONS: FUN STUFF: MIND—PHYSICS + MATHEMATICS + CONJECTURES 11 Gödel Theorems, Mind, and Machine 383 12 Toward Physics of Consciousness 391 LIST OF SYMBOLS 425 DEFINITIONS 429 BIBLIOGRAPHY 447 INDEX 461 v This page intentionally left blank CONTENTS PREFACE xix PART ONE: OVERVIEW: 2300 YEARS OF PHILOSOPHY, 100 YEARS OF MATHEMATICAL LOGIC, AND 50 YEARS OF COMPUTATIONAL INTELLIGENCE 1 Introduction: Concepts of Intelligence 3 1.1 CONCEPTS OF INTELLIGENCE IN MATHEMATICS, PSYCHOLOGY, AND PHILOSOPHY 3 1.1.1 What Is Intelligence? 3 1.1.2 Plato, Occam, and Neural Networks 4 1.1.3 Rule-Based Artificial Intelligence, Complexity, and Aristotle 6 1.1.4 Philosophy vs. Architecture of Intelligent Tracker 8 1.1.5 Summary 12 1.2 PROBABILITY, HYPOTHESIS CHOICE, PATTERN RECOGNITION, AND COMPLEXITY 13 1.2.1 Prerequisite: Basic Notions of the Theory of Probability 13 1.2.2 Classical Hypotheses Choice Paradigms and Definitions 20 1.2.3 Pattern Recognition 22 1.2.4 A Priori Information and Adaptation 24 1.2.5 Mathematical Formulation of Model-Based Recognition 27 1.2.6 Conundrum of Combinatorial Complexity 29 1.3 PREDICTION, TRACKING, AND DYNAMIC MODELS 29 1.3.1 Linear Regression 30 1.3.2 Regression as an Expectation 32 1.3.3 Autoregression 33 1.3.4 Tracking 35 vii viii Contents 1.3.5 Association Problem 37 1.4 PREVIEW: INTELLIGENCE, INTERNAL MODEL, SYMBOL, EMOTIONS, AND CONSCIOUSNESS 42 Notes 45 Bibliographical Notes 46 Problems 47 2 Mathematical Concepts of Mind 51 2.1 COMPLEXITY, ARISTOTLE, AND FUZZY LOGIC 52 2.1.1 Conundrum of Combinatorial Complexity 52 2.1.2 Adaptivity, Apriority, and Complexity 53 2.1.3 Fuzzy Logic and Complexity 55 2.2 NEAREST NEIGHBORS AND DEGENERATE GEOMETRIES 58 2.2.1 The Nearest Neighbor Concept 58 2.2.2 Mathematical Formulation 59 2.2.3 What Constitutes Simple and Complex Classification Problems? 59 2.2.4 Degenerate Geometry of Classification Spaces 60 2.3 GRADIENT LEARNING, BACK PROPAGATION, AND FEEDFORWARD NEURAL NETWORKS 62 2.3.1 Concept of Discriminating Surfaces and Gradient Learning 62 2.3.2 Mathematical Formulation 64 2.3.3 Learning Disability 67 2.4 RULE-BASED ARTIFICIAL INTELLIGENCE 68 2.4.1 Minsky, Apriority, and Adaptivity 68 2.4.2 Soar Production System 70 2.5 CONCEPT OF INTERNAL MODEL 73 2.5.1 Prolegomena: Parametric vs. Nonparametric Estimation 73 2.5.2 Model-Based Vision (MBV) 74 2.5.3 Adaptivity and MBV 75 2.6 ABDUCTIVE REASONING 76 2.6.1 Deduction, Induction, and Abduction 76 2.6.2 Abductive Reasoning Trees and Bayesian Networks 77 2.7 STATISTICAL LEARNING THEORY AND SUPPORT VECTOR MACHINES 79 2.7.1 Model Complexity: Risk Minimization vs. PDF Estimation 79 Contents ix 2.7.2 Consistency of ERM and VC Dimension 81 2.7.3 Support Vector Machines (SVM) 82 2.8 AI DEBATES PAST AND FUTURE 85 2.8.1 Arguments and Disagreements: An Overview 85 2.8.2 Can a Machine Think? 87 2.8.3 Rule-Based AI vs. Connectivism 90 2.8.4 Emerging Debates 91 2.9 SOCIETY OF MIND 94 2.9.1 Society of Agents 94 2.9.2 Types of Agents 95 2.9.3 Frames and Unity of Apperception 96 2.9.4 Limitations and What Is Next 96 2.10 SENSOR FUSION AND JDL MODEL 97 2.10.1 Sensor Fusion and Origins of JDL Model 97 2.10.2 Definitions, Issues, and Types of Fusion Problems 98 2.10.3 Sensor Fusion Levels 99 2.10.4 Hierarchy of JDL Model Organization 100 2.11 HIERARCHICAL ORGANIZATION 100 2.12 SEMIOTICS 104 2.13 EVOLUTIONARY COMPUTATION, GENETIC ALGORITHMS, AND CAS 106 2.13.1 Complex Adaptive Systems (CAS) 107 2.13.2 CAS: Complexity vs. Fuzziness 109 2.14 NEURAL FIELD THEORIES 110 2.14.1 Grossberg’s Method: Physics of Mind 110 2.14.2 ART Neural Network 111 2.14.3 Illusions and A Priori Contents of Vision 114 2.14.4 Motor Coordination and Sensorimotor Control 115 2.14.5 Emotions and Learning 116 2.14.6 Quantum Neurodynamics 118 2.14.7 Modeling Field Theory 119 2.15 INTELLIGENCE, LEARNING, AND COMPUTABILITY 120 2.15.1 Computability: Turing vs. Physics 120 2.15.2 Computational Methods of Intelligence: Summary 121 Notes 121 Bibliographical Notes 122 Problems 124 [...]... undergraduate Chapter 5 4 Model-Based Neural Networks: Dynamic Models Internal models of the world are considered an essential part of intelligence in AI, cognitive sciences, and psychology The course describes how to design neural networks with internal models Model-based neural networks combine domain knowledge with learning and adaptivity of neural networks Prerequisites: probability and signal processing... Or, use your favorite problems 3 Model-Based Neural Networks: Statistical Models Internal models of the world are considered an essential part of intelligence in AI, cognitive sciences, and psychology The course describes how to design neural networks with internal models Model-based neural networks combine domain knowledge with learning and adaptivity of neural networks Prerequisites: probability Level:... debates of apriority and adaptivity of mind in mathematics and philosophy 1.1.2 Plato, Occam, and Neural Networks A contemporary direction in the theory of intellect is based on modeling neural structures of the brain It was founded by McCulloch and co-workers beginning in the early 1940s McCulloch intended to create a mathematical theory of intellect on the basis of complicated a priori neural structures... concept called modeling field theory; demonstrates applications of neural networks based on this theory to a variety of problems; and analyzes relationships among mathematics, computational concepts in neural networks, and concepts of mind in psychology and philosophy Deep philosophical questions are discussed and related in detail to mathematics and the engineering of intelligence The book is directed toward... 256 5.7 6 254 MLANS, PHYSICS, BIOLOGY, AND OTHER NEURAL NETWORKS 257 Note 260 Bibliographical Notes 260 Problems 261 Einsteinian Neural Network 6.1 225 263 IMAGES, SIGNALS, AND SPECTRA 263 6.1.1 Definitions, Notations, and Simple Signal Models 263 6.1.2 Frequency Components, Spectrum, and Spectral Models 6.1.3 Model-Based Spectrum Estimation 269 265 Contents 6.2 6.3 NEURAL DYNAMICS OF ENN 271 6.3.1 Shannon’s... the a priori Eide of Plato were encoded in the complicated neural structures of brain In search of a mathematical theory unifying neural and cognitive processes, McCulloch and co-workers combined an empirical analysis of biological neural networks with information theory and mathematically formulated important properties of neurons McCulloch and Pitts (1943) reduced a complicated entanglement of a large... model-based neural networks for various applications of increasing complexity Chapter 5 discusses the Maximum Likelihood Adaptive Neural System (MLANS), based on Bayesian similarity, for pattern and image recognition applications Chapter 6 considers Shannon–Einsteinian similarity and discusses Modeling-field Einsteinian ANS (MEANS) for spectrum estimation of transient signals in the frequency domain and. .. Vinkovetsky, V Webb, M Xiarhos, L Zadeh, and G Zainiev NEURAL NETWORKS AND INTELLECT This page intentionally left blank part one OVERVIEW 2300 Years of Philosophy, 100 Years of Mathematical Logic, and 50 Years of Computational Intelligence This part of the book consists of three chapters: Chapter 1 is an introduction to the book and to the concepts of intelligence in philosophy and mathematics Chapter 2 reviews... MATHEMATICS, PSYCHOLOGY, AND PHILOSOPHY This section overviews the history of concepts of intellect and serves as an introduction to metaphysical and mathematical analysis in Chapters 2 and 3 1.1.1 What Is Intelligence? The human mind, intelligence and its limits, the range of spiritual human experiences and computers, artificial intelligence, robots, the Internet’s sea of information, and as yet unexhausted... classical mathematical concepts of intelligent algorithms, symbolic AI, and neural networks After analysis of successes and deficiencies of the classical techniques, new emergent concepts are introduced: evolutionary computation, hierarchical organization, and neural fields Prerequisites: probability Desirable: a course in neural networks or AI Level: graduate or advanced undergraduate Chapter 2 (Sec . Neural Networks and Intellect: Using Model-Based Concepts Leonid I. Perlovsky OXFORD UNIVERSITY PRESS NEURAL NETWORKS AND INTELLECT This page intentionally left blank NEURAL NETWORKS AND INTELLECT Using. Cataloging-in-Publication Data Perlovsky, Leonid I. Neural networks and intellect : using model-based concepts / Leonid I. Perlovsky. p. cm. Includes bibliographical references and index. ISBN 0-1 9-5 1116 2-1 1 of neural networks based on this theory to a variety of problems; and analyzes relationships among mathematics, computational concepts in neural networks, and concepts of mind in psychology and