William mcduff spears using neural networks and (bookfi)

USING NEURAL NETWORKS AND GENETIC ALGORITHMS AS HEURISTICS FOR NP-COMPLETE PROBLEMS by William McDuff Spears A Thesis Submitted to the Faculty of the Graduate School of George Mason University in Partial Fulfillment of the Requirements for the Degree of Masters of Science in Computer Science Committee: Director Department Chairperson Dean of the Graduate School Date: Fall 1989 George Mason University Fairfax, Virginia Using Neural Networks and Genetic Algorithms as Heuristics for NP-Complete Problems A thesis submitted in partial fulfillment of the requirements for the degree of Master of Science at George Mason University By William McDuff Spears Bachelor of Arts in Mathematics Johns Hopkins University, May 1984 Director: Kenneth A De Jong Associate Professor Department of Computer Science Fall 1989 George Mason University Fairfax, Virginia ii Acknowledgements There are a number of people who deserve thanks for making this thesis possible I especially wish to thank my parents, for encouraging and supporting my education throughout my life; Ken De Jong, for suggesting this project and for his sound advice; my committee members, Henry Hamburger and Eugene Norris, for their time and interest; and my friend Diana Gordon for the numerous hours she spent correcting every aspect of my work Finally, I wish to thank Frank Pipitone, Dan Hoey and the Machine Learning Group at the Naval Research Laboratory, for many valuable discussions Any remaining flaws are the sole responsibility of the author iii Table of Contents Introduction Genetic Algorithms Overview Representation Genetic Operators Evaluation Function Selection Analysis Applications Domain Knowledge Implementation/Connectionism Summary GAs and SAT Representation/Choosing a Payoff Function Possible Improvements to the Payoff Function Results Neural Networks Overview NNs and SAT Representation/System of Constraints Paradigm I Problems with Paradigm I Paradigm II Results NP-Completeness Hamiltonian Circuit Problems Results Summary and Future Work 4 10 11 12 13 13 13 18 19 27 28 32 32 35 38 42 45 49 49 51 58 iii iv List of Tables Table page Sample Payoff Function 15 Violation of Truth Invariance 17 Performance of GAs on the Two Peak Problems 20 Performance of GAs on the False Peak Problems 22 Energy of Satisfied System 40 Energy of Non-Satisfied System 40 Performance of NNs on the Two Peak Problems 46 Performance of NNs on the False Peak Problems 47 9: Performance of GAs on HC Problems 53 10: Performance of NNs on HC Problems 55 11: GA Performance (AVEˆp, p = 1) 69 12: GA Performance (AVEˆp, p = 2) 70 13: GA Performance (AVEˆp, p = 3) 71 14: GA Performance (AVEˆp, p = 4) 72 15: GA Performance (AVEˆp, p = 5) 73 16: NN Performance 74 v List of Figures Figure page Performance of GAs on the Two Peak Problems 21 Performance of GAs on the False Peak Problems 23 Performance of GAs using AVEˆp 24 Summary Performance of GAs using AVEˆ2 25 Example Parse Tree 33 Performance of NN on the Two Peak Problems 47 Performance of NN on the False Peak Problems 48 Sample Hamiltonian Circuit Problem 50 Another Hamiltonian Circuit Problem 51 10 Graph of HC7 Payoff Function for the GA 53 11 Performance of GAs on the HC Problems 54 12 Performance of GAs using AVEˆp 55 13 Comparison of GAs and NNs on the HC Problems 56 Abstract USING NEURAL NETWORKS AND GENETIC ALGORITHMS AS HEURISTICS FOR NP-COMPLETE PROBLEMS William M Spears, M.S George Mason University, 1989 Thesis Director: Dr Kenneth A De Jong Paradigms for using neural networks (NNs) and genetic algorithms (GAs) to heuristically solve boolean satisfiability (SAT) problems are presented Results are presented for two-peak and false-peak SAT problems Since SAT is NPComplete, any other NP-Complete problem can be transformed into an equivalent SAT problem in polynomial time, and solved via either paradigm This technique is illustrated for hamiltonian circuit (HC) problems INTRODUCTION One approach to discussing and comparing AI problem solving methods is to categorize them using the terms strong or weak Generally, a weak method is one that has the property of wide applicability but, because it makes few assumptions about the problem domain, can suffer from combinatorially explosive solution costs when scaling to larger problems State space search algorithms and random search are familiar examples of weak methods Frequently, scaling problems can be avoided by making sufficiently strong assumptions about the problem domain and exploiting these assumptions in the problem solving method Many expert systems fall into this category in that they require and use large amounts of domain- and problem-specific knowledge in order to efficiently find solutions in enormously complex spaces The difficulty with strong methods, of course, is their limited domain of applicability leading, generally, to significant redesign even when applying them to related problems These characterizations tend to make one feel trapped in the sense that one has to give up significant performance to achieve generality, and vice versa However, it is becoming increasingly clear that there are two methodologies that fall in between these two extremes and offer in similar ways the possibility of powerful, yet general problem solving methods These two methods are neural networks (NNs) and genetic algorithms (GAs) Neural networks and genetic algorithms are similar in the sense that they achieve both power and generality by demanding that problems be mapped into their own particular representation in order to be solved If a fairly natural mapping exists, impressive robust performance results On the other hand, if the mapping is awkward and strained, both approaches behave much like the more traditional weak methods, yielding mediocre, unsatisfying results when scaling These observations suggest two general issues that deserve further study First, we need to understand how severe the mapping problem is Are there large classes of problems for which effective mappings exist? Clearly, if we have to spend a large amount of time and effort constructing a mapping for each new problem, we are not any better off than we would be if we used the more traditional, strong methods The second major issue involves achieving a better understanding of the relationship between NNs and GAs Are the representation issues and/or performance characteristics significantly different? Are there classes of problems handled much more effectively by one approach than the other? This thesis is a first step in exploring these issues It focuses on the application of GAs and NNs to a large, well-known class of combinatorially explosive problems: NP-complete problems NP-Complete problems are problems that are not currently solvable in polynomial time However, they are polynomially equivalent in the sense that any NP-Complete problem can be transformed into any other in polynomial time Thus, if any NP-Complete problem can be solved in polynomial time, they all can [Garey79] An example of an NP-Complete problem is the boolean satisfiability (SAT) problem: given an arbitrary boolean expression of n variables, does there exist an assignment to those variables such that the expression is true? Other familiar examples include job shop scheduling, bin packing, and traveling salesman (TSP) problems GAs and NNs have been used as heuristics for some NP-Complete problems [Goldberg89, Tagliarini87] Unfortunately, the results have been mixed because although NP-complete problems are computationally equivalent in the complexity theoretic sense, they not appear to be equivalent at all with respect to how well they map onto NN or GA representations The TSP is a classic example of a problem that does not map naturally to either NNs [Gutzmann87] or GAs [De Jong89] 61 List of References Ackley, David H (1985) A Connectionist Algorithm for Genetic Search, Proc Int’l Conference on Genetic Algorithms and their Applications Aho, Hopcroft, and Ullman (1974) The Design and Analysis of Computer Algorithms, Addison-Wesley Anand, V (1989) The Application of Genetic Algorithms to an NP-Complete Problem, Unpublished work, Navy Center for Applied Research in Artificial Intelligence Antonisse, H J and K S Keller (1987) Genetic Operators for High-Level Knowledge Representation, Proc Int’l Conference on Genetic Algorithms and their Applications Axelrod, Robert (1987) The Evolution of Strategies in the Iterated Prisoner’s Dilemma, Genetic Algorithms and Simulated Annealing, Lawrence Davis, ed., Morgan Kaufmann Publishers Baker, James E (1985) Adaptive Selection Methods for Genetic Algorithms, Proc Int’l Conference on Genetic Algorithms and their Applications Baker, James E (1987) Reducing Bias and Inefficiency in the Selection Algorithm, Proc Int’l Conference on Genetic Algorithms and their Applications Bethke, A D (1981) Genetic Algorithms as Function Optimizers, Doctoral dissertation, University of Michigan 62 Bickel, Arthur S and Riva Wenig Bickel (1987) Tree Structured Rules in Genetic Algorithms, Proc Int’l Conference on Genetic Algorithms and their Applications Booker, Lashon B (1987) Improving Search in Genetic Algorithms, Genetic Algorithms and Simulated Annealing, Lawrence Davis, ed., Morgan Kaufmann Publishers Bridges, Clayton L and David E Goldberg (1987) An Analysis of Reproduction and Crossover in a Binary-Encoded Genetic Algorithm, Proc Int’l Conference on Genetic Algorithms and their Applications Brown, C and P Purdom (1982) An Empirical Comparison of Backtracking Algorithms, IEEE Trans PAMI Vol 4, No 3, 309-316 Cervantes, J H and Richard Hildebrant (1987) Comparison of Three NeuronBased Computation Schemes, IEEE First International Conference on Neural Networks, pg III-657 Cohoon, J P., et al (1987) Punctuated Equilibria: a Parallel Genetic Algorithm, Proc Int’l Conference on Genetic Algorithms and their Applications Davis, M and H Putnam (1960) A Computing Procedure for Quantification Theory, J Assoc Comput Mach 7, 201-215 Davis, Lawrence (1985) Job Shop Scheduling with Genetic Algorithms, Proc Int’l Conference on Genetic Algorithms and their Applications Davis, Lawrence and Susan Coombs (1987) Genetic Algorithms and Communication Link Speed Design, Proc Int’l Conference on Genetic Algorithms and their Applications 63 Davis, Lawrence (1987) Genetic Algorithms and Simulated Annealing, Morgan Kaufman Publishers, Inc., Los Altos, CA De Jong, K A (1975) An Analysis of the Behavior of a Class of Genetic Adaptive Systems, Doctoral dissertation, Dept Computer and Communication Sciences, University of Michigan, Ann Arbor De Jong, K A (1985) Genetic Algorithms: a 10 Year Perspective, Proc Int’l Conference on Genetic Algorithms and their Applications De Jong, K A (1987) On Using Genetic Algorithms to Search Program Spaces, Proc Int’l Conference on Genetic Algorithms and their Applications De Jong, K A & William M Spears (1989) Using Genetic Algorithms to Solve NP-Complete Problems, Proc Int’l Conference on Genetic Algorithms and their Applications Dolan, Charles P and Michael G Dyer (1987) Toward the Evolution of Symbols, Proc Int’l Conference on Genetic Algorithms and their Applications Fourman, Michael P (1985) Compaction of Symbolic Layout Using Genetic Algorithms, Proc Int’l Conference on Genetic Algorithms and their Applications Franco, John (1986) On the Probabilistic Performance of Algorithms for the Satisfiability Problem, Information Processing Letters 23, 103-106 Fujiko, Cory and John Dickinson (1987) Using the Genetic Algorithms to Generate LISP Source Code to Solve the Prisoner’s Dilemma, Proc Int’l Conference on Genetic Algorithms and their Applications Garey, Michael R & David S Johnson (1979) Computers and Intractability: A Guide to the Theory of NP-Completeness, W H Freeman and Company, San 64 Francisco, CA Geman, S and D Geman (1984) Stochastic Relaxation, Gibbs Distributions, and Bayesian Restoration of Images, IEEE Trans., Vol PAMI-6, No 6, p 721, November 1984 Glover, David E (1987) Solving a Complex Keyboard Configuration Problem Through Generalized Adaptive Search, Genetic Algorithms and Simulated Annealing, Lawrence Davis, ed., Morgan Kaufmann Publishers Goldberg, David E (1985) Genetic Algorithms and Rule Learning in Dynamic Systems Control, Proc Int’l Conference on Genetic Algorithms and their Applications Goldberg, David E and Robert Lingle, Jr (1985) Alleles, Loci, and the Traveling Salesman Problem, Proc Int’l Conference on Genetic Algorithms and their Applications Goldberg, David E and Philip Segrest (1987) Finite Markov Chain Analysis of Genetic Algorithms, Proc Int’l Conference on Genetic Algorithms and their Applications Goldberg, David E and Jon Richardson (1987) Genetic Algorithms with Sharing for Multimodal Function Optimization, Proc Int’l Conference on Genetic Algorithms and their Applications Goldberg, David E and Robert E Smith (1987) Nonstationary Function Optimization Using Genetic Algorithms with Dominance and Diploidy, Proc Int’l Conference on Genetic Algorithms and their Applications Goldberg, David E (1987) Simple Genetic Algorithms and the Minimal, Deceptive Problem, Genetic Algorithms and Simulated Annealing, Lawrence Davis, ed., Morgan Kaufmann Publishers 65 Goldberg, David E (1989) Genetic Algorithms in Search, Optimization & Machine Learning, Addison-Wesley Publishing Company, Inc Grefenstette, John J and J Michael Fitzpatrick (1985) Genetic Search with Approximate Function Evaluations, Proc Int’l Conference on Genetic Algorithms and their Applications Grefenstette, John J., et al (1985) Genetic Algorithms for the Traveling Salesman Problem, Proc Int’l Conference on Genetic Algorithms and their Applications Grefenstette, John J (1985) Proceedings of an International Conference on Genetic Algorithms and their Applications, Pittsburg, PA Grefenstette, John J (1987) Genetic Algorithms and Their Applications: Proceedings of the Second International Conference on Genetic Algorithms, Cambridge, MA Grefenstette, John J (1987) Incorporating Problem Specific Knowledge into Genetic Algorithms, Genetic Algorithms and Simulated Annealing, Lawrence Davis, ed., Morgan Kaufmann Publishers Gunn, Janet P and Robert B Weidlich (1989) A Derivative of the Hopfield- Tank Neural Network Model that Reliably Solves the Traveling Salesman Problem, IJNNS, Washington, DC., June 1989 Gutzmann, Kurt M (1987) Combinatorial Optimization Using a Continuous State Boltzmann Machine, IEEE First International Conference on Neural Networks, pg III-721 Hammer, Peter L and Sergiu Rudeanu (1968) Boolean Methods in Operations Research, Springer-Verlag New York Inc 1968 66 Hoey, Dan J Navy Center for Applied Research in Artificial Intelligence Private Communication Holland, John H (1975) Adaptation in Natural and Artificial Systems, The University of Michigan Press Hopfield, J J (1982) Neural Networks and Physical Systems with Emergent Collective Computational Abilities, Proceedings of the National Academy of Sciences, USA, 79, 2554-2558 Hopfield, J J and D W Tank (1985) "Neural" Computation of Decisions in Optimization Problems, Biological Cybernetics, 52, 141-152 Korst, Jan H M and Emile H L Aarts (1989) Combinatorial Optimization on a Boltzmann Machine, Journal of Parallel and Distributed Computing, pg 331 Kuchinski, M J (1985) Battle Management Systems Control Rule Optimization Using Artificial Intelligence, Technical Note, Naval Surface Weapons Center, Dahlgren, VA Levy, B C and Milton Adams (1987) Global Optimization with Stochastic Neural Networks, IEEE First International Conference on Neural Networks, pg III-681 Liepins, G E., et al (1987) Greedy Genetics, Proc Int’l Conference on Genetic Algorithms and their Applications McClelland, James L and David E Rumelhart (1988) Explorations in Parallel Distributed Processing, The MIT Press, Cambridge, MA Oliver, I M., Smith, D J and J R C Holland (1987) A Study of Permutation Crossover Operators on the Traveling Salesman Problem, Proc Int’l Conference on Genetic Algorithms and their Applications 67 Pettey, Chrisila B., et al.(1987) A Parallel Genetic Algorithm, Proc Int’l Conference on Genetic Algorithms and their Applications Rivest, R L., et al (1978) A Method for Obtaining Digital Signatures and Public-key Cryptosystems, CACM, 21, 2, 120-6 Robertson, George G (1987) Parallel Implementation of Genetic Algorithms in a Classifier System, Proc Int’l Conference on Genetic Algorithms and their Applications Rumelhart, David E and James L McClelland (1986) Parallel Distributed Processing, The MIT Press, Cambridge, MA Sannier, Adrian V II and Erik D Goodman (1987) Genetic Learning Procedures in Distributed Environments, Proc Int’l Conference on Genetic Algorithms and their Applications Schaffer, J David (1985) Multiple Objective Optimization with Vector Evaluated Genetic Algorithms, Proc Int’l Conference on Genetic Algorithms and their Applications Schaffer, J David and Amy Morishima (1987) An Adaptive Crossover Distribution Mechanism for Genetic Algorithms, Proc Int’l Conference on Genetic Algorithms and their Applications Sirag, David J and Paul T Weisser (1987) Toward a Unified Thermodynamics Genetic Operator, Proc Int’l Conference on Genetic Algorithms and their Applications Smith, Gerald H (1979) Adaptive Genetic Algorithms and the Boolean Satisfiability Problem, Unpublished Work 68 Smith, S F (1980) A Learning System Based on Genetic Adaptive Algorithms, Doctoral Thesis, Department of Computer Science, University of Pittsburg Smith, Derek (1985) Bin Packing with Adaptive Search, Proc Int’l Conference on Genetic Algorithms and their Applications Suh, Jung Y and Dirk Van Gucht (1987) Incorporating Heuristic Information into Genetic Search, Proc Int’l Conference on Genetic Algorithms and their Applications Tagliarini, Gene A and Edward W Page (1987) Solving Constraint Satisfaction Problems with Neural Networks, IEEE First International Conference on Neural Networks, pg III-741 Tanese, Reiko (1987) Parallel Genetic Algorithms for a Hypercube, Proc Int’l Conference on Genetic Algorithms and their Applications Van Gelder, Allen (1988) A Satisfiability Tester for Non-clausal Propositional Calculus, Information and Computation 79, 1-21 Whitley, Darrell (1987) Using Reproductive Evaluation to Improve Genetic Search and Heuristic Discovery, Proc Int’l Conference on Genetic Algorithms and their Applications 69 ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ Appendix _ Problem Vars Mean St Dev Iterations Trials _ TP1 10 278 132 50 TP2 20 963 239 50 TP3 30 1753 360 50 TP4 40 2855 676 50 TP5 50 4216 641 50 TP6 60 5485 947 50 TP7 70 7095 1077 50 TP8 80 11167 4977 11 10 TP9 90 13080 5491 11 10 _ FP1 10 439 338 12 10 FP2 20 1209 730 11 10 FP3 30 4805 5289 17 10 FP4 40 8031 8039 20 10 FP5 50 12167 12797 20 10 FP6 60 18387 21268 44 20 FP7 70 15617 13390 34 20 FP8 80 18605 16209 32 20 FP9 90 35153 31731 22 10 _ HC4 65 24 10 HC5 10 848 2041 10 HC6 15 1022 1083 10 HC7 21 5028 2016 10 HC8 28 21894 36391 20 10 HC9 36 70577 49946 10 HC10 45 259876 227254 10 HC11 55 838522 123350 _ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ Table 11: GA Performance (AVE p , p = 1) 70 ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ Appendix _ Problem Vars Mean St Dev Iterations Trials _ TP1 10 164 45 10 10 TP2 20 696 91 10 10 TP3 30 1257 294 10 10 TP4 40 2283 847 11 10 TP5 50 2741 393 10 10 TP6 60 4060 518 10 10 TP7 70 4966 2238 11 10 TP8 80 6973 2516 11 10 TP9 90 10208 7075 13 10 _ FP1 10 288 157 11 10 FP2 20 1879 2110 18 10 FP3 30 3608 2155 21 10 FP4 40 4219 2065 17 10 FP5 50 6517 3197 19 10 FP6 60 10162 9649 22 10 FP7 70 12929 11687 22 10 FP8 80 17868 14615 25 10 FP9 90 17569 13706 21 10 _ HC4 106 14 10 10 HC5 10 239 72 10 10 HC6 15 803 512 12 10 HC7 21 3559 3281 18 10 HC8 28 8680 9355 25 10 HC9 36 34417 29668 61 10 HC10 45 174706 182521 200 10 HC11 55 640478 477833 595 10 _ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ Table 12: GA Performance (AVE p , p = 2) 71 ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ Appendix _ Problem Vars Mean St Dev Iterations Trials _ TP1 10 175 74 10 10 TP2 20 687 285 12 10 TP3 30 1198 717 11 10 TP4 40 2123 1073 12 10 TP5 50 2828 1816 12 10 TP6 60 3575 2367 12 10 TP7 70 6567 4094 16 10 TP8 80 9838 5671 19 10 TP9 90 14524 9490 23 10 _ FP1 10 238 153 11 10 FP2 20 987 509 15 10 FP3 30 2432 1614 19 10 FP4 40 3229 2293 17 10 FP5 50 6554 4724 22 10 FP6 60 10537 10873 28 10 FP7 70 7814 3438 19 10 FP8 80 22412 13671 39 10 FP9 90 16644 15119 26 10 _ HC4 106 13 10 10 HC5 10 222 70 10 10 HC6 15 594 755 11 10 HC7 21 1897 1096 18 10 HC8 28 12247 8444 52 10 HC9 36 46558 51804 148 10 HC10 45 155141 138554 377 10 _ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ ✍ Table 13: GA Performance (AVE p , p = 3) 72 ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ Appendix Problem Vars Mean St Dev Iterations Trials TP1 10 174 62 10 10 TP2 20 497 96 11 10 TP3 30 1062 478 12 10 TP4 40 1501 311 10 10 TP5 50 2930 1030 15 10 TP6 60 4566 3618 17 10 TP7 70 5649 2176 16 10 TP8 80 10601 7545 25 10 TP9 90 17426 6991 32 10 FP1 10 314 265 12 10 FP2 20 780 450 14 10 FP3 30 1771 1021 18 10 FP4 40 3015 2786 19 10 FP5 50 5477 4165 26 10 FP6 60 11224 9881 36 10 FP7 70 9479 5374 26 10 FP8 80 13340 7674 30 10 FP9 90 31299 23323 57 10 HC4 106 13 10 10 HC5 10 234 82 10 10 HC6 15 697 728 12 10 HC7 21 1993 2460 23 10 HC8 28 10506 8753 73 10 HC9 36 69114 77931 342 10 ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ ✎ Table 14: GA Performance (AVE p , p = 4) 73 ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ Appendix Problem Vars Mean St Dev Iterations Trials TP1 10 165 40 10 10 TP2 20 433 84 10 10 TP3 30 961 266 12 10 TP4 40 1532 875 13 10 TP5 50 2402 1060 14 10 TP6 60 6167 4738 25 10 TP7 70 8637 4914 28 10 TP8 80 11774 9073 30 10 TP9 90 18892 11420 41 10 FP1 10 254 92 12 10 FP2 20 559 241 12 10 FP3 30 2030 1584 20 10 FP4 40 3988 2751 27 10 FP5 50 5240 4680 27 10 FP6 60 10850 5653 43 10 FP7 70 10663 6493 33 10 FP8 80 23963 23687 58 10 FP9 90 48010 34600 97 10 HC4 106 13 10 10 HC5 10 221 80 10 10 HC6 15 771 798 16 10 HC7 21 1784 1543 26 10 HC8 28 10605 9259 101 10 HC9 36 97837 90413 687 10 ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ Table 15: GA Performance (AVE p , p = 5) 74 ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ Appendix Problem Vars Mean St Dev Iterations Trials TP1 10 50 50 TP2 20 12 50 50 TP3 30 24 11 50 50 TP4 40 35 14 50 50 TP5 50 51 14 50 50 TP6 60 64 15 50 50 TP7 70 78 15 50 50 TP8 80 97 18 50 50 TP9 90 113 19 50 50 TP10 100 132 23 50 50 FP1 10 49 106 57 50 FP2 20 18 10 50 50 FP3 30 31 16 50 50 FP4 40 44 18 50 50 FP5 50 76 127 51 50 FP6 60 77 25 50 50 FP7 70 90 26 50 50 FP8 80 108 27 50 50 FP9 90 126 30 50 50 FP10 100 171 256 51 50 HC4 15 10 10 10 HC5 10 51 34 10 10 HC6 15 169 84 10 10 HC7 21 426 265 10 10 HC8 28 1120 1058 12 10 HC9 36 6698 5927 26 10 HC10 45 99431 100052 99 10 HC11 55 1417388 1341593 384 10 ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ ✑ Table 16: NN Performance 75 Vita William M Spears was born on April 3, 1962, in Providence, Rhode Island, and is an American citizen He graduated from Saratoga Springs High School, Saratoga Springs, New York, in 1980 He received his Bachelor of Arts in Mathematics from Johns Hopkins University, Baltimore, Maryland, in 1984 He has been employed at the Naval Research Laboratory, Washington D.C, since 1985 His publications include: De Jong, K.A and W Spears, "Using Genetic Algorithms to Solve NP-Complete Problems", International Conference on Genetic Algorithms, George Mason University, Fairfax, Virginia, 1989 Pipitone, F., K.A De Jong, and W Spears, "An Artificial Intelligence Approach to Analog Systems Diagnosis", NRL Report 9219, Naval Research Laboratory, Washington D.C., 1989 Pipitone, F., K.A De Jong, W Spears, and M Marrone, "The FIS Electronic Troubleshooting Project", Expert System Applications to Telecommunications, J Liebowitz, John Wiley, New York, 1988 Spears, W and K.A De Jong, "Using Genetic Algorithms as a Heuristic for NPComplete Decision Problems", Operations Research Society of America / The Institute of Management Sciences, New York City, New York, 1989 Spears, W., "Using Neural Networks and Genetic Algorithms as Heuristics for NP-Complete Problems", International Joint Conference on Neural Networks, Washington D.C, 1990