Artificial neural networks and deep learning

Artificial Neural Networks and Deep Learning Christian Borgelt Bioinformatics and Information Mining Dept of Computer and Information Science University of Konstanz, Universităatsstraòe 10, 78457 Konstanz, Germany christian.borgelt@uni-konstanz.de christian@borgelt.net http://www.borgelt.net/teach/nn eng.html Lecture in ZEuS: Exercises in ZEuS: Lecture in ILIAS: Christian Borgelt https://bit.ly/2qvmHjN https://bit.ly/2HpfGKZ https://bit.ly/2H1971H Artificial Neural Networks and Deep Learning Schedule and Exercises Lecture Exercises Exercises Day Monday Tuesday Thursday Time 15:15 – 16:45 10:00 – 11:30 10:00 – 11:30 Room A703 M631 M701 Start 16.04.2018 24.04.2018 26.04.2018 The slides for the lecture (beware of updates!) are available here: https://bit.ly/2H1971H http://www.borgelt.net/teach/nn eng.html The lecture is accompanied by sheets of exercises (one sheet per week), which should be worked on by you and which will be discussed afterward in the exercise lesson (conducted by Christoph Doell) The sheets of exercises can be downloaded as PDF files: https://bit.ly/2H1971H http://www.borgelt.net/teach/nn eng.html The first sheet is already available and is to be prepared for the first exercise lesson! Christian Borgelt Artificial Neural Networks and Deep Learning Exam Admission and Exam Exam admission is obtained via the exercise sheets At the beginning of each exercise lesson, a sheet will be passed round on which you can “vote” for exercises of the current exercise sheet Voting for an exercise means to declare oneself willing to present something about it A full solution would be perfect, but partial solutions or some approach that was tried are acceptable It should become clear that it was actually tried to solve the exercise In order to be admitted to the exam, you have to: • Vote for at least 50% of the exercises • Actually present something at least twice First exam (written): Tuesday, 24.07.2018, 11:00 to 13:00 hours, in Room R511 Second exam (written): to be determined Christian Borgelt Artificial Neural Networks and Deep Learning Textbooks This lecture follows the first parts of these books fairly closely, which treat artificial neural networks Textbook, 2nd ed Springer-Verlag Heidelberg, DE 2015 (in German) Christian Borgelt Textbook, 2nd ed Springer-Verlag Heidelberg, DE 2016 (in English) Artificial Neural Networks and Deep Learning Contents • Introduction Motivation, Biological Background • Threshold Logic Units Definition, Geometric Interpretation, Limitations, Networks of TLUs, Training • General Neural Networks Structure, Operation, Training • Multi-layer Perceptrons Definition, Function Approximation, Gradient Descent, Backpropagation, Variants, Sensitivity Analysis • Deep Learning Many-layered Perceptrons, Rectified Linear Units, Auto-Encoders, Feature Construction, Image Analysis • Radial Basis Function Networks Definition, Function Approximation, Initialization, Training, Generalized Version • Self-Organizing Maps Definition, Learning Vector Quantization, Neighborhood of Output Neurons • Hopfield Networks and Boltzmann Machines Definition, Convergence, Associative Memory, Solving Optimization Problems, Probabilistic Models • Recurrent Neural Networks Differential Equations, Vector Networks, Backpropagation through Time Christian Borgelt Artificial Neural Networks and Deep Learning Motivation: Why (Artificial) Neural Networks? • (Neuro-)Biology / (Neuro-)Physiology / Psychology: ◦ Exploit similarity to real (biological) neural networks ◦ Build models to understand nerve and brain operation by simulation • Computer Science / Engineering / Economics ◦ Mimic certain cognitive capabilities of human beings ◦ Solve learning/adaptation, prediction, and optimization problems • Physics / Chemistry ◦ Use neural network models to describe physical phenomena ◦ Special case: spin glasses (alloys of magnetic and non-magnetic metals) Christian Borgelt Artificial Neural Networks and Deep Learning Motivation: Why Neural Networks in AI? Physical-Symbol System Hypothesis [Newell and Simon 1976] A physical-symbol system has the necessary and sufficient means for general intelligent action Neural networks process simple signals, not symbols So why study neural networks in Artificial Intelligence? • Symbol-based representations work well for inference tasks, but are fairly bad for perception tasks • Symbol-based expert systems tend to get slower with growing knowledge, human experts tend to get faster • Neural networks allow for highly parallel information processing • There are several successful applications in industry and finance Christian Borgelt Artificial Neural Networks and Deep Learning Biological Background Diagram of a typical myelinated vertebrate motoneuron (source: Wikipedia, Ruiz-Villarreal 2007), showing the main parts involved in its signaling activity like the dendrites, the axon, and the synapses Christian Borgelt Artificial Neural Networks and Deep Learning Biological Background Structure of a prototypical biological neuron (simplified) terminal button synapse dendrites nucleus cell body (soma) axon myelin sheath Christian Borgelt Artificial Neural Networks and Deep Learning Biological Background (Very) simplified description of neural information processing • Axon terminal releases chemicals, called neurotransmitters • These act on the membrane of the receptor dendrite to change its polarization (The inside is usually 70mV more negative than the outside.) • Decrease in potential difference: excitatory synapse Increase in potential difference: inhibitory synapse • If there is enough net excitatory input, the axon is depolarized • The resulting action potential travels along the axon (Speed depends on the degree to which the axon is covered with myelin.) • When the action potential reaches the terminal buttons, it triggers the release of neurotransmitters Christian Borgelt Artificial Neural Networks and Deep Learning 10 NEFCLASS-J: Implementation in Java picture not available in online version Christian Borgelt Artificial Neural Networks and Deep Learning 443 Neuro-Fuzzy Systems in Finance Stock Index Prediction (DAX) [Siekmann 1999] • Prediction of the daily relative changes of the German stock index (Deutscher Aktienindex, DAX) • Based on time series of stock indices and other quantities between 1986 and 1997 Input Variables: ◦ DAX (Germany) ◦ German month interest rate ◦ Dow Jones industrial index (USA) ◦ US treasure bonds ◦ Composite DAX (Germany) ◦ Nikkei index (Japan) ◦ Morgan–Stanley index Germany ◦ Morgan–Stanley index Europe Christian Borgelt ◦ return Germany ◦ price to income ratio ◦ exchange rate DM / US-$ ◦ gold price Artificial Neural Networks and Deep Learning 444 DAX Prediction: Example Rules • trend rule: if DAX is decreasing and US-$ is decreasing then DAX prediction is decreasing with high certainty • turning point rule: if DAX is decreasing and US-$ is increasing then DAX prediction is increasing with low certainty • delay rule: if DAX is stable and US-$ is decreasing then DAX prediction is decreasing with very high certainty general form: if x1 is à1 and x2 is µ2 and and xn is µn then y is ν with certainty c Initial rules may be provided by financial experts Christian Borgelt Artificial Neural Networks and Deep Learning 445 DAX Prediction: Architecture membership functions rules increasing stable x1 decreasing output y input increasing x2 stable consequents increasing decreasing stable decreasing antecedents Christian Borgelt Artificial Neural Networks and Deep Learning 446 DAX Prediction: From Rules to Neural Network • Finding the membership values (evaluate membership functions): 1 2 0 zero negative decreasing positive stable increasing • Evaluating the rules (computing the rule activation for r rules): d ∀j ∈ {1, , r} : a˜j (x1, , xd) = i=1 • Accumulation of r rule activations, normalization: r wj y= j=1 Christian Borgelt cj a˜j (x1, , xd) , r ˜ c a (x , , x ) d k=1 k k (i) µj (xi) r wj = where Artificial Neural Networks and Deep Learning j=1 447 DAX Prediction: Training the Network • Membership degrees of different inputs share their parameters, e.g Advantage: number of free parameters is reduced (Composite DAX) (DAX) àstable = àstable Membership functions of the same input variable must not “pass each other”, but must preserve their original order: 1 2 0 negative zero positive µnegative < µzero < µpositive decreasing stable increasing Advantage: optimized rule base remains interpretable àdecreasing < àstable < àincreasing The parameters of the fuzzy sets, the rule certainties, and the rule weights are optimized with a backpropagation approach • Pruning methods are employed to simplify the rules and the rule base Christian Borgelt Artificial Neural Networks and Deep Learning 448 DAX Prediction: Trading Performance • Different prediction/trading models for the DAX: naive, Buy&Hold, linear model, multi-layer perceptron, neuro-fuzzy system • Profit and loss obtained from trading according to prediction • Validation period: March 1994 to April 1997 pictures not available in online version Christian Borgelt Artificial Neural Networks and Deep Learning 449 Neuro-Fuzzy Systems in Quality Control Surface Control of Car Body Parts (BMW) • Previous Approach: ◦ surface control is done manually ◦ experienced employee treats surface with a grinding block ◦ human experts classify defects by linguistic terms picture not available in online version ◦ cumbersome, subjective, error-prone, time-consuming • Suggested Approach: ◦ digitization of the surface with optical measurement systems ◦ characterization of the shape defects by mathematical properties (close to the subjective features) Christian Borgelt Artificial Neural Networks and Deep Learning 450 Surface Control: Topometric Measurement System picture not available in online version Christian Borgelt Artificial Neural Networks and Deep Learning 451 Surface Control: Data Processing picture not available in online version Christian Borgelt Artificial Neural Networks and Deep Learning 452 Surface Control: Color Coded Representation picture not available in online version Christian Borgelt Artificial Neural Networks and Deep Learning 453 Surface Control: 3D Representation picture not available in online version picture not available in online version sink mark slight flat-based sink inwards uneven surface several neighboring sink marks picture not available in online version picture not available in online version press mark local smoothing of the surface wavy surface several severe foldings in serie Christian Borgelt Artificial Neural Networks and Deep Learning 454 Surface Control: Defect Classification Data Characteristics • master pieces with a total of 99 defects were analyzed • 42 features were computed for each defect • Defect types are fairly unbalanced, rare types were dropped • Some extremely correlated features were dropped ⇒ 31 features remain • Remaining 31 features were ranked by their importance • Experiment was conducted with 4-fold stratified cross validation Accuracy of different classifiers: accuracy DC NBC DT NN NEFCLASS training 46.8% 89.0% 94.7% 90.0% 81.6% test 46.8% 75.6% 75.6% 85.5% 79.9% Christian Borgelt Artificial Neural Networks and Deep Learning 455 Surface Control: Fuzzy Rule Base R1: if and and then max dist to cog is fun extrema is fun max extrema is fun type is press mark R5: if and and then max dist to cog is fun all extrema is fun extrema is fun type is press mark R2: if and and then max dist to cog is fun all extrema is fun max extrema is fun type is sink mark R6: if and and then max dist to cog is fun all extrema is fun max extrema is fun type is uneven surface R3: if and and then max dist to cog is fun extrema is fun max extrema is fun type is uneven surface R7: if max dist to cog is fun and extrema is fun then type is uneven surface R4: if and and then max dist to cog is fun extrema is fun max extrema is fun type is uneven surface Christian Borgelt NEFCLASS rules for surface defect classification Artificial Neural Networks and Deep Learning 456 Neuro-Fuzzy Systems: Summary • Neuro-fuzzy systems can be useful for discovering knowledge in the form of rules and rule bases • The fact that they are interpretable, allows for plausibility checks and improves acceptance • Neuro-fuzzy systems exploit tolerances in the underlying system in order to find near-optimal solutions • Training procedures for neuro-fuzzy systems have to be able to cope with restrictions in order to preserve the semantics of the original model • No (fully) automatic model generation ⇒ A user has to work and interact with the system • Simple training methods support exploratory data analysis Christian Borgelt Artificial Neural Networks and Deep Learning 457 ... Borgelt Artificial Neural Networks and Deep Learning 10 Recording the Electrical Impulses (Spikes) pictures not available in online version Christian Borgelt Artificial Neural Networks and Deep Learning. .. neural networks Christian Borgelt Artificial Neural Networks and Deep Learning 14 Threshold Logic Units Christian Borgelt Artificial Neural Networks and Deep Learning 15 Threshold Logic Units... Borgelt Artificial Neural Networks and Deep Learning Motivation: Why (Artificial) Neural Networks? • (Neuro-)Biology / (Neuro-)Physiology / Psychology: ◦ Exploit similarity to real (biological) neural