Course Overview. before 1943: Neurobiology. before 1943: Neurobiology. before 1943: Neurobiology. The Start of Artificial Neural Nets

Transcription

1 Course Overview Introduction to Artificial Neural Networks Kristof Van Laerhoven mis.tu-darmstadt.de We will: Learn to interpret common ANN formulas and diagrams Get an extensive overview of the major ANNs Understand mechanisms behind ANNs Introduce it with historic background Left as exercises: Implementations of algorithms and details Evaluations and proofs /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks The Start of Artificial Neural Nets Standard ANN introduction courses just mention: McCulloch, W. and Pitts, W. (94. A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics, 7:5 -. Warren McCulloch ( Walter Pitts (9-99 real neuron simple model some authors But lets give a few more details /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks before 94: Neurobiology Trepanation: Brain = cranial stuffing ~4BC: Hippocrates: Brain = Seat of intelligence ~5BC: Aristotle: Brain cools blood : Galen: Brain damage ~ mental functioning 54: Vesalius: anatomy 78: Galvani: electrical excitability of muscles, neurons /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 4 before 94: Neurobiology before 94: Neurobiology 898: Camillo Golgi: Golgi Stains 9: Golgi + Santiago Ramón y Cajal: Nobel prize ~9: DuBois-Reymond, Müller, and von Helmholtz: neurons electrically excitable their activity affects electrical state of adjacent neurons real neurons One would assume, I think, that the presence of a theory, however strange, in a field in which no theory had previously existed, would have been a spur to the imagination of neurobiologists. But this did not occur at all! The whole field of neurology and neurobiology ignored the structure, the message, and the form of McCulloch s and Pitts s theory. Instead, those who were inspired by it were those who were destined to become the aficionados of a new venture, now called Artificial Intelligence, which proposed to realize in a programmatic way the ideas generated by the theory -- Lettvin, 989 In 94 there already existed a lively community of biophysicists doing mathematical work on neural nets Dissections, Golgi stains, and microscopes, but not: Church-Turing for the brain /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 5 /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks

2 Back to McCulloch-Pitts Warren McCulloch: Neurophysiologist (philosophy, psychology, MD The McCulloch group Walter Pitts Self-taught logic and maths, Homeless, non-enrolled as student, eccentric Manuscript on D networks Their seminal 94 paper: Mathematical (logic model for neuron 959 Nervous system can be considered a kind of universal computing device as described by Leibniz What the frog's eye tells the frog's brain» Recommended Reading: Dark Hero Of The Information Age: In Search of Norbert Wiener, the Father of Cybernetics, and search the web for Jerome Lettvin /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 7 (Hebbian Learning Previously: Pavlov, 97: Classical Conditioning Thorndike, 898: Operant Conditioning Donald Hebb (psychology, teacher The Organization of Behavior What fires together, wires together: When an axon of cell A is near enough to excite cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A's efficiency, as one of the cells firing B, is increased -- D. Hebb Others: Steven Grossberg, Gail Carpenter: Adaptive Resonance Theory /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 8 x w w w x.. The Perceptron b y y = i=,,,... (w i x i + b x: inputs w: weights b: bias < < : Sign(y : (usually - or y T : target (usually - or 957: Frank Rosenblatt The perceptron: A perceiving and recognizing automaton (project PARA. /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 9 x w w w x.. The Perceptron b y T? y y = i=,,,... (w i x i + b x: inputs w: weights b: bias < < : Sign(y : y T : target Learning / Training Phase: (Widrow-Hoff, or Delta Rule. Start with random w, w, w,, b (usually near. Calculate y for the current input x,, x,. Update weights and bias: w i ' + (y T yx i 4. Go to step /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks The Perceptron: Example Teach the neuron to spot male authors of s on a mailing list, given two inputs x : counts of { around, what, more } minus counts of { with, if, not } : counts of { at, it, many } minus counts of { myself, hers, was } Learning by examples: Negative samples: Positive samples: Sender: Gordon Christine Kristof (loosely based on and Gender, Genre, and Writing Style in Formal Written Texts, Shlomo Argamon, Moshe Koppel, Jonathan Fine, Anat Rachel Shimoni /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks Hans Cath Alan Subject: Camera Equipme Meeting room info Visitors on Mon/Tue Printer on B floor i Re: UN petition fo Coffee machine h (x, (5,- (-,- (5,4 (-, (, (, The Perceptron: Example Teach the neuron to spot male authors of s on a mailing list, given two inputs x : counts of { around, what, more } minus counts of { with, if, not } : counts of { at, it, many } minus counts of { myself, hers, was } Learning by examples: Negative samples: Positive samples: hans alan kristof x gordon (loosely based on and Gender, Genre, and Writing Style in Formal Written Texts, Shlomo Argamon, Moshe Koppel, Jonathan Fine, Anat Rachel Shimoni /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks cath christine Instead of using the whole mails, we use only keyword counts. These type of pre-processed values are called features in machine learning literature.

3 The Perceptron: Example The Perceptron: Example y = x w w i=,,,... Initial Values: w =/ w =-/ b = b y (w i x i + b sign(y=- y = x + = /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks y = w x +w +b = (/ x + (-/ + x sign(y=+ y = x w w i=,,,... w =/5 w =-/ b = -/7 b y (w i x i + b y T =- Values after training for all and (with =.: /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 4 y = w x +w +b = (/5 x + (-/ + (-/7 y = 5 x 7 = x y T =+ The Perceptron: Example The Perceptron s Delta Rule y = x w w i=,,,... y b (w i x i + b x y = 9 x = Delta Rule or Gradient Descent: Error: E = ( y T y Calculate partial derivative of the Error w.r.t. each weight: E = (y T yx w Values after training 5 times for all and (with =.: w =9/ w =-/ b = - y = w x +w +b = (9/ x + (-/ + (- /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 5 w = w' w = (y T yx Local Minima only! /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks The Perceptron Perceptron has phases: Training/Learning Phase: adaptive weights Testing Phase: for checking performance and generalisation Neurons and their s are grouped in : x x w y w w b w w y w b x b x x n b : : : b m y y y m Output Functions Output function per neuron: g(y limits and scales Simple threshold Sigmoid Gaussian Piecewise Linear... < x x+ g(x =... < x <... x < g(x = + e x g(x = e g(x = tanh(x /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 7 /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 8

4 Parameters Bias can be seen as weight to constant input, or input with constant weight Output parameters (e.g., steepness for sigmoid Momentum term to avoid local minima Initialisation: Use a Committee of networks to avoid random init effects (many networks training on same data, different initialised weights /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 9 y = The Perceptron s Issues x w w i=,,,... b (w i x i + b 99: Marvin Minsky and Seymour Papert, Perceptrons, Cambridge, MA: MIT Press => Non-linearly separable functions not solvable! y /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks XOR? x x = = -> = x = = -> = x = = -> = x = = -> = Backpropagation hidden Network: Multi- perceptron: Same neurons, but: Hidden s Training propagates per, from back to input weights P. Werbos in 974 Compiled in 98: D.E. Rumelhart, J.L. McClelland Parallel Distributed Processing: Explorations in the Microstructure of Cognition /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks Backpropagation Test Phase hidden Inputs are entered in network /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks Backpropagation Test Phase Backpropagation Test Phase hidden Inputs are entered in network Weighted sums calculate hidden Inputs are entered in network Weighted sums calculate Outputs of hidden neurons /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 4 4

5 Backpropagation Test Phase Backpropagation Test Phase hidden Inputs are entered in network Weighted sums calculate Outputs of hidden neurons Which are weighted again to calculate hidden Inputs are entered in network Weighted sums calculate Outputs of hidden neurons Which are weighted again to calculate The s of the neurons in the.. finally! /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 5 /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks Backpropagation Training Backpropagation Training hidden In the training phase, we also know the target s for the input hidden In the training phase, we also know the target s for the input Which can be used to re-adjust the weights going to the /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 7 /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 8 Backpropagation Training Backpropagation Training hidden In the training phase, we also know the target s for the input Which can be used to re-adjust the weights going to the Which lead to target s for the middle neurons hidden In the training phase, we also know the target s for the input Which can be used to re-adjust the weights going to the Which lead to target s for the middle neurons Which re-adjust the remaining weights (just like in the perceptron /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 9 /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 5

6 Backpropagation Backpropagation XOR works! In formulas: y y 4. y x Note: This network could be re-trained for OR, AND, NOR, or NAND /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks Multi Perceptron Apps Multi Perceptron Apps 988: Gorman and Sejnowski Analysis of Hidden Units in a Layered Network Trained to Classify Sonar Targets ALVINN (Pomerleau, 99 Image Compression using Backprop (Paul Watta, Brijesh Desaie, Norman Dannug, Mohamad Hassoun, 99 /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 4 Underfitting and Overfitting Overview so far Underfitting: ANN that is not sufficiently complex to correctly detect the pattern in a noisy data set Overfitting: ANN that is too complex so it reacts on noise in the data Techniques to avoid them: Model selection Jittering Weight decay Early stopping Bayesian estimation These are the most popular neural network architectures and their training algorithms: Perceptron - gradient descent / delta rule Only linearly separable functions One Multi perceptron - backpropagation Hidden Layer(s More powerful Training: errors are propagated back towards input weights Many applications They are SUPERVISED: we train them with inputs plus known target s /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 5 /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks

7 The Kohonen Map Also: Self-Organising (Feature Map Teuvo Kohonen (98 Self-organized formation of topologically correct feature maps Competition among neurons for the input: Topology among neurons Unsupervised x w x x 4 x 5 w w w 4 w 5 + (x i w i /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 7 Randomly initialised grid of (weight- vectors: (7,9 (,9 (4,9 (, (5, (95,7 (, (4,9 (,9 (8, (, (4,74 (9, (7,5 (75,55 (,4 + (x i w i /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 8 Randomly initialised grid of (weight- vectors: Input: (8,4 (7,9 (,9 (4,9 (, (5, (95,7 (, (4,9 (,9 (8, (, (4,74 (9, (7,5 (75,55 (,4 + (x i w i /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 9 Randomly initialised grid of (weight- vectors: Input: (8,4 minimum distance (7,9 (,9 (4,9 (, (5, (95,7 (, (4,9 (,9 (8, (, (4,74 (9, (7,5 (75,55 (,4 + (x i w i /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 4 Winner Takes All : for closest neuron Step : (7,9 (,9 (4,9 (, Neighbours: is smaller for surrounding Step : (7,9 (,9 (4,9 (, (8,4 (5, (95,7 (, (4,9 (,9 (8,9 (, (4,74 (9, (7,5 (75,55 (,4 (4,77 (9,5 (5, (4,9 (8,8 (8,9 (4, (4,74 (88,9 (78,4 (77,7 (,4 + (x i w i /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 4 + (x i w i /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 4 7

8 Neighbours: gets smaller and smaller Step : (74, (,89 (7,9 (8,7 (4,77 (9,5 (5, (8, (8,8 (8,9 (4, (,9 (88,9 (78,4 (77,7 (,4 Different inputs activate different regions Step : New (74, (,89 (7,9 (8,7 input: (7, (4,77 (9,5 (5, (8, (8,8 (8,9 (4, (,9 (88,9 (78,4 (77,7 (,4 + (x i w i /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 4 + (x i w i /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 44 Different inputs activate different regions Step : New (74, (,79 (,7 (4, input: (7, (44,7 (9,47 (57, (8, (8,8 (79,7 (9, (8,47 (88,9 (7,4 (77,7 (58,7 The map organises along input space Step : New input: (7, (74, (4, (7,9 (4, (5,5 (89,45 (,4 (8, (79,7 (78, (4, (8,47 (88,9 (7,4 (77,7 (58,7 + (x i w i /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 45 + (x i w i /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 4 The Kohonen Map The and are controlled in phases: Self-Organising / Ordering phase: topological ordering of the weight vectors (, large Convergence Phase: fine-tuning of the map (, small Kohonen Map: Applications Detecting structure in input data (NNRC, 997 input large small Input: 9 indicators describing various quality-of-life factors, such as state of health, nutrition, educational services /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 47 /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 48 8

9 Kohonen Map: Applications Traveling Salesman Problem (Fritzke & Wilke, 99 D Ring structure Kohonen Map: Applications Visualising sensor data from different contexts (Van Laerhoven, 999 /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 49 /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 5 The Hopfield Network The Hopfield Network 98: John Hopfield, "Neural networks and physical systems with emergent collective computational abilities Associative Memory: pony Simple threshold units Outputs are - or + (sometimes or : x w w y y = + if (w ix i > i=,,,... otherwise Connections are typically symmetric! States: state neuron new state if fires fires fires /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 5 /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 5 Energy Functions The Hopfield Network Energy function of the network is a scalar value used for finding local minima, regarded as solutions E = w ij s i s j + i s i for Hopfield net i< j i States for: state neuron new state if fires fires fires 4 solutions : memorised states Popular method for neural network design /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 5 /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 54 9

10 Hopfield Network Apps Pattern recognition: Elman Network Jeff Elman 99: Finding structure in time Sequences in the input: implicit representation of time Hidden neuron(s copied Output neuron as an input Hidden neurons Backpropagation as usual Input neurons Context neuron /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 55 /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 5 Elman Network Elman Network Hidden neurons Output neuron Input is given Hidden neurons Output neuron Input is given Hidden neurons functions are calculated Input neurons Context neuron Input neurons Context neuron /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 57 /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 58 Hidden neurons Input neurons Elman Network Output neuron Context neuron Input is given Hidden neurons functions are calculated Output is calculated and hidden neuron s is copied Hidden neurons Input neurons Elman Network Output neuron Context neuron Input is given Hidden neurons functions are calculated Output is calculated and hidden neuron s is copied The next input /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 59 /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks

11 Elman Network Apps Terminology Learning formal grammars Which strings belong to a language? Speech recognition: Recognition of Auditory Sound Sources and Events (Stephen McAdams, 99 Music composition: Neural network music composition by prediction: Exploring the benefits of psychophysical constraints and multiscale processing (Michael Mozer, 994 Activity recognition: Architectures: McCulloch-Pitts Adaptive Resonance Theory Perceptron Multi Perceptron Kohonen Map Hopfield Network Elman Network Learning: Hebbian Delta rule Backpropagation (UnSupervised Competitive Learning Associative Memory Energy /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks Architectures Training perceptron multi perceptron Hopfield net perceptron Delta rule / Gradient descent Supervised multi perceptron Backpropagation Supervised Hopfield net Associative memory Unsupervised Kohonen map Elman network Kohonen map Elman network Competitive Learning Backpropagation Unsupervised Supervised /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 4 Number of Inputs Number of Weights perceptron multi perceptron Hopfield net perceptron multi perceptron Hopfield net Kohonen map Elman network Kohonen map Elman network 5 9 /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 5 /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks

12 Number of Outputs perceptron multi perceptron Hopfield net 4 Further Research Spiking / Pulsed Neural Networks Spike trains : timings of pulses is crucial Closer to biological neuron Action potential p Kohonen map Elman network t, t, t t, /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 7 /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 8 Further Research Principal Component Analysis Dimension reduction Independent Component Analysis Cocktail party problem blind source detection Further Research Support Vector Machines x /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 9 /4/7 Kristof Van Laerhoven - Introduction in Artificial Neural Networks 7