Neural Nets in PR NM P F Outline Motivation: Pattern Recognition XII human brain study complex cognitive tasks Michal Haindl Faculty of Information Technology, KTI Czech Technical University in Prague Institute of Information Theory and Automation Academy of Sciences of the Czech Republic Prague, Czech Republic Evropský sociální fond. Praha & EU: Investujeme do vaší budoucnosti MI-ROZ 2011-2012/Z Neural Nets in PR 2 c M. Haindl MI-ROZ - 12 3/15 NM P F Outline c M. Haindl MI-ROZ - 12 1/15 January 16, 2012 Outline studied since the late Middle Ages structure discovered Santiago Ramon y Cajal (1888) & introduced the idea of neuron chemical transmission of nerve signals Sir Henry Dale (1936 Nobel prize in medicine) electrical signal transmission in the nervous system Sir John Eccles, A. L. Hodgkin, A. Huxley (1963 Nobel prize in medicine) W. McCulloch, W. Pitts (1943) Hebb s learning of McCulloch-Pitts neural network E. Caianiello (1961) c M. Haindl MI-ROZ - 12 4/15 1 Neural Nets in PR Neuron Models Neural Nets Properties Feedback c M. Haindl MI-ROZ - 12 2/15
Neural Nets in PR 2 NM P F Neural Nets in PR 2 NM P F W. McCulloch, W. Pitts (1943) Hebb s learning of McCulloch-Pitts neural network E. Caianiello (1961) perceptron type (feed-forward) of F. Rosenblatt (1960) analogy between McCulloch-Pitts and Ising model W. Little, J. Hopfield (1978,1982) Does the artificial neural net computing paradigm model biology or not? c M. Haindl MI-ROZ - 12 4/15 c M. Haindl MI-ROZ - 12 4/15 Machine System NM P F Neural Nets in PR 2 NM P F rely on speed and accuracy of execution of vast amount of instructions are easily overhelmed by task of exponential or greater complexity (most directly approached PR tasks) search whole system space for acceptable solution limited progress (little previously unknown knowledge) few -based algorithms superior to those previously known sleep (REM) eliminates undesirable memory Crick, Mitchison (1983) W. McCulloch, W. Pitts (1943) Hebb s learning of McCulloch-Pitts neural network E. Caianiello (1961) perceptron type (feed-forward) of F. Rosenblatt (1960) analogy between McCulloch-Pitts and Ising model W. Little, J. Hopfield (1978,1982) c M. Haindl MI-ROZ - 12 5/15 c M. Haindl MI-ROZ - 12 4/15
Neuron Models NM P F Neural Nets NM P F x 1 a i,1 x 2 a i,2. x l a i,l v i = vi f(v i ) y i l a i,j x i j=1 y i = f(v i +ϑ i ) ϑ i bias ϑ i threshold (affine tr. on output) x j inputs c M. Haindl MI-ROZ - 12 7/15 motivation - biological neurones neurocomputers, connectionist networks, parallel distributed processors, intelligent computing nodes neurons links synapses input node output node hidden node (interlayer) Net properties - activation function f, weights a j, net architecture c M. Haindl MI-ROZ - 12 6/15 Neuron Models 2 NM P F Neural Nets NM P F y i = f(v i ϑ i ) y i = f(v i +ϑ i ) motivation - biological neurones neurocomputers, connectionist networks, parallel distributed processors, intelligent computing Net properties - activation function f, weights a j, net architecture Neural nets: Recurrent - feedback, fixed weights, input pattern X as an initial condition, net converges to the closest template stored in its memory. Feed-forward - synaptic connections start with random weights and change during iterative learning, in the classification stage X - input, ω - output. c M. Haindl MI-ROZ - 12 8/15 c M. Haindl MI-ROZ - 12 6/15
NM P F Neuron Models 3 NM P F 1 if v 1 2 f(v) = 1 v 2 > v > 1 2 0 v 1 2 equivalent model x 0 = 1 a i,0=ϑ i x 1 a i,1 x 2 a i,2. a i,l (X 0 = 1) x l v i f(vi ) y i v i = l a i,j x i j=0 y i = f(v i ) x 0 = 1 a i,0 = ϑ i c M. Haindl MI-ROZ - 12 9/15 NM P F NM P F f(v) = 1 1+exp{ av} { 1 if v 0 f(v) = 0 otherwise
Neural Nets Properties NM P F NM P F 1 Each neuron is represented by a set of linear synaptic links, externally applied threshold ϑ i, nonlinear activation link f(). 2 The synaptic links of a neuron weight their resp. input signals. 3 The weighted sum of input signals defines the total internal activity of the neuron. 4 The activation link transforms the internal activity into an output (state variable of the neuron). 1 if v > 0 f(v) = 0 if v = 0 1 if v < 0 c M. Haindl MI-ROZ - 12 11/15 Neural Nets Properties 2 NM P F NM P F massive parallel distributed structure supervised learning nonlinearity (neuron is a nonlinear device) input-output mapping contextual behaviour of neurons fault tolerance (fault neuron impaired quality only) uniformity of analysis and design neurons common to all learning can be shared in different applications seamless integration of models hyperbolic tangent ( v f(v) = tanh = 2) 1 exp{ v} 1+exp{ v} c M. Haindl MI-ROZ - 12 12/15
Signal-Flow Graph NM P F Signal-Flow Graph NM P F 1 rule a signal flows only in the direction links synaptic x j a k,j y k = a k,j x j (linear) activation x j f() y k = f(x j ) (nonlinear) 2 rule signal = signals from incoming links y i ց y j ր y k = y i +y j (synaptic convergence, fan-in) 3 rule signal is transmitted independently on f i () of targets x j ր x j ց x j (synaptic divergence, fan-out) 1 rule a signal flows only in the direction links synaptic x j a k,j y k = a k,j x j (linear) activation x j f() y k = f(x j ) (nonlinear) 2 rule signal = signals from incoming links y i ց y j ր y k = y i +y j (synaptic convergence, fan-in) 3 rule signal is transmitted independently on f i () of targets x j ր x j ց x j (synaptic divergence, fan-out) c M. Haindl MI-ROZ - 12 13/15 c M. Haindl MI-ROZ - 12 13/15 Signal-Flow Graph 2 NM P F Signal-Flow Graph NM P F Neural network model is defined as a directed graph: A real state variable (input signal) x i is associated with each i. A real-valued (synaptic) weight a ik is associated with each link i k. A real-valued biased ϑ i (threshold) is associated with each i. A transfer function f i (x k,a ik,ϑ i,(k i)) is defined for each i, which determines its state as a function of its bias, incoming links weights and states of connected nodes. 1 rule a signal flows only in the direction links synaptic x j a k,j y k = a k,j x j (linear) activation x j f() y k = f(x j ) (nonlinear) 2 rule signal = signals from incoming links y i ց y j ր y k = y i +y j (synaptic convergence, fan-in) 3 rule signal is transmitted independently on f i () of targets x j ր x j ց x j (synaptic divergence, fan-out) c M. Haindl MI-ROZ - 12 14/15 c M. Haindl MI-ROZ - 12 13/15
Feedback NM P F ouput input (recurrent ) single-loop feedback x j (t) x j (t) B y i (t) A y i (t) = A[ x j (t)] z 1 unit delay x j (t) = x j (t)+b[y i (t)] y i (t) = A 1 AB [x j(t)] c M. Haindl MI-ROZ - 12 15/15