Neural Networks. Understanding the Brain

Transcription

1 Neural Neworks Threshold unis Neural Neworks Gradien descen Mulilayer neworks Backpropagaion Hidden layer represenaions Example: Face Recogniion Advanced opics And, more Neworks of processing unis (neurons) wih connecions (synapses) beween hem Large number of neurons: Large conneciiviy: 5 Parallel processing Disribued compuaion/memory Robus o noise, failures Blue slides: from Michell Orange slides: from Alpaydin Lecure Noes for E Alpaydın 24 Inroducion o Machine Learning The MIT Press (V) 3 Undersanding he Brain Levels of analysis (Marr, 982) Compuaional heory 2 Represenaion and algorihm 3 Hardware implemenaion Reverse engineering: From hardware o heory Parallel processing: SIMD vs MIMD Neural ne: SIMD wih modifiable local memory Learning: Updae by raining/experience Biological Neurons and Neworks Neuron swiching ime second ( ms) Number of neurons Connecions per neuron 4 5 Scene recogniion ime second ( ms) processing seps doesn seem like enough [ ] much parallel compuaion 4 2

2 Arificial Neural Neworks Biologically Moivaed (or Accurae) Neural Neworks k w kj oupu Spiking neurons j w ji hidden Complex morphological models i inpu Deailed dynamical models Many neuron-like hreshold swiching unis (real-valued) Many weighed inerconnecions among unis Highly parallel, disribued process Emphasis on uning weighs auomaically: New learning algorihms, new opimizaion echniques, new learning principles Conneciviy eiher based on or rained o mimic biology Focus on modeling nework/neural/subneural processes Focus on naural principles of neural compuaion Differen forms of learning: spike-iming-dependen plasiciy, covariance learning, shor-erm and long-erm plasiciy, ec 3 4 When o Consider Neural Neworks Example Applicaions (more laer) Sharp Lef Sraigh Ahead Sharp Righ Inpu is high-dimensional discree or real-valued (eg raw sensor inpu) Oupu is discree or real valued Oupu is a vecor of values Possibly noisy daa Long raining ime (may need occasional, exensive reraining) Form of arge funcion is unknown Fas evaluaion of learned arge funcion Human readabiliy of resul is unimporan 5 Examples: 4 Hidden Unis (a) ALVINN Speech synhesis 3 Oupu Unis 3x32 Sensor Inpu Reina (b) hp://yannlecuncom Handwrien characer recogniion (from yannlecuncom) Financial predicion, Transacion fraud deecion (Big issue laely) Driving a car on he highway 6

3 Perceprons Hypohesis Space of Perceprons x x 2 x n w w 2 w n x w Σ n Σ wi x i i n if Σ w > o { i x i i - oherwise x x 2 x n w w 2 w n x w Σ n Σ wi x i i n if Σ w > o { i x i i - oherwise 8 < o(x,, x n ) : if w + w x + + w n x n > oherwise The unable parameers are he weighs w, w,, w n, so he space H of candidae hypoheses is he se of all possible combinaion of real-valued weigh vecors: Someimes we ll use simpler vecor noaion: H { w w R (n+) } 8 < o( x) : if w x > oherwise 7 8 Boolean Logic Gaes wih Percepron Unis W W AND OR W NOT W2 W2 Wha Perceprons Can Represen w w W Oupu Slope W W Russel & Norvig Perceprons can represen basic boolean funcions Oupufs Thus, a nework of percepron unis can compue any Boolean funcion Wha abou OR or EQUIV? Perceprons can only represen linearly separable funcions Oupu of he percepron: W I + W I >, hen oupu is W I + W I, hen oupu is The hypohesis space is a collecion of separaing lines 9

4 Geomeric Inerpreaion w w W Oupufs Oupu Slope W W w w The Role of he Bias W Slope W W Rearranging W I + W I >, hen oupu is, we ge (if W > ) I > W W I + W, where poins above he line, he oupu is, and - for hose below he line Compare wih y W x + W W Wihou he bias ( ), learning is limied o adjusmen of he slope of he separaing line passing hrough he origin Three example lines wih differen weighs are shown 2 Limiaion of Perceprons w w W Oupufs Oupu Slope W W x Generalizing o n-dimensions z (x,y,z) n [a b c] T (x,y,z ) y x y z a b c d hp://mahworldwolframcom/planehml Only funcions where he - poins and poins are clearly separable can be represened by perceprons The geomeric inerpreaion is generalizable o funcions of n argumens, ie percepron wih n inpus plus one hreshold (or bias) uni 3 n (a, b, c), x (x, y, z), x (x, y, z ) Equaion of a plane: n ( x x ) In shor, ax + by + cz + d, where a, b, c can serve as he weigh, and d n x as he bias For n-d inpu space, he decision boundary becomes a (n )-D hyperplane (-D less han he inpu space) 4

5 Linear Separabiliy Linear Separabiliy (con d) Linearly separable No Linearly separable For funcions ha ake ineger or real values as argumens and oupu eiher - or Lef: linearly separable (ie, can draw a sraigh line beween he classes) Righ: no linearly separable (ie, perceprons canno represen such a funcion) AND OR OR Perceprons canno represen OR! Minsky and Paper (969)? 5 6 # I I OR OR in Deail w w W Oupufs W I + W I >, hen oupu is : 2 W > W > 3 W > W > 4 W + W W + W Oupu 2 < W + W < (from 2, 3, and 4), bu (from ), a conradicion 7 Slope W W x x 2 x n w w 2 w n Learning: Percepron Rule x w Σ n Σ wi x i i n if Σ w > o { i x i i - oherwise The weighs do no have o be calculaed manually We can rain he nework wih (inpu,oupu) pair according o he following weigh updae rule: w i w i + η( o)x i where η is he learning rae parameer Proven o converge if inpu se is linearly separable and η is small 8

6 Learning in Perceprons (Con d) w i w i + η( o)x i When o, weigh says When and o, change in weigh is: η( ( ))x i > if x i are all posiive Thus w x will increase, hus evenually, oupu o will urn o When and o, change in weigh is: η( )x i < if x i are all posiive Thus w x will decrease, hus evenually, oupu o will urn o - 9 x y Learning in Percepron: Anoher Look w(a,b) q p a + b The percepron on he lef can be represened as a line shown on he righ (why? see page 4) Learning can be hough of as adjusmen of w urning oward he inpu vecor x: w w + η( o) x Adjusmen of he bias moves he line closer or away from he origin 2 y x Anoher Learning Rule: Dela Rule The percepron rule canno deal wih noisy daa Gradien Descen The dela rule will find an approximae soluion even when inpu se is no linearly separable E[w] 5 2 Use linear uni wihou he sep funcion: o( x) w x Wan o reduce he error by adjusing w: E( w) 2 2 ( d o d ) 2 d D Wan o minimize by adjusing w: E( w) 2 w w Pd D ( d o d ) 2 Noe: he error surface is defined by he raining daa D A differen daa se will give a differen surface E(w, w ) is he error funcion above, and we wan o change (w, w ) o posiion under a low E

7 Gradien Descen (Con d) Gradien Descen (Example) Gradien line 2-2 Training rule: E[ w]» E w, E w, w η E[ w] E w n ie, w i η E Gradien poins in he maximum increasing direcion Gradien is prependicular o he level curve (uphill direcion) E(w, w ) is he error funcion above, so E ( E, E ), a vecor on a 2D plane w w E E Gradien Descen (Con d) 2 2 d d 2 d d ( d o d ) 2 d ( d o d ) 2 2( d o d ) ( d o d ) ( d o d ) ( d w x d ) ( d o d )( x i,d ) Gradien Descen: Summary Gradien-Descen (raining examples, η) Each raining example is a pair of he form x,, where x is he vecor of inpu values, and is he arge oupu value η is he learning rae (eg, 5) Iniialize each w i o some small random value Unil he erminaion condiion is me, Do Iniialize each w i o zero For each x, in raining examples, Do Inpu he insance x o he uni and compue he oupu o For each linear uni weigh w i, Do For each linear uni weigh w i, Do w i w i + η( o)x i Since we wan w i η E, w i η P d ( d o d )x i,d 25 w i w i + w i 26

8 Gradien Descen Properies Gradien descen is effecive in searching hrough a large or infinie H: H conains coninuously parameerized hypoheses, and he error can be differeniaed wr he parameers Limiaions: Sochasic Approximaion o Grad Desc Avoiding local minima: Incremenal gradien descen, or sochasic gradien descen Insead of weigh updae based on all inpu in D, immediaely updae weighs afer each inpu example: w i η( o)x i, convergence can be slow, and finds local minima (global minumum no guaraneed) insead of w i η d D( d o d )x i, Can be seen as minimizing error funcion E d ( w) 2 ( d o d ) Sandard and Sochasic Grad Desc: Differences Summary In he sandard version, error is defined over enire D In he sandard version, more compuaion is needed per weigh updae, bu η can be larger Sochasic version can someimes avoid local minima Percepron raining rule guaraneed o succeed if Training examples are linearly separable Sufficienly small learning rae η Linear uni raining rule using gradien descen Asympoic convergence o hypohesis wih minimum squared error Given sufficienly small learning rae η Even when raining daa conains noise Even when raining daa no separable by H 29 3

9 Exercise: Implemening he Percepron x w x Mulilayer Neworks I is fairly easy o implemen a percepron You can implemen i in any programming language: C/C++, ec Look for examples on he web, and JAVA apple demos x 2 x n w 2 w n w Σ n ne Σ w i x i i Differeniable hreshold uni: sigmoid σ(y) + exp( y) o σ(ne) + ē ne Ineresing propery: dσ(y) dy Oupu: σ(y)( σ(y)) o σ( w x) Oher funcions: anh(y) exp( 2y) exp( 2y) Mulilayer Neworks and Backpropagaion head hid who d hood Error Gradien for a Sigmoid Uni F F2 Nonlinear decision surfaces Oupu Inpu sigm(x+y-) (a) One oupu Anoher example: OR 6 8 Inpu Oupu 2 4 Inpu sigm(sigm(x+y-)+sigm(-x-y+3)-) Inpu 2 (b) Two hidden, one oupu E 2 2 d ( d o d ) 2 d D ( d o d ) 2 2( d o d ) 2 d d d ( d o d ) ( d o d ) «o d o d ( d o d ) ne d 34 ne d

10 From he previous page: Bu we know: So: Error Gradien for a Sigmoid Uni E E d o d ( d o d ) ne d ne d o d σ(ne d) o d ( o d ) ne d ne d ne d ( w x d) d D x i,d ( d o d )o d ( o d )x i,d Backpropagaion Algorihm Iniialize all weighs o small random numbers Unil saisfied, Do For each raining example, Do Inpu he raining example o he nework and compue he nework oupus 2 For each oupu uni k δ k o k ( o k )( k o k ) 3 For each hidden uni h δ h o h ( o h ) P k oupus w khδ k 4 Updae each nework weigh w i,j w ji w ji + w ji where w ji ηδ j x i Noe: w ji is he weigh from i o j (ie, w j i ) For oupu uni: For hidden uni: The δ Term δ k o k ( o k ) ( k o k ) {z } {z } σ (ne k ) Error δ h o h ( o h ) w kh δ k {z } σ k oupus (ne h ) {z } Backpropagaed error In sum, δ is he derivaive imes he error Derivaion o be presened laer Wan o updae weigh as: where error is defined as: Derivaion of w E d ( w) 2 Given ne j P j w jix i, w ji η, k oupus Differen formula for oupu and hidden ( k o k )

11 Derivaion of w: Oupu Uni Weighs From he previous page, Firs, calculae : 2 k oupus 2 ( j o j ) 2 ( k o k ) ( j o j ) ( j o j ) ( j o j ) 39 Derivaion of w: Oupu Uni Weighs From he previous page, Nex, calculae ( j o j ) : : Since o j σ(ne j ), and σ (ne j ) o j ( o j ), Puing everyhing ogeher, o j ( o j ) ( j o j )o j ( o j ) 4 Derivaion of w: Oupu Uni Weighs From he previous page: Since ( j o j )o j ( o j ) P k w jkx k x, ji ( j o j )o j ( o j ) {z } δ j error σ (ne) x i {z} inpu Derivaion of w: Hidden Uni Weighs Sar wih x i : k Downsream(j) k Downsream(j) k Downsream(j) k Downsream(j) k Downsream(j) ne k ne k δ k ne k δ k ne k δ k w kj δ k w kj o j ( o j ) {z } σ (ne) () 4 42

12 Finally, given and Derivaion of w: Hidden Uni Weighs x i, k Downsream(j) w ji η η [o j ( o j ) {z } σ (ne) δ k w kj o j ( o j ), {z } σ (ne) k Downsream(j) δ k w kj ] {z } error {z } δ j x i Exension o Differen Nework Topologies k w kj j w ji i oupu hidden inpu Arbirary number of layers: for neurons in layer m: δ r o r ( o r ) Arbirary acyclic graph: δ r o r ( o r ) s layer m+ w sr δs w sr δs s Downsream(r) Backpropagaion: Properies Gradien descen over enire nework weigh vecor Easily generalized o arbirary direced graphs Will find a local, no necessarily global error minimum: In pracice, ofen works well (can run muliple imes wih differen iniial weighs) Ofen include weigh momenum α w i,j (n) ηδ j x i,j + α w i,j (n ) Represenaional Power of Feedforward Neworks Boolean funcions: every boolean funcion represenable wih wo layers (hidden uni size can grow exponenially in he wors case: one hidden uni per inpu example, and OR hem) Coninous funcions: Every bounded coninuous funcion can be approximaed wih an arbirarily small error (oupu unis are linear) Arbirary funcions: wih hree layers (oupu unis are linear) Minimizes error over raining examples: Will i generalize well o subsequen examples? Training can ake housands of ieraions slow! Using he nework afer raining is very fas 45 46

13 H-Space Search and Inducive Bias Learning Hidden Layer Represenaions H-space n-d weigh space (when here are n weighs) The space is coninuous, unlike decision ree or general-o-specific concep learning algorihms Inducive bias: Smooh inerpolaion beween daa poins Inpus Oupus Inpu Oupu Learned Hidden Layer Represenaions Learned Hidden Layer Represenaions Inpus Oupus Inpu Hidden Oupu Values Learned encoding is similar o sandard 3-bi binary code Auomaic discovery of useful hidden layer represenaions is a key feaure of ANN Noe: The hidden layer represenaion is compressed 5

14 Error Error versus weigh updaes (example ) Training se error Validaion se error Number of weigh updaes Overfiing Error Error versus weigh updaes (example 2) Training se error Validaion se error Number of weigh updaes Penalize large weighs: E( w) 2 Alernaive Error Funcions d D k oupus( kd o kd ) 2 + γ i,j w 2 ji Train on arge slopes as well as values (when he slope is available): Error in wo differen robo percepion asks Training se and validaion se error Early sopping ensures good performance on unobserved samples, bu mus be careful Weigh decay, use of validaion ses, use of k-fold cross-validaion, ec o overcome he problem 2 E( w) 4( kd o kd ) 2 + kd 2 d D k oupus j inpus x j d Tie ogeher weighs: eg, in phoneme recogniion nework, or handwrien characer recogniion (weigh sharing) o kd x j d A Recurren Neworks Recurren Neworks (Con d) oupu hidden delay Sequence recogniion Sore ree srucure (nex slide) Can be rained wih plain inpu sack inpu, sack inpu sack delay A A (A, B) B B C (A, B) (C, A, B) C (A, B) delay inpu conex backpropagaion Generalizaion may no be perfec Auoassociaion (inpu oupu) Represen a sack using he hidden layer represenaion Accuracy depends on numerical precision 53 54

15 36 37 Learning Time Time-Delay Neural Neworks Applicaions: Sequence recogniion: Speech recogniion Sequence reproducion: Time-series predicion Sequence associaion Nework archiecures Time-delay neworks (Waibel e al, 989) Recurren neworks (Rumelhar e al, 986) ecure Noes for E Alpaydın 24 Inroducion o Machine Learning The MIT Press (V) 34 Lecure Noes for E Alpaydın 24 Inroducion o Machine Learning The MIT Press (V) 35 Recurren Neworks Unfolding in Time

16 Some Applicaions: NETalk NETalk: Sejnowski and Rosenberg (987) Learn o pronounce English ex Demo Daa available in UCI ML reposiory NETalk daa aardvark a-rdvark <<<>2<< aback xb@k-><< <>< abaf xb@f ><< 2<>> abandon xb@ndxn ><>< abase xbes-><< abash xb@s-><< abae xbe-><< Word Pronunciaion Sress/Syllable abou 2, words Backpropagaion Exercise URL: hp://wwwcsamuedu/faculy/choe/src/backprop-6argz Unar and read he README file: gzip -dc backprop-6argz ar xvf - Run make o build (on deparmenal unix machines) Run /bp conf/xorconf ec Backpropagaion: Example Resuls Error Backprop OR AND OR , Epochs Epoch: one full cycle of raining hrough all raining inpu paerns OR was easies, AND he nex, and OR was he mos difficul o learn Nework had 2 inpu, 2 hidden and oupu uni Learning rae was 57 58

17 27 28 Backpropagaion: Example Resuls (con d) Backpropagaion: Things o Try Error Backprop OR AND OR OR, Epochs AND How does increasing he number of hidden layer unis affec he () ime and he (2) number of epochs of raining? How does increasing or decreasing he learning rae affec he rae of convergence? How does changing he slope of he sigmoid affec he rae of convergence? Differen problem domains: handwriing recogniion, ec Oupu o (,), (,), (,), and (,) form each row OR 59 6 Srucured MLP Weigh Sharing (Le Cun e al, 989)

18 Tuning he Nework Size Bayesian Learning Desrucive Weigh decay: w i E' E E η w λ + 2 i i w λw 2 i i Consrucive Growing neworks (Ash, 989) (Fahlman and Lebiere, 989) Consider weighs w i as random vars, prior p(w i ) p ( w ) p p logp ( w ) p( w ) wˆ MAP arg max p( ) w ( w ) logp( w ) + logp( w ) ( w ) p( wi ) where p( wi ) i E' E + λ w ( w ) 2 w i c exp 2( / 2λ) Weigh decay, ridge regression, regularizaion cosdaa-misfi + λ complexiy 2 logp + C ecure Noes for E Alpaydın 24 Inroducion o Machine Learning The MIT Press (V) 3 Lecure Noes for E Alpaydın 24 Inroducion o Machine Learning The MIT Press (V) 3 Summary ANN learning provides general mehod for learning real-valued funcions over coninuous or discree-valued aribued ANNs are robus o noise H is he space of all funcions parameerized by he weighs H space search is hrough gradien descen: convergence o local minima Backpropagaion gives novel hidden layer represenaions Overfiing is an issue More advanced algorihms exis 6