Neural Networks. Understanding the Brain

Transcription

1 Threshold uns Graden descen Mullayer neworks Backpropagaon Hdden layer represenaons Example: Face Recognon Advanced opcs Neural Neworks Neural Neworks Neworks of processng uns (neurons) wh connecons (synapses) beween hem Large number of neurons: 1 1 Large connecvy: 1 5 Parallel processng Dsrbued compuaon/memory Robus o nose, falures And, more Blue sldes: from Mchell Turquose sldes: from Alpaydn 1 Lecure Noes for E Alpaydın 21 Inroducon o Machne Learnng 2e The MIT Press (V1) 3 Undersandng he Bran Levels of analyss (Marr, 1982) 1 Compuaonal heory 2 Represenaon and algorhm 3 Hardware mplemenaon Reverse engneerng: From hardware o heory Parallel processng: SIMD vs MIMD Neural ne: SIMD wh modfable local memory Learnng: Updae by ranng/experence Bologcal Neurons and Neworks Neuron swchng me 1 second (1 ms) Number of neurons 1 1 Connecons per neuron Scene recognon me 1 second (1 ms) 1 processng seps doesn seem lke enough [ ] much parallel compuaon Lecure Noes for E Alpaydın 21 Inroducon o Machne Learnng 2e The MIT Press (V1) 4 2

2 Arfcal Neural Neworks Bologcally Movaed (or Accurae) Neural Neworks k w kj oupu Spkng neurons j w j hdden Complex morphologcal models npu Dealed dynamcal models Many neuron-lke hreshold swchng uns (real-valued) Many weghed nerconnecons among uns Hghly parallel, dsrbued process Emphass on unng weghs auomacally: New learnng algorhms, new opmzaon echnques, new learnng prncples Connecvy eher based on or raned o mmc bology Focus on modelng nework/neural/subneural processes Focus on naural prncples of neural compuaon Dfferen forms of learnng: spke-mng-dependen plascy, covarance learnng, shor-erm and long-erm plascy, ec 3 4 When o Consder Neural Neworks Example Applcaons (more laer) Sharp Lef Sragh Ahead Sharp Rgh Inpu s hgh-dmensonal dscree or real-valued (eg raw sensor npu) Oupu s dscree or real valued Oupu s a vecor of values Possbly nosy daa Long ranng me (may need occasonal, exensve reranng) Form of arge funcon s unknown Fas evaluaon of learned arge funcon Human readably of resul s unmporan 5 Examples: 4 Hdden Uns (a) ALVINN Speech synhess 3 Oupu Uns 3x32 Sensor Inpu Rena (b) hp://yannlecuncom Handwren characer recognon (from yannlecuncom) Fnancal predcon, Transacon fraud deecon (Bg ssue laely) Drvng a car on he hghway 6

3 Perceprons Hypohess Space of Perceprons x 1 x 2 x n w 1 w 2 w n x 1 w Σ n Σ w x n 1 f Σ w > o { x -1 oherwse x 1 x 2 x n w 1 w 2 w n x 1 w Σ n Σ w x n 1 f Σ w > o { x -1 oherwse o(x 1,, x n ) 1 1 f w + w 1 x w n x n > oherwse The unable parameers are he weghs w, w 1,, w n, so he space H of canddae hypoheses s he se of all possble combnaon of real-valued wegh vecors: Somemes we ll use smpler vecor noaon: H { w w R (n+1) } o( x) 1 1 f w x > oherwse 7 8 Boolean Logc Gaes wh Percepron Uns AND OR 1 NOT W21 W21 1 Wha Perceprons Can Represen w w1 Oupu 1 Slope W Russel & Norvg npu: { 1, 1} Oupufs Perceprons can represen basc boolean funcons Thus, a nework of percepron uns can compue any Boolean funcon Wha abou XOR or EQUIV? Perceprons can only represen lnearly separable funcons Oupu of he percepron: W I + W 1 I 1 >, hen oupu s 1 W I + W 1 I 1, hen oupu s 1 The hypohess space s a collecon of separang lnes 9 1

4 1 Geomerc Inerpreaon w w1 Oupufs Oupu 1 Slope W 1 w w1 The Role of he Bas Slope W Rearrangng W I + W 1 I 1 >, hen oupu s 1, we ge (f W 1 > ) I 1 > W W 1 I + W 1, where pons above he lne, he oupu s 1, and -1 for hose below he lne Compare wh y W x + W 1 11 W 1 Whou he bas ( ), learnng s lmed o adjusmen of he slope of he separang lne passng hrough he orgn Three example lnes wh dfferen weghs are shown 12 1 Lmaon of Perceprons w w1 Oupufs Oupu 1 Slope W x Generalzng o n-dmensons z (x,y,z) 1 n [a b c] T (x,y,z ) y x y z a b c d hp://mahworldwolframcom/planehml Only funcons where he -1 pons and 1 pons are clearly separable can be represened by perceprons The geomerc nerpreaon s generalzable o funcons of n argumens, e percepron wh n npus plus one hreshold (or bas) un 13 n (a, b, c), x (x, y, z), x (x, y, z ) Equaon of a plane: n ( x x ) In shor, ax + by + cz + d, where a, b, c can serve as he wegh, and d n x n n as he bas For n-d npu space, he decson boundary becomes a (n 1)-D hyperplane (1-D less han he npu space) 14

5 Lnear Separably Lnear Separably (con d) Lnearly separable No Lnearly separable For funcons ha ake neger or real values as argumens and oupu eher -1 or 1 Lef: lnearly separable (e, can draw a sragh lne beween he classes) Rgh: no lnearly separable (e, perceprons canno represen such a funcon) AND OR XOR Perceprons canno represen XOR! Mnsky and Paper (1969) 1? # I I 1 XOR XOR n Deal 1 w w1 Oupufs W I + W 1 I 1 >, hen oupu s 1: 1 2 W 1 > W 1 > 3 W > W > 4 W + W 1 W + W 1 Oupu 1 2 < W + W 1 < (from 2, 3, and 4), bu (from 1), a conradcon 17 Slope W x 1 x 2 x n w 1 w 2 w n Learnng: Percepron Rule x 1 w Σ n Σ w x n 1 f Σ w > o { x -1 oherwse The weghs do no have o be calculaed manually We can ran he nework wh (npu,oupu) par accordng o he followng wegh updae rule: w w + η( o)x where η s he learnng rae parameer Proven o converge f npu se s lnearly separable and η s small 18

6 Learnng n Perceprons (Con d) w w + η( o)x When o, wegh says When 1 and o 1, change n wegh s: η(1 ( 1))x > f x are all posve Thus w x wll ncrease, hus evenually, oupu o wll urn o 1 When 1 and o 1, change n wegh s: η( 1 1)x < f x are all posve Thus w x wll decrease, hus evenually, oupu o wll urn o x y Learnng n Percepron: Anoher Look w(a,b) q p a + b The percepron on he lef can be represened as a lne shown on he rgh (why? see page 14) Learnng can be hough of as adjusmen of w urnng oward he npu vecor x: w w + η( o) x Adjusmen of he bas moves he lne closer or away from he orgn 2 y x Anoher Learnng Rule: Dela Rule The percepron rule canno deal wh nosy daa Graden Descen The dela rule wll fnd an approxmae soluon even when npu se s no lnearly separable E[w] Use lnear un whou he sep funcon: o( x) w x Wan o reduce he error by adjusng w: E( w) ( d o d ) 2 d D Wan o mnmze by adjusng w: E( w) w w1 d D ( d o d ) 2 Noe: he error surface s defned by he ranng daa D A dfferen daa se wll gve a dfferen surface E(w, w 1 ) s he error funcon above, and we wan o change (w, w 1 ) o poson under a low E

7 Graden Descen (Con d) Graden Descen (Example) Graden lne Tranng rule: E[ w] [ E w, E w 1, w η E[ w] E w n ] e, w η E Graden pons n he maxmum ncreasng drecon Graden s prependcular o he level curve (uphll drecon) E(w, w 1 ) s he error funcon above, so E ( E, E ), a vecor on a 2D plane w w E E Graden Descen (Con d) d d 1 2 d d ( d o d ) 2 d ( d o d ) 2 2( d o d ) ( d o d ) ( d o d ) ( d w x d ) ( d o d )( x,d ) Graden Descen: Summary Graden-Descen (ranng examples, η) Each ranng example s a par of he form x,, where x s he vecor of npu values, and s he arge oupu value η s he learnng rae (eg, 5) Inalze each w o some small random value Unl he ermnaon condon s me, Do Inalze each w o zero For each x, n ranng examples, Do Inpu he nsance x o he un and compue he oupu o For each lnear un wegh w, Do For each lnear un wegh w, Do w w + η( o)x Snce we wan w η E, w η d ( d o d )x,d 25 w w + w 26

8 Graden Descen Properes Graden descen s effecve n searchng hrough a large or nfne H: H conans connuously parameerzed hypoheses, and he error can be dfferenaed wr he parameers Lmaons: Sochasc Approxmaon o Grad Desc Avodng local mnma: Incremenal graden descen, or sochasc graden descen Insead of wegh updae based on all npu n D, mmedaely updae weghs afer each npu example: w η( o)x, convergence can be slow, and fnds local mnma (global mnumum no guaraneed) nsead of w η d D( d o d )x, Can be seen as mnmzng error funcon E d ( w) 1 2 ( d o d ) Sandard and Sochasc Grad Desc: Dfferences Summary In he sandard verson, error s defned over enre D In he sandard verson, more compuaon s needed per wegh updae, bu η can be larger Sochasc verson can somemes avod local mnma Percepron ranng rule guaraneed o succeed f Tranng examples are lnearly separable Suffcenly small learnng rae η Lnear un ranng rule usng graden descen Asympoc convergence o hypohess wh mnmum squared error Gven suffcenly small learnng rae η Even when ranng daa conans nose Even when ranng daa no separable by H 29 3

9 Exercse: Implemenng he Percepron x 1 w 1 x 1 Mullayer Neworks I s farly easy o mplemen a percepron You can mplemen n any programmng language: C/C++, ec Look for examples on he web, and JAVA apple demos x 2 x n w 2 w n w Σ n ne Σ w x Dfferenable hreshold un: sgmod σ(y) exp( y) o σ(ne) ē ne Ineresng propery: dσ(y) dy Oupu: σ(y)(1 σ(y)) o σ( w x) Oher funcons: anh(y) exp( 2y) 1 exp( 2y) Mullayer Neworks and Backpropagaon head hd who d hood Error Graden for a Sgmod Un F1 F2 Nonlnear decson surfaces Oupu Inpu sgm(x+y-11) (a) One oupu Anoher example: XOR Inpu Oupu 2 4 Inpu 1 sgm(sgm(x+y-11)+sgm(-x-y+113)-1) Inpu 2 (b) Two hdden, one oupu E d ( d o d ) 2 d D ( d o d ) 2 1 2( d o d ) 2 d d d ( d o d ) ( d o d ) ) ( o d o d ( d o d ) ne d 34 ne d

10 From he prevous page: Bu we know: So: Error Graden for a Sgmod Un E E d o d ( d o d ) ne d ne d o d σ(ne d) o d (1 o d ) ne d ne d ne d ( w x d) d D x,d ( d o d )o d (1 o d )x,d Backpropagaon Algorhm Inalze all weghs o small random numbers Unl sasfed, Do For each ranng example, Do 1 Inpu he ranng example o he nework and compue he nework oupus 2 For each oupu un k δ k o k (1 o k )( k o k ) 3 For each hdden un h δ h o h (1 o h ) k oupus w khδ k 4 Updae each nework wegh w,j w j w j + w j where w j ηδ j x Noe: w j s he wegh from o j (e, w j ) For oupu un: For hdden un: The δ Term δ k o k (1 o k ) ( k o k ) }{{}}{{} σ (ne k ) Error δ h o h (1 o h ) w kh δ k }{{} σ k oupus (ne h ) }{{} Backpropagaed error In sum, δ s he dervave mes he error Dervaon o be presened laer Wan o updae wegh as: where error s defned as: Dervaon of w E d ( w) 1 2 Gven ne j j w jx, w j η, k oupus Dfferen formula for oupu and hdden ( k o k )

11 Dervaon of w: Oupu Un Weghs From he prevous page, Frs, calculae : 1 2 k oupus 1 2 ( j o j ) 2 ( k o k ) ( j o j ) ( j o j ) ( j o j ) 39 Dervaon of w: Oupu Un Weghs From he prevous page, Nex, calculae ( j o j ) : : Snce o j σ(ne j ), and σ (ne j ) o j (1 o j ), Pung everyhng ogeher, o j (1 o j ) ( j o j )o j (1 o j ) 4 Dervaon of w: Oupu Un Weghs From he prevous page: Snce ( j o j )o j (1 o j ) k w jkx k x, j ( j o j )o j (1 o j ) }{{} δ j error σ (ne) x }{{} npu Dervaon of w: Hdden Un Weghs Sar wh x : k Downsream(j) k Downsream(j) k Downsream(j) k Downsream(j) k Downsream(j) ne k ne k δ k ne k δ k ne k δ k w kj δ k w kj o j (1 o j ) }{{} σ (ne) (1) 41 42

12 Fnally, gven and Dervaon of w: Hdden Un Weghs x, k Downsream(j) w j η η [o j (1 o j ) }{{} σ (ne) δ k w kj o j (1 o j ), }{{} σ (ne) k Downsream(j) δ k w kj ] }{{} error } {{ } δ j x Exenson o Dfferen Nework Topologes k w kj j w j oupu hdden npu Arbrary number of layers: for neurons n layer m: δ r o r (1 o r ) Arbrary acyclc graph: δ r o r (1 o r ) s layer m+1 w sr δs w sr δs s Downsream(r) Backpropagaon: Properes Graden descen over enre nework wegh vecor Easly generalzed o arbrary dreced graphs Wll fnd a local, no necessarly global error mnmum: In pracce, ofen works well (can run mulple mes wh dfferen nal weghs) Ofen nclude wegh momenum α w,j (n) ηδ j x,j + α w,j (n 1) Represenaonal Power of Feedforward Neworks Boolean funcons: every boolean funcon represenable wh wo layers (hdden un sze can grow exponenally n he wors case: one hdden un per npu example, and OR hem) Connous funcons: Every bounded connuous funcon can be approxmaed wh an arbrarly small error (oupu uns are lnear) Arbrary funcons: wh hree layers (oupu uns are lnear) Mnmzes error over ranng examples: Wll generalze well o subsequen examples? Tranng can ake housands of eraons slow! Usng he nework afer ranng s very fas 45 46

13 H-Space Search and Inducve Bas Learnng Hdden Layer Represenaons H-space n-d wegh space (when here are n weghs) The space s connuous, unlke decson ree or general-o-specfc concep learnng algorhms Inducve bas: Smooh nerpolaon beween daa pons Inpus Oupus Inpu Oupu Learned Hdden Layer Represenaons Learned Hdden Layer Represenaons Inpus Oupus Inpu Hdden Oupu Values Learned encodng s smlar o sandard 3-b bnary code Auomac dscovery of useful hdden layer represenaons s a key feaure of ANN Noe: The hdden layer represenaon s compressed 5

14 Error Error versus wegh updaes (example 1) Tranng se error Valdaon se error Number of wegh updaes Overfng Error Error versus wegh updaes (example 2) Tranng se error Valdaon se error Number of wegh updaes Penalze large weghs: E( w) 1 2 Alernave Error Funcons d D k oupus( kd o kd ) 2 + γ,j w 2 j Tran on arge slopes as well as values (when he slope s avalable): Error n wo dfferen robo percepon asks Tranng se and valdaon se error Early soppng ensures good performance on unobserved samples, bu mus be careful Wegh decay, use of valdaon ses, use of k-fold cross-valdaon, ec o overcome he problem E( w) 1 ( kd o kd ) 2 + µ kd 2 d D k oupus j npus x j d Te ogeher weghs: eg, n phoneme recognon nework, or handwren characer recognon (wegh sharng) o kd x j d Recurren Neworks Recurren Neworks (Con d) oupu hdden delay Sequence recognon Sore ree srucure (nex slde) Can be raned wh plan npu sack npu, sack npu sack delay A A (A, B) B B C (A, B) (C, A, B) C (A, B) delay npu conex backpropagaon Generalzaon may no be perfec Auoassocaon (npu oupu) Represen a sack usng he hdden layer represenaon Accuracy depends on numercal precson 53 54

15 Learnng Tme Tme-Delay Neural Neworks Applcaons: Sequence recognon: Speech recognon Sequence reproducon: Tme-seres predcon Sequence assocaon Nework archecures Tme-delay neworks (Wabel e al, 1989) Recurren neworks (Rumelhar e al, 1986) Lecure Noes for E Alpaydın 21 Inroducon o Machne Learnng 2e The MIT Press (V1) 34 Lecure Noes for E Alpaydın 21 Inroducon o Machne Learnng 2e The MIT Press (V1) 35 Recurren Neworks Unfoldng n Tme Lecure Noes for E Alpaydın 21 Inroducon o Machne Learnng 2e The MIT Press (V1) 36 Lecure Noes for E Alpaydın 21 Inroducon o Machne Learnng 2e The MIT Press (V1) 37

16 Some Applcaons: NETalk NETalk: Sejnowsk and Rosenberg (1987) Learn o pronounce Englsh ex Demo Daa avalable n UCI ML reposory NETalk daa aardvark a-rdvark 1<<<>2<< aback xb@k->1<< 1<>< abaf xb@f >1<< 2<>1> abandon xb@ndxn >1<>< abase xbes->1<< abash xb@s->1<< abae xbe->1<< Word Pronuncaon Sress/Syllable abou 2, words Backpropagaon Exercse URL: hp://wwwcsamuedu/faculy/choe/src/backprop-16argz Unar and read he README fle: gzp -dc backprop-16argz ar xvf - Run make o buld (on deparmenal unx machnes) Run /bp conf/xorconf ec Backpropagaon: Example Resuls Error Backprop OR AND XOR , Epochs Epoch: one full cycle of ranng hrough all ranng npu paerns OR was eases, AND he nex, and XOR was he mos dffcul o learn Nework had 2 npu, 2 hdden and 1 oupu un Learnng rae was

17 Backpropagaon: Example Resuls (con d) Backpropagaon: Thngs o Try Error Backprop OR AND XOR OR 1, Epochs AND How does ncreasng he number of hdden layer uns affec he (1) me and he (2) number of epochs of ranng? How does ncreasng or decreasng he learnng rae affec he rae of convergence? How does changng he slope of he sgmod affec he rae of convergence? Dfferen problem domans: handwrng recognon, ec Oupu o (,), (,1), (1,), and (1,1) form each row XOR 59 6 Srucured MLP Wegh Sharng (Le Cun e al, 1989) Lecure Noes for E Alpaydın 21 Inroducon o Machne Learnng 2e The MIT Press (V1) 27 Lecure Noes for E Alpaydın 21 Inroducon o Machne Learnng 2e The MIT Press (V1) 28

18 Tunng he Nework Sze Desrucve Consrucve Wegh decay: Growng neworks w E w E' E 2 w w 2 (Ash, 1989) (Fahlman and Lebere, 1989) Bayesan Learnng Consder weghs w as random vars, pror p(w ) p X w p w p w X wˆ MAP argmaxlog p w X p X w log p w X log p X w log p w C p w p w where p w E' E w 2 w c exp ( / ) Wegh decay, rdge regresson, regularzaon cosdaa-msf + λ complexy More abou Bayesan mehods n chaper 14 2 Lecure Noes for E Alpaydın 21 Inroducon o Machne Learnng 2e The MIT Press (V1) 3 Lecure Noes for E Alpaydın 21 Inroducon o Machne Learnng 2e The MIT Press (V1) 31 Summary ANN learnng provdes general mehod for learnng real-valued funcons over connuous or dscree-valued arbued ANNs are robus o nose H s he space of all funcons parameerzed by he weghs H space search s hrough graden descen: convergence o local mnma Backpropagaon gves novel hdden layer represenaons Overfng s an ssue More advanced algorhms exs 61