Supervised Learning! B." Neural Network Learning! Typical Artificial Neuron! Feedforward Network! Typical Artificial Neuron! Equations!

Transcription

1 Part 4B: Neura Networ earg 10/22/08 Superved earg B. Neura Networ earg Produce dered output for trag put Geeraze reaoaby appropratey to other put Good exampe: patter recogto Feedforward mutayer etwor 10/22/ /22/08 2 Feedforward Networ Typca Artfca Neuro coecto weght put output put ayer hdde ayer output ayer threhod 10/22/ /22/08 4 Typca Artfca Neuro ear combato actvato fucto et put (oca fed Net put: Neuro output: Euato h = h = * = h = h w j j * ' 10/22/ /22/08 6 1

2 Part 4B: Neura Networ earg 10/22/08 Sge-ayer Perceptro Varabe x 1 w 1 w j w h y x 10/22/ /22/08 8 Sge ayer Perceptro Euato Bary threhod actvato fucto : ( h = ( h = 1, f h > 0 ' 0, f h 0 j Hece, y = 1, f w j > 0, otherwe 1, f w' x > = 0, f w' x ( 10/22/08 9 w x = w x co co = v x w x = w v w x > w v > v > w 2D eght Vector w 2 10/22/08 10 v w x + w w 1 N-Dmeoa eght Vector eparatg hyperpae + w orma vector Goa of Perceptro earg Suppoe we have trag patter x 1, x 2,, x P wth correpodg dered output y 1, y 2,, y P where x p {0, 1}, y p {0, 1} e wat to fd w, uch that y p = (w'x p for p = 1,, P 10/22/ /22/

3 Part 4B: Neura Networ earg 10/22/08 Treatg Threhod a eght x 1 x w 1 w j w 10/22/08 13 h h = w j * ' = * + w j y Treatg Threhod a eght x 0 = 1 x 1 x w 1 = w 0 w j w 10/22/08 14 h h = w j * ' = * + w j y et x 0 = 1 ad w 0 = h = w 0 x 0 + w j = w j = w x j= 0 Augmeted Vector ( w w = 1 ( ( ( ' w 1 p( x x p 1 = ( ( p( x ' e wat y p = ( w x p, p =1,,P 10/22/08 15 Reformuato a Potve Exampe e have potve (y p =1 ad egatve (y p = 0 exampe at w x p > 0 for potve, w x p 0 for egatve et z p = x p for potve, z p = x p for egatve at w z p 0, for p =1,,P Hyperpae through org wth a z p o oe de 10/22/08 16 Adjutmet of eght Vector Oute of Perceptro earg Agorthm z 2 z 5 z 6 z 9 z 10 z 11 z 3 z 1 z 8 z 4 1. taze weght vector radomy 2. ut a patter cafed correcty, do: a for p = 1,, P do: z 7 1 f z p cafed correcty, do othg 2 ee adjut weght vector to be coer to correct cafcato 10/22/ /22/

4 Part 4B: Neura Networ earg 10/22/08 eght Adjutmet w z p p z w w z p Improvemet Performace If w z p < 0, z p w z p = w + z p = w z p + z p z p = w z p + z p 2 > w z p 10/22/ /22/08 20 Perceptro earg Theorem If there a et of weght that w ove the probem, the the PA w evetuay fd t (for a uffcety ma earg rate Note: oy appe f potve egatve exampe are eary eparabe Netogo Smuato of Perceptro earg Ru Perceptro.ogo 10/22/ /22/08 22 Cafcato Power of Mutayer Perceptro Perceptro ca fucto a ogc gate Therefore MP ca form terecto, uo, dfferece of eary-eparabe rego Cae ca be arbtrary hyperpoyhedra My Papert crtcm of perceptro No oe ucceeded deveopg a MP earg agorthm 10/22/08 23 Credt Agmet Probem How do we adjut the weght of the hdde ayer? put ayer hdde ayer output ayer Dered output 10/22/

5 Part 4B: Neura Networ earg 10/22/08 Netogo Demotrato of Bac-Propagato earg Adaptve Sytem Sytem Evauato Fucto (Fte, Fgure of Mert S F Ru Artfca Neura Net.ogo 10/22/08 25 P 1 P P m Cotro Parameter Cotro Agorthm C 10/22/08 26 Gradet Gradet Acet o Fte Surface F P meaure how F atered by varato of P F = F P1 F P F Pm ' ( F pot drecto of maxmum creae F + gradet acet (F 10/22/ /22/08 28 Gradet Acet by Dcrete Step Gradet Acet oca But Not Shortet + (F + 10/22/ /22/

6 Part 4B: Neura Networ earg 10/22/08 Gradet Acet Proce Chage fte : P = F( P F = df dt = m F d P m = F =1P d t =1 F = F P F = F F = F 2 0 Therefore gradet acet creae fte (ut reache 0 gradet 10/22/08 31 P Geera Acet Fte Note that ay adaptve proce P( t w creae fte provded : 0 < F = F P = F P co where age betwee F ad P Hece we eed co > 0 or < 90 10/22/08 32 Geera Acet o Fte Surface Fte a Mmum Error Suppoe for Q dfferet put we have target output t 1,,t Q Suppoe for parameter P the correpodg actua output + (F are y 1,,y Q Suppoe D( t,y [ 0, meaure dfferece betwee target actua output et E = D( t,y be error o th ampe Q [ ] et F( P = E ( P = D t,y ( P =1 Q =1 10/22/ /22/08 34 Gradet of Fte ( F = ' E * = E E = D t,y P P = j y j D t,y = d D t,y d y y j P y P = y D( t,y y P 10/22/08 35 Jacoba Matrx y 1 P y 1 1 P m ( Defe Jacoba matrx J = ( y P y ( 1 P ( m ' Note J m ad D( t,y 1 Sce (E = E E = ( J T D( t,y = y j D t,y P P y, j j 10/22/

7 Part 4B: Neura Networ earg 10/22/08 Dervatve of Suared Eucdea Dtace Suppoe D t,y D( t y = y j d D t,y dy = t y 2 = ( t y 2 ( t y 2 = y j 2 y j t y 10/22/08 37 = d ( t y j j 2 = 2( t j y j d y j = 2( y t Gradet of Error o th Iput (,y E = d D t P d y y P = 2( y t y P j y j = 2 y j t j E = 2( J T ( y t P 10/22/08 38 P = Recap To ow how to decreae the dfferece betwee actua dered output, we eed to ow eemet of Jacoba, y j P, whch ay how jth output vare wth th parameter (gve the th put ( J T t y The Jacoba deped o the pecfc form of the ytem, th cae, a feedforward eura etwor 10/22/08 39 Mutayer Notato x y /22/08 40 Notato ayer of euro abeed 1,, N euro ayer = vector of output from euro ayer put ayer 1 = x (the put patter output ayer = y (the actua output = weght betwee ayer ad +1 Probem: fd how output y vary wth weght j ( = 1,, Typca Neuro h 1 j j 1 N 1 N 10/22/ /22/

8 Part 4B: Neura Networ earg 10/22/08 Error Bac-Propagato e w compute E tartg wth at ayer ( = 1 j ad worg bac to earer ayer ( = 2,,1 Deta Vaue Coveet to brea dervatve by cha rue : E = E 1 1 j j et = E E So = 1 j 1 j 10/22/ /22/08 44 Output-ayer Neuro Output-ayer Dervatve (1 1 1 j j N N h = y E t 2 = E h = t h 2 = d t d h = 2 t ' h d = 2 t d h 10/22/ /22/08 46 Output-ayer Dervatve (2 Hdde-ayer Neuro = = j j j E = 1 1 j j where = 2 t ' h 1 1 j 1 N j 1 N 1 h N N +1 N E 10/22/ /22/

9 Part 4B: Neura Networ earg 10/22/08 Hdde-ayer Dervatve (1 E Reca = 1 j 1 j = E = E +1 h +1 h h h = h +1 h m m h = m = d h = = h dh = +1 h = h +1 Hdde-ayer Dervatve (2 = 1 1 j j E = 1 1 j j where = ' h 1 1 j 1 1 = d j 1 d j +1 ( 1 = j 10/22/ /22/08 50 Dervatve of Sgmod 1 Suppoe = ( h = (ogtc gmod 1+ exp (h D h = D h [ 1+ exp (h] 1 = [ 1+ exp (h] 2 D h 1+ e h e h = ( 1+ e h 2 (e h = ( 1+ e h 2 1 e h 1+ eh = = 1+ e h h 1+ e 1+ e 1 ' h 1+ e h ( = (1 Summary of Bac-Propagato Agorthm Output ayer : = 2 ( 1 ( t E = 1 1 j j Hdde ayer : = 1 E = 1 1 j j +1 10/22/ /22/08 52 Output-ayer Computato 1 1 j j. j N + - N j 1 = j 1 h 10/22/ * + = 2 ( 1 t ( 2, = y t 1 1 Hdde-ayer Computato j 1 j. 1 j 1 N N = 1 1 j = 1 j h 1 * N N +1 * 1 +1 * +1 * N +1 10/22/08 54, E 9

10 Part 4B: Neura Networ earg 10/22/08 Trag Procedure Batch earg o each epoch (pa through a the trag par, weght chage for a patter accumuated weght matrce updated at ed of epoch accurate computato of gradet Oe earg weght are updated after bac-prop of each trag par uuay radomze order for each epoch approxmato of gradet Doe t mae much dfferece 10/22/08 55 Summato of Error Surface E E 1 E 2 10/22/08 56 Gradet Computato Batch earg Gradet Computato Oe earg E E E 1 E 1 E 2 E 2 10/22/ /22/08 58 Tetg Geerazato Probem of Rote earg error error o tet data Avaabe Data Trag Data Tet Data Doma error o trag data epoch top trag here 10/22/ /22/

11 Part 4B: Neura Networ earg 10/22/08 Improvg Geerazato Avaabe Data Vadato Data Trag Data Tet Data Doma 10/22/08 61 A Few Radom Tp Too few euro ad the ANN may ot be abe to decreae the error eough Too may euro ca ead to rote earg Preproce data to: tadardze emate rreevat formato capture varace eep reevat formato If tuc oca m., retart wth dfferet radom weght 10/22/08 62 Beyod Bac-Propagato Adaptve earg Rate Adaptve Archtecture Add/deete hdde euro Add/deete hdde ayer Rada Ba Fucto Networ Etc., etc., etc. The Gode Rue of Neura Net Neura Networ are the ecod-bet way to do everythg 10/22/ /22/08 64 V 11