The Backpopagaton Algothm Achtectue of Feedfowad Netwok Sgmodal Thehold Functon Contuctng an Obectve Functon Tanng a one-laye netwok by teepet decent Tanng a two-laye netwok by teepet decent Copyght Robet R. Snapp 202 CS 295 (UVM) The Backpopagaton Algothm Fall 203 / 20
Feedfowad Netwok: Nomenclatue Conde a feedfowad netwok f W R n! R N, one wth n eal nput and N output unt. Aume thee ae L laye of lnea thehold unt, wth n unt n laye n 2 unt n laye 2 : n L N unt n laye L Let n 0 n, and let y.`/ denote the output value of unt n laye `, denote the ynaptc weght of unt n laye ` that appled to the output of unt fom laye `, w.`/ ; w.`/ ;0 denote the ntenal ba of unt n laye `. Note, fo ; 2; : : : ; n`, y.`/ gn n` X w.`/.` / ; y C w.`/ ;0 ; wth, y.0/ x : CS 295 (UVM) The Backpopagaton Algothm Fall 203 2 / 20
Feedfowad netwok (cont.) (0) y =x w (),2 y () w,0 () w (2) 2, y (2) w,0 (2) y (l) w,0 (l) w (L),0 y (L) (0) y 2 =x 2 y () w 2,0 () 2 y (2) w 2,0 (2) 2 y (l) w 2,0 (l) 2 w (L) 2,0 y (L) 2 (0) y 3 =x 3 y () w 3,0 () 3 y (2) w 3,0 (2) 3 y (l) w 3,0 (l) 3 w (L) 3,0 y (L) 3 (0) y n0 =x n0 w () n,0 w (2) n 2,0 y () n y (2) n 2 w (l) n l,0 y (l) n l w (L) n L,0 y (L) n L Numbe of bae and weght =. C n 0 /n C. C n /n 2 C : : : C. C n L /n L. Fo ` ; 2; : : : ; n L, y.`/ gn n` X w.`/.` / ; y C w.`/ ;0 ; fo ; 2; : : : ; n`. : CS 295 (UVM) The Backpopagaton Algothm Fall 203 3 / 20
Supeved Leanng Paadgm Let, denote a tanng et of m patten, wth X m.x./ ; t./ /; : : : ;.x.m/ ; t.m/ /g x.p/ 2 R n 0 and 2 R n L ; fo p ; 2; : : : ; m. n o n o Goal: Fnd bae and weght W ; : : : ; o that the netwok w./ ; output unt decbe a vecto n R n L that le uffcently cloe to the taget vecto, wheneve the nput unt coepond to the nput vecto x.p/, fo p ; 2; : : : ; m. Method: Fnd the weght that mnmze the LMS obectve functon E.W/ w.l/ ; mx E p.w/; whee, p E p.w/ def 2 y.l/ x.p/ 2 ; W 2 n L X 2 y.l/ x.p/ ; W : CS 295 (UVM) The Backpopagaton Algothm Fall 203 4 / 20
Weght pace fo the Neual Netwok The numbe of bae and weght n the neual netwok gven by W. C n 0 /n C. C n /n 2 C C. C n L /n L : The tate of the netwok at epoch t can thu be epeented a a pont w.t/ 2 R n the -dmenonal weght pace. Let @ @ @ def ; ; : : : ; @w @w 2 @w denote the -dmenonal gadent. The netwok can be taned by gadent decent: Intalze the netwok wth weght w.0/. 2 Select two potve paamete, ; > 0. 3 Whle E.t/ >, update the weght by the ule, T w.t C / w.t/ E w.t/ : CS 295 (UVM) The Backpopagaton Algothm Fall 203 5 / 20
ffeent Mode of Leanng The pevouly defned leanng ule w.t C / w.t/ E w.t/ ; wth E.w/ mx E p.w/; whee, p E p.w/ def 2 y.l/ x.p/ 2 ; w 2 n L X 2 y.l/ x.p/ ; w ; called a batch update ule. The equental update ule defned, altenatvely, by w.t C / w.t/ E p.t/ w.t/ ; whee p.t/ denote the patten ndex that elected at epoch t. Fo example, p.t/ t mod m C, fo a equental cyclc update ule, o p.t/ andœ; m, fo a equental andom update ule. CS 295 (UVM) The Backpopagaton Algothm Fall 203 6 / 20
Sgmodal Functon The applcaton of ethe ule eque that the gadent of E p be defned. We thu need to eplace the dcontnuou gnum (o gn) functon gn n each LTU wth a mooth appoxmaton, whch we call a gmodal functon. We uually eque that be monotonc, wth 0.u/ > 0; fo < u < C, and lm.u/ ; lm.u/ C; and,.0/ 0: u! u!c.u/ u CS 295 (UVM) The Backpopagaton Algothm Fall 203 7 / 20
Sgmodal Functon (cont.) Fo example, we let whence, e u.u/ eu tanh u; e u C e u 0.u/ d du.eu C e u /.e u e u / 0.e u e u /.e u C e u / 0.e u C e u / 2 4.e u C e u / 2 ech2 u tanh 2 u 2.u/: CS 295 (UVM) The Backpopagaton Algothm Fall 203 8 / 20
Sgmodal Functon (cont.) Altenatvely, f one dee the output ange to be wthn the unt nteval, let, fo whch,.u/ e u e u C e u C e 2u ; lm.u/ 0;.0/ u! 2 ; lm.u/ u!c 0.u/ 2 e 2u C e 2u 2 2.u/.u/ :.u/ O u CS 295 (UVM) The Backpopagaton Algothm Fall 203 9 / 20
Sgmodal Fucnton (cont.) Anothe ueful choce.u/ 2 tan u; wth, 0.u/ 2 a th 0.u/ (uually) eman nonzeo n dgtal mulaton. C u 2 ; When t deable to model a geneal analytc functon, W R n 0! R n L t often ueful to employ lnea thehold functon n the output laye:.u/ u; 0.u/ : CS 295 (UVM) The Backpopagaton Algothm Fall 203 0 / 20
Notatonal Conventon It advantageou to defne y.0/ y.`/ ( f 0 x f n 0 8 < f 0 : Pn` w.`/.` / f n` 0 ; y fo l ; 2; : : : ; L. Notaton alo mplfed by defnng S.`/ X n` def w.`/.` / ; y 0 CS 295 (UVM) The Backpopagaton Algothm Fall 203 / 20
One-Laye Netwok Ft, conde the cae L, o that the netwok cont of n 0 nput unt, and n output unt. Then, @E p 2 Xn @ Xn Xn y./ x.p/ I w 2 y./ x.p/ I w @ y./ x.p/ I w @y./ : y./ x.p/ I w CS 295 (UVM) The Backpopagaton Algothm Fall 203 2 / 20
One-Laye Netwok (cont.) @y./ @ S./ 0 X n S./ 0 0 0 0./ S./ @S ; y.0/ 0 X n S./ 0 ı ; ı ; y.0/ 0 S./ ı ; y.0/ whee, ı ;ˇ ( f ˇ 0 f ˇ, denote the Konecke delta functon. CS 295 (UVM) The Backpopagaton Algothm Fall 203 3 / 20
One-Laye Netwok (cont.) Subttuton nto the pevou equaton yeld @E p Xn y./ x.p/ I w 0 S./ ı ; y.0/ y./ x.p/ I w 0 S./ y.0/ ;./ y.0/ whee./ def y./ x.p/ I w 0 S./ : Thu, fo ; 2; : : : ; n and 0; ; 2; : : : ; n 0, w././.t C / w.t/ C./ y.0/ CS 295 (UVM) The Backpopagaton Algothm Fall 203 4 / 20
Two-Laye Netwok: econd-laye weght Now, let L 2. Thu the netwok ha n 0 nput unt, one laye of n hdden unt, and one laye of n 2 output unt. The numbe of bae and weght.n 0 C /n C.n C /n 2. By the pecedng analy, t can be hown that fo ; : : : ; n 2, and 0; ; 2; : : : ; n, @E p whee,.2/ @w.2/.2/ y./ ; def y.2/ x.p/ I w 0 S.2/ : CS 295 (UVM) The Backpopagaton Algothm Fall 203 5 / 20
Two-Laye Netwok: ft-laye weght @E p 2 Xn 2 @ Xn 2 Xn 2 y.2/ x.p/ I w 2 y.2/ x.p/ I w @ y.2/ x.p/ I w @y.2/ y.2/ x.p/ I w We now evaluate @y.2/ @ S.2/ 0 X n S.2/ 0.2/ S.2/ @S w.2/ ; @y./ 0 X n S.2/ : (Why?) 0 w.2/ ; @y./ ; CS 295 (UVM) The Backpopagaton Algothm Fall 203 6 / 20
Two-Laye Netwok: ft-laye weght (cont.) Fom the analy of the one-laye netwok @y./ 0 S./ ı ; y.0/ : Thu, by ubttuton, @y.2/ 0 X n S.2/ w.2/ ; @y./ 0 X n S.2/ w.2/ ; 0 S./ ı ; y.0/ ; 0 S.2/ w.2/ ; 0 S./ ; y.0/ : CS 295 (UVM) The Backpopagaton Algothm Fall 203 7 / 20
Two-Laye Netwok: ft-laye weght (cont.) Recall that Thu, whee,./ @E p def 0 @y.2/.2/ Xn 2 Xn 2 Xn 2 S./ Pn2 0 S.2/ w.2/ ; 0 S./ y.0/ ; and y.2/ x.p/ I w 0 S.2/ y.2/ x.p/ I w @y.2/ y.2/ x.p/ I w 0 S.2/.2/ w.2/ ; 0 S./ y.0/./ y.0/ ;.2/ w.2/ ;. w.2/ ; 0 S./ y.0/ CS 295 (UVM) The Backpopagaton Algothm Fall 203 8 / 20
L-laye feedfowad netwok The equental update ule eadly genealzed to L laye: At epoch t peent patten p p.t/, x.p/ to the nput unt: ( y.0/ f 0 x.p/ f n 0 2 Fo ` ; 2; : : : ; L, compute y.`/ whee S.`/ P n` 0 w.`/.` / ; y ( ; f 0 ; f n`; S.`/ 3 Compute the eo of each output unt,.l/ y.l/ fo ; 2; : : : ; n L.. (The gnal popagate fowad.) x.p/ I w 0 S.L/ ; CS 295 (UVM) The Backpopagaton Algothm Fall 203 9 / 20
L-laye feedfowad netwok (cont.) 4 Popagate the eo backwad though the netwok by computng fo ` L ; L 2; : : : ;,.`/ 0 S.`/ n`c X fo ; 2; : : : ; n`. (Eo popagate backwad.) 5 Update the weght: w.`/ ;.`C/ w.`c/ ; ;.t C / w.`/.t/ C.`/ ;.` / y ; fo ` ; 2; : : : ; L; ; 2; : : : n`; and 0; ; : : : ; n`. CS 295 (UVM) The Backpopagaton Algothm Fall 203 20 / 20