Multilayer neural networks

Transcription

1 Topic 3 Artificial nural ntwors: Suprvisd larning Multi nural ntwors Acclratd larning in multi nural ntwors Multi nural ntwors A multi prcptron is a fdforward nural ntwor with on or mor hiddn s. Th ntwor consists of an input of sourc nurons, at last on middl or hiddn of computational nurons, and an output of computational nurons. Th input signals ar propagatd in a forward dirction on a -by by- basis. Multi nural ntwors Gnrally much mor vrsatil than singl nurons No linar sparability rquirmnt. Training is lss obvious and potntially mor tim consuming. Svral varitis, th most common of which is nown as: MLP (Multi-Lvl Prcptron) Bacpropagation Ntwor (alluding to a common mthod of training ths ntwors; othr training mthods could concivably b usd.) 3 Multi prcptron with two hiddn s What dos th middl hid? A hiddn hids its dsird output. Nurons in th hiddn cannot b obsrvd through th input/output bhaviour of th ntwor. Thr is no obvious way to now what th dsird output of th hiddn should b. Commrcial ANNs incorporat thr and somtims four s, including on or two hiddn s. Each can contain from to nurons. Exprimntal nural ntwors may hav fiv or vn six s, including thr or four hiddn s, and utilis millions of nurons. How to train a MLP? With a singl nuron, it is not too hard to s how to adjust th wights basd upon th rror valus. With a multi- ntwor, it is lss obvious. For on thing, what is th rror for th nurons in nonfinal s? Without ths, w don t now how to adjust. This is calld th crdit assignmnt problm (mayb should b blam assignmnt ). 6

2 Bacpropagation Wrbos, in his Harvard PhD thsis in 974 found a mthod. Rumlhart and McCllland, in 985 discovrd th mthod, prsumably indpndntly, and popularizd it undr th currnt nam. In mathmatics, such mthods ar in th catgory of optimization. Th tchniqu is gradint dscnt, as for Adalins. Howvr, th computation of th gradint is lss clar. Bac-propagation nural ntwor Larning in a multi ntwor procds th sam way as for a prcptron. A training st of input pattrns is prsntd to th ntwor. Th ntwor computs its output pattrn, and if thr is an rror or in othr words a diffrnc btwn actual and dsird output pattrns th wights ar adjustd to rduc this rror. In a bac-propagation nural ntwor, th larning algorithm has two phass. First, a training input pattrn is prsntd to th ntwor input. Th ntwor propagats th input pattrn from to until th output pattrn is gnratd by th output. If this pattrn is diffrnt from th dsird output, an rror is calculatd and thn propagatd bacwards through th ntwor from th output to th input. Th wights ar modifid as th rror is propagatd. 7 Thr- bac-propagation nural ntwor x x i x n i n Input w ij Input signals j m Hiddn Error signals w j l Output y y y y l Bacpropagation training cycl Forward propagation: Driv th activation valus (th inputs to th activation functions) at ach nuron, and th final output. Comput th rror in th output. Bacpropagat th rror through th ntwor to gt snsitivitis at ach nuron. (Th gradint approximation is drivabl from th snsitivitis.) Us th snsitivitis to driv wight changs. Apply th wight changs. Bacpropagat is mathmatically a lot li forward propagat. Snsitivitis ar usd instad of signal valus. Th snsitivitis ar th partial drivativs of th MSE with rspct to th activation valus. Basically both ar itratd matrix multiplications. Bacpropagation Givn an input vctor, can comput th outputs. Givn a sampl, can comput th rrors in output. Knowing gradint, can adjust th wights. Big Qustion: How to comput th gradint? Rcall that th gradint consists of componnts J/ w whr J is th man-squard rror and w is som wight (or bias) in th ntwor. For th Adalin, alrady drivd: J/ wi = - ε xi f (n), whr xi is th input corrsponding to wight wi, and n(nt) is th wightd sum. This wors as is for th multi- cas at th output.

3 Insid on nuron Bacward propagation of snsitivity Bacward propagation of snsitivity Th bac-propagation training algorithm Stp : Initialisation St all th wights and thrshold lvls of th ntwor to random numbrs uniformly distributd insid a small rang:.4.4, + F i F i whr F i is th total numbr of inputs of nuron i in th ntwor. Th wight initialisation is don on a nuron-by by-nuron basis. Stp : Activation Activat th bac-propagation nural ntwor by applying inputs x (p), (p),, x n (p)) and dsird outputs y d, (p), y d, (p),, y d,n (p). (a)) Calculat th actual outputs of th nurons in th hiddn : n y j = sigmoid xi wij θ j i= whr n is th numbr of inputs of nuron j in th hiddn, and sigmoid is th sigmoid activation function. Stp : Activation (continud) (b)) Calculat th actual outputs of th nurons in th output : m y = sigmoid x j w j θ j= whr m is th numbr of inputs of nuron in th output.

4 Stp 3: Wight training Updat th wights in th bac-propagation ntwor propagating bacward th rrors associatd with output nurons. (a)) Calculat th rror gradint for th nurons in th output : δ = y [ y ] whr = yd, y Calculat th wight corrctions: w j = α y j Updat th wights at th output nurons: w j ( p + ) = w j + w j Stp 3: Wight training (continud) (b)) Calculat th rror gradint for th nurons in th hiddn : l δ j = y j [ y j ] δ w j = Calculat th wight corrctions: wij = α xi j Updat th wights at th hiddn nurons: wij ( p + ) = wij + wij Stp 4: Itration Incras itration p by on, go bac to Stp and rpat th procss until th slctd rror critrion is satisfid. As an xampl, w may considr th thr- bac-propagation ntwor. Suppos that th ntwor is rquird to prform logical opration Exclusiv-OR OR.. Rcall that a singl- prcptron could not do this opration. Now w will apply th thr- nt. Thr- ntwor for solving th Exclusiv-OR opration w 3 x 3 w 3 Input θ 3 w 35 w 4 4 w 45 w 4 θ 4 Hiddn 5 Output θ 5 y 5 Th ffct of th thrshold applid to a nuron in th hiddn or output is rprsntd by its wight, θ, connctd to a fixd input qual to. Th initial wights and thrshold lvls ar st randomly as follows: w 3 =.5, w 4 =.9, w 3 =.4, w 4 =., w 35 =., 45 =., θ 3 =.8, θ 4 =. and θ 5 =.3. w 45 W considr a training st whr inputs x and ar qual to and dsird output y d,5 is. Th actual outputs of nurons 3 and 4 in th hiddn ar calculatd as ( ) [ + ]. 55 ( ) [ + ]. 888 y3 = sigmoid ( xw3 + xw3 θ3) = / = y4 = sigmoid ( xw4 + xw4 θ4) = / = Now th actual output of nuron 5 in th output is dtrmind as: Thus, th following rror is obtaind: (.3) [ ] y5 = sigmoid( y3w35 + y4w45 θ5) = / = = yd, 5 y5 =.597 =.597

5 Th nxt stp is wight training. To updat th wights and thrshold lvls in our ntwor, w propagat th rror,,, from th output bacward to th input. First, w calculat th rror gradint for nuron 5 in th output : δ5 = y5 ( y5) =.597 (.597) (.597) =.74 Thn w dtrmin th wight corrctions assuming that th larning rat paramtr, α,, is qual to.: w35 = α y3 5 =..55 (.74) =.67 w45 = α y4 5 =..888 (.74) =. θ5 = α ( ) 5 =. ( ) (.74) =.7 Nxt w calculat th rror gradints for nurons 3 and 4 in th hiddn : δ3 = y3( y3) 5 w35 =.55 (.55) (.74) (.) =.38 δ 4 = y4( y4) 5 w45 =.888 (.888) (.7 4). =.47 W thn dtrmin th wight corrctions: w3 = α x 3 =..38 =.38 w3 = α x 3 =..38 =.38 θ3 = α ( ) 3 =. ( ).38 =.38 w4 = α x 4 =. (.47) =.5 w4 = α x 4 =. (.47) =.5 θ4 = α ( ) 4 =. ( ) (.47 ) =.5 At last, w updat all wights and thrshold: w3 = w3 + w3 = =.538 w4 = w4 + w4 =.9.5 =.8985 w3 = w3 + w3 = =.438 w4 = w4 + w4 =..5 =.9985 w35 = w35 + w35 =..67 =.67 w45 = w45 + w45 =.. =.888 θ3 = θ3 + θ3 =.8.38 =.796 θ4 = θ4 + θ4 =. +.5 =.985 θ5 = θ5 + θ5 = =.37 Th training procss is rpatd until th sum of squard rrors is lss than.. Larning curv for opration Exclusiv-OR Sum-Squard Error Sum-Squard Ntwor Error for 4 s Final rsults of thr- ntwor larning Inputs x Dsird output y d Actual output y Error Sum of squard rrors. Ntwor rprsntd by McCulloch-Pitts modl for solving th Exclusiv-OR opration +.5 x y

6 x +.5 = x +.5 = (a) Dcision boundaris x (b) (a)) Dcision boundary constructd by hiddn nuron 3; (b)) Dcision boundary constructd by hiddn nuron 4; (c)) Dcision boundaris constructd by th complt thr- ntwor x x (c) Acclratd larning in multi nural ntwors A multi ntwor larns much fastr whn th sigmoidal activation function is rprsntd by a hyprbolic tangnt: tan h a Y = a bx + whr a and b ar constants. Suitabl valus for a and b ar: a =.76 and b =.667 W also can acclrat training by including a momntum trm in th dlta rul: w j = β w j ( p ) + α y j whr β is a positiv numbr ( β < ) calld th momntum constant.. Typically, th momntum constant is st to.95. This quation is calld th gnralisd dlta rul. Larning with momntum for opration Exclusiv-OR Larning Rat Sum-Squard Error Training for 6 s Larning with adaptiv larning rat To acclrat th convrgnc and yt avoid th dangr of instability, w can apply two huristics: Huristic If th chang of th sum of squard rrors has th sam algbraic sign for svral consqunt pochs, thn th larning rat paramtr, α,, should b incrasd. Huristic If th algbraic sign of th chang of th sum of squard rrors altrnats for svral consqunt pochs, thn th larning rat paramtr, α,, should b dcrasd. Adapting th larning rat rquirs som changs in th bac-propagation algorithm. If th sum of squard rrors at th currnt poch xcds th prvious valu by mor than a prdfind ratio (typically.4), th larning rat paramtr is dcrasd (typically by multiplying by.7) and nw wights and thrsholds ar calculatd. If th rror is lss than th prvious on, th larning rat is incrasd (typically by multiplying by.5).

7 Sum-Squard Error Larning Rat Larning with adaptiv larning rat Training for 3 s Epo ch Epo ch Larning with momntum and adaptiv larning rat Sum-Squard Error Larning Rat Training for 85 s BacProp Tchniqu & Trics (Som of ths apply to Gnral Nural Ntwors) (Two Rfrncs: Nural Ntwors Trics of th Trad, Orr and Mullr, ds. Choos xampls with maximum information contnt Shuffl th training st so that succssiv sampls rarly blong to th sam class. Prsnt input xampls that produc a larg rror mor frquntly than ons that produc a small rror. 39 Tchniqu and trics Normaliz th inputs Bttr if man of a particular variabl is nar. Thn wight changs ar lss lily to b synchronizd, sinc som will b positiv, othrs ngativ. Thrfor, subtract th actual man from th variabl bfor training. Bttr if th variabls ar scald to hav similar auto-covariancs, dfind as (sum-of-squars of variabl)/(numbr of sampls) Thn th wights will larn at similar rats. Excption: Whn som variabls ar nown in advanc to b of lss significanc. 4 Tchniqu and trics Dcorrlat th inputs Bttr if no two input variabls ar corrlatd. Corrlatd inputs analogous to having linarly dpndnt variabls in a linar systm. A tchniqu calld PCA (Principal Componnts Analysis), aa Karhunn-Lov Expansion, can b usd to rmov linar corrlations. W will loo at PCA latr; PCA itslf can b don by a PCA nural ntwor. 4 Summary of input normalization 4

8 Tchniqu and trics Prfr tansig (hyprbolic tangnt) rathr than logsig for innr s. tansig output is symmtric about origin, logsig is not. tansig will mor lily produc outputs clos to for th nxt stag of th ntwor Som rcommnd adding a small linar constant to th output of tansig to avoid flat spots Picwis quadratic approximation to tanh Choic of targt valus Choosing targt valus of +, - for a tansig causs th nuron to b drivn toward th saturation rgion. To gt into this rgion, th wights ar larg and may bcom stuc bcaus small gradint valus will not chang thm sufficintly. It may b bttr to choos th targts offst from ths saturation valus, or to scal th tansig to gt th sam ffct,.g. f(x) =.759 tanh(x/3), which has a maximum nd drivativ whr th function s valu is +/-. Wight initialization Assuming that th training st has bn normalizd and th prvious sigmoid is usd, Draw th initial wights from a distribution, such as a uniform distribution, with man and standard dviation /sqrt(m) whr m is th fan-in (numbr of inputs to th nod). Incrass lilihood that th input to th sigmoid will hav a standard dviation of (sinc th lattr is th sqrt of th sum of th squars of th wights, for normalizd input) Larning rats Idally, ach wight should hav its own larning rat. S th Nural Ntwors Trics of th Trad, Orr and Mullr, ds., for how to choos larning rat basd on nd drivativs. As a substitut, ach nuron, or ach could hav its own larning rat. Larning rats should b proportional to th sqrt of th numbr of inputs to th nuron. Wights in arlir s should b largr than thos in latr s, sinc th arlir s tnd to hav a smallr nd drivativ of th MSE. Validation Tchniqu ( Cross- Validation ) & Early Stopping Split th training st into training and validation substs,.g. : or 5: ratio. Train only on th training subst; us th validation st for MSE, vry so oftn (.g. vry 5 pochs). For arly stopping: Stop training as soon as th validation rror gos up. Us th wights bfor th rror wnt up. Rational: Evn though a lowr minimum might hav bn rachd, th local minima tnd to b fairly clos in valu in practic. Ovr-fitting It is possibl for a ntwor to ovr-fit th data, maning that it larns small variations in th data which might actually b du to nois. Anothr way of saying this is that th ntwor dos not gnraliz wll; it is too spcializd. Validation is on tchniqu usd to hlp avoid ovr-fitting. Ovr-fitting can rsult if th ntwor has too many nurons at its disposal

9 Sizing a ntwor Givn a problm: How many s? How many nurons pr? What activation functions? Thortically, any function can b mulatd ovr a givn rang by a ntwor with just on hiddn and on output (two s total), with sufficint nurons in that. Practically, -3 s suffic for larg familis of problms, although mor may b usd, spcially whn spcial fatur-slction s ar usd, as in th zipcod rcognition ntwor. 49 Nurons Choos numbr of nurons basd on th assssd complxity within a (numbr of crsts and vallys of a function, for xampl). Two approachs for xprimntal dtrmination: Start with a larg numbr of nurons and prun. Start with a small numbr of nurons and build up. Ngligibl wights can b liminatd (st to ). If all input wights to a nod ar, th nod can b liminatd. If all wights a nod fds ar, th nod itslf can b liminatd. Vary wights w to s whthr J / w is significant; if not, prun th wight. 5 Doubling Start with a small numbr of nurons in th innr. If at th conclusion of a training cycl, th MSE is inadquat, rpat with doubl th numbr of nurons. 5 Numbr of training sampls Baum-Hauslr rul (989): Ncssary condition: (numbr of sampls) > W / (-a) whr W is th numbr of wights in th ntwor and a is th dsird accuracy on th tst st. Sufficint condition: (numbr of sampls) > log(n / (-a)) * W/(-a) whr N is th numbr of nurons. 5