Pattern Recognition 2015 Neural Networks (2)

Transcription

1 Pttern Reognition 2015 Nerl Networks (2) Ad Feelders Universiteit Utreht Ad Feelders ( Universiteit Utreht ) Pttern Reognition 1 / 37

2 Error k-propgtion How to ompte the weights in nerl network? Evlte derivtives of the error fntion with respet to the weights. Apply grdient sed optimiztion lgorithm to find the weight vles t lol minimm of the error fntion. Most simple grdient sed optimiztion lgorithm: grdient desent. Bk-propgtion is ompttionlly effiient lgorithm for evlting derivtives of the error fntion in mlti-lyered nerl network. Ad Feelders ( Universiteit Utreht ) Pttern Reognition 2 / 37

3 Generl k-propgtion Consider n ritrry feed-forwrd topology, ritrry differentile tivtion fntion h, nd ritrry differentile error fntion E. Weighted inpt of neron j j = i: i j w ji z i Otpt / Non-liner tivtion fntion of neron j: z j = h( j ) Error fntion (sm over ll ptterns n): E = n E n Define for eh neron j the lol error δ j : δ j E n j Ad Feelders ( Universiteit Utreht ) Pttern Reognition 3 / 37

4 Generl network topology: nottion k... w kj j = i:i j w ji z i j z j = h( j ) w ji i z i... Ad Feelders ( Universiteit Utreht ) Pttern Reognition 4 / 37

5 Generl Bk-Propgtion (ontined) We wnt to evlte the error grdient: E n = E n j (hin rle: E n depends on w ji only throgh j ) w ji j w ji = δ j j w ji = δ j z i (δ j En j ) ( j w ji = z i ) We hve ese j w ji = z i, j = i: i j w ji z i. Ad Feelders ( Universiteit Utreht ) Pttern Reognition 5 / 37

6 Generl Bk-Propgtion (ontined) Only remins to ompte δ j for eh neron. We onsider otpt nerons nd hidden nerons seprtely. Otpt neron: δ k E n k = E n h( k ) h( k ) k (E n only depends on k throgh h( k )) = h ( k ) E n y k (h ( k ) h( k) k ; h( k ) y k ) This is how fr we n go withot mking speifi hoies for the tivtion fntion nd error fntion. Ad Feelders ( Universiteit Utreht ) Pttern Reognition 6 / 37

7 Generl Bk-Propgtion (ontined) Hidden neron: δ j E n = E n h( j ) j h( j ) j = h ( j ) E n (h ( j ) h( j ) z j ; h( j ) z j ) j = h ( j ) E n k (hin rle) k z j k: j k = h ( j ) k: j k = h ( j ) k: j k E n k w kj w kj δ k ( k z j = w kj ) ( En k δ k ; 5.56) Sine the forml for δ j only ontins terms in lter lyers, the lol errors n e llted from otpt to inpt on the network. Hene the term k-propgtion. Ad Feelders ( Universiteit Utreht ) Pttern Reognition 7 / 37

8 Illstrtion of hin rle E n = f( 5, 6, 7 ) E n 5 E n 6 E n z 4 7 z 4 z 4 = w 74 z 4 z 4 z 4 4 E n z 4 = E n 5 5 z 4 + E n 6 6 z 4 + E n 7 7 z 4 Ad Feelders ( Universiteit Utreht ) Pttern Reognition 8 / 37

9 Error Bk-Propgtion proedre 1 Forwrd propgtion of inpt vles: tivtions of ll nerons 2 Compte δ k for eh otpt neron k 3 Bkwrd propgtion: ompte δ j for eh hidden neron j 4 Compte derivtive for eh weight w ji E n w ji = δ j z i Ad Feelders ( Universiteit Utreht ) Pttern Reognition 9 / 37

10 Bk-Propgtion for speifi se Consider two-lyer network with logisti hidden nerons nd liner otpt nerons: ( M D ) y k = w k0 + w kj σ w ji x i j=1 i=0 with the sm-of-sqres error fntion: E n = 1 2 K (y k t k ) 2 k=1 Ad Feelders ( Universiteit Utreht ) Pttern Reognition 10 / 37

11 Bk-Propgtion for speifi se (ontined) Lol error t the otpt nerons: sine h( k ) = k (liner otpt) nd δ k = h ( k ) E n y k = y k t k, 1 2 (y k t k ) 2 y k = y k t k Lol error t hidden nerons: K K δ j = σ ( j ) w kj δ k = z j (1 z j ) w kj δ k k=1 k=1 Ad Feelders ( Universiteit Utreht ) Pttern Reognition 11 / 37

12 Derivtive of logisti tivtion fntion σ() = e 1 + e = (1 + e ) 1 σ () = d (1 + e ) 1 d = (1 + e ) 2 e e = (1 + e ) 2 1 = 1 + e e 1 + e = σ()(1 σ()) Ad Feelders ( Universiteit Utreht ) Pttern Reognition 12 / 37

13 Logisti density (red) nd mltive density (le) f(z) nd F(z) z Ad Feelders ( Universiteit Utreht ) Pttern Reognition 13 / 37

14 Bk-Propgtion for speifi se (ontined) First-lyer derivtives: Seond-lyer derivtives: E n w ji = δ j x i E n w kj = δ k z j Ad Feelders ( Universiteit Utreht ) Pttern Reognition 14 / 37

15 Error Bk-Propgtion proedre: speifi se 1 Apply inpt vetor x, forwrd propgte tivtions throgh network 2 Compte δ k for eh otpt neron k : δ k = y k t k 3 Bkwrd propgte δ k s to ompte δ j for eh hidden neron j: δ j = z j (1 z j ) K k=1 w kjδ k 4 Compte derivtive En w ji = δ j z i for eh weight w ji : 1 First-lyer derivtives: E n = δ j x i w ji 2 Seond-lyer derivtives: E n w kj = δ k z j Ad Feelders ( Universiteit Utreht ) Pttern Reognition 15 / 37

16 Exmple Bk-Propgtion with skip-lyer onnetions x x x y Otpt neron: liner; hidden neron: logisti Trining pttern: (x 1, x 2 ) = (2, 3), t = 0 Skip-lyer onnetions! Ad Feelders ( Universiteit Utreht ) Pttern Reognition 16 / 37

17 Forwrd propgtion: ompte predited trget To ompte the predited trget vle, we strt t the inpt lyer nd work towrd the otpt lyer: 3 = w 30 + w 31 x 1 + w 32 x 2 = = z 3 = 1 + e = 1 = e = w 40 + w 41 x 1 + w 42 x 2 + w 43 z 3 = = 1.07 y = z 4 = 4 = 1.07 Ad Feelders ( Universiteit Utreht ) Pttern Reognition 17 / 37

18 Bkwrd propgtion To ompte the lol errors we strt t the otpt lyer, nd work towrd the inpt lyer: δ 4 = y t = = 1.07 δ 3 = z 3 (1 z 3 )w 43 δ 4 = 0.67(1 0.67)( 0.2)1.07 = 0.05 (otpt nit) (hidden nit) Ad Feelders ( Universiteit Utreht ) Pttern Reognition 18 / 37

19 Error derivtives E n w 40 = δ 4 1 = 1.07 E n w 41 = δ 4 z 1 = = 2.14 E n w 42 = δ 4 z 2 = = 3.21 E n w 30 = δ 3 1 = 0.05 E n w 31 = δ 3 z 1 = = 0.10 E n w 32 = δ 3 z 2 = = 0.15 E n w 43 = δ 4 z 3 = = 0.72 Ad Feelders ( Universiteit Utreht ) Pttern Reognition 19 / 37

20 Weight Updte With lerning rte η = 0.1, we hve nd w (τ+1) 42 = w (τ) 42 η E n w 42 = = 0.02 w (τ+1) 43 = w (τ) 43 η E n w 43 = = 0.27 Other weight pdtes re ompted nlogosly. Ad Feelders ( Universiteit Utreht ) Pttern Reognition 20 / 37

21 Error fntion for inry lsses Cross-entropy error fntion: E(w) = N t n ln y n + (1 t n ) ln(1 y n ) n=1 with y n = y(x n, w). We only onsider single pttern, so E n = t ln y (1 t) ln(1 y), where for simpliity we hve dropped the ssript n on the right-hnd side. Ad Feelders ( Universiteit Utreht ) Pttern Reognition 21 / 37

22 Lol error Lol error for logisti otpt with entropy error: δ o = σ ( o ) E y = y(1 y) E y Left to determine E y [ t ln y (1 t) ln(1 y)] = y = 1 t 1 y t y Ad Feelders ( Universiteit Utreht ) Pttern Reognition 22 / 37

23 Lol error (ontined) Therefore y(1 y) E y [ 1 t = y(1 y) 1 y t ] y = (1 t)y t(1 y) = y t Finlly, we onlde tht δ o = y t for logisti otpt nits nd entropy error fntion. Ad Feelders ( Universiteit Utreht ) Pttern Reognition 23 / 37

24 Mltinomil logisti regression Let t {1,..., K} (where K is the nmer of lsses). The mltinomil logisti regression ssmption is p(t = k x) = exp(w k x) K j=1 exp(w j x) (4.104 nd 4.105) where we now hve weight vetor w k for eh lss. Ad Feelders ( Universiteit Utreht ) Pttern Reognition 24 / 37

25 Non-inry lsses in nerl networks Rther thn tking liner fntions k = wk x we n generlize this model to k (x, w): p(t = k x) = exp( k (x, w)) K j=1 exp( j(x, w)) In prtilr, the k n e tivtions proded y nerl network. Ad Feelders ( Universiteit Utreht ) Pttern Reognition 25 / 37

26 Softmx tivtion fntion Implementtion in nerl network. Crete K otpt nits: y 1 y 2 y K K With Let (1-of-K oding) y k = t j = exp( k ) K j=1 exp( j) { 1 if t = j 0 otherwise Ad Feelders ( Universiteit Utreht ) Pttern Reognition 26 / 37

27 Error fntion for non-inry lsses Negtive log-likelihood E(w) = N n=1 k=1 For single oservtion (pttern) we hve (proof omitted) K t kn ln y k (x n, w) (5.24) δ k = E n k = y k t k Ad Feelders ( Universiteit Utreht ) Pttern Reognition 27 / 37

28 Reglriztion: weight dey Pnish lrge weights y tking Ẽ(w) = E(w) + λ w 2 ij (5.112) to otin smooth fit (nd therey void overfitting). Also my help optimiztion: less lol minim nd fster onvergene. Ad Feelders ( Universiteit Utreht ) Pttern Reognition 28 / 37

29 Exmple: Cshing s Syndrome Hypertensive disorder ssoited with over-seretion of ortisol y the drenl glnd. The oservtions re rinry exretion rtes of two steroid metolites. The Cshings dt frme (in lirry MASS) hs 27 rows nd 3 olmns: Tetrhydroortisone: rinry exretion rte (mg/24hr). Pregnnetriol: rinry exretion rte (mg/24hr). Type: nderlying type of syndrome (denom) (ilterl hyperplsi) (rinom) for nknown (not sed in fitting models) Ad Feelders ( Universiteit Utreht ) Pttern Reognition 29 / 37

30 Cshing s Syndrome: Mltinomil Logit Tetrhydroortisone Pregnnetriol Ad Feelders ( Universiteit Utreht ) Pttern Reognition 30 / 37

31 Cshing s Syndrome: NN1 (λ = 0.001) Tetrhydroortisone Pregnnetriol Ad Feelders ( Universiteit Utreht ) Pttern Reognition 31 / 37

32 How to in R > lirry(nnet) > lirry(mass) > sh <- log(s.mtrix(cshings[,-3])) > tp <- ftor(cshings$type[1:21]) > tp [1] Levels: > tpi <- lss.ind(tp) > tpi[(1,10,20),] [1,] [2,] [3,] > set.seed(54321) > sh.nnet1 <- nnet(sh[1:21,], tpi, skip=t, softmx=t, size=2, dey=0.001, mxit=1000) Ad Feelders ( Universiteit Utreht ) Pttern Reognition 32 / 37

33 How to in R # weights: 21 initil vle iter 10 vle iter 20 vle iter 30 vle iter 40 vle iter 50 vle iter 60 vle iter 70 vle iter 80 vle iter 90 vle iter 100 vle iter 110 vle iter 120 vle iter 130 vle iter 140 vle iter 150 vle finl vle onverged Ad Feelders ( Universiteit Utreht ) Pttern Reognition 33 / 37

34 Cshing s Syndrome: NN2 (λ = 0.001) Tetrhydroortisone Pregnnetriol Ad Feelders ( Universiteit Utreht ) Pttern Reognition 34 / 37

35 Cshing s Syndrome: NN3 (λ = 0) Tetrhydroortisone Pregnnetriol Ad Feelders ( Universiteit Utreht ) Pttern Reognition 35 / 37

36 Model Seletion: Forensi Glss Mesrements (refrtive index nd weight perent of oxides of N, Mg, Al, Si, K, C, B, nd Fe) on 214 frgments of glss. Glss types 1 Window flot glss (70) 2 Window non-flot glss (76) 3 Vehile window glss (17) 4 Continers (13) 5 Tlewre (9) 6 Vehile hedlmps (29) Ad Feelders ( Universiteit Utreht ) Pttern Reognition 36 / 37

37 Model Seletion: Forensi Glss Use 10-fold ross vlidtion to selet nmer of hidden nits nd λ. Use 10 different sets of strt weights for eh CV-rn nd verge posterior proilities of fitted networks. (tle displys error rtes) # hidden nits λ Ad Feelders ( Universiteit Utreht ) Pttern Reognition 37 / 37