Multilayer Kerceptron

Transcription

1 Mutiayer Kerceptron Zotán Szabó, András Lőrincz Department of Information Systems, Facuty of Informatics Eötvös Loránd University Pázmány Péter sétány 1/C H-1117, Budapest, Hungary e-mai: Abstract Mutiayer Perceptrons (MLP) are formuated within Support Vector Machine (SVM) framework by constructing mutiayer networks of SVMs The couped approximation scheme can take advantage of generaization capabiities of the SVM and the combinatory feature of the hidden ayer of MLP The network, the Mutiayer Kerceptron (MLK) assumes its own backpropagation procedure that we sha derive here Tuning rue wi be provided for quadratic cost function, with reguarization capabiity as we A further appeaing property of our approach is that by the aid of the so caed kerne trick the MLK computations can be performed in the dua space 1 Introduction Mutiayer Perceptrons (MLP) and Support Vector Machines (SVM) have been extensivey studied in the iterature For an exceent review, see [3, 2] and references therein Here we extend SVMs to muti-ayer structures and provide the backpropagation tuning rues for this system By appying the so caed kerne-trick, we embed the probem into a space having scaar product For other approaches using the same trick, see, eg, [4, 6, 7] 2 Network Architecture 21 Notations Numbers (a), vectors (a), and matrices (A) are denoted by different etter types A T denotes the transpose of matrix A Extension of vector a by component a is Journa of Appied Mathematics, 24: ,

2 written as [a; a] R stands for rea numbers 2 indicates the L 2 norm induced by the scaar product, of Eucidean space E, ie, e 2 e, e (e E) 22 Buiding Bocks 221 SVM SVMs are popuar approximation toos [9, 10, 8, 6, 5] SVMs approximate {x(t), d(t)} t1t sampe pairs, where each x(t) input is in input space X and d(t) R The approximation is inear, but it occurs in feature space H Inputs x(t) are mapped to feature space by ϕ : x X H (1) One can interpret ϕ(x) as the representation of input x The form of the SVM approximation is Formay, the SVM-task is defined as f w : x X w, ϕ(x) H (w H) (2) min w T H[w] : C V [d(t), f w (x(t))] w 2 H t1 (C > 0), (3) where V [, ] is the so caed oss function, which can assume quadratic, ɛ-insensitive, or other forms [4] That is, SVMs are reguarized inear approximators [2] Instead of using the expicit ϕ mapping, feature space H can be expoited through kerne k, that is H H(k) [11], where ϕ(x) k(, x) Kerne k assumes the reproducing property [1, 11] f( ), k(, x) H f(x) (x X, f H), (4) and H is caed Reproducing Kerne Hibert Space (RKHS) Scaar product of any function with kerne k(, x) evauates the function at x in RKHS H Scaar product in feature space can be computed impicity by means of the kerne k(u, v) ϕ(u), ϕ(v) H (u, v X) (5) In particuar, for w N α j ϕ(z j ) j1 (α j R, z j X) we have N N f w (x) w, ϕ(x) H α j ϕ(z j ), ϕ(x) H α j k(z j, x) (6) j1 Thus, function f w can be evauated by means of coefficients α j, sampes z j and the kerne k without expicit reference to representation ϕ(x) This is caed the kerne trick j1 2

3 222 MLP An MLP network has mutipe ayers, each performing non-inear mapping x g(w x) (7) Here, g is a differentiabe non-inear function In the MLP task, we tune matrix W for a ayers so that the network approximates the samped input-output mapping given by input-output training pairs {x(t), d(t)} That is, the objective is to minimize the squared error ε 2 (t) : d(t) y(t) 2 2 min, (8) W 1,W 2, where the output of the network at time t is y(t), for a times The MLP task is soved by the we-known backpropagation agorithm 23 The MLK Architecture The mapping of a genera MLP ayer (ie, Eq (7)) can be written as x g w i, x, (9) where wi T denotes the i th row of matrix W The SVM can aso be inserted into the MLP: Let a genera ayer of the network assume the form 1 w 1, ϕ(x) H x g (10) w N, ϕ(x) H A network made of such ayers, see Fig 1, wi be caed Mutiayer Kerceptron (MLK) The input (x ) of each ayer is the output of the preceding ayer (y 1 ) The externa word is the 0 th ayer providing input to the first ayer of the MLK x y 1 R N I, where N I is the dimension of the th ayer Inputs x to ayer are mapped by features ϕ and are mutipied by the weights wi This twostep process can be accompished impicity by making use of kerne k and the expansion property for wi s The resut is vector s R N S, which undergoes non-inear processing g, where function g is differentiabe The output of this non-inear function is the input to the next ayer, ie, ayer x +1 The output of the ast ayer (ayer L, the output of the network) wi be referred to as y Given that y x +1 R N o, the output dimension of ayer is N o Beow, we show that (i) a backpropagation rue can be derived for MLKs, which (ii) requires the kernes ony, so computations can be accompished in dua space 1 For the sake of simpicity et us choose sampe space X as finite dimensiona Eucidean space, ie, R n 3

4 <w,> i H g i -1 y x (x) s +1 yx Figure 1: The th ayer of the MLK, 1, 2, L The input (x ) of each ayer is the output of the preceding ayer (y 1 ) The externa word is the 0 th ayer providing input to the first ayer of the MLK Inputs x to ayer are mapped by features mapping ϕ undergo scaar mutipication by the weights (w i ) of the ayer in RKHS H H (k ) The resut is vector s, which undergoes noninear processing g, with a differentiabe function The output of this non-inear function is the input to the next ayer, ayer x +1 The output of the network is the output of the ast ayer 3 MLK Backpropagation A sighty more genera task, which incorporates reguarizing terms, too, is formaized beow: c(t) : ε 2 (t) + r(t) where ε 2 (t) d(t) y(t) 2 2 and r(t) L min {H w i : 1,,L; i1,,n S }, (11) N S 1 i1 λ i w i (t) 2 H (λ i 0) are the approximation and the reguarization terms of the cost function, respectivey, and y(t) denotes the output of the network for the t th input Parameters λ i contro the trade-off between approximation and reguarization For λ i 0 the best approximation is searched ike in the MLP task (Eq (8)) Increasing the λ i vaues, the smoothness of the approximation wi increase With the notations introduced above, the foowing statements can be proven Theorem 1 (expicit case) Let us suppose that the x w, ϕ (x) and the H g functions are a differentiabe ( 1,, L) Then, backpropagation rue can be derived for MLK if the cost function has the form c(t) ε 2 (t) + N L S λ i w i(t) 2 (λ H i 0) (12) 1 i1 Theorem 2 (impicit case) Assume that the foowing hods 4

5 1 Constraint on differentiabiity: Kernes k are differentiabe with respect to both arguments, functions g are aso differentiabe ( 1,, L) 2 Expansion property: The initia weights wi (1) of the network can be expressed in the dua representation, ie, H w i(1) N i (1) j1 α i,j(1) ϕ (z i,j(1)) ( 1,, L; i 1,, N S) Then backpropagation appies for MLK if the cost function has the form c(t) ε 2 (t) + (13) N L S λ i w i(t) 2 (λ H i 0) (14) 1 i1 This procedure spares the expansion property (13), which then remains vaid for the tuned network The agorithm is impicit in the sense that it can be reaized in the dua space The pseudo codes of the MLK backpropagation agorithms are provided in Tabe 1 and Tabe 2, respectivey Derivations of these agorithms, both for the expicit and for the impicit forms, are provided in the next subsection MLK-backpropagation can be envisioned as foows (see Tabe 1 and 2 simutaneousy): 1 backpropagated error δ (t) starts from δ L (t) and is computed by a backward recursion via the differentia expression d[s+1 (t)] d[s (t)] 2 expression d[s+1 (t)] d[s (t)] can be determined by means of feature mapping ϕ +1, or, in an impicit fashion, through kernes k +1 3 two components pay roes in the tuning of ws: (a) forgetting ( is accompished ) by scaing the weights wi with mutipier 1 2µ i (t) λ i, where λ i is the reguarization coefficient (b) adaptation occurs through the backpropagated error Weights at ayer are tuned by feature space representation of x (t), the actua input arriving at ayer Tuning is weighted by the backpropagated error 31 Derivation of the Backpropagation Agorithm for MLKs d[c(t)] Gradient d[wi (t)] is derived first Then it is embedded into steepest descent tuning 2 The c(t) error has two terms, the approximation and the reguarization 2 Steepest descent is used to iustrate the concepts Other types of gradient optimizations beyond steepest descent may be utiized For exampe, different versions of the momentum method or the conjugate gradient procedure coud have their respective advantages 5

6 Tabe 1: Pseudocode of the expicit MLK backpropagation agorithm Inputs sampe points: {x(t), d(t)} t1,,t,t cost function: λ i 0 ( 1,, L; i 1,, N S ) earning rates: µ i (t) > 0 ( 1,, L; i 1,, N S ; t 1,, T ) Network initiaization size: L (number of ayers), N I, N S, N o ( 1,, L) parameters: w i (1) ( 1,, L; i 1,, N S ) Start computation Choose sampe x(t) Feedforward computation x (t) ( 2,, L + 1), s (t) ( 2,, L) a Backpropagation of error L whie 1 if ( L) δ L (t) 2 [y(t) d(t)] T (g L) (s L (t)) ese d[s +1 (t)] d[ w +1 i (t),ϕ +1 (u) H +1] [ (g d[s (t)] d[u] ux +1 (t) ) (s b (t))] δ (t) δ +1 (t) d[s+1 (t)] d[s (t)] Weight update for a i: 1 i NS wi (t + 1) (1 2µ i (t) λ i ) w i (t) µ i (t) δ i (t) ϕ (x (t)) 1 End computation a The output of the network, ie, y(t) x L+1 (t) is aso computed b Here: i 1,, N +1 S 6

7 Tabe 2: Pseudocode of the impicit MLK backpropagation agorithm Inputs sampe points: {x(t), d(t)} t1,,t,t cost function: λ i 0 ( 1,, L; i 1,, N S ) earning rates: µ i (t) > 0 ( 1,, L; i 1,, N S ; t 1,, T ) Network initiaization size: L (number of ayers), N I, N S, N o ( 1,, L) parameters: w i (1)-expansions ( 1,, L; i 1,, N S ) coefficients: α i (1) RN i (1) ancestors: z i,j (1), where j 1,, N i (1) Start computation Choose sampe x(t) Feedforward computation x (t) ( 2,, L + 1), s (t) ( 2,, L) a Backpropagation of error L whie 1 if ( L) δ L (t) 2 [y(t) d(t)] T (g L) (s L (t)) ese d[s +1 N (t)] +1 i (t) d[s (t)] α +1 ij (t) [k +1 ] y(z +1 ij (t), x +1 (t)) j1 δ (t) δ +1 (t) d[s+1 (t)] d[s (t)] Weight update for a i: 1 i NS Ni (t + 1) N i (t) + 1 α i (t + 1) [( 1 2µ i (t) ) λ i α i (t); µ i (t) δ i (t)] z i,j (t + 1) z i,j (t) (j 1,, N i (t)) z i,j (t + 1) x (t) (j Ni (t + 1)) 1 End computation a The output of the network, ie, y(t) x L+1 (t) is aso computed b i 1,, N +1 S Note aso that (k ) y denotes the derivative of kerne k according to its second argument [ (g ) ] (s b (t)) 7

8 terms: c(t) ε 2 (t) + r(t) (15) 311 Gradient of the Approximation Term First, we ist basic reations, invoved by the MLK structure For the case of simpicity, beow, index t sha be dropped [precise form: x x (t), y y (t), s s (t),w i w i (t)] x y 1 R N I ( 1,, L + 1) (16) x +1 g (s ) ( 1,, L) (17) w 1, ϕ (x ) H s w i, ϕ (x ) H ( 1,, L; i 1,, N S) (18) w 1, ϕ (g 1 (s 1 )) H w i, ϕ (g 1 (s 1 )) H ( 2,, L; i 1,, N S) (19) w +1 1, ϕ +1 (g (s )) H +1 s +1 w +1 i, ϕ +1 (g (s )) H +1 ( 1,, L 1; i 1,, N +1 S ) Let the backpropagated error for ayer be defined as δ (t) : d[ε2 (t)] d[s (t)] (20) ( 1,, L) (21) The specia case of the ast ayer is as foows: [ δ L (t) d[ε2 (t)] d d(t) g L (s L (t)) ] 2 d[s L (t)] 2 d[s L (t)] (22) 2 [g L ( s L (t) ) d(t) ] T ( g L ) ( s L (t) ) (23) 2 [y(t) d(t)] T (g L) (s L (t)) (24) Here we used the chain rue and made use of the rue vaid for vectors d[ d y 2 2 ] dy 2(y d) T, (25) 8

9 and inserted the reation imposed by the MLK architecture Expression d[s +1 (t)] d[s (t)] y(t) g L ( s L (t) ), (26) ( 1,, L 1) (27) can be computed by using Eq (20) It is sufficient to consider terms ike d[ w, ϕ(g(s)) H ] (28) and then to compie the fu derivative from them The vaue of (28) is computed by means of the foowing emma Lemma 1 Let w H H(k) be a point in the RKHS Let us assume the foowing 1 Let kerne k be differentiabe wrt both arguments and et k y denote the derivative of the kerne according to its second argument 2 In the impicit case we aso assume that w is within the image space of the feature space representation of a finite number of points z i That is w Im (ϕ(z 1 ), ϕ(z 2 ),, ϕ(z N )) H (29) Let this expansion be w N α j ϕ(z j ), where α j R Then we have two cases: 1 Expicit case: 2 Impicit case: Proof j1 d[ w, ϕ(g(s)) H ] d[ w, ϕ(g(s)) H ] d [ w, ϕ(u) H ] d[u] g (s) (30) ug(s) N α j k y(z j, g(s)) g (s) (31) j1 1 Expicit case: the statement foows from the chain rue 9

10 2 Impicit case: d[ w, ϕ(g(s)) H ] j [ ] d j α j ϕ(z j ), ϕ(g(s)) H [ ] d j α j ϕ(z j ), ϕ(g(s)) H [ ] d j α j k (z j, g(s)) (32) (33) (34) α j k y(z j, g(s)) g (s) (35) The first equation has the expansion of w and the inear property of the scaar product was utiized Then, the reation k(u, v) ϕ(u), ϕ(v) H (36) between feature mapping and the kerne was appied The ast step foows from the chain rue Let us turn back to the computation of Eq (27): 1 Expicit case: According to the emma we have d[s +1 (t)] d[ w d[s +1 i (t),ϕ +1 (u) H +1] (t)] d[u] d[ w +1 i (t),ϕ +1 (u) H +1] d[u] ug (s (t)) ux +1 (t) ( 1,, L 1; i 1,, N +1 S ) (g ) (s (t)) (37) [ (g ) (s (t))] (38) In the second equation (i) we used identity (17) and (ii) pued out the term ( g ) ( s (t) ) according to the matrix mutipication rues 2 Impicit case: For terms w +1 i (t) we have the expansion property expressed by Eq (13) This was our starting assumption In subsection 313, we sha see that this feature is inherited from time to time Thus, w +1 i (t) N +1 i (t) j1 α +1 ij (t) ϕ +1 (z +1 ij (t)) ( 1,, L 1; i 1,, N +1 S ) (39) 10

11 and the derivative (27) we need assumes the form d[s +1 (t)] d[s (t)] N +1 i (t) α +1 j1 N +1 i (t) α +1 j1 ij (t) [k +1 ] y(z +1 ij (t), g (s (t))) (g ) (s (t)) ij (t) [k +1 ] y(z +1 ij (t), x +1 (t)) ( 1,, L 1; i 1,, N +1 S ) (40) [ (g ) ] (s (t)) (41) Here, the second equation is based on identity (17) Matrix term ( g ) ( s (t) ) was pued out according to the matrix mutipication rues Appying the chain rue and the definition of δ +1 (t), we have δ (t) d[ε2 (t)] d[s (t)] d[ε2 (t)] d[s +1 (t)] d[s+1 (t)] d[s δ +1 (t) d[s+1 (t)] (t)] d[s (t)] ( 1,, L 1) (42) One can appy the chain rue once again and can make use the definitions of δ (t) and s (t) to show that d[ε 2 (t)] d[wi (t)] d[ε2 (t)] d[s i (t)] d[s i (t)] d[wi (t)] δ i(t) ϕ (x (t)) ( 1,, L; i 1,, NS), (43) which is the desired derivative Note that the derivative can be expressed by using number δi (t) and by the feature representation of the input x (t) arriving to the th ayer, ie, by ϕ (x (t)) 312 Reguarization Term This term is reativey simpe: [ L N S d λ i d[r(t)] w ] i (t) 2 H d[wi (t)] 1 i1 d[wi (t)] 2λ i wi(t) ( 1,, L; i 1,, NS) (44) Note that the respective terms of the derivative are scaed actua weights [wi (t)] This form enabes our impicit tuning 11

12 313 Cost Term Using identity d[c(t)] d[wi (t)] d[ε2 (t)] d[r(t)] d[wi + (t)] d[wi (t)] ( 1,, L; i 1,, NS) (45) as we as our resuts on the approximation and the reguarization terms [ie, Eqs (43), and (44)], we arrive to the steepest descent form w i(t + 1) w i(t) µ i(t) So we have d[c(t)] d[w i (t)] ( 1,, L; i 1,, N S) (46) w i(t + 1) w i(t) µ i(t) (δ i(t) ϕ (x (t)) + 2λ i w i(t) ) (47) The same in dua form is as foows (1 2µ i(t) λ i) w i(t) µ i(t) δ i(t) ϕ (x (t)) (48) ( 1,, L; i 1,, N S) α i(t + 1) [ ( 1 2µ i(t) λ i) α i (t); µ i(t) δi(t)] ( 1,, L; i 1,, NS) (49) In turn, the expansion property of the weight vectors of the network [ie, Eq (13)] is inherited from time to time In particuar, the expansion is vaid for parameter set wi received at the end of the computation In summing up, MLK can be tuned by the backpropagation procedure The derived expicit and impicit procedures are summarized in Tabe 1 and Tabe 2, respectivey 4 Concusions Theoretica description of a nove mutiayer mode, the Mutiayer Kerceptron was provided This network unifies the advantages of Mutiayer Perceptrons and Support Vector Machines: (i) It earns the weights and earning is subject to reguarization (ii) MLK aows for feature representations (iii) MLK computes the output quicky through earned weights (iv) MLK can have hidden ayers, and thus, it can combine SVM partitionings Advantages and disadvantages of the approach for different databases remain to be seen References [1] N Aronszajn Theory of Reproducing Kernes Trans of Am Math Soc, 68: , 1950 [2] T Evgeniou, M Ponti, and T Poggio Reguarization Networks and Support Vector Machines Advances in Computationa Mathematics, 13(1):1 50,

13 [3] S Haykin Neura Networks Prentice Ha, New Jersey, USA, 1999 [4] R Herbrich Learning Kerne Cassifiers MIT Press, 2002 [5] K-R Müer, A Smoa, G Rätsch, B Schökopf, J Kohmorgen, and V Vapnik Predicting Time Series with Support Vector Machines In Advances in Kerne Methods, pages MIT Press, 1999 [6] B Schökopf and AJ Smoa Learning with Kernes MIT Press, Cambridge, MA, 2002 [7] J Shawe-Tayor and N Cristianini Kerne Methods for Pattern Anaysis Cambridge University Press, 2004 [8] V Vapnik, S Goowich, and A Smoa Support Vector Method for Function Estimation, Regression Estimation and Signa Processing, voume Vo 9 MIT Press, Cambridge, MA, neura information processing systems edition, 1997 [9] VN Vapnik The Nature of Statistica Learning Theory Springer-Verag New York, Inc, 1995 [10] VN Vapnik Statistica Learning Theory Wiey, Chichester, GB, 1998 [11] G Wahba Support Vector Machines, Reproducing Kerne Hibert Spaces, and Randomized GACV In Advances in Kerne Methods, pages MIT Press,