Sensitivity of Nonlinear Network Training to Affine Transformed Inputs

Transcription

1 Sensitiity of Nonlinear Network raining to Affine ransformed Inuts Changhua Yu, Michael Manry, Pramod Lakshmi Narasimha Deartment of Electrical Engineering Uniersity of exas at Arlington, X changhua yu@utaedu, manry@utaedu, l ramod@utaedu Abstract In this aer, the effects of nonsingular affine transforms on arious nonlinear network training algorithms are analyzed It is shown that gradient related methods, are quite sensitie to an inut affine transform, while Newton related methods are inariant hese results gie a connection between re-rocessing techniques and weight initialization methods hey also exlain the adantages of Newton related methods oer other algorithms Numerical results alidate the theoretical analyses Introduction Nonlinear networks, such as multi-layer ercetron (MLP) and radial basis function (RBF) networks, are oular in the signal rocessing and attern recognition area he networks arameters are adjusted to best aroximate the underlying relationshi between inut and outut training data (Vanik 1995) Neural net training algorithms are time consuming, in art due to ill-conditioning roblems (Saarinen, Bramley, & Cybenko 1993) Pre-rocessing techniques, such as inut feature-decorrelation, whitening transform (LeCun et al 1998), (Brause & Ril 1998) are suggested to alleiate ill-conditioning and thus accelerate the network training rocedure (Š Raudys 2001) Other inut transforms, including inut re-scaling (Rigler, Irine, & Vogl 1991), unbiasing and normalization, are also widely used to equalize the influence of the inut features In addition, some researchers think of the MLP as a nonlinear adatie filter (Haykin 1996) Linear re-rocessing techniques, such as noise cancellation, can imroe the erformance of this non-linear filter Although widely used, the effects of these linear transforms on MLP training algorithms haen t been analyzed in detail Interesting issues, such as (1)whether these re-rocessing techniques hae same effects on different training algorithms, (2) whether the benefits of these rerocessing can be dulicated or cancelled out by other strategies, ie, adanced weight initialization method, still need his work was suorted by the Adanced echnology Program of the state of exas, under grant number Coyright c 2005, American Association for Artificial Intelligence (wwwaaaiorg) All rights resered further research In a reious aer (Yu, Manry, & Li 2004), the effects of an inut orthogonal transform on the conjugate gradient algorithm was analyzed using the concet of equialent states We show that the effect of inut orthogonal transform can be absorbed by roer weight initialization strategy Because all linear transforms can be exressed as an affine transform, in this aer, we analyze the influence of the more general affine transform on tyical training algorithms First, the conentional affine transform is described Second, tyical training algorithms are briefly reiewed hen, the sensitiity of arious training algorithms to inut affine transforms are analyzed Numerical simulations are resented to erify the theoretical results General Affine ransform Suggested re-rocessing techniques, such as feature decorrelation, whitening, inut unbiasing and normalization can all be ut into the form of nonsingular affine transform: z = Ax + b (1) where x is the original th inut feature ector, z is the affine transformed inut, and b = b 1 b 2 b N For examle, unbiasing the inut features is modelled as z = x m x (2) An affine transform of articular interest in this aer can be exressed as a linear transform by using extended inut ectors as: Z = A e X (3) where Z = z 1 z N, 1, X = x 1 x N, 1 and a 11 a 1N b 1 A e = a N1 a NN b, (4) N Conentional raining Algorithms Generally, training algorithms for nonlinear networks can be classified into three categories: gradient descent methods, conjugate gradient methods, and Newton related methods (Haykin 1999), (Møller 1997) In the following, we gie a brief reiew for each method

2 Figure 1: oology of a three-layer MLP Notation In this aer, we inestigate a three-layer fully connected MLP, as shown in figure 1 he training data consists of training atterns {(x, t )}, where x and the th desired outut ector t hae dimensions N and M, resectiely hresholds in the hidden and outut layers are handled by letting x,(n+1) = 1 For the jth hidden unit, the net function net j and the outut actiation O j for the th training attern are net j = N+1 w hi (j, n) x n (5) n=1 O j = f(net j ) 1 j N h where x n denotes the nth element of x, w hi (j, n) denotes the weight connecting the nth inut to the jth hidden unit and N h denotes the number of hidden units he actiation function f is sigmoidal 1 f(net j ) = 1 + e net j he kth outut y k for the th training attern is y k = N+1 n=1 (6) N h w oi (k, n) x n + w oh (k, j) O j (7) j=1 k = 1,, M, where w oi (k, n) denotes the weight connecting the nth inut node to the kth outut unit he network s training rocedure is to minimize the following global Mean Square Error (): E = 1 m=1 M t m y m 2 = 1 (t y )(t y ) (8) where t m denotes the mth element of t Many methods hae been roosed to sole this otimization roblem Gradient Descent Method he idea of gradient descent method is to minimize the linear aroximation of in (8): E(w + w) E(w) + w w = E(w) + w g (9) where g is the gradient ector of E(w) ealuated at current w he weight udating strategy is: w = Z g (10) where the learning factor Z is usually set to a small ositie constant or determined by an adatie or otimal method(magoulas, Vrahatis, & Androulakis 1999) he back roagation (BP) algorithm is actually a gradient descent method (Haykin 1999), (Rumelhart, Hiton, & Williams 1986) When using an otimal learning factor, the gradient descent method is also known as steeest descent (Fletcher 1987) Conjugate Gradient Method Peole hae looked for some comromise between slow conerging gradient methods and comutationally exensie Newton s methods (Møller 1997) Conjugate gradient (CG) is such an intermediate method he basic idea of CG is to search the minimum of the error function in conjugate directions (Fletcher 1987) In CG, the directions P(i) for the ith searching iteration and P(j) for the jth iteration hae to be conjugate with resect to the Hessian matrix: P (i)hp(j) = 0 for i j (11) he conjugate directions can be constructed by (Fletcher 1987): P(k) = g(k) + B 1 P(k 1) (12) Here, g(k) reresents the gradient of E with resect to corresonding weight and threshold for the kth searching iteration Initial ectors are g(0) = 0, P(0) = 0 B 1 in Eq (12) is the ratio of the gradient energies between the current iteration and the reious iteration he corresonding weight changes in CG are: w = B 2 P(k) (13) where B 2 is an otimal learning factor Newton s Method he goal in Newton s method is to minimize the quadratic aroximation of the error function E(w) around the current weights w: E(w + w) E(w) + w g w H w (14) Here, the matrix H is called the Hessian matrix, which is the second order artial deriaties of E with resect to w Eq (14) is minimized when w = H 1 g (15) For Newton s method to work, the Hessian matrix H has to be ositie definite, which is not guaranteed in most situations Many modified Newton algorithms hae been deeloed to ensure a ositie definite Hessian matrix (Battiti 1992) Sensitiity of raining to Affine ransform For affine transforms, the requirements to ensure equialent states (Yu, Manry, & Li 2004) in the network with original inuts and the network with transformed inuts are: w hi = u hi A e (16) w oh = u oh, w oi = u oi A e (17) where u denote the corresonding weights and thresholds in the transformed network

3 Sensitiity of CG to Affine ransform In CG, for the kth training iteration, the weight udating law in the transformed network is: u hi u hi + B 2 P dhi (k), u oi u oi + B 2 P doi (k) u oh u oh + B 2 P doh (k) (18) he weights and thresholds in the original network are udated in the same way If the two networks hae the same learning factors B 2, the conjugate directions P must satisfy following conditions to ensure equialent states after weights modification: P hi = P dhi A e, P oh = P doh, P oi = P doi A e (19) he gradients in the transformed network can be found: u oi(1,1) u oi(1,n+1) g doi = u oi = u oi (M,1) u oi (M,N+1) e d1 ( Z N ) e d X A e N e dm (20) where e dm = t m y dm m = 1, M g dhi = u hi = and g doh is: δ d1 δ dnh u hi (1,1) u hi (1,N) u hi (N h,1) g doh Z ( N u hi (N h,n) ) δ d X A e (21) e d O d (22) In the original network, the matrices are: g oi e X, g oh g hi δ X (23) e O From Eq (12), after the first training iteration in CG, the conjugate directions are: P(1) = g(1) (24) Assuming the two networks start from same initial states, for the first training iteration, we hae from (20) that: P doi = g doi = g oi A e = P oi A e (25) As A e is not orthogonal, condition (19) can t be satisfied So the training cures in the two networks dierge from the ery beginning B 1 and B 2 in the two networks won t be the same either herefore the CG algorithm is sensitie to inut affine transforms Affine ransforms and OWO-BP In outut weight otimization (OWO), which is a comonent of seeral training algorithms (Chen, Manry, & Chandrasekaran 1999), we sole linear equations for MLP outut weights In the following, we will use W o = w oi w oh and an augmented inut ector: Ô = x 1 x N 1 O 1 O Nh (26) For the three-layer network in figure 1, the outut weights can be found by soling linear equations, which result when gradients of E with resect to the outut weights are set to zero Using equation (7) and (8), we find the gradients of E with resect to the outut weights as ĝ o (m, j) = 1 2 L t m w o (m, i)ôi i=1 L C(m, j) w o (m, i)r(i, j) i=1 Ôj (27) where the autocorrelation matrix R has the elements: R(i, j) = 1 Ô i Ô j (28) the cross-correlation matrix C has the elements: C(m, j) = 1 t m Ô j (29) Setting ĝ o (m, j) to zero, the jth equation in the mth set of equations is L w o (m, i)r(i, j) = C(m, j) (30) i=1 So the OWO rocedure in the two networks means to sole W o R x = C x U o R z = C z (31) If before OWO the two networks hae equialent states, we hae Ô Ae 0 d = Ô (32) Nh N h And in the original network, R x = 1 Ô N Ô C x = 1 t N Ô (33) For the network with transformed inuts, these matrices are: R z = 1 Ae 0 = Ô dôd R x A e 0 C z = 1 t A N Ôd = C e 0 x (34) (35)

4 Inariance of OWO to Nonsingular Affine ransform In the following lemma, we describe the effects of affine transforms on the OWO rocedure Lemma 1 For any nonsingular matrix A e, if outut weights satisfy the conditions of Eq (17), then after the OWO rocedure it is still alid Proof As the two networks start from equialent states, U o R z = C z in Eq (31) can be rewritten as: Ae 0 A U o R e 0 x it can be simlified further as: U o Ae 0 = C x A e 0 (36) R x = C x (37) Comaring it with W o R x = C x, so we hae Ae 0 W o = U o which means condition (17) is still alid (38) he Sensitiity of BP to Affine ransform For BP rocedure, in order to kee equialent states in the two networks, the gradients must satisfy: g hi = g dhi A e (39) As we know, when B 1 in (12) is set to zero, CG degrades into BP algorithm hus the analyses for CG in is also alid for BP Condition (39) won t be satisfied after BP So BP is also sensitie to the inut affine transforms In OWO-BP training (Chen, Manry, & Chandrasekaran 1999), we alternately use OWO to find outut weights and use BP to modify hidden weights Because of the sensitiity of BP to affine transform, OWO-BP has a similar sensitiity Effects on Newton Related Method Generally, it is much more difficult to find the hidden weights in neural networks Due to its high conergence rate, Newton s method is referred for small scale roblems (Battiti 1992) he roosed hidden weight otimization (HWO) in (Chen, Manry, & Chandrasekaran 1999) can be considered as a Newton related method o ensure equialent states in the two networks, the hidden weights and thresholds must satisfy (16) and (17), and corresondingly, the hidden weight changes in the two networks must satisfy: w hi = u hi A e (40) Rearrange w hi in the form of a column ector: W = w hi (1, 1) w hi (N h, N + 1) (41) and the corresonding column ector in the transformed network is denoted as U So condition (40) is equialent to: W = à U (42) where A e A à = e A e q q (43) with q = N h (N + 1) he hidden weights are udated by the law of (15) he Hessian matrix of the original network is: H = 2 E whi 2 (1,1) 2 E 2 E 2 E whi 2 (N h,n+1) (44) he gradient ector used in the Newton method is: g = w hi (1,N+1) δ1 x 1 δ 1 x (N+1) δ Nh x (N+1) X δ 1 X δ Nh Λ x δ (45) where Λ x = diag{x 1 x (N+1) x 1 x (N+1) } is a q q matrix, and δ = δ 1 δ 1 δ Nh δ Nh (46) is a q 1 ector Similarly, in the transformed network, g d = u hi (1,1) u hi (N h,n+1) δd1 z 1 δ dnh z (N+1) Z δ d1 Z δ dnh Λ x δ d Ã Λ x δd (47) with δ d = δ d1 δ d1 δ dnh δ dnh (48) From Eq (44) and (45), the q q Hessian matrix in the original network is H Λ x δ 1 δ 1 δ Nh δ Nh δ 1 δ 1 δ Nh δ Nh (49)

5 In the rightmost matrix of (49), eery (N + 1) rows from the (i 1)(N + 1) + 1th row to the i (N + 1)th row are exactly same, with i = 1,, N h he elements δ i w hi (j, n) = γ ijx n (50) where i, j = 1,, N h, i j, n = 1,, (N + 1), and M γ ij = w oh (m, j)w oh (m, i)f jf i (51) m=1 It is easy to erify that γ ij = γ ji For the case i = j, with γ ii = M m=1 δ i w hi (i, n) = γ iix n (52) w 2 oh(m, i)(f i) 2 t m y m w oh (m, i)f i (53) Here, f i, f i are shorthand for the first order and second order deriaties of the sigmoid function f(net i ) So, we hae H = 2 Λ x Ψ (54) where γ 11 X γ 1Nh X Ψ = (55) γ Nh 1X γ Nh N h X where eery row in the q q matrix X is just the ector X Similarly, for the transformed network, the corresonding Hessian matrix is: H d = 2 à γ d11 Z γ d1nh Z Λ x N γ dnh 1Z γ dnh N h Z (56) where eery row in the q q matrix Z is just the ector Z Using the fact that Z = XA e (57) Eq (56) becomes with H d = { } 2 Ã Λ x Ψ d à (58) γ d11 Z γ d1nh Z Ψ d = (59) γ dnh 1Z γ dnh N h Z heorem 1 If two networks are initially equialent, and their inut ectors are related ia a nonsingular affine transform, the networks are still equialent after Newton training Proof When the two networks are in equialent states, conditions (16) is satisfied and we hae: δ = δ d, Ψ = Ψ d (60) so the change of the hidden weights and thresholds in the transformed network are { } ) 1 1 U = H 1 d g d = (Ã Λ x Ψ d { } à 1 Ã Λ x δ d (61) { } ) 1 { } 1 = (Ã Λ x Ψ Λ x δ ) 1 ) 1 = (à H 1 g = (à W So the required condition (42) would be satisfied after the Newton training rocedure, which means that the conditions (16) would be alid again Numerical Results In the exeriments, we use the unbiasing rocedure as an examle of the affine transform, and erify the theoretical results for two remote sensing data sets Inuts for training data set Oh7tra are V V and HH olarization at L 30, 40 deg, C 10, 30, 40, 50, 60 deg, and X 30, 40, 50 deg (Oh, Sarabandi, & Ulaby 1992) he corresonding desired oututs are Θ = {σ, l, m }, where σ is the rms surface height; l is the surface correlation length; m is the olumetric soil moisture content in g/cm 3 here are 20 inuts, 3 oututs, training atterns We use 20 hidden units raining data set wodtra contains simulated data based on models from backscattering measurements (Dawson, Fung, & Manry 1993) his training file is used in the task of inerting the surface scattering arameters from an inhomogeneous layer aboe a homogeneous half sace, where both interfaces are randomly rough he arameters to be inerted are the effectie ermittiity of the surface, the normalized rms height, the normalized surface correlation length, the otical deth, and single scattering albedo of an in-homogeneous irregular layer aboe a homogeneous half sace from back scattering measurements he data set has 8 inuts, 7 oututs, and 1768 atterns here are 10 hidden units (a) oh7dat (b) twodtra Figure 2: he sensitiity of CG to affine transforms

6 (a) oh7dat (b) twodtra Figure 3: he effect of affine transform on OWO-BP 18 (a) oh7dat (b) twodtra Figure 4: he inariance of OWO-HWO to affine transforms For both data sets, the original and transformed networks start from equialent states and are trained for 200 iterations From fig 2 and fig 3, we can see that CG and BP are quite sensitie to the affine transform Een though the two networks start from same initial states, the difference of their training cures are quite large For the Newton related OWO-HWO method, the training cures for the two networks in fig 4 are almost the same he slight differences are caused by the finite numerical recision roblem Summary and Discussion In this aer, we analyze the effects of the affine transform on different training algorithms Using inut unbiasing as an examle, we show that CG and BP are quite sensitie to inut affine transforms As a result, the inut unbiasing strategy hels to accelerate the training of these gradient algorithms As for Newton methods, the network with original inuts and the network with transformed inuts hae the same states during the training rocedure as long as they start from equialent states his imlies that OWO-HWO is more owerful than non-newton algorithms when dealing with correlated/biased data hat is, in Newton methods, the benefits of feature de-correlation or unbiasing hae already been realized by roer initial weights his gies a connection between the re-rocessing and the weight initialization rocedure Howeer, this also indicates a trade-off Sometimes the effects of re-rocessing may be cancelled out by the initial weights Naturally, further inestigation can be done on whether there is a similar connection between the rerocessing and other techniques, for examle, the learning factor setting With theoretical analyses on these techniques, better neural network training strategy can be deeloed References Battiti, R 1992 First- and second-order methods for learning: between steeest descent and newton s method Neural Comutation 4(2): Brause, R W, and Ril, M 1998 Noise suressing sensor encoding and neural signal orthonormalization IEEE rans Neural Networks 9(4): Š Raudys 2001 Statistical and Neural Classifiers: An Integrated Aroach to Design Berlin, Germany: Sringer- Verlag Chen, H H; Manry, M ; and Chandrasekaran, H 1999 A neural network training algorithm utilizing multile sets of linear equations Neurocomuting 25(1-3):55 72 Dawson, M S; Fung, A K; and Manry, M 1993 Surface arameter retrieal using fast learning neural networks Remote Sensing Reiews 7(1):1 18 Fletcher, R 1987 Practical Methods of Otimization Chichester, NY: John Wiley & Sons, second edition Haykin, S 1996 Adatie Filter heory Englewood Cliffs, NJ: Prentice Hall, third edition Haykin, S 1999 Neural Networks: A Comrehensie Foundation Englewood Cliffs, NJ: Prentice Hall, second edition LeCun, Y; Bottou, L; Orr, G B; and Muller, K R 1998 Efficient backro In Orr, G B, and Muller, K R, eds, Neural Networks: ricks of the rade Berlin, Germany: Sringer-Verlag Magoulas, G D; Vrahatis, M N; and Androulakis, G S 1999 Imroing the conergence of the backroagation algorithm using learning adatation methods Neural Comutation 11(8): Møller, M 1997 Efficient raining of Feed-forward Neural Networks PhD Dissertation, Aarhus Uniersity, Denmark Oh, Y; Sarabandi, K; and Ulaby, F 1992 An emirical model and an inersion technique for radar scattering from bare soil surfaces IEEE rans Geosci Remote Sensing 30: Rigler, A K; Irine, J M; and Vogl, P 1991 Rescaling of ariables in back roagation learning Neural Networks 4: Rumelhart, D E; Hiton, G E; and Williams, R J 1986 Learning internal reresentation by error roagation, olume 1 of Parallel Distributed Processing Cambridge, MA: MI Press Saarinen, S; Bramley, R; and Cybenko, G 1993 Illconditioning in neural network training roblems SIAM Journal on Scientific Comuting 14: Vanik, V N 1995 he Nature of Statistical Learning heory New York: Sringer-Verlag Yu, C; Manry, M ; and Li, J 2004 Inariance of ml training to inut feature de-correlation In Proceedings of the 17th International Conference of the Florida AI Research Society,