* o * * o o 1.5. o o. o * 0.5 * * o * * o. o o. oo o. * * * o * 0.5. o o * * * o o. * * oo. o o o o. * o * * o* o

Transcription

1 Pseudinverse Learning Algrithm fr Feedfrward Neural Netwrs Λ PING GUO and MICHAEL R. LYU Department f Cmputer Science and Engineering, The Chinese University f Hng Kng, Shatin, NT, Hng Kng. SAR f P.R. CHINA pgu@cse.cuh.edu.h, lyu@cse.cuh.edu.h Abstract: - A supervised learning algrithm (Pseudinverse Learning Algrithm, PIL) fr feedfrward neural netwrs is develped. The algrithm is based n generalized linear algebraic methds and it adpts matrix inner prducts and pseudinverse peratins. The algrithm eliminates learning errrs by adding hidden layers and will give a perfect learning. Unlie the gradient descent algrithm, the PIL is a feedfrwardnly, fully autmated algrithm, including n critical user-dependent parameters such as learning rate r mmentum cnstant. Key-Wrds: - Feedfrward neural netwrs, Supervised learning, Generalized linear algebra, Pseudinverse learning algrithm, Fast perfect learning. Intrductin Several adaptive learning algrithms fr multilayer feedfrward neural netwrs have been prpsed[][2]. Mst f these algrithms are based n variatins f the gradient descent algrithm, fr example, Bac Prpagatin (BP) algrithm[]. They usually have a pr cnvergence rate and smetimes fall int lcal minima[3]. Cnvergence t lcal minima can be caused by the insufficient number f hidden neurns as well as imprper initial weight settings. Hwever, slw cnvergence rate is a cmmn prblem f the gradient descent methds, including the BP algrithm. Varius attempts have been made t speed up learning, such as prper initializatin f weight taviding lcal minima, an adaptive least square algrithm using the secnd rder terms f errr fr weight updating[4]. There is anther drawbac fr mst gradient descent algrithms, namely, learning factrs prblem", such as learning rate, mmentum cnstant. The values f these parameters are ften crucial fr the success f the algrithm. Mst gradient descent methds depend n these parameters which have t be specified by the user, as n theretical basis fr chsing them exists. Furthermre, fr applicatins which require high precisin utput, such as the predictin f chatic time series, the nwn algrithms are ften still t slw and inefficient. Fr example, lie staced general- Λ This research wr was fully supprted by a grant frm the Research Grants Cuncil f the Hng Kng Special Administrative Regin (Prject N. CUHK493/E). izatin[5], which needs t train a lt f netwrs t get level training samples, it is very cmputatintime cnsuming when using BP algrithm t perfrm the required tas. In rder t reduce training time and investigate the generalizatin prperties f learned neural netwr, in this paper a Pseudinverse Learning algrithm (PIL) is prpsed, which is a feedfrwardnly algrithm. Learning errrs are transferred frward and the netwr architecture established. The trained weights previusly in the netwr are nt changed. Hence, the learning errr is minimized n each layer separately and nt glbally fr the netwr as a whle. By adding layers t eliminate errrs, all examples f a training set can be perfect learned. 2 The Netwr Structure and Learning Algrithm 2. The Netwr Structure Let us cnsider a multilayer feedfrward neural netwr. The netwr has ne input layer, ne utput layer and several hidden layers. While the number f hidden layer depend n the desired learning accuracy and the examples f a training set t be learned in this paper. The weight matrix W l cnnects layer l and layer l + with elements wi;j. l Element wi;j l cnnects neurns i f layer l with neurns j f layer l +. Nte that the W matrix cnnects the input layer and the first hidden layer, the W L matrix cnnects

2 the last hidden layer and the utput layer. We assume nly the input layer has bias neurn, while the hidden layer(s) and the utput layer have n bias neurn. The nnlinear activate functin is ff( ), fr example, we can use s called sigmidal functin, ff(x) = +e x () which utput is in a range f (,), r a hyperblic functin tanh(x) = ex e x e x +e x (2) which utput is in a range f (-,) as an activate functin. Given a training data set D = fx i ; i g N i= ; Let (x i ; i ) be the i th input-utput training pair, where x i = (x ;x 2 ;:::;x n ) 2 R n is the input signal vectr and i = ( ; 2 ;:::; m ) 2 R m is the crrespnd target utput vectr. Fr given N sets f input utput vectr pairs as examples t be learned, we can summarize all given input vectrs int a matrix X with N rws and n + clumns. Here the last clumn f X is a bias neurn f cnstant value Each rwfx cntains the signals f ne input vectr. X =[Xj ], where matrix X cnsist f all signal x i as rw vectrs. All desired target utput vectrs are summarized int a matrix O with N rws and m clumns. Each rwf the matrix O cntains the signals f ne utput vectr i. In the designed netwr structure, the activate functin is nt applied t the utput layer, s the last layer is linear. Basically, the tas f training the netwr means trying t find the weight matrix which minimizes the sum-square-errr functin, E = 2N NX mx i= j= jjg j (x i ; ) i jjj 2 : (3) Where g(x; ) is a netwr mapping functin and is the netwr parameter set. In three layer structure case, g j (x; ) = NX i= w i;j ff i ( nx l= w i;l x l + i ): (4) where i is a bias value fr netwr input. Fr simplifying, we can write the system cst functin in matrix frm, E = 2N Trace[(G O)T (G O)]; (5) Prpagating the given examples thrugh the netwr, multiplying the utput f layer l with the weights between layers l and l +, and applying the nnlinear activate functin t all matrix elements, we get: Y l+ = ff(y l W l ); (6) and the netwr utput shuld be G = Y L W L : (7) where we use superscript L t dnate the last hidden layer utput and the last weight matrix. By examining the abve equatins, refrmulating the tas f training, the prblem becmes; minimize jjy L W L Ojj 2 : (8) This is a linear least square prblem. If we can find the netwr weight parameter such that maes jjy L W L Ojj 2 =,wewill have trained the neural netwr t learn all given examples exactly, that is, a perfect learning. Withut lss generalizatin, in the fllwing discussin we drp superscript index L in equatin (7). 2.2 Pesudinverse Slutin Nw let us discuss the equatin YW = O; W 2 R p m ; Y 2 R N p ; O 2 R N m (9) When p<n;the system is underdetermined system. Ntice that such a system either has n slutin r has an infinity f slutins. If Y 2 R N N is invertible and has been learned in L layer, the system f equatin (9) is, in principle, easy t slve. The unique slutin fr last layer weight matrix is W = Y O. If Y is an arbitrary matrix in R N p ; then it becmes mre difficult t slve equatin (9). There may be nne, ne r an infinite number f slutins depending n where O 2 R(Y) and whether N ran(y) >. One wuld lie t be able t find a matrix (r matrices) C, such that slutin f 9 are f the frm CO. But if O =2 R(Y); then equatin (9) has n slutin. Frm linear algebra therem, it has: Therem The system YW = O has a slutin if and nly if ran([y; O]) = ran(y); () Prf: See reference [6]. We intend t use pseudinverse slutin fr finding weight matrix, the reasn is that the therem frm linear algebra states that pseudinverse slutin is the best apprximatin. Therem 2 Suppse that X 2 R p m ; A 2 R N p ; B 2 R N m ; The best apprximate slutin f the equatin AX = B is X = A + B (The superscript + dentes the pseudinverse matrix) Prf: Frm reference [6], it is easy prved. Frm the therem 2, it has, Crllary The best apprximate slutin f AX = I is X = A + :

3 Frm abve analysis, we try t find the utput layer weight in this way: Let W = Y + O, the learning prblem becmes jjyy + O Ojj 2 =, where Y + is the pseudinverse f Y. This is equal t find the matrix Y s that YY + I =, where I is the identity matrix. S the tas f training the netwr becmes that f managing t raise the ran f matrix Y up t full ran. As sn as Y becmes a full ran matrix, the YY + will becme the identity matrix I. Nte that since we multiply Y n the right by Y +, it needs nly requiring the right inverse f Y t exist, nt necessarily fr Y + t be a tw-sided inverse f Y. This means that Y needs nt be a square matrix, but its number f clumn shuld nt be less than its number f rw. This cnditin requires that hidden neurn numbers shuld be greater than r equal t N. If the cnditin is satisfied, we can find an exact slutin fr weight matrix. When we chse hidden neurn number t be equal t N, with such a netwr structure, we can find the weight matrix which can exactly mapping the training set. 2.3 Psudinverse Learning Algrithm Based n the abve discussin, we first let weight matrix W be equal t Yp which isann Nmatrix. Then we apply nnlinear activate functin, that is t cmpute Y = ff(y W ), then cmpute the (Y ) +, the pseudinverse f Y, and s n. Because the algrithm is feedfrward nly, n errr will prpagate bac t preceding layer f neural netwr. We cannt use standard errr frm E = Trace[(G 2N O)T (G O)] t judge whether the trained netwr has reached the desired accuracy during training prcedure. Instead, we use the criterin jjy l (Y l ) + Ijj 2 < E. At each layer, we cmpute jjy l Y l+ Ijj 2, If it is less than the desired errr, we set W L =(Y L ) + Oand stp the training prcedure. Otherwise, let W l = (Y l ) +, add anther layer, feed frward this layer utput t next layer again, until we reach the required learning accuracy. T use any nnlinear activate functin in the hidden ndes is t utilize the nnlinearity f the functin and t increase the linear independency amng the clumn vectrs r, equivalently, the ran f the matrix. It is prved that sigmidal functins can raise the dimensin f the input space up t the number f the hidden neurns[7]. S thrugh nnlinear activate actin, the ran f the weight matrix will be raised layer by layer. With abve discussin, we prpse a feedfrward-nly algrithm which reduce learning errrs n every layer. First we establish a tw layer neural netwr. If the given precisin cannt be reached, a third layer is added t eliminate the remained errr. If the third layer added still cannt satisfy the desired accuracy, then anther hidden layer is added again t reduce the learning errrs until the required accuracy is achieved. Frm a mathematical pint f view, we can summarize the algrithm int the fllwing steps: Step. Set hidden neurn number ass N; and let Y = X. Step 2. Cmpute (Y ) + =Pseudinverse(Y ). Step 3. Cmpute jjy l (Y l ) + Ijj 2. If it is less than the given errr E, g t step 6. If nt, g n t the next step. Step 4. Let W l = (Y l ) +. Feed frward the result t the next layer, cmpute Y l+ = ff(w l Y l ). Step 5. Cmpute (Y l+ ) + = Pseudinverse (Y l+ ), l ψ l +, and g t step 3. Step 6. Let W L =(Y L ) + O. Step 7. Stp training, the netwr mapping functin is g = ff(:::ff(w ff(w Y ))) W L : 3 Add and Delete Sample The prpsed algrithm is a batch way learning algrithm, in which we assume that all the input data are available at the time f training. Hwever, in real-time applicatin, as a new input vectr is given t the netwr, the weight matrix must be updated. Or, we need t delete a sample frm the learned weight matrix. It is nt efficient at all if we recmpute the pseudinvese f a new weight matrix with PIL algrithm. When we assign the hidden neurn number is equal t the number f training samples, add r delete the sample is equivalent with add r delete hidden neurn number. Here we use add r delete neurn algrithm t efficiently cmpute the pseudinverse matrix. Accrding t Griville's therem[8], the first clumns f the Y matrix cnsist f a submatrix, the pseudinverse f this submatrix can be calculated frm the previus ( )-th pseudinverse submatrix.» Y Y + + = (I y b T ) b T () where the vectr y is the -th clumn vectr f the matrix Y, while b = 8 < : (I Y Y + )y ; if CD 6= (Y + )T Y + y +jjy + y jj 2 ; therwise (2) where CD = jji Y Y + y jj. It needs at mst N times iterative cycle t btain the pseudinverse f a matrix if there are N clumns in this matrix.

4 With this therem, we can add the hidden neurns relative easy t calculate the pseudinverse matrix. When a hidden neurn is deleted, the matrix need t be update. It is nt efficient at all if we cmpute the pseudinverse matrix frm beginning. Here we cnsider using brdering algrithm[9] t cmpute the inverse f the matrix. Given the inverse t a matrix, the methd shws hw t find the inverse f a ( +) (+ ) matrix, which is the same ld matrix with an additinal rw and clumn attached t its brders. If the clumn vectr y i in Y is linearly independent each ther, by definitin, Y + =(Y T Y) Y T (3) Let V = Y T Y;we efficiently calculate V + frm the prir V withut inverting a matrix. The algrithm is V V = + ff vvt v ff + ff vt ff (4) where v = V Y T y + ; and ff = V YT y + When delete a vectr frm the matrix, cnsider the riginal matrix cntaining + vectr pairs. The ey step is t cmpute V frm V + : When the ( +) th pair is deleted frm the matrix, we rewrite V + as the fur partitins: V b T c + = A b (5) where A is ; b is ; and c is a scalar. By cmparing equatin 4, it is apparent that A = V + ff vvt ; b=(/ff)v; and c=/ff: Frm thee expressins, we find that the desired result is V = A c bbt (6) The inverse f matrix nw can be calculated frm ( +) ( + ) matrix. This is equivalent with deleting the last hidden neurn and updating the weight matrix. This will be very useful in the case f leave ne ut crss-validatin partitin training samples (CVPS). Because in each CVPS data set there nly ne sample is different with ttal sample set. We can first cmpute the inverse f matrix which is learned based n ttal sample set, then at each time, mve nly ne sample t the last clumn (rw) psitin, and use the abve algrithm t delete this sample. In this way we can btain the learned weight matrices with CVPS data sets efficiently. 4 Numerical Examples The algrithm is tested with the fllwing functin mapping examples. Example. Cnsider a nnlinear mapping prblem f Sine functin by neural netwr. Fr the training set, 5 input-utput signals (x i ;y i ) pairs were generated with x i = 2ß Λ i=49, fr i = ; ; 2; ;49, and crrespnd y i were cmputed using y i = sin(x i ). The given learning errr is E = 7. If learning errr E < 7, we regard that perfect learning has been reached. Fr this prblem, input neurn number is n + = 2 including bias ne, utput neurn is m = and hidden layer neurn number is N = 5. After using the PIL algrithm prpsed abve, we reach the perfect learning when tw hidden layers are added. The trained netwr altgether has 4 layers including input and utput layer. The actual learning errr is E =7: Example 2. The nnlinear mapping f 8 input quantities x i int three utput quantities y i prblem, defined by Biegler-König and Bärmann in []: y = (x Λ x 2 + x 3 Λ x 4 + x 5 Λ x 6 + x 7 Λ x 8 )=4: y 2 = (x + x 2 + x 3 + x 4 + x 5 + x 6 + x 7 + x 8 )=8: y 3 = ( y ) :5 (7) All three functins are defined fr values between and and prduce values in this range. Fr the training set, 5 sets f input signals x i were randmly generated in the range f t, the crrespnding y i were cmputed using the abve equatin. The desired learning errr we give is E = : 7. When training is finished, nly ne hidden layer is added, and the actual learning errr is E =3: fr this prblem. Example 3. Anther functinal mapping prblem is y = sin(x)cs(3x) + x=3. Lie example, we use 5 examples with x i in the regin f t ß t train the netwr. Perfect learning is reached after tw hidden layers are added. Actual learning errr is E =4: Generalizatin What is the generalizatin abilities f trained netwrs? We als tested the ability f trained netwrs t frecast functin values f examples nt belnging t the training set. Fr Sine functinal mapping, we train the netwr using 2 examples with x i =2ßΛi=9, fr i =;;2; ;9, and the crrespnding y i were cmputed using y i = sin(x i ). After the netwr is trained, N = input signals x i randmly generated within the range ft2ßare used t test the netwr, the crrespnding y i were cmputed using trained netwr. Figure a and b shw the results fr example

5 Output Output (a) (c) Output Output (b) (d) Fig.. The trained netwr utput fr (a) y = sin(x) functin mapping, (b) y = sin(x)cs(x) + x=3 functin mapping, (c) fr functin defined in equatin (8) with 2 learning examples, and (d) fr functin defined in equatin (8) with nly 5 learning examples. Λ" is training data, " is test data. and 3. It is reasnable gd. We have als tested examples 2 and 3 with 2 examples training netwr and using randmly generated input signals fr testing. Fr further investigating the prpsed netwr architecture and learning algrithm's respnse t unlearned data, let us see anther example. Example 4. A sin(x) lie nnlinear functin is defined by: y = 8 < : x; if» x<ß=2; ß x; if ß=2» x<3ß=2; x 2ß; if 3ß=2» x» 2ß: (8) First, 2 examples with x i = 2ß Λ i=9, fr i = ; ; 2; ;9, and the crrespnding y i were cmputed using abve equatin are used t train the netwr, then randm input signals generated in the range f t 2ß are used t test the trained netwr. The result is shwn in Figure c. When using 5 set examples f(,), (ß=2; ), (ß; ); (3ß=2; ); (2ß; )g t train the netwr, we get a netwr structure which has ne hidden layer with 5 hidden neurns. The learning errr is E =3: Afterward, sets f input signals x i which were randmly generated within the range f t 2ß are used t test the netwr. The result is shwn in Figure d. Frm the Figure d, it can be seen that the netwr acts lie asine functin. It shuld be reminded that the architecture and weight matrices is the same as in example and example 4 when using the abve 5 set examples. Frm this result, it can be nwn that the netwr frecast unlearned data ability is better fr smth functin when the data in the range f training input space. When 5 set examples with x i =2ßΛi=49, fr i =;;2; ;49, and the crrespnding y i were cmputed using crrespnding equatin are used t train the netwr, then randmly generated input signals in the range f t 2ß are used t test the trained netwr. In example and 4, nly W L matrix is different. The ther matrices are the same while 5 set examples are used t train the netwr. But the netwr's respnse t the same input matrix is ttally different. The methd f staced generalizatin[5] prvides a way f cmbining trained netwrs tgether which uses partitining f the data set t find an verall system with usually imprved generalizatin perfrmance. The experiments shw that with smthed functin r piece wise smthed functin, the trained netwr generalizatin perfrmance is gd with staced generalizatin. Hwever, fr nise data set, if the netwr is vertrained, the generalizatin ability will be pr. Using staced generalizatin can nt imprve the netwr perfrmance when ver trained netwrs are used. When verfitting t the nise ccur, staced generalizatin nt a suitable technique fr imprving netwr generalizatin perfrmance. We shuld see ther generalizatin techniques suchas ensemble netwrs[][2] t imprve the netwr perfrmance, but this tpic is beynd the scpe f this paper. 5 Discussin On examining the algrithm, it can be seen that we d nt need t cnsider the questin f hw the weight matrix shuld be initialized t avid lcal minima. We just feedfrward examples t get a weight matrix and the slutin will nt cnverge t lcal minima. This is different frm the BP algrithm. It can als be seen that training prcedure is in fact the prcessing f raising the ran f weight matrix. When a matrix f sme hidden layer utput becmes full ran, the right inverse f the matrix can be btained, and we end the training prcedure. Frm the learning prcedure, it is bvius that n differentiable activate functin is needed. We nly require that the activate functin can perfrm nnlinear transfrm t raise the ran f the weight matrix. Because the PIL algrithm is based n the nnlinear functin transfrmatin t raise the matrix ran, it will fail if there tw r mre input vectrs are identity in the input matrix. But this case can be eliminated thrugh preprcessing input patterns.

6 The BP as well as ther gradient algrithm requires user selected parameter, such as step size r mmentum cnstant. These parameters have effect n the learning speed. There is n theretical basis which guides us hw t select these parameters t speed learning. In PIL, such a prblem des nt exist. Anther characteristics is that if the input matrix has ran N then a right inverse exists, and we will get a linear netwr with nly tw layers. Fr mst prblems, with tw hidden layers, the netwr can reach the perfect learning. Frm the examples, we see that netwr layer number is nt nly dependent n learning accuracy, but als n the examples t be learned. The algrithm is suitable fr sme applicatins which require high precisin utput, in which case the netwr structure is less imprtant than precisin utput. One f the algrithm's imprtant feature is that desired utput matrix T is embedded in the weight matrix W L which cnnects last hidden layer and utput layer. This give us a very easy and fast way t get the weight matrix fr different target utput, as lng as input matrix is the same. Fr example, after we have trained the netwr t learning Sine functin mapping in the regin frm t 2ß, we nly need recalculate the W L, in rder t get Csine functin mapping prblem in the same regin with Sine functin. Fr BP algrithm, it is necessary t train whle netwr again t get all weight matrices fr Csine functin mapping, thugh input matrix is the same with Sine functin. We have nt cmpared the verall perfrmance f this algrithm with thers. Obviusly, the number f iteratins is nt avalid metric cnsidering the fact that the calculatin cmplexity per iteratin is nt the same fr any f the algrithms. Hwever, if we cnsider the CPU time cst n training netwr t reach the high learning accuracy using the same machine, the PIL algrithm is bviusly fast than ther gradient descent algrithms in its learning speed. 6 Summary The pseudinverse learning algrithm was intrduced in this paper. The algrithm is mre effective than the standard BP and ther gradient descent algrithm fr mst prblems. The algrithm des nt cntain any user-dependent parameters whse values are crucial fr the success f the algrithm. The mathematical peratins are simple, it is nly based n generalized linear algebra and adpt pseudinverse and matrix inner prduct peratins. On cnsidering its learning speed and accuracy, the PIL algrithm is mst cmpetitive t ther gradient descent algrithms in real r near real time practical use. With the PIL algrithm, it allws us t investigate the cmputatin-intensive techniques such as staced generatin mre efficiently. References: [] D. E. Rumelhurt, G. E. Hintn, and R. J. Williams, Learning Internal Representatin by Errr Prpagatin", in Parallel Distributed Prcessing, Vl., D.E.Rumelhart and J.L.McClelland, Eds. (MIT Press, Cambridge, MA), Chapter 8, 986. [2] E. Barnard Optimizatin fr training neural nets". IEEE Transactins n Neural Netwrs, Vl.3, N.2, pp232-24, 992. [3] F. A. Zdewy, Wesswls and B. Etienne, Avid False lcal minima by prper Initializatins f cnnectins", IEEE Transactin n Neural netwrs, Vl. 3, pp899-95, 992. [4] S. Kllias, and D. Anastassiu, An adaptive least squares algrithm fr the efficient training f artificial neural netwrs", IEEE Transactin n Circuit and System, CAS-36, pp92-,989. [5] D. H. Wlpert, Staced Generalizatin", Neural Netwrs, Vl.5, pp24-259, 992. [6] Thmas L. Bullin and Patric L. Odell, Generalized Inverse Matrices", Jhn Wiley and Sns, Inc. (New Yr), 97. [7] S. Tamura, Capabilities f a Tree Layer Feedfrward Neural Netwr", Prceedings f Internatinal Jint Cnference n Neural Netwrs, pp , (Seattle), 99. [8] C. R. Ra and S. K. Mitra, Generalized Inverse f Matrices and Its Applicatins, Wiley, (New Yr), 97. [9] Jn F. Claerbut, Fundamentals f Gephysical Dada Prcessing with applicatins t petrleum prspecting", McGraw-Hill Inc.,(USA), (TN 27, P4C6), 976. [] F. Biegler-König, and F. Bärmann,, A learning Algrithm fr Multilayered Neural Netwrs Based n Linear Least Squares Prblems", Neural Netwrs, vl. 6, pp27-3, 993. []M. P. Perrne and L. N. Cper, When netwrs disagree: ensemble methds fr hybrid neural netwrs", in R. J. Mammne Ed., Artificial Neural Netwrs fr Speech and Visin, Chapman and Hall, pp26-42, (Lndn), 993. [2] Ping Gu, Averaging ensemble neural netwrs in parameter space", In Prceedings f fifth internatinal cnference n neural infrmatin prcessing, pp , (Japan), 998.