PERFORMANCE COMPARISON BETWEEN BACK PROPAGATION, RPE AND MRPE ALGORITHMS FOR TRAINING MLP NETWORKS

Transcription

1 PERFORMANCE COMPARISON BETWEEN BACK PROPAGATION, RPE AND MRPE ALGORITHMS FOR TRAINING MLP NETWORKS Mohd Yusoff Mashor School of Electrcal and Electronc Engneerng, Unversty Scence Malaysa, Pera Branch Campus, 3750 Tronoh, Pera, Malaysa. Emal: ABSTRACT Ths paper presents the performance comparson between bac propagaton, recursve predcton error (RPE) and modfed recursve predcton error (MRPE) algorthms for tranng multlayered perceptron networs. Bac propagaton s a steepest descent type algorthm that normally has slow convergence rate and the search for the global mnmum often becomes trapped at poor local mnma. RPE and MRPE are based on Gaussan-Newton type algorthm that generally provdes better performance. The current study nvestgates the performance of three algorthms to tran MLP networs. Two real data sets were used for the comparson. Its was found that the RPE and MRPE are much better than the bac propagaton algorthm.. INTRODUCTION Nowadays, artfcal neural networs are studed and appled n varous dscplnes such as neurobology, psychology, computer scence, cogntve scence, engneerng, economcs, medcne, etc. Tan et al. (992) used neural networs for forecastng US-Sngapore dollar exchange. Lnens and Ne (993) appled a neuro-fuzzy controller to a problem of multvarable blood pressure control. Yu et al. (993) used neural networs to solve the travellng salesman problem and the map-colourng problem. Arad et al. (994) used RBF networs to recognse human facal expressons based on 2D and 3D models of the face. Rosenblum and Davs (994) have appled RBF networs for a vehcle vsual autonomous road followng system. Many applcatons of artfcal neural networs are nspred by the

2 ablty of the networs to demonstrate bran-le behavour. Applcatons of artfcal neural networs n these dverse felds have made t possble to tacle some problems that were prevously consdered to be very dffcult or unsolvable. Multlayered perceptron (MLP) networs traned usng bac propagaton (BP) algorthm are the most popular choce n neural networ applcatons. It has been shown that the networ can provde satsfactory results. However, MLP networ and BP algorthm can be consdered as the bass to the neural networ studes. For examples RBF and HMLP networs have been proved to provde much better performance than MLP networ (Chen, 992; Mashor, 999a). The current study compares the performance of BP, RPE and MRPE to tran MLP networs. The comparson was carred out by usng the MLP networs that were traned usng the three algorthms to perform non-lnear system dentfcaton. 2. MULTILAYERED PERCEPTRON NETWORKS MLP networs are feed forward neural networs wth one or more hdden layers. Cybeno (989) and Funahash (989) have proved that the MLP networ s a general functon approxmator and that one hdden layer networs wll always be suffcent to approxmate any contnuous functon up to certan accuracy. A MLP networ wth two hdden layers s shown n Fgure. The nput layer acts as an nput data holder that dstrbutes the nput to the frst hdden layer. The outputs from the frst hdden layer then become the nputs to the second layer and so on. The last layer acts as the networ output layer. A hdden neuron performs two functons that are the combnng functon and the actvaton functon. The output of the j-th neuron of the -th hdden layer s gven by n v j ( t ) F wj v = ( t ) + b j ; for j n () = and f the m-th layer s the output layer then the output of the l-th neuron $y l of the output layer s gven by m m $y ( t ) = w v ( t ) l n m j = ; for l n o (2) where n, n o w s, b s and F(.) are the number of neurons n -th layer, number of neurons n output layer, weghts, thresholds and an actvaton functon respectvely.

3 $y $y no Output Layer 2nd Hdden Layer st Hdden Layer Input Layer v v2 vn Fgure : Multlayered perceptron networs In the current study, the networ wth a sngle output node and a sngle hdden layer s used,.e m = 2 and n o =. Wth these smplfcatons the networ output s: n n n r 2 2 $y ( t ) = w v ( t ) = w F wjv j ( t ) + b j = = j= (3) where n r s the number of nodes n the nput layer. The actvaton functon F(.) s selected to be F( v( t )) = + ( ) e v t (4) The weghts w and threshold b j are unnown and should be selected to mnmse the predcton errors defned as ε( t ) = y( t ) y$ ( t ) (5) where y(t) s the actual output and ( ) $y t s the networ output.

4 3. TRAINING ALGORITHMS Ths secton brefly presents bac propagaton, RPE and MRPE algorthms. Bac propagaton and RPE algorthms are based on paper by Chen et. al. (990) and MRPE s based on paper by Mashor (999b). 3. Bac Propagaton Algorthm Bac propagaton algorthm was ntally ntroduced by Werbos (974) and further developed by Rumelhart and McClelland (986). Bac propagaton s the steepest decent type algorthm where the weght connecton between the j-th neuron of the (-)-th layer and the -th neuron of the -th layer are respectvely updated accordng to w ( t) = w ( t ) + w ( t ) j j j b ( t ) = b ( t ) + b ( t ) (6) wth the ncrement wj ( t ) and b ( t ) gven by w ( t ) = η ρ ( t) v ( t ) + α w ( t ) j w j w j b ( t ) = η ρ ( t) + α b ( t ) b b (7) where the subscrpts w and b represent the weght and threshold respectvely, α w and α b are momentum constants whch determne the nfluence of the past parameter changes on the current drecton of movement n the parameter space, η w and η b represent the learnng rates and ρ ( t ) s the error sgnal of the -th neuron of the -th layer whch s bac propagated n the networ. Snce the actvaton functon of the output neuron s lnear, the error sgnal at the output node s ρ m ( t ) = y( t ) y$ ( t) (8) and for the neurons n the hdden layer ( ) ( ) ( ) ( ) ρ ( ) ( ) ρ + + t = F v t j t w j t = m-,..., 2, (9) j ( ) where F v ( t ) s the frst dervatve of F v ( t ) wth respect to v ( t ). Snce bac propagaton algorthm s a steepest decent type algorthm, the algorthm suffers from a slow convergence rate. The search for the global mnma may become trapped at local mnma and the algorthm can be senstve to the user selectable parameters.

5 3.2 RPE and MRPE Algorthms Recursve predcton error algorthm (RPE) was orgnally derved by Ljung and Soderstrom (983) and modfed by Chen et al. (990) to tran MLP networs. RPE algorthm s a Gauss-Newton type algorthm that wll generally gve better performance than a steepest descent type algorthm such as bac propagaton algorthm. In the present study, the convergence rate of the RPE algorthm s further mproved by usng the optmsed momentum and learnng rate RPE (Mashor, 999b). The momentum and learnng rate n ths research are vared compared to the constant values n Chen et al. (990). Ths modfed RPE s orgnally proposed by Mashor (999b) and referred as modfed recursve predcton error (MRPE) algorthm. The RPE algorthm modfed by Chen et al. (990) mnmses the followng cost functon, J N T ( Θ) = ε ( t, Θˆ ) Λ ε( t, Θˆ ) ˆ (0) 2N t= by updatng the estmated parameter vector, Θˆ (conssts of w s and b s), recursvely usng Gauss-Newton algorthm: and ( t) = Θˆ ( t ) + P( t) ( t) Θ ˆ () ( t) = α ( t) ( t ) + α ( t) ψ( t) ε( t) (2) m where ε ( t) and Λ are the predcton error and m respectvely, and m s the number of output nodes; and m( t) αg ( t) and learnng rate respectvely. m( t) αg ( t) between 0 and and the typcal value of m( t) αg ( t) In the present study, ( t) α ( t) g m symmetrc postve defnte matrx α and are the momentum α and can be arbtrarly assgned to some values α and are closed to and 0 respectvely. αm and g are vared to further mprove the convergence rate of the RPE algorthm accordng to (Mashor, 999b): and ( t) = α ( t ) a α (3) m m + g ( t) = α ( t) ( α ( t) ) α (4) m where a s a small constant (typcally a = 0.0); m( t) 0 α m ( 0) <. ( t) m α s normally ntalsed to ψ represents the gradent of the one step ahead predcted output wth respect to the networ parameters: ( t, Θ) ( ) dyˆ ψ t, Θ = dθ (5)

6 P ( t) n equaton () s updated recursvely accordng to: P ( t) = λ ( t) P T ( ) ψ ( t) P( t ) T ( t ) P( t ) ψ( t) λ( t) I + ψ ( t) P( t ) ψ( t) (6) where λ ( t) s the forgettng factor, 0 λ( ) < < t, and normally been updated usng the followng scheme, Ljung and Soderstrom (983): λ ( t) = λ λ( t ) + ( λ ) 0 0 where 0 λ 0 are the desgn values. Intal value of P(t) matrx, P(0) s normally set to α I where I s the dentty matrx and α s a constant, typcally between 00 to Small value of α wll cause slow learnng however too large α may cause the estmated parameters do not converge properly. Hence, t should be selected to compromse between the two ponts, α = 000 s adequate for most cases. λ and the ntal forgettng factor ( ) ψ can be modfed to accommodate the extra lnear connectons for one-hdden-layer HMLP networ model by dfferentatng equaton (3) wth respect to the parameters, θ c, to yeld: The gradent matrx ( t) (7) ψ ( t) dy = dθ ( ) j t v j ( v j) = c v ( v ) j v 0 j w 2 j 2 0 jv w f θ c f θ f θ c c = w = b = w otherwse 2 j j j j n j n j n h h h, n (8) The above gradent matrx s derved based on sgmod functon therefore, f other actvaton functons were used the matrx should be changed accordngly. The modfed RPE algorthm for one hdden layer MLP networ can be mplemented as follows (Mashor, 999b): Intalse weghts, thresholds, P (0), a, b, m( 0), λ 0 and λ 0. Present nputs to the networ and compute the networ outputs accordng to equaton (3). Calculate the predcton error accordng to equaton (5) and compute matrx ψ( t) accordng to equaton (8). Note that elements of ψ ( t) should be calculated from the output layer down to the hdden layer. v Compute matrx λ ( t) and P (t) accordng to equaton (2) and () respectvely. v If α m ( t) < b, update α m ( t) accordng to equaton (3). v Update α g ( t) and then ( t) accordng to equaton (4) and (2) respectvely. v Update parameter vector Θˆ ( t) accordng to equaton (). v Repeat steps () to (v) for each tranng data sample. α ( )

7 The desgn parameter b n step (v) s the upper lmt of momentum that has typcal value between 0.8 to 0.9. So momentum wll be ncreased for each data sample from a small value (normally close to 0) to ths value. 4. MODELLING NON-LINEAR SYSTEMS USING HMLP NETWORKS Modellng usng MLP networs can be consdered as fttng a surface n a multdmensonal space to represent the tranng data set and usng the surface to predct over the testng data set. Therefore, MLP networs requre all the future data of the system to le wthn the doman of the ftted surface to ensure a correct mappng so that good predctons can be acheved. Ths s normal for the non-lnear modellng where the model s only vald over a certan ampltude range. A wde class of non-lnear systems can be represented by non-lnear auto-regressve movng average wth exogenous nput (NARMAX) model, Leontarts and Bllngs (985). The NARMAX model can be expressed n terms of a non-lnear functon expanson of lagged nput, output and nose terms as follows: y ( t) = f ( y( t ),, y( t n ), u( t ), L, u( t n ), e( t ), L, e( t n )) e( t) L (9) s y u e + where y ( t) u ( t ) e ( t ) y( t ) = M, u( t) = M and e( t) = M y ( t) u ( t ) e ( t ) m r m are the system output, nput and nose vector respectvely; ny, nu and ne are the maxmum lags n the output, nput and nose vector respectvely. The non-lnear functon, f s ( ) s normally very complcated and rarely nown a pror for practcal systems. If the mechansms of a system are nown the functon f s ( ) can be derved from the functons that govern those mechansms. In the case of an unnown system, f s ( ) s normally constructed based on the observaton of the nput and output data. In the present study, MLP networs wll be used to model the nput-output relatonshp. In other words, f s ( ) wll be approxmated by usng equaton (3) where F ( ) s selected to be sgmod functon. The networ nput vector, v( t ) s formed from lagged nput, output and nose terms, whch are denoted as u( t ) L u( t n u ), y( t ) L y ( t n y ) and e( t ) L e( t n e ) respectvely n equaton (9). The fnal stage n system dentfcaton s model valdaton. There are several ways of testng a model such as one step ahead predctons (OSA), model predcted outputs (MPO), mean squared error (MSE), correlaton tests and ch-squares tests. In the present study, OSA,

8 MPO, MSE and correlaton tests were used to justfy the performance of the ftted networ models. In the present study, only OSA test and MSE test wll be used snce t s not easy to see the performance dfferent usng other tests. OSA s a common measure of predctve accuracy of a model that has been consdered by many researchers. OSA can be expressed as: ( ) ( ) = ( ) L ( ) ( ) L ( ) ε( θ) L ε( θ) y $ t f u t,, u t n, y t,, y t n, t, $,, t n, $ s u y ε (20) and the resdual or predcton error s defned as: ( t θ) y( t ) y( t ) ε$, $ = $ (2) where ( ) f s s a non-lnear functon, n ths case the MLP networ. A good model wll normally gve a good predcton, however, a model that has a good one step ahead predcton and model predcted output may not always be unbased. The model may be sgnfcantly based and predcton over a dfferent set of data often reveals ths problem. Splttng the data nto two sets can test ths condton, a tranng set and a testng set. MSE s an teratve method of model valdaton where the model s tested by calculatng the mean squared errors after each tranng step. MSE test wll ndcate how fast a predcton error or resdual converges wth the number of tranng data. The MSE at the t-th tranng step, s gven by: n d 2 ( ) E ( ( )) ( ) ( ( ) MSE t,θ t = y y$, Θ t ) (22) n ( ) ( ( )) d = where E t ( t ) MSE, Θ and y$, Θ t are the MSE and OSA for a gven set of estmated parameters Θ( t ) after t tranng steps respectvely, and n d s the number of data that were used to calculate the MSE. 5. PERFORMANCE COMPARISON The performance of MLP networs traned usng the BP, RPE and MRPE algorthms presented n secton 3 were compared. Two real data sets were used for ths comparson. The networs were used to perform system dentfcaton and the resultng models were used to produce OSA and MSE tests. Example The frst data set was taen from a heat exchanger system and conssts of 000 samples. The frst 500 data were used to tran the networ and the remanng 500 data were used to test the ftted networ model. The networ has been traned usng the followng specfcaton: v t = u t u t 2 y t y t 4 e t 3 e t 4 e t 5 and bas nput. ( ) [ ( ) ( ) ( ) ( ) ( ) ( ) ( )]

9 All the networ models have the same structure but dfferent tranng algorthm. The desgn parameters for BP, RPE and MRPE algorthms were set as follows, BP algorthm: ηw = ηb = and αw = αb = RPE algorthm: P(0) = 000I, αm = 0.85, αg = 0., 0 = MRPE algorthm: P = 000 ( t ) I, ( 0 ) = 0.6, λ and ( 0 ) = α m a = 0.0, b = 0.85, 0 = λ. λ and ( 0 ) = λ. The MSE calculated over both tranng and testng data sets for the networ models traned usng BP, RPE and MRPE algorthms are shown n Fgure (2) and (3) respectvely. Both fgures ndcated that MRPE produces MSE that s sgnfcantly better than RPE and much better than BP algorthm. These fgures also suggest that the networ traned usng BP algorthm do not have good generalsaton snce the performance dfferent over the testng data set s larger than the one over the tranng data set. Fgure 2: MSE calculated over tranng data set Fgure 3: MSE calculated over testng data set

10 Fgure 4: OSA test for MLP networ traned usng BP algorthm Fgure 5: OSA test for MLP networ traned usng RPE algorthm OSA tests over the tranng and testng data sets for the networ models traned usng BP, RPE and MRPE are shown n Fgure (4), (5) and (6) respectvely. The result n Fgure (4) reconfrms that the networ traned usng BP algorthm cannot predct properly and do not have good generalsaton where the predcton over testng data set s not satsfactory. The networs traned usng RPE and MRPE algorthms on the other hand predct very well over both the tranng and testng data sets. Referrng to Fgure (5) and (6), t can be sad that the networ traned usng MRPE algorthm gves sgnfcantly better predcton (OSA test) compared to the networ traned usng RPE algorthm. Fgure 6: OSA test for MLP networ traned usng MRPE algorthm

11 Example 2 The second system s a tenson legs data consst of 000 data samples where the frst 600 data samples were used for tranng and the next 400 data samples were used for testng. The networ has been traned usng the followng specfcaton: v ( t) = [ u( t ) L u( t 8) y( t ) L y( t 4) e( t 3) e( t 5) ] BP algorthm: ηw = ηb = 0.00 and αw = αb = RPE algorthm: P(0) = 000I, αm = 0.85, αg = 0. 07, 0 = MRPE algorthm: P = 000 ( t ) I, ( 0 ) = 0, λ and ( 0 ) = α m a = 0.0, b = 0.85, 0 = λ. λ and ( 0 ) = λ. and bas nput The MSE calculated over both tranng and testng data sets for the networ models traned usng BP, RPE and MRPE algorthms are shown n Fgure (7) and (8) respectvely. In ths example the networ models traned usng MRPE and RPE algorthms produced MSE that are much better than the one traned usng BP algorthm. These fgures also ndcate that the MRPE s sgnfcantly better than RPE algorthm. Fgure (7) and (8) also suggest that the networ traned usng BP algorthm do not have good generalsaton snce the performance dfferent over the testng data set s larger than the one over the tranng data set. Fgure 7: MSE calculated over tranng data set Fgure 8: MSE calculated over testng data set

12 Fgure 9: OSA test for MLP networ traned usng BP algorthm Fgure 0: OSA test for MLP networ traned usng RPE algorthm OSA tests over the tranng and testng data sets for the networ models traned usng BP, RPE and MRPE are shown n Fgure (9), (0) and () respectvely. For ths example t s qute hard to dstngush the performance advantage between the networs traned usng BP and RPE algorthms. However, the OSA test produced by the networ traned usng MRPE s sgnfcantly better than the networ models that have been traned usng BP and RPE algorthms. Fgure : OSA test for MLP networ traned usng MRPE algorthm

13 6. CONCLUSION BP, RPE and MRPE algorthms are brefly dscussed and used to tran MLP networs to perform system dentfcaton. Two real data sets were used to test the performance of the algorthms. The MSE and OSA tests for both examples ndcated that the MRPE has sgnfcantly mproved the performance of RPE algorthm and both RPE and MRPE algorthm are much better than BP algorthm. The results also suggest that the networ traned usng BP algorthm does not normally have good generalsaton where the predcton over the testng data set s not as good as over the tranng data set. REFERENCES [] Arad, N., Dyn, N., Resfeld, D., and Yeshurun, Y., 994, Image warpng by radal bass functons: applcaton to facal expressons, CVGIP: Graphcal Models and Image Processng, 56 (2), [2] Chen, S., Cowan, C.F.N., Bllngs, S.A., and Grant, P.M., 990, A parallel recursve predcton error algorthm for tranng layered neural networs, Int. J. Control, 5 (6), [3] Chen, S., and Bllngs, S.A., 992, Neural networs for non-lnear dynamc system modellng and dentfcaton, Int. J. of Control, 56 (2), [4] Cybeno, G., 989, Approxmatons by superposton of a sgmodal functon, Mathematcs of Control, Sgnal and Systems, 2, [5] Funahash, K., 989, On the approxmate realsaton of contnuous mappngs by neural networs, Neural Networs, 2, [6] Leontarts, I.J., and Bllngs, S.A., 985, Input-output parametrc models for nonlnear systems. Part I - Determnstc non-lnear systems. Part II - Stochastc non-lnear systems, Int. J. Control, 4, [7] Lnens, D.A., and Ne, J., 993, Fuzzfed RBF networ-based learnng control: Structure and self-constructon, IEEE Int. Conf. on Neural Networs, 2, [8] Ljung, L., and Soderstrom, T., 983, Theory and Practce of Recursve Identfcaton, MIT Press, Cambrdge. [9] Mashor, M.Y., 999a, Performance Comparson Between HMLP and MLP Networs, Int. Conf. on Robotcs, Vson & Parallel Processng for Automaton (ROVPIA 99), pp [0] Mashor, M.Y., 999b, Hybrd multlayered perceptron networs, accepted for publcaton by Int. J. of System and Scence.

14 [] Rosenblum, M., and Davs, L.S., 994, An mproved radal bass functon networ for vsual autonomous road followng, Proc. of the SPIE - The Int. Socety for Optcal Eng., 203, [2] Rumelhart, D.E., and McClelland, J.L., 986, Parallel dstrbuted processng: exploratons n the mcrostructure of cognton, I & II, MIT Press, Cambrdge, MA [3] Tan, P.Y., Lm, G., Chua, K., Wong, F.S., and Neo, S., 992, Comparatve studes among neural nets, radal bass functons and regresson methods, ICARCV '92. Second Int. Conf. on Automaton, Robotcs and Computer Vson,, NW-3.3/-6. [4] Yu, D.H., Ja, J., and Mor, S., 993, A new neural networ algorthm wth the orthogonal optmsed parameters to solve the optmal problems, IEICE Trans. on Fundamentals of Electroncs, Comm. and Computer Scences, E76-A (9), [5] Werbos, P.J., 974, Beyond Regresson: New Tools for Predcton and Analyss n the Behavoural Scences, Ph.D. Thess, Harvard Unversty.