Modified Recursive Prediction Error Algorithm For Training Layered Neural Network

Transcription

1 Modfed Recursve Predcton Error Alorth For Trann Layered Neural Netor Mohd Yusoff Mashor Centre for ELectronc Intellent Syste (CELIS) School of Electrcal and Electronc Enneern, Unversty Sans Malaysa, Pulau Pnan, MALAYSIA. Eal: ABSTRACT Bac propaaton s a steepest descent type alorth that norally has slo learnn rate and the search for the lobal nu often becoes trapped at poor local na. Ths paper proposes an alorth called odfed recursve predcton error (MRPE) alorth for trann ultlayered perceptron netors. MRPE s a odfed verson of recursve predcton error (RPE) alorth. RPE and MRPE are based on Gaussan-Neton type alorth that enerally provdes better perforance than a steepest type alorth such as bac propaaton. The current study nvestates the perforance of MRPE alorth to tran MLP netors and copares ts perforance to the faous bac propaaton alorth. Three data sets ere used for the coparson. It s found that the proposed MRPE s uch better than bac propaaton alorth.. INTRODUCTION Noadays, artfcal neural netors are studed and appled n varous dscplnes such as neuroboloy, psycholoy, coputer scence, contve scence, enneern, econocs, edcne, etc. Tan et al. (992) used neural netors for forecastn US- Snapore dollar exchane. Lnens and Ne (993) appled a neuro-fuzzy controller to a proble of ultvarable blood pressure control. Yu et al. (993) used neural netors to solve the travelln salesan proble and the ap-colourn proble. Arad et al. (994) used RBF netors to reconse huan facal expressons based on 2D and 3D odels of the face. Rosenblu and Davs (994) have appled RBF netors for a vehcle vsual autonoous road follon syste. Many applcatons of artfcal neural netors are nspred by the ablty of the netors to deonstrate branle behavour. Applcatons of artfcal neural netors n these dverse felds have ade t possble to tacle soe probles that ere prevously consdered to be very dffcult or unsolved. Multlayered perceptron (MLP) netor traned usn bac propaaton (BP) alorth s the ost popular choce n neural netor applcatons. It has been shon that the netor can provde satsfactory results. Hoever, MLP netor and BP alorth can be consdered as the 24

2 Modfed Recursve Predcton Error Alorth basc to the neural netor studes. For exaples RBF and HMLP netors have been proved to provde uch better perforance than MLP netor (Chen, 992; Mashor, 999). Bac propaaton s a steepest descent type alorth that norally has slo converence rate and the search for the lobal nu often becoe trapped at poor local na. Ths paper proposes an alorth called odfed recursve predcton error (MRPE) alorth for trann ultlayered perceptron netors. MRPE s a odfed verson of recursve predcton error (RPE) alorth. RPE and MRPE are based on Gaussan-Neton type alorth that enerally provdes better perforance than a steepest type alorth such as bac propaaton. The current study nvestates the perforance of MRPE alorth to tran MLP netors and copared ts perforance to RPE alorth and the faous bac propaaton alorth. The coparson as carred out by usn the MLP netors that ere traned usn the three alorths to perfor non-lnear syste dentfcaton. becoe the nputs to the second layer and so on. The last layer acts as the netor output layer. A hdden neuron perfors to functons that are the cobnn functon and the actvaton functon. The output of the -th neuron of the -th hdden layer, s ven by n v ( t) F v ( t) = + b ; = for and f the -th layer s the output layer then the output of the l-th neuron ŷ l of the output layer s ven by $y ( t) = v ( t) ; for l (2) l n = here n, n o s, b s and F(.) are the nuber of neurons n -th layer, nuber of neurons n output layer, ehts, thresholds and actvaton functon respectvely. n n o ŷ ŷ n o 2. MULTILAYERED PERCEPTRON NETWORKS MLP netor s a feed forard neural netor th one or ore hdden layers. Cybeno (989) and Funahash (989) have proved that the MLP netor s a eneral functon approxator and one hdden layer netors ll alays be suffcent to approxate any contnuous functon up to certan accuracy. A MLP netor th to hdden layers s shon n Fure. The nput layer acts as an nput data holder that dstrbutes the nput to the frst hdden layer. The outputs fro the frst hdden layer then v v 2 v n Output Layer 2nd Hdden Layer st Hdden Layer Input Layer Fure : Multlayered perceptron netors Internatonal Journal of The Coputer, The Internet and Manaeent, Vol., No.2, 2003, pp

3 In the current study, the netor th a snle output node and a snle hdden layer as used,.e = 2 and n o =. Wth these splfcatons the netor output s: n n n r 2 2 $y( t) = v( t) F v ( t) = + b = = = (3) here n r s the nuber of nodes n the nput layer. The actvaton functon F(.) s selected to be Fvt ( ( )) = ( ) + e vt (4) The ehts and threshold b are unnon and should be selected to nsed the predcton errors defned as = y yˆ ε (5) here y(t) s the actual output and netor output. 3. TRAINING ALGORITHMS $y( t) s the Ths secton brefly presents bac propaaton and the proposed MRPE alorths. The bac propaaton alorth th oentu has been used n ths study. It s ell non that ths verson of bac propaaton has better learnn rate copared to the ornal bac propaaton. 3. Bac Propaaton Alorth Bac propaaton alorth as ntally ntroduced by Werbos (974) and further developed by Ruelhart and McClelland (986). Bac propaaton s a steepest decent type alorth here the eht connecton beteen the -th neuron of the (-)-th layer and the -th neuron of the -th layer are respectvely updated accordn to ( t) = ( t ) + ( t) b ( t) = b ( t ) + b ( t) (6) th the ncreent ( ) t and b ( ) t ven by ( t) = η ρ ( t) v ( t) + α ( t ) b ( t) = ηρ ( t) + α b ( t ) b ) b (7) here the subscrpts and b represent the eht and threshold respectvely, α and α b are oentu constants hch deterne the nfluence of the past paraeter chanes on the current drecton of oveent n the paraeter space, η and η b represent the learnn rates and ρ ( t) s the error snal of the -th neuron of the -th layer hch s bac propaated n the netor. Snce the actvaton functon of the output neuron s lnear, the error snal at the output node s ρ ( t) = y( t) y$ ( t) (8) and for the neurons n the hdden layer + + ρ ( t) = F ( v ( t) ) ρ ( t) ( t ; ) here ( ( )) ( F v = -,..., 2, (9) ( t) Fv t th respect to v ( t). s the frst dervatve of Snce bac propaaton alorth s a steepest decent type alorth, the alorth suffers fro a slo converence rate. The search for the lobal na ay becoe trapped at local na and the alorth can be senstve to the user selectable paraeters. 26

4 Modfed Recursve Predcton Error Alorth 3.2 Modfed Recursve Predcton Error Alorth Recursve predcton error alorth (RPE) as ornally derved by Lun and Soderstro (983) and odfed by Chen et al. (990) to tran MLP netors. RPE alorth s a Gauss-Neton type alorth that ll enerally ve better perforance than a steepest descent type alorth such as bac propaaton alorth. In the present study, the converence rate of the RPE alorth s further proved by usn the optsed oentu and learnn rate. The oentu and learnn rate n ths research are vared copared to the constant values n Chen et al. (990). The RPE alorth odfed by Chen et al. (990) nses the follon cost functon, J N T ( Θ) = ε ( t, Θˆ ) Λ ε( ˆ t, Θˆ ) 2N t= (0) by updatn the estated paraeter vector, Θˆ (conssts of s and b s), recursvely usn Gauss-Neton alorth: and = Θˆ ( t ) + P t Θ ˆ = α ( t ) + α ψ t ε t (2) here ε t and Λ are the predcton error and syetrc postve defnte atrx respectvely, and s the nuber of output nodes; and α t and α t are the oentu and learnn rate respectvely. t and α t can be arbtrarly assned α to soe values beteen 0 and and the typcal value of t and α t are closed α to and 0 respectvely. In the present study, α and α are vared to further prove the converence rate of the RPE alorth accordn to: and α α = α (t α ( 0) 0 α < ψ = α (t ) + a (3) ( t, Θ) = )( ) α (4) here a s a sall constant (typcally a = 0.0); s norally ntalsed to. ψ t represents the radent of the one step ahead predcted output th respect to the netor paraeters: ( ) dyˆ t, Θ dθ (5) P t n equaton s updated recursvely accordn to: T ( t ) ψ ψ P( ) P t P = P( t ) ; λ t γ T γ = λ t I + ψ t P t ψ t (6) ( ) here λ t s the forettn factor, 0 < λ( t) <, and norally been updated usn the follon schee, Lun and Soderstro (983): here ( ) = λ λ( t ) + ( λ ) λ t (7) λ and the ntal forettn factor λ 0 are the desn values. Intal value of P(t) atrx, P(0) s norally set to α I here I s the dentty atrx and α s a constant, typcally beteen 00 to Sall value of α ll cause slo learnn hoever too lare α ay cause the estated paraeters do not convere properly. Hence, t should be selected to coprose beteen the to ponts, α = 000 s adequate for ost cases. Internatonal Journal of The Coputer, The Internet and Manaeent, Vol., No.2, 2003, pp

5 The radent atrx ψ t for one-hddenlayer MLP netor can be obtaned by dfferentatn equaton (3) th respect to the paraeters, θ, to yeld: ψ dyˆ = dθ v = v v ( v ) ( v ) v f θ = c 2 f θ = b c f θ = c otherse n n n, n h h h (8) The above radent atrx s derved based on sod functon therefore, f other actvaton functons ere used the atrx should be chaned accordnly. The odfed recursve predcton error alorth (MRPE) alorth for one hdden layer MLP netor can be pleented as follos: ) Intalse ehts, thresholds, P(0), a, b, α ( 0), λ 0 and λ( 0). ) Present nputs to the netor and copute the netor outputs accordn to equaton (3). ) Calculate the predcton error accordn to equaton (5) and copute atrx ψ t accordn to equaton (8). Note that, eleents of ψ t should be calculated fro the output layer don to the hdden layer. v) Copute atrx λ t and P(t) accordn to equaton (7) and (6) respectvely. < b α v) If α t, update t accordn to equaton (3). v) Update α t and then t accordn to equaton (4) and (2) respectvely. v) Update paraeter vector accordn to equaton. Θˆ v) Repeat steps to (v) for each trann data saple. The desn paraeter b n step (v) s the upper lt of oentu that has typcal value beteen 0.8 to 0.9. So oentu ll be ncreased for each data saple fro a sall value (norally close to 0) to ths value. 4. MODELLING NON-LINEAR SYSTEMS USING MLP NETWORKS Modelln usn MLP netors can be consdered as fttn a surface n a ultdensonal space to represent the trann data set and usn the surface to predct over the testn data set. A de class of nonlnear systes can be represented by nonlnear auto-reressve ovn averae th exoenous nput (NARMAX) odel, Leontarts and Bllns (985). The NARMAX odel can be expressed n ters of a non-lnear functon expanson of laed nput, output and nose ters as follos: y (, = f s y( t ), L, y( t ny ), u( t ), L, u( t nu ) e( t ),, e( t ne )) + e( t ) here y y = M y L (9), u u = M ur and e e = M e t are the syste output, nput and nose vector respectvely; ny, n u and n e are the axu las n the output, nput and nose vector respectvely. 28

6 Modfed Recursve Predcton Error Alorth The non-lnear functon f s ( ) s norally very coplcated and rarely non a pror for practcal systes. If the echanss of a syste are non the functon f ( ) s can be derved fro the functons that overn those echanss. In the case of an unnon syste, f s ( ) s norally constructed based on the observaton of the nput and output data. In the present study, MLP netors ll be used to odel the nput-output relatonshp. In other ords, f s ( ) ll be approxated by usn equaton (3) here F s selected to be sod functon. The netor nput vector, v( t ) s fored fro laed nput, output and nose ters, hch are denoted as u ( t ) L u( t n u ), y( t ) L y( t n y ) and et ( ) L et ( n e equaton (9). ) respectvely The fnal stae n syste dentfcaton s odel valdaton. There are several ays of testn a odel such as one step ahead predctons (OSA), odel predcted outputs (MPO), ean squared error (MSE), correlaton tests and ch-squares tests. In the present study, only OSA and MSE tests ll be use snce t s not easy to see the perforance dfferent usn other tests. OSA s a coon easure of predctve accuracy of a odel that has been consdered by any researchers. OSA can be expressed as: yˆ = fs ( u( t ), L, u( t nu ), y( t ), L, y( t ny ), ε t, θˆ, L, ε t n, θˆ (20) ( ) ( )) and the resdual or predcton error s defned as: ( t θ) y( t) y( ) ε$, $ = $ t (2) ε n here f s ( ) s a non-lnear functon, n ths case the MLP netor. A ood odel ll norally ve a ood predcton, hoever a odel that has a ood one step ahead predcton ht not alays be unbased. The odel ay be snfcantly based and predcton over a dfferent set of data often reveals ths proble. Splttn the data nto to sets, a trann set and a testn set, can norally detect ths condton. MSE s an teratve ethod of odel valdaton here the odel s tested by calculatn the ean squared errors after each trann step. MSE test ll ndcate ho fast a predcton error or resdual converes th the nuber of trann data. The MSE at the t-th trann step, s ven by: E MSE n d 2, (22) n ( t Θ t ) = ( y yˆ (, Θ t )) d = ( ) ( ( )) here E ( ) MSE t, Θ t and y$, Θ t are the MSE and OSA for a ven set of estated paraeters Θ( t) after t trann steps respectvely, and nd s the nuber of data that ere used to calculate the MSE. 5. SIMULATION RESULTS The perforance of MLP netors traned usn the BP, RPE and MRPE alorths presented n secton 3 ere copared. One sulated and to real data sets ere used for ths coparson. The netors ere used to perfor syste dentfcaton and the resultn odels ere used to produce OSA and MSE tests. Internatonal Journal of The Coputer, The Internet and Manaeent, Vol., No.2, 2003, pp

7 Exaple The frst data set s a sulated syste defned by the follon dfference equaton: y 3 ( t) = 0.3y( t ) + 0.6y( t 2) + u ( t ) 2 0.3u ( t ) 0.4u( t ) + e( t) here et ( ) s a Gaussan hte nose sequence th zero ean and varance and the nput, u(t) s a unforly rando sequence beteen (-,+). Ths syste as used to enerate 000 pars of data nput and output. The frst 600 data ere used to tran the netor and the reann 400 data ere used to test the ftted odel. The netor as traned based on the follon confuraton: v ( t) = [ u( t ) y( t ) y( t 2) ] + alorths produced uch better MSE than the one traned usn BP alorth. These fures also ndcate that the MRPE and RPE alorths produced about the sae perforance. OSA tests over the trann and testn data sets for the netor odels traned usn BP, RPE and MRPE are shon n Fure (4), (5) and (6) respectvely. These plots aan sho that RPE and MRPE have slar perforance. Hoever, the OSA test produced by the netor traned usn BP alorth s uch orse than the netor odels that have been traned usn RPE and MRPE alorths. The OSA test n Fure (4) shos that BP traned-netor fal to predct properly. All the netor odels have the sae nput v(t) and 7 hdden nodes but dfferent trann alorth. The desnn paraeters for BP, RPE and MRPE alorths ere set as follos, BP alorth: η η = and α = α = = b b RPE alorth: P(0) = 000I, α = 0.85, α = 0., ( ) 95 λ 0 = 0.99 and λ 0 = 0.. Fure 2: MSE calculated over trann data set MRPE alorth: = I P 000 b = 0.85, λ0 ( 0 ) = 0.2,, α a = 0.99 and λ 0 = ( ) = 0.0, The MSE calculated over both the trann and testn data sets for the netor odels traned usn BP, RPE and MRPE alorths are shon n Fure (2) and (3) respectvely. In ths exaple the netor odels traned usn MRPE and RPE Fure 3: MSE calculated over testn data set 30

8 Modfed Recursve Predcton Error Alorth Exaple 2 The second data set as taen fro a heat exchaner syste and conssts of 000 saples. The frst 500 data ere used to tran the netor and the reann 500 data ere used to test the ftted netor odel. The netor has been traned usn the follon specfcaton: Fure 4: OSA test for BP alorth v ( t) = [ u( t ) u( t 2) y( t ) y( t 4) e( t 3) e( t 4) e( t 5)] and bas nput All the netor odels have the sae structure but dfferent trann alorth. The desn paraeters for BP, RPE and MRPE alorths ere set as follos, BP alorth: η η = and α = b RPE alorth: = α b = 0.85 P(0) = 000I, α = 0.85, α = 0., ( ) 95 λ 0 = 0.99 and λ 0 = 0.. Fure 5: OSA test for RPE alorth MRPE alorth: = I P 000 a = 0.0, λ( 0 ) = b, α ( 0 ) = 0. 6, = 0. 85, λ 0 = and The MSE calculated over both the trann and testn data sets for the netor odels traned usn BP and MRPE alorths are shon n Fure (7) and (8) respectvely. These fures sho that MRPE alorth produced snfcantly better MSE than RPE alorth and uch better than BP alorth. Fure 6: OSA test for MRPE alorth Internatonal Journal of The Coputer, The Internet and Manaeent, Vol., No.2, 2003, pp

9 Fure 7: MSE calculated over the trann data set Fure 9: OSA test for BP alorth Fure 8: MSE calculated over the testn data set OSA tests over the trann and testn data sets for the netor odels traned usn BP, RPE and MRPE are shon n Fure (9), (0) and respectvely. The result n Fure (9) reconfrs that the netor traned usn BP alorth cannot predct properly here the predcton over both trann and testn data set are not satsfactory. The netors traned usn RPE and MRPE alorths on the other hand predct very ell over both the trann and testn data sets. Referrn to Fure (0) and, t can be sad that the netor traned usn MRPE alorth ve snfcantly better predcton (OSA test) copared to the netor traned usn RPE alorth. Fure 0: OSA test for RPE alorth Fure : OSA test for MRPE alorth 32

10 Modfed Recursve Predcton Error Alorth Exaple 3 A data set of 000 nput-output saples ere taen fro a tenson le platfor. Descrpton of the process can be found n Mashor (995). The data set conssts of 000 data saples here the frst 600 data saples ere used for trann and the next 400 data saples ere used for testn. The netor has been traned usn the follon specfcaton: v ( t) = [ u( t ) L u( t 8) y( t ) L y( t 4) e ( t 3) e( t 5)] and bas nput Fure 2: MSE calculated over trann data set BP alorth: η = ηb = 0.00 and α = α = RPE alorth: P(0) =000I, α = 0.85, α = 0.07, 0 = 0.99 and λ 0 = 0.. λ ( ) 95 MRPE alorth: b I P t = 000, b = 0.85, λ0 ( 0 ) = 0.4, α a = 0.0, = 0.99 and λ 0 = 0.. ( ) 95 The MSE calculated over both trann and testn data sets for the netor odels traned usn BP, RPE and MRPE alorths are shon n Fure (2) and (3) respectvely. In ths exaple the netor odels traned usn MRPE and RPE alorths produced uch better MSE than the one traned usn BP alorth. These fures also ndcate that the MRPE s snfcantly better than RPE alorth. Fure (2) and (3) also suestn that the netor traned usn BP alorth cannot learn properly here the converence of ts MSE values are not snfcant. Fure 3: MSE calculated over testn data set Fure 4: OSA test for BP alorth Internatonal Journal of The Coputer, The Internet and Manaeent, Vol., No.2, 2003, pp

11 Fure 7: MSE for syste n exaple Fure 5: OSA test for RPE alorth OSA tests over the trann and testn data sets for the netor odels traned usn BP, RPE and MRPE are shon n Fure (4), (5) and (6) respectvely. For ths exaple t s qute hard to dstnush the perforance advantae beteen the netors traned usn MRPE and RPE alorths. Hoever, the OSA tests produced by the to netors are uch better than the one produced by the netor odel that have been traned usn BP alorth. All the three exaples sho that the MLP netor traned usn BP alorth could not learn properly. Ths s because the learnn rate of BP s very slo. Thus further analyss has been carred out to chec ho ood s BP th ore trann epochs. Fure (7), (8) and (9) sho the MSE plots over the trann data set produced by BP alorth after 00 epochs for the syste n exaple, 2 and 3 respectvely. Each fure has three MSE plots for dfferent learnn rates. The netor specfcaton for the netors ere the sae as n prevous analyss except for the learnn rates that ere assn as ndcated n the respectve fures. Coparn these results th the ones n Fure (2), (7) and (2) t as found that BP alorth cannot tran the MLP netor as ood as RPE and MPRE even after 00 trann epochs. Therefore, t can be deduced that the learnn rate of BP alorth s very slo copare to MRPE and RPE alorths. Fure 6: OSA test for MRPE alorth 34

12 Modfed Recursve Predcton Error Alorth learnn rate than BP alorth and does not requre ultple trann epoch. REFERENCES Fure 8: MSE for syste n exaple 2 [] Arad, N., Dyn, N., Resfeld, D., and Yeshurun, Y., 994, Iae arpn by radal bass functons: applcaton to facal expressons, CVGIP: Graphcal Models and Iae Processn, 56 (2), [2] Chen, S., Coan, C.F.N., Bllns, S.A., and Grant, P.M., 990, A parallel recursve predcton error alorth for trann layered neural netors, Int. J. Control, 5 (6), Fure 9: MSE for syste n exaple 3 6. CONCLUSION MRPE alorth s proposed to tran MLP netor and ts perforance as copared to RPE and BP alorths. Three data sets ere used to test the perforance of the alorths. The MSE and OSA tests for the exaples ndcated that the MRPE has snfcantly proved the perforance of RPE alorth especally for the to real data sets. The results also proved that both RPE and MRPE alorths are uch better than BP alorth. The perforance of BP alorth th 00 trann epochs stll cannot copete th MRPE alorth th one trann epoch. Hence, t can be concluded that MRPE has uch faster [3] Chen, S., and Bllns, S.A., 992, Neural netors for non-lnear dynac syste odelln and dentfcaton, Int. J. of Control, 56 (2), [4] Cybeno, G., 989, Approxatons by superposton of a sodal functon, Matheatcs of Control, Snal and Systes, 2, [5] Funahash, K., 989, On the approxate realsaton of contnuous appns by neural netors, Neural Netors, 2, [6] Leontarts, I.J., and Bllns, S.A., 985, Input-output paraetrc odels for non-lnear systes. Part I - Deternstc non-lnear systes. Part II - Stochastc non-lnear systes, Int. J. Control, 4, Internatonal Journal of The Coputer, The Internet and Manaeent, Vol., No.2, 2003, pp

13 [7] Lnens, D.A., and Ne, J., 993, Fuzzfed RBF netor-based learnn control: Structure and selfconstructon, IEEE Int. Conf. on Neural Netors, 2, [8] Lun, L., and Soderstro, T., 983, Theory and Practce of Recursve Identfcaton, MIT Press, Cabrde. Trans. on Fundaentals of Electroncs, Co. and Coputer Scences, E76-A (9), [5] Werbos, P.J., 974, Beyond Reresson: Ne Tools for Predcton and Analyss n the Behavoural Scences, Ph.D. Thess, Harvard Unversty. [9] Mashor, M.Y., 995, Syste dentfcaton usn radal bass functon netor, PhD Thess, Unversty of Sheffeld, Unted Kndo. [0] Mashor, M.Y., 999, Perforance Coparson Beteen HMLP and MLP Netors, Int. Conf. on Robotcs, Vson & Parallel Processn for Autoaton (ROVPIA 99), pp [] Rosenblu, M., and Davs, L.S., 994, An proved radal bass functon netor for vsual autonoous road follon, Proc. of the SPIE - The Int. Socety for Optcal En., 203, [2] Ruelhart, D.E., and McClelland, J.L., 986, Parallel dstrbuted processn: exploratons n the crostructure of conton, II MIT Press Cabrde, MA, I. [3] Tan, P.Y., L, G., Chua, K., Won, F.S., and Neo, S., 992, Coparatve studes aon neural nets, radal bass functons and reresson ethods, ICARCV '92. Second Int. Conf. on Autoaton, Robotcs and Coputer Vson,, NW-3.3/-6. [4] Yu, D.H., Ja, J., and Mor, S., 993, A ne neural netor alorth th the orthoonal optsed paraeters to solve the optal probles, IEICE 36