A Hybrid Learning Algorithm for Locally Recurrent Neural Networks

Transcription

1 Contemporary Engneerng Scences, Vo. 11, 2018, no. 1, 1-13 HIKARI Ltd, A Hybrd Learnng Agorthm for Locay Recurrent Neura Networks Dmtrs Varsams and Evangeos Outsos Department of Informatcs Engneerng Technoogca Educatona Insttute of Centra Macedona - Serres 62124, Serres, Greece Pars Mastorocostas Department of Computer Systems Engneerng, Praeus Unversty of Apped Scences, 12244, Egaeo, Greece Copyrght c 2018 Dmtrs Varsams, Evangeos Outsos and Pars Mastorocostas. Ths artce s dstrbuted under the Creatve Commons Attrbuton Lcense, whch permts unrestrcted use, dstrbuton, and reproducton n any medum, provded the orgna work s propery cted. Abstract In ths work a fast and effcent tranng method for bock-dagona recurrent neura networks s proposed. The method modfes and extends the Smuated Anneang RPROP agorthm, orgnay deveoped for statc modes, by takng nto consderaton the archtectura characterstcs and the tempora nature of ths category of recurrent neura modes. The performance of the proposed agorthm s evauated through a comparatve anayss wth a seres of agorthms and recurrent modes. Keywords: bock-dagona recurrent neura network, nterna feedback, resent back-propagaton, smuated anneang, ordered dervatves 1 Introducton Recurrent neura networks has become a popuar research fed of Computatona Integence durng the ast. Due to ther tempora capabtes, they

2 2 Dmtrs Varsams, Pars Mastorocostas and Evangeos Outsos have been extensvey empoyed n rea-word appcatons ke system dentfcaton and pattern recognton. The ocay recurrent neura networks wth nterna feedback connectons consttute a speca subcass, where the feedback nternks are mted excusvey between neghbourng neurons. Thus, these neura modes have sgnfcanty reduced compexty wth respect to fuy recurrent networks [13]. A speca subcass of ocay recurrent networks s the Dagona Recurrent Neura Network (DRNN) [2], where there are no nternks among neurons n the hdden ayer. A modfed DRNN s the Bock-Dagona Recurrent Neura Network (BDRNN) [10], where dynamcs s ntroduced between pars of neurons n the hdden ayer. The BDRNNs have been proved to be an effcent modeng too [4], [5], [11]. Due to the tempora reatons of BDRNN, a parameter optmzaton method shoud unfod n tme. Most of the tmes these tempora reatons are negected and parameter earnng s attempted by consderng a few prevous tme steps. The Back Propagaton Through Tme agorthm (BPTT) [7] s the most common agorthm for tranng BDRNNs. However, the BPTT exhbts two major dsadvantages: (a) t shows a ow speed of convergence, (b) most often t becomes trapped to oca mnma of the error surface. In order to overcome the above fangs of gradent-based methods ke BPTT, the Resent Propagaton agorthm (RPROP, [8]) has been proved to be one of the best performng earnng methods for statc neura networks [1]. However, n RPROP the probem of poor convergence to oca mnma s not fuy emnated. Hence, n an attempt to aevate ths drawback, a hybrd scheme combnng the goba search technque of Smuated Anneang (SA) and RPROP was ntroduced n [12]. The resuted agorthm, named SARPROP, was proved to be an effcent earnng method for statc neura networks. Stem from the above deveopments n tranng statc neura modes, ths work proposes an extenson of the standard SARPROP method that takes nto consderaton the tempora reatons exstng n a ocay recurrent neura networks. Snce the agorthm s adapted to the speca features of BDRNN, t s entted Hybrd Learnng Agorthm for BDRNNs (HLA-BDRNN). The rest of ths paper s organzed as foows: In Secton 2 the structure and characterstcs of the BDRNN are ustrated. The earnng agorthm s deveoped n Secton 3. In Secton 4 a comparatve anayss of the proposed method wth other earnng schemes and recurrent modes s conducted. The paper concudes wth a bref dscusson of the proposed method.

3 Hybrd earnng agorthm 3 2 The Bock-Dagona Recurrent Neura Network The BDRNN s a speca case of recurrent neura networks. It s not fuy connected and t beongs to the cass of ocay-recurrent-gobay-feedforward neura networks [13]. It conssts of two ayers, where the output ayer s statc and the hdden ayer s dynamc. The hdden ayer conssts of pars of neurons (bocks); there are feedback connectons between the neurons of each par, ntroducng dynamcs to the network. For the sake of smpcty, a sngenputsngeoutput BDRNN wth four bocks of neurons s shown n Fgure 1. Fgure 1: Confguraton of BDRNN wth four bocks of neurons n the hdden ayer A BDRNN wth m nputs, r outputs, and N neurons at the hdden ayer operates accordng to the foowng state equatons: where x(k) f a (W x(k 1) + B u(k)) (1a) y(k) f b (C x(k)) (1b) f a, f b are the neuron actvaton functons of the hdden and the output ayers, respectvey. In the foowng, the actvaton functons are both chosen to be the sgmod functon f(z) 1 e an z 1 + e an z.

4 4 Dmtrs Varsams, Pars Mastorocostas and Evangeos Outsos u(k) [u (k)] s a m-eement nput vector, wth k beng the tme varabe. x(k) [x (k)] s a N-eement vector, comprsng the outputs of the hdden ayer. In partcuar, x (k) s the output of the -th hdden neuron at tme k. y(k) [y (k)] s a r-eement output vector. B [b,j ] and C [c,j ] are N m and r N nput and output weght matrces, respectvey. W [w,j ] s the N N bock dagona feedback matrx. In partcuar, 0 f j 0 f j and j 1 and s odd w,j 0 f j and j + 1 and s even 0 otherwse (2) The feedback matrx, W, s bock dagona: N W dag W (1),..., W ( 2 ) (3) Each dagona eement, correspondng to a bock of recurrent neurons, has a bock sub-matrx n the form: [ ] W () w2,2 w 2,2+1 1, 2,... N (4) w 2+1,2 w 2+1,2+1 2 Equaton (4) descrbes the genera case of BDRNN, whch s caed BDRNN wth free-form sub-matrces. A speca case of BDRNN conssts of scaed orthogona sub-matrces n the form [ ] [ W () w2,2 w 2,2+1 w 2,2+1 w 2,2 w (1) w (2) w (2) w (1) ] 1, 2,... N 2 From (4) and (5) t s concuded that the Free-Form BDRNN conssts of feedback sub-matrces wth four dstnct eements and provdes a greater degree of freedom compared to the Scaed Orthogona BDRNN, whch has two weghts at each feedback sub-matrx. Nevertheess, as dscussed n [10], the atter network exhbts superor modeng capabtes than the Free-Form BDRNN, and the forthcomng earnng method w be deveoped for ths network. (5)

5 Hybrd earnng agorthm 5 Combnng (1)-(5), the state equatons for the Scaed Orthogona BDRNN can take the foowng form: m x 2 1 (k) f a b 2 1,j u j (k) + w (1) x 2 1 (k 1) + w (2) x 2 (k 1), 1,..., N 2 j1 j1 (6a) m x 2 (k) f a b 2,j u j (k) w (2) x 2 1 (k 1) + w (1) x 2 (k 1), 1,..., N 2 N y (k) f b c,j x j (k), 1,..., r j1 where w (1), w (2) are the feedback weghts at the hdden ayer. 3 The HLA-BDRNN Agorthm (6b) In gradent-based optmzaton methods, the weght changes are proportona to the sze of the gradent of an error functon E: (6c) w (t) µ E(t) (7) where E s the parta dervatve of E wth respect to a weght w and t represents the epoch ndex. When t comes to recurrent modes, the common parta dervatve shoud be substtuted by the ordered parta dervatve, due to the exstence of tempora reatons through the feedback connectons of BDRNN, as w be dscussed n the seque. The term µ n (7) s the earnng rate, whch n BPTT s kept fxed throughout the earnng process and s common to a weght updates. Therefore, an approprate seecton of the earnng rate s cruca to the evouton of the earnng process and consttutes a sgnfcant constrant. The RPROP earnng scheme attempted to aevate ths dsadvantage of BPTT by aowng each fttng parameter to have ts ndvdua step sze, whch s adjusted durng the earnng process based on the sgn of the respectve parta dervatve at the current and the prevous epoch. Therefore, the effect of the adaptaton process w not burred by the nfuence of the sze of the parameter gradent but s ony dependent on the tempora behavor of the gradent ([8]). Partcuary, et E(t) and E(t 1) denote the dervatves of E wth respect to w at the present and the precedng epochs, respectvey. RPROP s descrbed n pseudo-code as foows:

6 6 Dmtrs Varsams, Pars Mastorocostas and Evangeos Outsos Some very ong text... (a) For a weghts w ntaze the step szes (1) 0 Repeat (b) For a weghts w compute the error gradent: E(t) (c) For a weghts w, update step szes: (c.1) If E(t) (c.2) Ese f E(t) E(t 1) E(t 1) > 0 Then (t) < 0 Then (t) mn η + (t 1), max max η (t 1), mn (c.3) Ese (t) (t 1) ( ) (d) Update the weghts w : w (t) sgn E(t) (t) Unt convergence where the step szes are bounded by mn, max. The nta vaues of the step szes (1) 0 are chosen rather moderatey (e.g. 0.1), snce these vaues drecty determne the szes of the frst parameter changes. The ncrease and attenuaton factors are set to n + [1.01, 1.3] and n [0.5, 0.9], respectvey. The HLA-BDRNN agorthm performs the foowng modfcatons: () Substtutes the error gradents wth the ordered dervatves + E(t) ([14]), n order to take nto consderaton the tempora dependences exstng n a dynamc mode. () Modfes steps (b) and (c.2) of RPROP as shown: (b ) Compute the HLA-BDRNN error gradent: + E(t) 0.01 SA (c.2 ) Ese f + E(t) If ( (t) Ese (t) + E(t 1) < 0 Then < 0.4 SA 2 ) Then (t) max max η (t 1), mn w 1+w 2 η (t 1) 0.8 r SA 2, mn where SA 2 t T emp s the smuated anneang term, parameter r takes random vaues wthn the nterva [0, 1] and T emp s the temperature. The modfed step (c.2 ) ams at addng nose to the weghts, accordng to the concept of smuated anneang, n order to ncrease the convergence speed of the earnng process. In HLA-BDRNN nose s added to the weght update vaues when the error gradent changes sgn n two successve epochs, and the magntude of the update vaue s ess than a vaue that s proportona to the SA term. In ths way, the weght update s modfed by nose ony when t has a reatvey sma vaue, thus aowng the weght to move out of oca mnma, whe mnmzng the dsturbance to the adaptaton process. In the modfed step (b ), a weght decay term s added to the error gradent, as proposed n [1]. The effect of ths form of weght decay s to modfy the error surface such that ntay weghts wth ower vaues are favoured. As tranng proceeds, the magntude of weght decay s reduced, factatng the ncrease of bgger weghts and aowng the mode to expore regons of the error surface that were prevousy unavaabe. Addtonay, as mentoned n [1], the use of weght decay has been proved to mprove the generazaton capabty of the

7 Hybrd earnng agorthm 7 mode. The adaptaton mechansm descrbed above has the advantage of correatng the step szes not to the sze of the dervatves but to ther sgns. Hence, whenever a parameter moves aong a drecton reducng E (the dervatves at successve epochs have the same sgn), ts step sze s ncreasng ndependenty of the sze of the dervatve. In ths way, the step szes can suffcenty ncrease when needed, even at the fna stage of the earnng process when the szes of the dervatves are rather sma. Addtonay, when changes n the sgn of the dervatve occur, the step sze s dmnshng to prevent the error measure from oscatng. A key ssue n the case of ocay recurrent networks ke BDRNN s the accurate extracton of the error gradents. In the proposed agorthm the tme dependences are fuy taken nto consderaton, and no approxmaton pocy to a few steps back s apped. The error measure used s the Mean Squared Error (MSE), defned by E 1 k f k f k1 [y (k) ŷ (k)] 2 (8) where y (k) s the -th mode output, ŷ (k) s the -th desred (actua) output of the system at tme step k. Contrary to statc networks, the extracton of the ordered parta dervatves n BDRNN s not straghtforward and s accompshed va a set of recursve equatons. In order to determne the error gradents of the dynamc part of BDRNN, et us ntroduce (a) the state vector st(t), defned as: st(k) [x 1 (k),..., x N (k), y 1 (k),..., y r (k)] T comprsng the outputs of the hdden and the output ayer. (b) the contro vector θ comprsng the synaptc and feedback weghts (N (m + r + 1) weghts) [ ] T θ b 1,1,..., b N,m, w (1) 1, w (1) N, w (2) 1, w (2) N, c 1,1,..., c r,n 2 For a data set ncudng k f pars, the state equatons are wrtten f (st(k), θ(k)) 0, k 1,..., k f wth f (1) 2 1 (k) 1,..., N 2 (k f N 2 equatons): f a ( m j1 b 2 1,j u j (k) + w (1) x 2 1 (k 1) + w (2) x 2 (k 1) 2 ) x 2 1 (k) 0 (9a)

8 8 Dmtrs Varsams, Pars Mastorocostas and Evangeos Outsos f (1) 2 (k) 1,..., N 2 (k f N 2 equatons): f a ( m j1 b 2,j u j (k) w (2) x 2 1 (k 1) + w (1) x 2 (k 1) f (2) (k) 1,..., r (k f r equatons): ( N ) f b c,j x j (k) y (k) 0 j1 ) x 2 (k) 0 (9b) (9c) where the extracton of the La- f + λt 0. θ The error gradents are gven by + E λ T f θ θ grange mutpers λ s based on the formua E After cacuatons are conducted n (19), the mutpers are determned through the foowng recursve equatons: x λ (2) (k) 1 k f [y (k) ŷ (k)] (10a) λ (1) 2 1 (k) 1 k f λ (1) 2 (k) 1 k f c,2 1 f b () (k) [y (k) ŷ (k)] + (2 1) (2) () λ E (k) c,2 1 f b (k) + λ (1) 2 1 (k + 1) w(1) f a (k + 1) λ (1) 2 (k + 1) w(2) f a (k + 1) (10b) () c,2 f b (k) [y (k) ŷ (k)] + + λ (1) 2 1 (k + 1) w(2) f a (2 1) (k + 1) + λ (1) 2 λ (2) (k + 1) w(1) (2) () (k) c,2 f b (k) (2) + f a (k + 1) (10c) where 1,..., N, 1,..., r and f 2 b (() k ), f 1( k (2 1) + j), f 1( k (2) + j) are the dervatves of y j (k + ) and x 2 1 (k + j), x 2 (k + j), respectvey, wth respect to ther arguments. Equatons (10) are backward dfference equatons that can be soved for k k f, k f 1,..., 1 usng the foowng boundary condtons: λ (1) 2 1 (k f) 1 k f λ (2) (k f ) 1 k f [y (k f ) ŷ (k f )] (11a) c,2 1 f b () (k f ) [y (k f ) ŷ (k f )] + λ (2) () (k f ) c,2 1 f b (k f ) (11b)

9 Hybrd earnng agorthm 9 λ (1) 2 (k f) 1 k f c,2 f b () (k f ) [y (k f ) ŷ (k f )] + λ (2) () (k f ) c,2 f b (k f ) (11c) Substtutng (17) and (21) to (18), and takng nto consderaton (12), the error gradents are gven by + E c + E b j k f k1 k f k1 λ (2) () (k) x (k) f b (k) 1,..., r, 1,..., N λ (1) () (k) u j (k) f a (k) 1,..., N, j 1,..., m (12a) (12b) + E w (1) + E w (2) k f k1 k f k1 λ (1) 2 1 (k) x 2 1(k 1) f a (2 1) (k) +λ (1) 2 (k) x 2(k 1) f a (2) (k), 1,..., N 2 (12c) λ (1) 2 1 (k) x 2(k 1) f a (2 1) (k) λ (1) 2 (k) x 2 1(k 1) f a (2) (k), 1,..., N 2 (12d) 4 Performance Tests and Resuts The stabzng propertes and the performance of the M-SARPROP approach are hghghted by use of a benchmark dentfcaton probem of a dynamca system [6]. The actua system s descrbed by the dfference equaton: y p (k + 1) y p(k) y p (k 1) y p (k 2) u(k 1) [y p (k 2) 1] + u(k) 1 + y 2 p(k 1) + y 2 p(k 2) A parae dentfcaton system s consdered, wth the nput u(k) beng the soe nput to the network. The BDRNN comprses four bocks of neurons n the hdden ayer and has a tota number of 32 weghts. The frst objectve of the expermentaton s to compare the performance of the HLA-BDRNN method to BPTT [5], n tranng of the BDRNN. A second objectve s compare the BDRNN and the earnng agorthm wth other recurrent modes, n terms of approxmaton accuracy and generazaton capabtes. In compance wth prevous resuts reported n the terature, the tranng data set contans ten batches of 900 patterns. For each data batch, the nput u(k) s an ndependent and dentcay dstrbuted unform sequence for the frst haf of the 900 tme steps and a snusod gven by 1.05 sn(πk/45)

10 10 Dmtrs Varsams, Pars Mastorocostas and Evangeos Outsos for the rest of the tme nstants. The checkng data set s composed of 1000 sampes wth a sgna descrbed by sn(πk/25) k < k < 500 u(k) k < sn(πk/25) sn(πk/32) sn(πk/10) 750 k < 1000 In order to seect the tranng parameters of HLA-BDRNN and BPTT, severa runs are performed wth the same nta weghts but dfferent parameter combnatons. Then, the parameter combnaton that has exhbted the fastest convergence and ow vaues of the error functon s seected. Thus, we are ed to , η and η 0.85, T emp 1.15 for HLA-BDRNN, and a earnng rate of µ for BPTT, respectvey. Next, a seres of 100 ndependent tras wth dfferent weght ntazatons are attempted. Partcuary, the feedback weghts W and the weght matrces B and C are randomy seected wthn the range [ 0.5, 0.5]. The sope pertanng to the actvaton functons of the network neurons s set to 2. For each partcuar tra and for far comparson, the same weght ntazatons are used for HLA-BDRNN and BPTT. The competng rvas are recurrent neura networks and ther archtectures and earnng parameters are determned as foows: The IIR-MLP s a mutayered network [13], where the synaptc connectons are mpemented through IIR fters, ncudng a movng average (MA) and an auto-regressve (AR) part. We seected a 1x8x1 IIR-MLP mode wth unt deays n the MA and the AR parts, both for the nputto-hdden and the hdden-to-output synaptc fters, respectvey. The dagona recurrent neura network (DRNN, [3]) has one hdden ayer, contanng sef-recurrent neurons. A 1x8x1 DRNN mode s seected. The weghts of the IIR-MLP and DRNN are randomy ntazed n that range [ 0.5, 0.5]. The earnng rate s set to 0.01, seected as best vaue after severa tranng runs. The IIR-MLP and DRNN modes are traned by BPTT, whe the memory neura network (MNN, [9]) s traned usng the rea tme recurrent earnng (RTRL) method [7]. A network modes and earnng schemes are traned foowng a parae mode approach, wth the excepton of the MNN where the seres-parae confguraton s adopted, as reported n [9].

11 Hybrd earnng agorthm 11 Tabe 1 hosts the comparatve resuts attaned after fve tranng epochs of the entre data set, wth the weght updates takng pace at the end of each one of the ten batches. The resuts for the MNN are taken from [9]. As shown, the BDRNN traned by the HLA-BDRNN method exhbts the best performance among the competng schemes, wth regard to both the average and, especay, the standard devaton of the checkng data sets error. The former crteron ndcates the accuracy of the HLA-BDRNN agorthm whe the ater one shows ts robustness to weght ntazatons. The BPTT scheme for the BDRNN s consderaby nferor to the HLA-BDRNN method regardng the accuracy and the generazaton property. Furthermore t has an error standard devaton amost twce as arge compared to the one attaned by HLA-BDRNN, eadng to the concuson that HLA-BDRNN acceerates the earnng process for the BDRNN sgnfcanty, whe exhbtng nsenstvty to nta weght settngs. Tabe 1: Resuts of the comparatve anayss, averaged over 100 ndependent tras wth dfferent weght ntazatons Network Tranng Checkng Checkng No. of type method MSE Avg MSE StD weghts BDRNN HLA-BDRNN BDRNN BPTT IIR-MLP BPTT DRNN BPTT MNN RTRL It shoud be ponted out that the proposed agorthm has been deveoped for Scaed Orthogona BDRNNs. However, t can be easy modfed to take nto consderaton the archtectura dfferences of the Free-Form BDRNN (.e. four tunabe feedback weghts at each bock of neurons of the hdden ayer). 5 Concuson A nove earnng agorthm for tranng the speca cass of Bock-Dagona Recurrent Neura Networks has been proposed, entted HLA-BDRNN. The hybrd method combnes gradent descent and the random search technque of Smuated Anneang, and s taored to BDRNN, by takng nto account the tempora reatons exstng n ths partcuar dynamc system. It shoud be mentoned that the agorthm can be easy adapted wth moderate modfcatons to reevant archtectures ke the tranguar recurrent neura network [6].

12 12 Dmtrs Varsams, Pars Mastorocostas and Evangeos Outsos The earnng scheme has been compared wth a seres of agorthms and recurrent networks n the context of a nonnear system dentfcaton benchmark probem, where ts earnng characterstcs have been hghghted. Acknowedgements. The authors wsh to acknowedge fnanca support provded by the Research Commttee of the Technoogca Educaton Insttute of Centra Macedona, under grant SAT/IC/ /12. References [1] C. Ige, H. Husken, Emprca Evauaton of the Improved RPROP Learnng Agorthms, Neurocomputng, 50 (2003), [2] C.-C. Ku, K.Y. Lee, Dagona Recurrent Neura Networks for Dynamc Systems Contro, IEEE Transactons on Neura Networks, 6 (1995), no. 1, [3] R. Kumar, S. Srvastava, J.R.P. Gupta, Dagona Recurrent Neura Network Based Adaptve Contro of Nonnear Dynamca Systems Usng Lyapunov Stabty Crtera, ISA Transactons, 67 (2017), [4] P. Mastorocostas, C. Has, D. Varsams, S. Dova, A Recurrent Neura Network-based Forecastng System for Teecommuncatons Ca Voume, Apped Mathematcs & Informaton Scences, 7 (2013), no. 5, [5] P. Mastorocostas, J.B. Theochars, A Stabe Learnng Agorthm for Bock-Dagona Recurrent Neura Networks: Appcaton to the Anayss of Lung Sounds, IEEE Transactons on Systems, Man, and Cybernetcs, Part B: Cybernetcs, 36 (2006), no. 2, [6] K.S. Narendra, K. Parthasarathy, Identfcaton and Contro of Dynamca Systems usng Neura Networks, IEEE Transactons on Neura Networks, 1 (1990), no. 1, [7] S. Pche, Steepest Descent Agorthms for Neura Network Controers and Fters, IEEE Transactons on Neura Networks, 5 (1994), no. 2,

13 Hybrd earnng agorthm 13 [8] M. Redmer, H. Braun, A Drect Adaptve Method for Faster Backpropagaton Learnng: The RPROP Agorthm, IEEE Internatona Conference on Neura Networks, (1993), [9] P.S. Sastry, G. Santharam, K.P. Unnkrshnan, Memory Neuron Networks for Identfcaton and Contro of Dynamca Systems, IEEE Transactons on Neura Networks, 5 (1994), no. 2, [10] S. Svakumar, W. Robertson, W.J. Phps, Onne Stabzaton of Bock- Dagona Recurrent Neura Networks, IEEE Transactons on Neura Networks, 10 (1999), no. 1, [11] S. Svakumar, Sh. Svakumar, Margnay Stabe Tranguar Recurrent Neura Network Archtecture for Tme Seres Predcton, IEEE Transactons on Cybernetcs, (2017), [12] N.K. Treadgod, T.D. Gedeon, Smuated Anneang and Weght Decay n Adaptve Learnng: The SARPROP Agorthm, IEEE Transactons on Neura Networks, 9 (1998), no. 4, [13] A.C. Tso, A.D. Back, Locay Recurrent Gobay Feedforward Networks: A Crtca Revew of Archtectures, IEEE Transactons on Neura Networks, 5 (1994), no. 2, [14] P. Werbos, Beyond Regresson: New Toos for Predcton and Anayss n the Behavora Scences, Ph.D. Thess, Harvard Unv., Receved: December 9, 2017; Pubshed: January 5, 2018