Simplified Gating in Long Short-term Memory (LSTM) Recurrent Neural Networks

Transcription

1 Simplified Gaing in Long Shor-erm Memory (LSTM) Recurren Neural Neworks Yuzhen Lu and Fahi M. Salem Circuis, Sysems, and Neural Neworks (CSANN) Lab Deparmen of Biosysems and Agriculural Engineering Deparmen of Elecrical and Compuer Engineering Michigan Sae Universiy Eas Lansing, Michigan 48824, USA Absrac The sandard LSTM recurren neural neworks while very powerful in long-range dependency sequence applicaions have highly complex srucure and relaively large (adapive) parameers. In his work, we presen empirical comparison beween he sandard LSTM recurren neural nework archiecure and hree new parameer-reduced varians obained by eliminaing combinaions of he inpu signal, bias, and hidden uni signals from individual gaing signals. The experimens on wo sequence daases show ha he hree new varians, called simply as LSTM1, LSTM2, and LSTM3, can achieve comparable performance o he sandard LSTM model wih less (adapive) parameers. Index Terms Recurren Neural Neworks (RNN), Long Shor-erm Memory (LSTM), Sochasic Gradien Descen I. INTRODUCTION Recurren neural neworks (RNN) have recenly shown grea promise in ackling various sequence modeling asks in machine learning, such as auomaic speech recogniion [1-2], language ranslaion [3-4], and generaion of language descripions for images [5-6]. Simple RNNs, however, are difficul o rain using he sochasic gradien decen and have been repored o exhibi he so-called vanishing gradien and/or exploding gradien phenomena [7-8]. This has limied he abiliy of simple RNN o learn sequences wih relaively long dependencies. To address his limiaion, researchers have developed a number of echniques in nework archiecures and opimizaion algorihms [9-11], among which he mos successful in applicaions is he Long Shor-erm Memory (LSTM) unis in RNN [9, 12]. A LSTM uni uilizes a memory cell ha may mainain is sae value over a long ime, and a gaing mechanism ha conains hree non-linear gaes, namely, an inpu, an oupu and a forge gae. The gaes inended role is o regulae he flow of signals ino and ou of he cell, in order o be effecive in regulaing long-range dependencies and achieve successful RNN raining. Since he incepion of he LSTM uni, many modificaions have been inroduced o improve performance. Gers e al. [13] have inroduced peephole connecions o he LSTM uni ha connecs he memory cell o he gaes so as o infer precise iming of he oupus. Sak e al. [14-15] inroduced wo recurren and non-recurren projecion layers beween he LSTM unis layer and he oupu layer, which resuled in significanly improved performance in a large vocabulary speech recogniion ask. Adding more componens in he LSTM unis archiecure, however, may complicae he learning process and hinder undersanding of he role of individual componens. Recenly, researchers proposed a number of simplified varians of he LSTM-based RNN. Cho e al. [3] proposed a wo-gae relaed archiecure, called Gaed Recurren Uni (GRU) RNN, in which he inpu, forge, and oupu gaes are replaced by an updae gae and a rese gae. Chung e al. [16] presened performance comparisons beween LSTM and GRU RNNs, and observed ha he laer performed comparably or even exceeded he former on he specific daase used. These conclusions, however, sill are being furher evaluaed using more experimens and over broader daases. In exploring eigh archiecural varians of he LSTM RNN, Greff e al. [17] found ha coupling he inpu and forge gaes, as in he GRU model, and removing peephole connecions, did no significanly impair performance. Furhermore, hey repor ha he forge gae and he oupu acivaion are criical componens. These findings were corroboraed by he work of Jozefowicz e al. [18] who evaluaed an exensive archiecural designs of en housand differen RNNs. In [18], he auhors observed ha he oupu gae was he leas imporan compared o he inpu and forge gaes, and suggesed adding a bias of 1 o he forge gae o improve he performance of he LSTM RNN. Zhou e al. [19] proposed a Minimal Gae Uni (MGU), which has a minimum of one gae, namely, he forge gae archiecure, creaed by merging he updae and rese gaes in he GRU model. Through evaluaions on four differen sequence daa, he auhors found ha an RNN wih he fewer parameers MGU model was a par wih he GRU model in erms of (esing) accuracy. The auhors, however, did no explicily perform comparisons agains he sandard LSTM RNN. Recenly, Salem [20] inroduced a simple approach o simplifying he sandard LSTM model focusing only on he gaing signal generaion. The gaing signals can be used as general conrol signals o be specified by minimizing he loss funcion/crierion. Specifically, all hree gaing equaions were reained bu heir parameers were reduced by eliminaing one or more of he signals driving he gaes. For simpliciy, we shall call hese hree varians, LSTM1, LSTM2, and LSTM3 and will be deailed in secion III below. The paper presens a comparaive evaluaion of he sandard LSTM RNN model wih hree new LSTM model varians. The evaluaion and es resuls have been demonsraed on wo public daases which reveal ha he LSTM model varians are comparable o he sandard LSTM

2 RNN model in esing accuracy performance. We remark ha hese are iniial ess and furher evaluaions and comparisons need o be conduced among he sandard LSTM RNN and he hree LSTM varians. The remainder of he paper is organized as follows. Secion II specifies he sandard LSTM RNN archiecure wih is hree gaing signals. Secion III describes he hree LSTM varians, called LSTM1, LSTM2, and LSTM3, respecively. Secion IV presens he experimens considered in his sudy. Secion V deails he comparaive performance resuls. Finally, secion VI summarizes he main conclusions. II. THE RNN LSTM ARCHITECTURE The LSTM archiecure considered here is similar o ha in Graves e al. [2, 16-19] bu wihou peep-hole connecions. I is referred o as he sandard LSTM archiecure and will be used for comparison wih is simplified LSTM varians [20]. The (dynamic) equaions for he LSTM memory blocks are given as follows: i = σ ( U h + W x + b ) (1) i 1 i i f = σ ( U h + W x + b ) (2) f 1 f f o = σ ( U h + W x + b ) (3) o 1 o o c = f c + i anh( U h + W x + b ) (4) 1 c 1 c c h = o anh( c ) (5) In hese equaions, he (n-d vecors) i, f and o are he inpu gae, forge gae, oupu gae a ime, eqns (1)-(3). Noe ha hese gae signals include he logisic nonlineariy,, and hus heir signals ranges is beween 0 and 1. The n-d cell sae vecor,, and is n-d acivaion hidden uni, h, a he curren ime, are in eqns (4)-(5). The inpu vecor,, is an m-d vecor, anh is he hyperbolic angen funcion, and * in eqns (4)-(5), denoes a poin-wise (Hadamard) muliplicaion operaor. Noe ha he gaes, cell and acivaion all have he same dimension (n). The parameers of he LSTM model are he marices (, ) and biases ( ) in eqns (1)-(5). The oal number of parameers (i.e., he number of all he elemens in W, U and b ), say N, for he sandard LSTM, can be calculaed o be 2 N = 4 ( m n+ n + n) (6) where, again, m is he inpu dimension, and n is he cell dimension. This consiues a four-fold increase in parameers in comparison o he simple RNN [16-20]. III. THE RNN LSTM VARIANTS While he LSTM model has demonsraed impressive performance in applicaions involving sequence-o-sequence relaionships, a criicism of he sandard LSTM resides in is relaively complex model srucure wih 3 gaing signals and he number of is relaively large number of parameers [see eqn (6)]. The gaes in fac replicae he parameers in he cell. I is observed ha he gaes serve as conrol signals and he forms in eqns (1)-(3) are redundan [20]. Here, hree simplificaions o he sandard LSTM resul in hree LSTM varians we refer o hem here as simply, LSTM1, LSTM2, and LSTM3. There varians are obained by removing signals, and associaed parameers in he gaing eqns (1)-(3). For uniformiy and simpliciy, we apply he changes o all he gaes idenically: 1) The LSTM1 model: No Inpu Signal Here he inpu signal and is associaed parameer marix are removed from he gaing signals (1)-(3). We hus obain he new gaing equaions: i = σ ( U h + b ) (7) i 1 i f = σ ( U h + b ) (8) f 1 f o = σ ( U h + b ) (9) o 1 o 2) The LSTM2 model: No Inpu Signal and No Bias The gaing signals conain only he hidden acivaion uni in all hree gaes, idenically. i = σ ( U h ) (10) i 1 f = σ ( U h ) (11) f 1 o = σ ( U h ) (12) o 1 3) The LSTM3 model: No Inpu Signal and No Hidden Uni Signal The gaing signals conain only he bias erm. Noe ha, as he bias is adapive during raining, i will include informaion abou he sae via he backpropagaion learning algorihms or he co-sae [24]. i = σ ( b ) (13) i f = σ ( b ) (14) f o = σ ( b ) (15) Compared o he sandard LSTM, i can be seen ha he 2 hree varians resuls in 3mn, 3( mn+ n) and 3( mn+ n ) fewer parameers, respecively, and consequenly, reducing he compuaional expense. IV. EXPERIMENTS The effeciveness of he hree proposed varians were evaluaed using wo public daases, MNIST and IMDB. The focus here is o demonsrae he comparaive performance of he sandard LSTM RNN and he varians raher han o achieve sae-of-he-ar resuls. Only he sandard LSTM RNN o

3 [2, 16-19] was used as a base-line model and compared wih is hree varians. A. Experimens on he MNIST daase: This daase conains 60,000 raining se and 10,000 esing se of handwrien images of he digis (0-9). The raining se conains he labelled class of he image available for raining. Each image has he size of pixels. The image daa were pre-processed o have zero mean and uni variance. As in he work of Zhou e al. [19], he daase was organized in wo manners o be he inpu of an LSTM-based nework. The firs was o reshape each image as a onedimensional vecor wih pixels scanned row by row, from he op lef corner o he boom righ corner. This resuls in a long sequence inpu of lengh 784. The second requires no image reshaping bu reaed each row of an image as a vecor inpu, hus giving a much shorer inpu sequence of lengh 28. The wo ypes of daa organizaion were referred o as pixel-wise and row-wise sequence inpus, respecively. I is noed ha he pixel-wise sequence is more ime consuming in raining. In he wo raining asks, 100 hidden unis and 100 raining epochs were used for he pixel-wise sequencing inpu, while 50 hidden unis and 200 raining epochs were used for he row-wise sequencing inpu. Oher nework seings were kep he same hroughou, including he bach size se o 32, RMSprop opimizer, cross-enropy loss, dynamic learning rae (η) and early sopping sraegies. In paricular, for he learning rae, i was se o be an exponenial funcion of raining loss o speed up raining, specifically, η= η 0 exp(c), where η 0 is a consan coefficien, and C is he raining loss. For he pixelwise sequence, wo learning rae coefficiens η 0 =1e-3 and 1e-4 were considered as i akes relaively long ime o rain, while for he row-wise sequence, four η 0 values of 1e-2, 1e-3, 1e-4 and 1e-5 were considered. The dynamic learning rae is hus direcly relaed o he raining performance. A he iniial sage, he raining loss is ypically large, hus resuling in a large learning rae (η), which in urn increases he sepping of he gradien furher from he presen parameer locaion. The learning rae decreases only as he loss funcions decreases owards lower loss level and evenually owards an accepable minima in he parameer space. Thus was found o achieve faser convergence o an accepable soluion. For he early sopping crierion, he raining process would be erminaed if here was no improvemen on he es daa over consecuive epochs, in our case we chose 25 epochs. B. Experimens on he IMDB daase: This daase consiss of 50,000 movie reviews from IMDB, which have been labelled ino wo classes according o (he reviews) senimen, posiive or negaive. Boh raining and es ses conain 25,000 reviews. These reviews are encoded as a sequence of word indices based on he overall frequency in he daase. The maximum sequence lengh was se o 80 among he op 20,000 mos common words (longer sequences were runcaed while shorer ones were zero-padded a he end). Referring o an example in he Library Keras [21], an embedding layer wih he oupu dimension of 128 was added as an inpu o he LSTM layer ha conained 128 hidden unis. The dropou echnique [22] was implemened o randomly zero 20% of signals in he embedding layer and 20% of rows in he weigh marices (i.e., U and W) in he LSTM layer. The model was rained for 100 epochs. Oher seings remained he same as hose in he MNIST daa. Training for he wo daases were implemened by using he Keras package in conjuncion wih he Theano library (he implemenaion code and resuls are available a: hps://gihub.com/jingweimo/modified-lstm). V. RESULTS AND DISCUSSION A. The MNIST daase: Table I summarizes he accuracies on he es daase for he pixel-wise sequence. A η 0 =1e-3, he sandard LSTM produced he highes accuracy, while a η 0 =1e-4, boh LSTM1 and LSTM2 achieved accuracies slighly higher han ha by he sandard LSTM. LSTM3 performed he wors in boh cases. TABLE I THE BEST ACCURACIES OF DIFFERENT LSTM NETWORKS ON THE TEST SET AND CORRESPONDING PARAMETER SIZES OF THE LSTM LAYERS Learning coefficien (η 0) 1e-3 1e-4 #Params Sandard ,800 LSTM ,500 LSTM ,200 LSTM ,500 Examining he raining curves revealed he imporance of η 0 and he differen responses of he. As shown in Fig. 1, he sandard LSTM performed well in he wo cases; while LSTM1 and LSTM2 performed similarly poorly a η 0 =1e-3, where boh suffered serious flucuaions a beginning and dramaically lowered accuracies a he end. However, decreasing η 0 o 1e-4 circumvened he problem fairly well for LSTM1 and LSTM2. For LSTM3, boh η 0 =1e-3 and 1e-4 could no achieve successful raining because of he flucuaion issue, suggesing ha η 0 should be decreased furher. As shown in Fig. 2 where 200 raining epochs were execued, choosing η 0 =1e-5 provided a seadily increasing accuracy wih he highes es accuracy of Despie he accuracy ha was sill lower han oher varians. I is expeced ha he LSTM3 would achieve higher accuracies if longer raining ime was allowed. In essence LSTM3 has he lowes parameers bu needs more raining execuion epochs o improve (esing) accuracy. The flucuaion phenomenon observed above is a ypical issue caused by a large learning rae, and is likely due o numerical insabiliy where he (sochasic) gradien can no longer be approximaed, and i can readily be resolved by decreasing he learning coefficien-- however, a he price of slowing down raining. From he resuls, he sandard LSTM seemed more resisan o flucuaions in modeling longsequence daa han he hree varians, more likely due o he suiabiliy of he learning rae coefficien. LSTM3 was he mos suscepible o he flucuaion issue, however, and i

4 requires a lower coefficien. Is opimal coefficien in his sudy appears o be beween η 0 =1e-5 and η 0 =1e-4. Thus, furher invesigaion may lead o benefis of using he LSTM3 o reap he benefi of is dramaically reduced model parameers (see Table I). Overall, hese findings have showed ha he hree LSTM varians were capable of handling a long-range dependencies sequence comparable o he sandard LSTM. Due aenion should be paid o uning he learning rae o achieve higher accuracies. TABLE II THE BEST ACCURACIES OF DIFFERENT LSTM NETWORKS VARIANTS ON THE TEST SET AND THEIR CORRESPONDING PARAMETER SIZES Learning rae coefficien (η 0) 1e-2 1e-3 1e-4 1e-5 # Params Sandard ,800 LSTM ,600 LSTM ,450 LSTM ,100 Among he four η 0 values, he η 0 =1e-3 gave he bes resuls for all he excep LSTM2 ha performed he bes a η 0 =1e-2. Fig. 3 shows he corresponding learning curves a η 0 =1e-3. All he exhibied similar raining paern profiles, which demonsraed he efficacy of he hree LSTM varians. Fig. 1 Accuracies vs. epochs on he rain/es daases obained by he sandard LSTM (op), LSTM1 (middle) and LSTM2 (boom), wih he learning rae coefficiens η 0 = 1e-3 (lef) and η 0 = 1e-4 (righ). The difference in epochs is due o he response o he implemened early sopping crierion. Fig. 3 Accuracies vs. epochs on he rain/es daase obained by he sandard LSTM (a), LSTM1 (b), LSTM2 (c) and LSTM3 (d) wih he learning rae coefficien η 0 = 1e-3. The difference in epochs is due o he response o he early sopping crierion. From he resuls of he pixel-wise (long) and row-wise (shor) sequence daa, i is noed ha he hree LSTM varians, especially LSTM3, performed closely similar o he sandard LSTM in handling he shor sequence daa. Fig. 2 Accuracies vs. epochs on he rain/es daases obained by he LSTM3 wih he learning rae coefficiens η 0 = 1e-4 (lef) and η 0 = 1e-5 (righ). The difference in epochs is due o he response o he early sopping crierion. Compared o he pixel-wise sequence of lengh 784, he row-wise sequence form is 28 in lengh and was much easier (and faser) o rain. Table II summarizes he resuls. All he achieved high accuracies a four differen η 0. The sandard LSTM, LSTM1 and LSTM2 performed similarly, where hey all slighly ouperformed he LSTM3. No flucuaion issues were encounered in all he cases. These experimens have used neworks wih 50 hidden unis. B. The IMDB daase: For his daase, he inpu sequence from he embedding layer o he LSTM layer is of he inermediae lengh 128. Table III liss he esing resuls for various learning coefficiens. The sandard LSTM and he hree varians have show similar accuracies, excep ha LSTM1 and LSTM2 show slighly lower performance a η 0 =1e-2. Similar o he row-wise MNIST sequence case sudy, no noiceable flucuaions have been observed for any of he four values of η 0.

5 TABLE III THE BEST ACCURACIES OF DIFFERENT LSTM NETWORKS ON THE TEST SET AND CORRESPONDING PARAMETER SIZES OF THE LSTM LAYERS Learning rae coefficien (η 0) 1e-2 1e-3 1e-4 1e-5 # Params Sandard ,584 LSTM ,432 LSTM ,048 LSTM ,280 The case for η 0 =1e-5, consisenly produced he bes resuls for all he and exhibied very similar raining/esing profile curves as depiced in Fig. 4. Fig. 4 Accuracies agains epochs on he es daase obained by he sandard LSTM (a), LSTM1 (b), LSTM2 (c) and LSTM3 (d) wih he learning rae coefficien η 0 = 1e-5. The main benefi of he hree LSTM varians is o reduce he number of parameers involved, and hus reduce he compuaion expense. This has been confirmed from he experimens and as summarized in he hree ables above. The LSTM1 and LSTM2 show small difference in he number of parameers and boh conain he hidden uni signal in heir gaes, which explains heir similar performance. The LSTM3 has dramaically reduced parameers size since i only uses he bias, an indirecly conained delayed version of he hidden uni signal via he gradien descen updae equaions. This may explain heir relaive lagging performance, especially in long sequences. The acual reducion of parameers is dependen on he srucure (i.e., dimension) of inpu sequences and he number of hidden unis in he LSTM layer. VI. CONCLUSIONS In his paper, hree simplified ha were defined by eliminaing inpu signal, bias and/or hidden unis from heir he gae signals in he sandard LSTM RNN, were evaluaed on he asks of modeling sequence daa of varied lenghs. The resuls confirmed he uiliy of he hree LSTM varians wih reduced parameers, which a proper learning raes were capable of achieving he performance comparable o he sandard LSTM model. This work represens a preliminary sudy, and furher work is needed o evaluae he hree LSTM varians on more exensive daases of varied sequence lengh. REFERENCES [1] A. Graves, Supervised Sequence Labelling wih Recurren Neural Neworks, Berlin, Heidelbergz: Springer-Verlag, [2] A. Graves, Speech recogniion wih deep recurren neworks, in Acousics, Speech and Signal Processing (ICASSP), 2013 IEEE Inernaional Conference on, 2013, pp [3] K. Cho, B. van Merienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, Learning phrase represenaions using RNN encoder-decoder for saisical machine ranslaion, in Proceedings of he 2014 Conference on Empirical Mehods in Naural Language Processing, Associaion for Compuaional Linguisics, 2014, pp [4] I. Suskever, O. Vinyals, and Q. V. Le, Sequence o sequence learning wih neural neworks, in Proceedings of Advances in Neural Informaion Processing Sysems 27, NIPS, 2014, pp , [5] A. Karpahy, F. F. Li, Deep visual-semanic alignmens for generaing image descripions, in Proceedings of IEEE Inernaional Conference on Compuer Vision and Paern Recogniion, IEEE, 2015, pp [6] K. Xu, J. L. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. S. Zemel, and Y. Bengio. Show, aend and ell: Neural image capion generaion wih visual aenion, in Proceedings of he 32nd Inernaional Conference on Machine Learning, 2015, vol. 37, pp [7] Y. Bengio, P. Simard, and P. Frasconi, Learning long-erm dependencies wih gradien descen is difficul, IEEE Transacions on Neural Neworks, vol. 5, no. 2, pp , [8] R. Pascanu, T. Mikolov, and Y, Bengio, On he difficuly of raining recurren neural neworks, in Proceedings of he 30 h Inernaional Conference on Ma-chine Learning, JMLR: W&CP volume 28. [9] S. Hochreier, J. Schmidhuber. Long shor-erm memory, Neural Compuaion, vol. 9, no. 8, pp , [10] Q.V. Le, N. Jaily, G.E. Hinon, A simple way o iniialize recurren neworks of recified linear unis. arxiv preprin arxiv: , [11] J. Marens and I. Suskever, Training deep and recurren neural neworks wih Hessian-Free opimizaion. Neural Neworks: Tricks of he Trade, Springer, 2012, pp. pp [12] F. A. Gers, J. Schmidhuber, and F. Cummins. Learning o forge: Coninual predicion wih LSTM. in Proceedings of he 9 h Inernaional Conference on Arificial Neural Neworks, IEEE, 1999, vol. 2, pp [13] F. A. Gers, N. N. Schraudolph, and J. Schmidhuber. Learning precise iming wih LSTM recurren neworks, Journal of Machine Learning Research, vol. 3, pp , [14] H. Sak, A. Senior, and F. Beaufays, Long shor-erm memory recurren neural nework archiecures for large scale acousic modeling. ISCA, pp , [15] H. Sak, A. Senior, and F. Beaufays, Long shor-erm memory based recurren neural nework archiecures for large vocabulary speech recogniion, ArXiv e-prins, [16] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio. Empirical evaluaion of gaed recurren neural neworks on sequence modeling, arxiv: , [17] K. Greff, R. K. Srivasava, J. Kounk, B. R. Seunebrink, and J. Schmidhuber. LSTM: a search space odyssey, arxiv: , [18] R. Jozefowicz, W. Zaremba, and I. Suskever. An empirical exploraion of recurren nework archiecures, in Proceedings of he 32 nd Inernaional Conference on Machine Learning, 2015, vol. 37, pp , [19] G.B. Zhou, J. Wu, C.L. Zhang, Z.H. Zhou, Minimal gaed uni for recurren neural neworks, Inernaional Journal of Auomaion and Compuing, pp , [20] F. Salem, ` Reduced Parameerizaion in Gaed Recurren Neural Neworks, Memorandum , MSU, Nov [21] hps://gihub.com/fcholle/keras/blob/maser/examples/imdb_lsm.py. [22] W. Zaremba, I. Suskever, and O. Vinyals, Recurren neural nework regularizaion, arxiv preprin arxiv: , [23] G. Hinon, N. Srivasava, and K. Swersky Lecure 6a Overview of mini-bach gradien descen, Coursera Lecure slides. [24] F. Salem, A Basic Recurren Neural Nework Model, arxiv. Preprin arxiv: , Dec