Incorporating Side Information into Recurrent Neural Network Language Models

Size: px

Start display at page:

Download "Incorporating Side Information into Recurrent Neural Network Language Models"

Phillip Johnston
5 years ago
Views:

1 Incorporating Sid Information into Rcurrnt Nural Ntwork Languag Modls Cong Duy Vu Hoang Univrsity of Mlbourn Mlbourn, VIC, Australia Gholamrza Haffari Monash Univrsity Clayton, VIC, Australia Abstract Rcurrnt nural ntwork languag modls (RNNLM) hav rcntly dmonstratd vast potntial in modlling long-trm dpndncis for NLP problms, ranging from spch rcognition to machin translation. In this work, w propos mthods for conditioning RNNLMs on xtrnal sid information,.g., mtadata such as kywords or documnt titl. Our xprimnts show consistnt improvmnts of RNNLMs using sid information ovr th baslins for two diffrnt datasts and gnrs in two languags. Intrstingly, w found that sid information in a forign languag can b highly bnficial in modlling txts in anothr languag, srving as a form of cross-lingual languag modlling. 1 Introduction Nural ntwork approachs to languag modlling (LM) hav mad rmarkabl prformanc gains ovr traditional count-basd ngram LMs (Bngio t al., 2003; Mnih and Hinton, 2007; Mikolov t al., 2011). Thy offr svral dsirabl charactristics, including th capacity to gnralis ovr larg vocabularis through th us of vctor spac rprsntation, and for rcurrnt modls (Mikolov t al., 2011) th ability to ncod long distanc dpndncis that ar impossibl to includ with a limitd contxt windows usd in convntional ngram LMs. Ths arly paprs hav spawnd a cottag industry in nural LM basd applications, whr txt gnration is a ky componnt, including conditional languag modls for imag captioning (Kiros t al., 2014; Vinyals t al., 2015) and nural machin translation Trvor Cohn Univrsity of Mlbourn Mlbourn, VIC, Australia t.cohn@unimlb.du.au (Kalchbrnnr and Blunsom, 2013; Sutskvr t al., 2014; Bahdanau t al., 2015). Inspird by ths works for conditioning LMs on complx sid information, such as imags and forign txt, in this papr w invstigat th possibility of improving LMs in a mor traditional stting, that is whn applid dirctly to txt documnts. Typically corpora includ rich sid information, such as documnt titls, authorship, tim stamp, kywords and so on, although this information is usually discardd whn applying statistical modls. Howvr, this information can b highly informativ, for instanc, kywords, titls or dscriptions, oftn includ cntral topics which will b hlpful in modlling or undrstanding th documnt txt. W propos mchanisms for ncoding this sid information into a vctor spac rprsntation, and mans of incorporating it into th gnrating procss in a RNNLM framwork. Evaluating on two corpora and two diffrnt languags, w show consistntly significant prplxity rductions ovr th stat-of-thart RNNLM modls. Th contributions of this papr ar as follows: 1. W propos a framwork for ncoding structurd and unstructurd sid information, and its incorporation into a RNNLM. 2. W introduc a nw corpus, th RIE corpus, basd on th Europarl wb archiv, with rich annotations of svral typs of mta-data. 3. W provid mpirical analysis showing consistnt improvmnts from using sid information across two datasts in two languags.

2 2 Problm Formulation & Modl W first rviw RNNLM architctur (Mikolov t al., 2011) bfor dscribing our xtnsion in RNNLM Architctur Th standard RNNLM consists of 3 main layrs: an input layr whr ach input word has its mbdding via on-hot vctor coding; a hiddn layr consisting of rcurrnt units whr a stat is conditiond rcursivly on past stats; and an output layr whr a targt word will b prdictd. RNNLM has an advantag ovr convntional n-gram languag modl in modlling long distanc dpndncis ffctivly. In gnral, an RNN oprats from lft-to-right ovr th input word squnc; i.., h t = RU (x t, h t 1 ) = f (W (hh) h t 1 + W (ih) x t + b (h)) ( x t+1 softmax W (ho) h t + b (o)) ; whr f(.) is a non-linar function,.g., tanh, applid lmnt-wis to its vctor input; h t is th currnt RNN hiddn stat at tim-stp t; and matrics W and vctors b ar modl paramtrs. Th modl is traind using gradint-basd mthods to optimis a (rgularisd) training objctiv,.g. th liklihood function. In principl, a rcurrnt unit (RU) can b mployd using diffrnt variants of rcurrnt structurs such as: Long Short Trm Mmory (LSTM) (Hochritr and Schmidhubr, 1997), Gatd Rcurrnt Unit (GRU) (Cho t al., 2014), or rcntly dpr structurs,.g. Dpth Gatd Long Short Trm Mmory (DGLSTM) a stack of LSTMs with xtra connctions btwn mmory clls in dp layrs (Yao t al., 2015). It can b rgardd as bing a gnralisation of LSTM rcurrnc to both tim and dpth. Such dp rcurrnt structur may captur long distanc pattrns at thir most gnral. Empirically, w found that RNNLM with DGLSTM structur appars to b bst prformr across our datasts, and thrfor is usd prdominantly in our xprimnts. 2.2 Incorporating Sid Information Nowadays, many corpora ar archivd with sid information or contxtual mta-data. In this work, w a) b) h t-1 x t+1 h t x t h t-1 x t+1 Figur 1: Intgration mthods for auxiliary information, : a) as input to th RNN, or b) as part of th output softmax layr. argu that such information can b usful for languag modlling (and prsumably othr NLP tasks). By providing this auxiliary information dirctly to th RNNLM, w stand to boost languag modlling prformanc. Th first qustion in using sid information is how to ncod ths unstructurd inputs, y, into a vctor rprsntation, dnotd. W discuss svral mthods for ncoding th auxiliary vctor: BOW additiv bag of words, = t y t, and avrag th avrag mbdding vctor, = 1 t T y t, both inspird by (Hrmann and Blunsom, 2014a); bigram convolution with sum-pooling, = t tanh (y t 1 + y t ) (Hrmann and Blunsom, 2014b); and RNN a rcurrnt nural ntwork ovr th word squnc (Sutskvr t al., 2014), using th final hiddn stat(s) as. From th abov mthods, w found that BOW workd consistntly wll, outprforming th othr approachs, and morovr lad to a simplr modl with fastr training. For this rason w rport only rsults for th BOW ncoding. Not that whn using multipl auxiliary inputs, w us a wightd combination, = i W (ai) (i). Th nxt stp is th intgration of into th RNNLM. W considr two intgration mthods: as input to th hiddn stat (dnotd input), and connctd to th output softmax layr (output), as shown in Figur 1 a and b, rspctivly. In both cass, w compar xprimntally th following intgration stratgis: add adding th vctors togthr,.g., using x t + as th input to th RNN, such that h t x t

3 h t = RU (x t +, h t 1 ); stack concatnating th vctors,.g., using [ x t ] for gnrating th RNN hiddn stat, such that h t = RU ([ xt ], h t 1 ); and mlp fding both vctors into an xtra prcptron with singl hiddn layr, using a tanh nonlinarity and projcting th output to th rquird dimnsionality; i.., h t = tanh (W ) (hh ) h t + W (h) + b (h ) ( x t+1 softmax W (ho) h t + b (o)). Not that add rquirs th vctors to b th sam dimnsionality, whil th othr two mthods do not. Th stack mthod can b quit costly, givn that it incrass th siz of svral matrics, ithr in th rcurrnt unit (for input) or th output mapping for word gnration. This is a problm in th lattr cas: givn th larg siz of th vocabulary, th matrix W (ho) is alrady vry larg and making it largr (doubling th siz, to bcom W (h o) ) has a sizabl ffct on training tim (and prsumably also propnsity to ovr-fit). Th output+stack mthod dos howvr hav a complling intrprtation as a jointly traind product modl btwn a RNNLM and a unigram modl conditiond on th sid information, whr both modls ar formulatd as softmax classifirs. Considrd as a product modl (Hinton, 2002; Pascanu t al., 2013), th two componnts can concntrat on diffrnt aspcts of th problm whr th othr modl is not confidnt, and allowd ach modl th ability to vto crtain outputs, by assigning thm a low probability. 3 Exprimnts Datasts. W conductd our xprimnts on two datasts with diffrnt gnrs in two languags. As th first datast, w us th IWSLT2014 MT track on TED Talks 1 du to its slf-containd rich auxiliary information, including: titl, dscription, kywords, and author rlatd information. W chos th English-Frnch pair for our xprimnts. Th statistics of th training st is shown in Tabl 1. W 1 (IWSLT 14 MT Track) tokns (M) typs (K) docs snts (K) TED-n TED-fr RIE-n RIE-fr Tabl 1: Statistics of th training sts, showing in ach cll th numbr of word tokns, typs, documnts (talks or plnaris), and sntncs. usd dv2010 (7 talks/817 sntncs) for arly stopping of training nural ntwork modls. For valuation, w usd diffrnt tsting sts ovr yars, including tst2010 (10/1587), tst2011 (7/768), tst2012 (10/1083). As th scond datast, w crawld th ntir Europan Parliamnt 2 wbsit, focusing on plnary sssions. Such sssions contain usful structural information, namly multilingual txts dividd into spakr sssions and topics. W bliv that thos txts ar intrsting and challnging for languag modlling tasks. Our datast contains 724 plnary sssions ovr 12.5 yars until Jun 2011 with multilingual txts in 22 languags 3. W rfr to this datast by RIE 4 (Rich Information Europarl). W randomly slct 200/5/30 plnary sssions as th training/dvlopmnt/tst sts, rspctivly. Furthrmor, th sizs of our working datasts ar an ordr of magnitud largr than th standard Pnn Trbank st which is oftn usd for valuating nural languag modls. St-up and Baslins. W hav usd cnn 5 to implmnt our modls. W us th sam configurations for all nural modls: 512 input mbdding and hiddn layr dimnsions, 2 hiddn layrs, and vocabulary sizs as givn in Tabl 1. W usd th sam vocabulary for th auxiliary and modlld txt. W traind a convntional 5 gram languag modl using modifid Knsr-Ny smoothing, with th KnLM toolkit (Hafild, 2011). W usd th Wilcoxon signd-rank tst (Wilcoxon, 1945) to masur th statistical significanc (p < 0.05) on diffrncs btwn sntnc-lvl prplxity scors of improvd modls compard to th bst bas W ignord th priod from Jun 2011 onwards, as from this dat th EU stoppd crating manual human translations. 4 This datast will b rlasd upon publication. 5

4 Mthod tst2010 tst2011 tst gram LM RNNLM LSTM DGLSTM input+add+k input+mlp+k input+stack+k output+mlp+k output+mlp+t output+mlp+d output+mlp+k+t output+mlp+k+d output+mlp+t+d output+mlp+k+t+d Tabl 2: Prplxity scors basd on th English part of TED talks datast in IWSLT14 MT. +k, +t, +d: with kywords, titl, and dscription as auxiliary sid information rspctivly. bold: Statistically significant bttr than th bst baslin. lin. Throughout our xprimnts, punctuation, stop words and sntnc markrs ( s, /s, unk ) ar filtrd out in all auxiliary inputs. W obsrvd that this filtring was rquird for BOW to work rasonably wll. For ach modl, th bst prplxity scor on dvlopmnt st is usd for arly stopping of training modls, which was obtaind aftr 2-5 pochs on both datasts. Rsults & Analysis. Th prplxity rsults on TED Talks datast ar prsntd in Tabl 2 and 3. RNNLM variants consistntly achiv substantially bttr prplxitis compard to th convntional 5 gram languag modl baslin. 6 Of th basic RNNLM modls (middl), th DGLSTM works consistntly bttr than both th standard RNN and th LSTM. This may b du to bttr intractions of mmory clls in hiddn layrs. Sinc th DGLSTM outprformd othrs 7, w usd it for all subsqunt xprimnts. For TED Talks datast, thr ar thr kinds of sid information, including kywords, titl, dscription. W attmptd to injct thos into diffrnt RNNLM layrs, rsulting in modl variants as shown in Tabl 2. First, w chos kywords (+k) information as an anchor to figur out 6 For fair comparison, whn computing th prplxity with th 5-gram LM, w xclud all tst words markd as unk (i.., with low counts or OOVs) from considration. 7 This concurs with th finding in (Yao t al., 2015), who showd that DGLSTM producd th stat-of-th-art rsults ovr Pnn Trbank datast. Mthod tst2010 tst2011 tst gram LM LSTM DGLSTM output+mlp+t output+mlp+d output+mlp+t+d output+mlp+k output+mlp+d+k Tabl 3: Prplxity scors basd on th Frnch part of TED talks datast in IWSLT14 MT. Not that +k mans with kywords in English. which incorporation mthod works wll. Comparing input+add+k, input+mlp+k and input+stack+k, th largst dcras is obtaind by output+mlp+k consistntly across all tst sts (and dvlopmnt sts, not shown hr). W furthr valuatd th addition of othr sid information (.g., dscription (+d), titl (+t)), finding that +d has similar ffct as +k whras +t has a mixd ffct, bing dtrimntal for on tst st (tst2011). W suspct that it is du to oftn-tims short sntncs of titls in that tst, aftr our filtring stp, lading to a shortag of usful information fd into nural ntwork larning. Intrstingly, th bst prformanc is obtaind whn incorporating both +k and +d, showing that thr is complmntary information in th two auxiliary inputs. Furthr, w also achivd th similar rsults in th countrpart of English part (in Frnch) using output+mlp with both +t and +d as shown in Tabl 3. In Frnch data, no kywords information is availabl. For this rason, w run additional xprimnts by injcting English kywords as sid information into nural modls of Frnch. Intrstingly, w found that kywords sid information in English ffctivly improvs th modlling of Frnch txts as shown in Tabl 3, srving as a nw form of cross-lingual languag modlling. W furthr achivd similar rsults by incorporating th topic hadlin in th RIE datast. Th consistntly-improvd rsults (in Tabl 4) dmonstrat th robustnss of th output+mlp approach. 4 Conclusion W hav proposd an ffctiv approach to boost th prformanc of RNNLM using auxiliary sid information (.g. kywords, titl, dscription, topic had-

5 Mthod tst (n) tst (fr) 5-gram LM LSTM DGLSTM output+mlp+h Tabl 4: Prplxity scors basd on th sampld RIE datast. +h: topic hadlin. lin) of a txtual uttranc. W providd an mpirical analysis of various ways of injcting such information into a distributd rprsntation, which is thn incorporatd into ithr th input, hiddn, or output layr of RNNLM architctur. Our xprimntal rsults rval consistnt improvmnts ar achivd ovr strong baslins for diffrnt datasts and gnrs in two languags. Our futur work will invstigat th modl prformanc on a closly-rlatd task, i.., nural machin translation (Sutskvr t al., 2014; Bahdanau t al., 2015). Furthrmor, w will xplor larning mthods to combin uttrancs with and without th auxiliary sid information. Acknowldgmnts Cong Duy Vu Hoang was supportd by full scholarships of th Univrsity of Mlbourn, Australia. Dr Trvor Cohn was supportd by th ARC (Futur Fllowship). Rfrncs D. Bahdanau, K. Cho, and Y. Bngio Nural Machin Translation by Jointly Larning to Align and Translat. In Procdings of Intrnational Confrnc on Larning Rprsntations (ICLR 2015), Sptmbr. Yoshua Bngio, Réjan Ducharm, Pascal Vincnt, and Christian Janvin A Nural Probabilistic Languag Modl. Th Journal of Machin Larning Rsarch, 3: Kyunghyun Cho, Bart van Mrrinbor, Dzmitry Bahdanau, and Yoshua Bngio On th Proprtis of Nural Machin Translation: Encodr Dcodr Approachs. In Procdings of SSST-8, Eighth Workshop on Syntax, Smantics and Structur in Statistical Translation, pags , Doha, Qatar, Octobr. Association for Computational Linguistics. Knnth Hafild KnLM: Fastr and Smallr Languag Modl Quris. In Procdings of th EMNLP 2011 Sixth Workshop on Statistical Machin Translation, pags , Edinburgh, Scotland, Unitd Kingdom, July. K. M. Hrmann and P. Blunsom. 2014a. Multilingual Distributd Rprsntations without Word Alignmnt. In Procdings of Intrnational Confrnc on Larning Rprsntations (ICLR 2014), Dcmbr. Karl Moritz Hrmann and Phil Blunsom. 2014b. Multilingual Modls for Compositional Distributd Smantics. In Procdings of th 52nd Annual Mting of th Association for Computational Linguistics (Volum 1: Long Paprs), pags 58 68, Baltimor, Maryland, Jun. Association for Computational Linguistics. Goffry E Hinton Training Products of Exprts by Minimizing Contrastiv Divrgnc. Nural computation, 14(8): Spp Hochritr and Jurgn Schmidhubr Long Short-Trm Mmory. Nural Comput., 9(8): , Novmbr. Nal Kalchbrnnr and Phil Blunsom Rcurrnt Continuous Translation Modls. In Procdings of Empirical Mthods in Natural Languag Procssing (EMNLP 2013). Ryan Kiros, Ruslan Salakhutdinov, and Rich Zml Multimodal Nural Languag Modls. In Procdings of th 31st Intrnational Confrnc on Machin Larning (ICML-14), pags T. Mikolov, S. Kombrink, A. Doras, and J. H. Burgt, L.and Crnocky RNNLM - Rcurrnt Nural Ntwork Languag Modling Toolkit. In 2011 IEEE Workshop on Automatic Spch Rcognition & Undrstanding (ASRU). IEEE Automatic Spch Rcognition and Undrstanding Workshop, Dcmbr. Andriy Mnih and Goffry Hinton Thr Nw Graphical Modls for Statistical Languag Modlling. In Procdings of th 24th Intrnational Confrnc on Machin Larning, pags R. Pascanu, C. Gulchr, K. Cho, and Y. Bngio How to Construct Dp Rcurrnt Nural Ntworks. ArXiv -prints, Dcmbr. Ilya Sutskvr, Oriol Vinyals, and Quoc V L Squnc to Squnc Larning with Nural Ntworks. In Advancs in Nural Information Procssing Systms (NIPS 2014), pags Oriol Vinyals, Alxandr Toshv, Samy Bngio, and Dumitru Erhan Show and Tll: A Nural Imag Caption Gnrator. In Th IEEE Confrnc on Computr Vision and Pattrn Rcognition (CVPR), Jun. Frank Wilcoxon Individual Comparisons by Ranking Mthods. Biomtrics Bulltin, 1 (6):80 83, Dc. K. Yao, T. Cohn, K. Vylomova, K. Duh, and C. Dyr Dpth-Gatd LSTM. ArXiv -prints, August.

Learning Spherical Convolution for Fast Features from 360 Imagery

Learning Spherical Convolution for Fast Features from 360 Imagery Larning Sphrical Convolution for Fast Faturs from 36 Imagry Anonymous Author(s) 3 4 5 6 7 8 9 3 4 5 6 7 8 9 3 4 5 6 7 8 9 3 3 3 33 34 35 In this fil w provid additional dtails to supplmnt th main papr