Tensorial Recurrent Neural Networks for Longitudinal Data Analysis

Transcription

1 1 Tensorial Recurren Neural Neworks for Longiudinal Daa Analysis Mingyuan Bai Boyan Zhang and Junbin Gao arxiv: v1 [cs.lg] 1 Aug 2017 Absrac Tradiional Recurren Neural Neworks assume vecorized daa as inpus. However many daa from modern science and echnology come in cerain srucures such as ensorial ime series daa. To apply he recurren neural neworks for his ype of daa a vecorisaion process is necessary while such a vecorisaion leads o he loss of he precise informaion of he spaial or longiudinal dimensions. In addiion such a vecorized daa is no an opimum soluion for learning he represenaion for he longiudinal daa. In his paper we propose a new varian of ensorial neural neworks which direcly ake ensorial ime series daa as inpus. We call his new varian as Tensorial Recurren Neural Nework TRNN. The proposed TRNN is based on ensor Tucker decomposiion. I. INTRODUCTION In recen years he ineress in ime series wih sequenial effecs among he daa have been consanly growing in boh academic field and indusry. These ineress are from he developmen of echnology and social science including bu no limied o mulimedia social nework and economic and poliical nework especially inernaional relaionship sudy. Time series daa acquired from many discipline is no only in large volume in erms of ime bu also in more ever complicaed srucures such as in muli- and high-dimension. The rise of massive muli-dimensional daa has led o new demands for Machine Learning ML sysems o learn complex models wih millions o billions of parameers for new ypes of daa srucures ha promise adequae capaciy o diges massive daases and offer powerful predicive analyics hereupon. As ensorial daa come wih a special spaial srucure i is highly desired o mainain his srucure informaion in learning process. We have seen he mos recen developmen in deep learning archiecure for muli-dimensional ensor daa exending convenional vecor neural neworks o srucured daa such as he marix neural nework [1] [2] wo independen works on ensorial neural neworks [3] [4] graph daa [5] [6] and even neural neworks for manifold-valued daa [7]. The recurren neural neworks RNN as a commonly applied ool in longiudinal daa analysis have consanly been invesigaed in he las couple of decades wih many successful applicaions such as language processing [8] speech recogniion [9] and human acion recogniion [10] [11] ec. There are many differen archiecures for RNN such as he basic Mingyuan Bai and Junbin Gao are wih wih he Discipline of Business Analyics The Universiy of Sydney Business School The Universiy of Sydney NSW 2006 Ausralia. mbai8854@uni.sydney.edu.au junbin.gao@sydney.edu.au Boyang Zhang is School of Informaion Technologies The Universiy of Sydney NSW 2006 Ausralia. bzha8220@uni.sydney.edu.au recurren nework Elman neworks or Jordan neworks long shor-erm memory LSTM and gaed recurren uni GRU ec. The mos recen developmens of recurren neural neworks are generally focused on he LSTM model. For example he LSTM has been combined wih he convoluional neural neworks CNN for sequence represenaion learning [12]. However he radiional LSTM model can only deal wih vecorised daa which leads o he loss of some spaial informaion for mulidimensional ime series. Our inenion in his paper is o propose a fully ensorial conneced neural neworks for ensorial longiudinal daa. A recen paper has also considered ensorial srucure in he classical recurren neural neworks however he main purpose was o reduce he number of neworks parameers when vecorial daa are in very high-dimension [13]. The res of his paper is organized as follows. Secion II inroduces basic recurren neural nework and wo ypes of ensorial RNN i.e. ensorial LSTM LSTM and ensorial GRU GRU. In Secion III we derive he backpropagaion algorihms for he proposed LSTM and rgru. In Secion IV experimenal resuls are presened o evaluae he performance of he proposed models. Finally conclusions and fuure works are summarized in Secion V. II. TENSORIAL RECURRENT NEURAL NETWORKS The simple building block for RNN is expressed in he following forward mapping Elman model [14] h σ h W hx x + W hh h 1 + b h y σ y W hy h + b y or in Jordan model [15] h σ h W hx x + W hh y 1 + b h y σ y W hy h + b y Jordan model furher passes on he oupu informaion a ime o he nex ime + 1 as inpus. In his noe we will focus on Elman model 1. However we will consider he seing for ensorial longiudinal daa in general denoed by D {X Y } T 1. where each independen daa X is a ensor of D-ways he ensor dimension and he response daa Y could be a scalar a vecor or more general a ensor of he same dimension as X. The wo mos popular recurren neural nework archiecures are he Long Shor-Term Memory Unis LSTMs and he 1 2

2 2 Gaed Recurren Unis GRUs which will be exended for ensors daa. A. Tensorial LSTM LSTM LSTMs were inroduced in [16] and furher modernized by many people e.g. [17]. The applicaion has demonsraes ha LSTMs work remendously well on a large variey of problems and are now widely used. Based on he classic LSTMs we propose he following ensorial LSTM F σ g H 1 1 W f1 2 W f2 D W fd + X 1 U f1 2 U f2 D U fd + B f 3 I σ g H 1 1 W i1 2 W i2 D W id + X 1 U i1 2 U i2 D U id + B i 4 O σ g H 1 1 W o1 2 W o2 D W od + X 1 U o1 2 U o2 D U od + B o 5 Ĉ σ c H 1 1 W c1 2 W c2 D W cd + X 1 U c1 2 U c2 D U cd + B c 6 C F C 1 + I Ĉ 7 H O σ h C 8 where he operaor denoes he Hadamard produc i.e. he enry-wise produc and W d as well as W d are marices in relevan order applied on hidden ensorial variables and inpu ensorial variables in erms of ensorial mode produc [18] and all B are ensorial biases. Noe O is no he acual oupu of he LSTM. In fac O will be joinly regulaed by boh I and C o make he poenial oupu from he hidden variable H as in 8. Depending on he ype of response daa Y we may apply an exra layer of neural nework on he op of H o conver he ensor hidden H o he shape/srucure of Y. For he sake of noaion simpliciy we assume he ransformed oupu is denoed by Ô. B. Tensorial GRU GRU The Gaed Recurren Uni GRU was inroduced in [19] wih a slighly more dramaic variaion on he LSTM. Similar o GRU in our proposed ensorial GRU he forge F and he inpu I gaes are o be combined ino a single updae gae. GRU is simpler han he aforemenioned LSTM models. R σ g H 1 1 W r1 2 W r2 D W rd + X 1 U r1 2 U r2 D U rd + B r 9 Z σ g H 1 1 W z1 2 W z2 D W zd + X 1 U z1 2 U z2 D U zd + B z 10 R R H 1 11 Ĥ σ h R 1 W h1 2 W h2 D W hd + X 1 U h1 2 U h2 D U hd + B h 12 H Z H Z Ĥ. 13 Similar o LSTM we will add an addiional ransform mapping he hidden variables H o mach he response variable Y. A. Loss Funcion III. RECURRENT BP ALGORITHM According o he way how daa is presened we propose hree ypes of loss funcions. Loss funcion for single daa series. The raining daa is presened as a single ime series and we will apply he LTSM or GRU on he series and collec heir oupus a each ime poin. The simple loss a each ime is defined as l s ly Ô which is he building block for all he oher overall loss funcion. Please noe ha Ô is calculaed hrough he recurren neworks from he inpu {X 1 X 2... X }. l is a loss funcion such as he usual squared loss funcion for regression or he cross-enropy loss for classificaion. The overall loss is defined as l s l s ly Ô Loss funcion for muliple daa series wih same lengh/duraion. Mos of ime we will use a recurren nework srucure wih a cerain duraion. In his case we will assume ha he raining daa consis of a number of raining series D {X j1... X jt Y j } N The loss for he j-h case is only calculaed a ime T as l T j ly j ÔjT where ÔjT is he las oupu of LSTM or GRU from he inpu series X j1... X jt. Hence he overall loss is l T l T j ly j ÔjT. 15 Similar o he simple series case if he response is a series Y j1... Y jt hen he loss can be revised as l m l m j ly j Ôj Loss funcion for Panel Daa In many applicaion case paricularly for panel daa he duraion or period T for each series X j1... X jt may be differen hus he recurren nework will run hrough differen loops. Suppose he daa are D {X j1... X jtj Y j } N hen he loss can be defined as l T p l T p j ly j ÔjT j. 17 or 1 T j T j l mp l m j ly j Ôj Loss funcion 14 is a special case of loss funcion 16 when N 1. For he BP algorihm in he nex subsecion we will focus on losses 15 and 16. Similarly losses 17 and 18 can be processed in he similar way as for 15 and 16 respecively.

3 3 B. BP Algorihm The major difference beween he proposed ensorial RNN RNN and he vecorial RNN is ha he vecorial linear mapping has been replaced wih he ensor muliple linear mapping i.e. Tucker muliplicaion [18]. Le us denoe Tucker mapping by M α H 1 1 W α1 2 W α2 D W αd + X 1 U α1 2 U α2 D U αd + B α where α can be eiher f i o c r z or h. When α h we replace H 1 wih R in 12. Firs we inroduce he resuls from [20] wihou proof. Lemma 1 Denoe by M and H he oal sizes of ensors M α and H 1 respecively hen M α H 1 M H W αd W α1 19 where he maricized form has been applied and means he Kronecker produc of marices. And M α d M α d [W αd W αd 1 W αd+1 W α1 H T 1d ] I h d h d 20 [U αd U αd 1 U αd+1 U α1 Xd T ] I x d x d 21 M α M M I M M 22 where he subscrip d means he d-mode maricizaion of a ensor and boh h d and x d are he dimension of mode d of ensors H 1 and X respecively. Firs le us derive he BP algorihm for each LSTM sep. The compuaion flow defined by equaions 3-8 can be shown in Fig. 1. Along he ime C will be forwarded o he nex LSTM sep while he hidden ensorial H will be carried ono he nex sep and also oupu an exra layer o mach he response Y a. Hence in he BP algorihm on LSTM uni here could be wo pieces of informaion one from he nex LSTM uni denoed by H and he oher from he oupu loss l denoed by o H. Thus he combined derivaive informaion o be furher backpropagaed is We know ha + o. 23 h H H H o H 0 if here is no response Y a ime. According o he flows from C 1 F Ĉ I and O respecively o boh C and H and he chain rule i is easy o read from he diagram + σ C 1 C hc F 24 h H + σ F C hc C 1 25 h H + σ Ĉ C hc I 26 h H + σ I C hc h H Ĉ 27 σ h C 28 O h H Le us inroduce wo operaors: for any ensor M denoe he vecorizaion as m VecM while is inverse operaor ivecm M. Then we will have H 1 ivec Vec F gm f T M f H 1 +ivec Vec M σ gm c T c Ĉ H 1 +ivec Vec M I gm i T i H 1 +ivec Vec M O gm o T o H Similarly we have a ime ivec Vec A gm α T Mα d 30 ivec Vec A gm α T Mα d 31 [ ] sum A gm α cons ensor 32 where A α F f or Ĉ c or I i or O o. Finally he overall derivaives for he parameers are given by Le us consider loss funcions defined in 15 and 16. Firs we noe ha 0. In he case of 16 o H is calculaed according o he given layer and cos funcion for he response Y for T. In he case of 15 we only have he informaion o H when T oherwise 0. Hence 23 will be calculaed accordingly. We summarize he derivaive BP algorihm for LSTM in Algorihm 1

4 4 Fig. 1. LSTM Compuaion Flow. Algorihm 1 The Derivaive BP Algorihm for LSTM for a single raining daa Inpu: The raining case {X j1 X j2 X jt Y jt } or {X j1 X j2 X jt Y j1 Y j2 Y jt } Oupu: D. 1: Se and for α f c i o and d C T 0 0 and calculae o according o he oupu layer for Y jt a ime T 2: for T o 1 do 3: Use 23 o calculae h H while for < T o H 0 if here is no response a oherwise i is calculaed according o he oupu loss a 4: Use o calculae F Ĉ I and O 5: Use o calculae Mα d 6: Use o calculae and Mα d and 7: Use 19 o calculae Mα H 1 for α f c i o 8: Use 24 o updae C 1 and 29 for H 1 9: end for 10: Use o summarize and oupus. The BP algorihm for GRU can be derived in a similar way by looking a he compuaion flow defined by equaions 9-13 as shown in Fig. 2 Similarly he backpropagaed informaion from ime + 1 is as in 23. Hence h Ĥ 1 Z 36 Ĥ h H H 1 Z Ĥ 37 h H R H 1 ivec Vec Ĥ hm h T M h R for 38 ivec Vec M H 1 R gm r T r R + ivec Vec M Z gm z T z Z + Z 39 h H The derivaives wih respec o all hree ses of parameers can be obained from wih A α R r Z z and Ĥ h. The derivaive BP algorihm for GRU is summarized in Algorihm 2 Algorihm 2 The Derivaive BP Algorihm for GRU for a single raining daa Inpu: The raining case {X j1 X j2 X jt Y jt } or {X j1 X j2 X jt Y j1 Y j2 Y jt } Oupu: and D. 1: Se 0 and calculae o layer for Y jt a ime T 2: for T o 1 do 3: Use 23 o calculae for α r z h and d according o he oupu h H while for < T o H 0 if here is no response a oherwise i is calculaed according o he oupu loss a 4: Use 19 o calculae Mα H 1 for α r z and Mh 5: Use o calculae Ĥ 6: Use o calculae Mα d 7: Use o calculae 8: Use 39 o updae H 1 9: end for 10: Use o summarize oupus. A. Daa Descripion IV. EXPERIMENTS Z and R R and Mα d and and To assess he performance of RNN on he real world daa we conduc an empirical sudy. In his sudy he daa is for

5 5 Fig. 2. GRU Compuaion Flow. colleced from Inegraed Crisis Warning Sysem ICEWS 1 which is he same weekly daase applied in he sudy of MLTR [21] for he relaionship beween 25 counries in four ypes of acions: maerial cooperaion maerial conflic verbal cooperaion and verbal conflic from 2004 o mid Thus a any paricular ime poin he daa is a 3D ensor of dimensions Tha is each inpu X R To explore differen ypes of paerns ofen seen in relaional daa and social neworks we organise explanaory ensors X in he following differen ways. As done in [21] we consruc he arge ensor Y a ime as he lagged X 1. In oal we consruc an overall daase {X Y } of size 543 in which all X and Y are 3D ensors. Furher we ake T 7 as he period of ime series secions and use 90% daa for raining as defined in he following wo cases: Case I: We use LSTM in erms of a many-o-one recurren model wih he raining daase as follows D 1 {X j X j+1... X j+6 Y j+6 } 488 and he remaining 55 daa will be used for esing. Case II: We use LSTM in erms of many-o-many recurren model wih he raining daase as follows D 2 {X j X j+1... X j+6 Y j Y j+1... Y j+6 } 488 and he remaining 55 daa will be used for esing. B. Experimen Seing and Resuls Our inenion in hese wo experimens is o quickly demonsrae how LSTM works wih he mos possible convenience. GRU can give similar resuls. In Case I we se he size of he hidden nodes o be doubled he inpu size In Case II we use half he inpu size for he hidden nodes i.e We also use a regulariser for all he marix coefficiens W and U i.e. 1 hp:// adding he following o he loss funcion o have a regularised objecive funcion λ W 2 F + U 2 F all W all U where F is he Frobenius norm for marices and λ > 0 is a parameer o rade-off beween he loss and he regulariser. In our experimen we empirically se λ This parameer should be opimised by using a se of valid daase. Fig. 3 shows he convergence rends for boh cases in raining process wih 1000 epoches. The es errors are for Case I and for Case II. V. CONCLUSION In his paper we inroduced he new recurren neural neworks for high-order ensor daa. Two special recurren srucures i.e. LSTM and GRU are proposed wih deailed BP algorihm derivaion. Two simple experimens have demonsraed he performance of he new recurren neural neworks. More experimens shall be conduced o demonsrae is efficiency and accuracy agains he exising neural neworks. We also inend o explore more applicaions such as for video daa analysis. REFERENCES [1] J. Gao Y. Guo and Z. Wang Marix neural neworks in Proceedings of he14h Inernaional Symposium on Neural Neworks ISNN vol. Acceped Sapporo Japan 2017 pp [2] C. Ionescu O. Vanzos and C. Sminchisescu Marix backpropagaion for deep neworks wih srucured layers in Proceedings of he IEEE Inernaional Conference on Compuer Vision ICCV 2015 pp [3] M. Bai B. Zhang and J. Gao Tensorial neural neworks and is applicaion in longiudinal nework daa analysis in submied o he 24h Inernaional Conference On Neural Informaion Processing ICONIP [4] J.-T. Chien and Y.-T. Bao Tensor-facorized neural neworks IEEE Transacions on Neural Neworks and Learning Sysems vol. PP [Online]. Available: hp://ieeexplore.ieee.org/documen/ / [5] Y. Li D. Tarlow M. Brockschmid and R. Zemel Gaed graph sequence neural neworks in Proceedings of Inernaional Conference on Learning Represenaions ICLR 2016.

6 Case I 3.4 Case II Objecive Objecive Epoches a Epoches b Fig. 3. Convergence Trends for Two Cases. [6] Y. Seo M. Defferrard P. Vandergheyns and X. Bresson Srucured sequence modeling wih graph convoluional recurren neworks in Proceedings of Inernaional Conference on Learning Represenaion [7] Z. Huang and L. V. Gool A Riemannian nework for spd marix learning in Proceedings of he Thiry-Firs AAAI Conference on Arificial Inelligence AAAI [8] W. D. Mulder S. Behard and M.-F. Moens A survey on he applicaion of recurren neural neworks o saisical language modeling Compuer Speech & Language vol. 30 no. 1 pp [9] A. Graves A. rahman Mohamed and G. Hinon Speech recogniion wih deep recurren neural neworks in Proceedings of IEEE Inernaional Conference on Acousics Speech and Signal Processing ICASSP [10] J. Liu A. Shahroudy D. Xu and G. Wang Spaio-emporal LSTM wih rus gaes for 3D human acion recogniion in Proceedings of he European Conference on Compuer Vision ECCV 2016 pp [11] P. Zhang C. Lan J. Xing W. Zeng J. Xue and N. Zheng View adapive recurren neural neworks for high performance human acion recogniion from skeleon daa arxiv: vol [12] Z. Gany Y. Puy R. Henaoy C. Liy X. Hez and L. Carin Learning generic senence represenaions using convoluional neural neworks arxiv: vol [13] C. Jose M. Cissé and F. Fleure Kronecker recurren unis arxiv: [14] J. L. Elman Finding srucure in ime Cogniive Science vol. 14 no. 2 pp [15] M. I. Jordan Serial Order: A Parallel Disribued Processing Approach ser. Advances in Psychology: Neural-Nework Models of Cogniion. Elsevier 1997 vol. 121 ch. 25 pp [16] S. Hochreier and J. Schmidhuber Long shor-erm memory.. Neural Compuaion vol. 9 no. 8 pp [17] F. A. Gers J. Schmidhuber and F. Cummins Learning o forge: Coninual predicion wih LSTM Neural Compuaion vol. 12 no. 10 pp [18] T. G. Kolda and B. W. Bader Tensor decomposiions and applicaions SIAM Review vol. 51 no. 3 pp [19] K. Cho B. van Merriënboer Ç. Gülçehre D. Bahdanau F. Bougares H. Schwenk and Y. Bengio Learning phrase represenaions using rnn encoder decoder for saisical machine ranslaion in Proceedings of he 2014 Conference on Empirical Mehods in Naural Language Processing EMNLP. Doha Qaar: Associaion for Compuaional Linguisics Oc pp [Online]. Available: hp:// [20] T. G. Kolda Mulilinear Operaors for Higher-Order Decomposiions Sandia Naional Laboraories Technical repor [21] P. D. Hoff Mulilinear Tensor Regression for Longiudinal Relaional Daa Ann. Appl. Sa. vol. 9 no. 3 pp