Learning in the Deep-Structured Conditional Random Fields

Transcription

1 Learig i he Deep-Srucured Codiioal Radom Fields Dog Yu, Li Deg Shizhe Wag Microsof Research Uiversiy of Califoria Oe Microsof Way 405 Hilgard Aveue Redmod, WA 9805 Los Ageles, CA {dogyu, deg}@microsof.com szwag@ee.ucla.edu Absrac We have proposed he deep-srucured codiioal radom fields (CRFs) for sequeial labelig ad classificaio recely. The core of his model is is deep srucure ad is discrimiaive aure. This paper oulies he learig sraegies ad algorihms we have developed for he deep-srucured CRFs, wih a focus o he ew sraegy ha combies he layer-wise usupervised pre-raiig usig eropy-based muli-objecive opimizaio ad he codiioal lielihood-based bac-propagaio fie uig, as ispired by he rece developme i learig deep belief ewors. Iroducio Codiioal radom fields (CRFs) are discrimiaive models ha direcly esimae he probabiliies of he sae sequece codiioed o he whole observaio sequece. This is i coras o he geeraive models such as he hidde Marov models (HMMs) ha describe he joi probabiliy of he observaio ad he saes. Give heir discrimiaive aure ad heir high flexibiliy i choosig feaures, CRFs have bee widely ad successfully used o solve sequeial labelig problems, oably hose i aural laguage processig [] [] ad speech processig [3]. y0 y y y y y y T T x :T Figure. The graphical represeaio of he liear-chai CRF The liear-chai CRF depiced i Figure is he mos popular CRF due o is simpliciy ad efficiecy. Give a T -frame observaio sequece x = x, x,, x T, he codiioal probabiliy of he sae sequece y = y, y,, y T (which may be augmeed wih a special sar (y 0 ) ad ed (y T+ ) sae) is formulaed as Shizhe Wag coribued o his wor while he was a ier a Microsof Research.

2 p y x; Λ = exp λ if i y, y, x,,i Z x; Λ where we have used f i y, y, x, o represe boh he observaio feaures f i y, x, ad he sae rasiio feaures f i y, y,. The pariio fucio () Z x; Λ = exp λ i f i y, y, x, y,i is used o ormalize he expoeial form so ha i becomes a valid probabiliy measure. The model parameers Λ = λ i are ypically opimized o maximize he L regularized codiioal sae sequece log-lielihood J Λ, X = log p y x ; Λ () σ (3) where σ is a parameer ha balaces he log-lielihood ad he regularizaio erm ad is ypically ued usig a developme se. The derivaives of J Λ, X over he model parameers λ i are give by J Λ, X = E f i y, x E f i y, x λ i σ = f i y, x λ i σ y p y x ; Λ f i y, x which ca be efficiely esimaed usig he forward-bacward (sum-produc) algorihm [] [4]. The model parameers i he CRFs ca hus be opimized usig algorihms such as geeralized ieraive scalig (GIS) [5], gradie ad cojugae gradie (e.g. L-BFGS) asce [6], ad RPROP [7]. Alhough grea performace has bee observed usig he sigle-layer CRFs, limiaios associaed wih heir shallow srucure are also oiceable. For example, he sigle-layer CRFs ypically require maual cosrucio of may differe feaures o achieve good performace ad require a large amou of raiig daa o obai he geeralizaio abiliy. They lac he abiliy o auomaically geerae robus discrimiaive ieral feaures from he raw feaures. As a example, we have show [8][9] ha whe coiuous feaures are used, beer performace ca be achieved by imposig cosrais o he disribuio of he feaures, which is equivale o expadig each coiuous feaure f i y, y, x, io L feaures f il y, y, x, = a l f i y, y, x, f i y, y, x,, (5) where a l. is a weigh fucio whose defiiio ca be foud i [8][0][]. However, he sigle-layer CRFs cao lear hese expaded feaures auomaically. Moivaed by he rece advaces i deep learig developed by he eural ewor commuiy [][3][4][5][6], we have recely proposed he deep-srucured CRFs for sequeial labelig ad classificaio ad observed promisig resuls o he ex labelig [] ad laguage ideificaio ass [9]. I he deep-srucured CRFs, muliple layers of simple CRFs are saced ogeher o achieve a much more powerful modelig ad discrimiaio abiliy. Usig muliple layers of CRFs o improve he modelig power is o ew. Several flavors of hierarchical CRFs have bee proposed i he lieraure [3][7][8]. Those models ypically aim a aclig he graulariy problem a differe represeaio layers ad use he lower layer CRFs as he buildig blocs for he higher layer CRFs. The deep-srucured CRF discussed i his paper disiguishes iself from he coveioal hierarchical models i ha i aims a learig discrimiaive iermediae represeaios from he raw feaures ad a combiig all sources of iformaio o obai a superior classificaio abiliy. The purpose of his paper is o summarize he learig sraegies ad algorihms we have, (4)

3 ... developed for he deep-srucured CRFs, wih a focus o he ew sraegy ha combies he eropy-based layer-wise usupervised pre-raiig ad he codiioal lielihood-based bac-propagaio fie uig. We firs describe he archiecure of he deep-srucured CRF i Secio. We he illusrae he layer-wise supervised learig sraegy, ad he sraegy ha combies he layer-wise usupervised pre-raiig ad he lielihood bac-propagaio fie uig i Secios 3 ad 4, respecively. We provide some experimeal resuls i Secio 5 ad summarize he paper i Secio 6. Archiecure of Deep-Srucured CRF The archiecure of he deep-srucured CRF discussed i his paper is depiced i Figure, where he fial layer is a liear-chai CRF ad he lower layers are zero-h-order CRFs ha do o use sae rasiio feaures. Usig zero-h-order isead of liear-chai CRFs i he lower layers ca sigificaly reduce he compuaioal cos while oly slighly degrades he classificaio performace. I he deep-srucured CRF, he observaio sequece a layer j cosiss of wo pars: he previous layer s observaio sequece x j ad he frame-level margial poserior probabiliies p y j x j from he precedig layer j. This is ispired by he adem srucure used i some auomaic speech recogiio sysems [9]. Noe ha he feaures cosruced o he observaios may use oly par of he ipu iformaio hough. N N N N y0 y y y N y N yt y N T Layer N N N N N p y x x x... : T y y y y y T Layer x x p y x : T y y y y y T Layer x x p y 0 : T Figure. The graphical represeaio of he deep-srucured CRF. I he deep-srucured CRF, he sae sequece iferece is carried ou layer-by-layer i a boom-up maer so ha he compuaioal complexiy is limied o a mos liear o he umber of layers used. The model parameer esimaio is more complicaed. A he fial layer he umber of saes ca be direcly deermied by he problem o be solved ad he parameers ca be leared i he supervised way. However, parameer learig ca be ricy for he iermediae layers, which serve as absrac ieral represeaios of he origial observaio ad may have compleely differe umber of saes ha he fial layer. I he followig secios, we will describe wo learig sraegies for he deep-srucured CRFs. I he layer-wise supervised learig, we resric he umber of saes a iermediae layers o be he same as ha i he fial layer. so ha he same label used o rai he fial layer 3

4 ca be used o rai all he iermediae layers. I he secod sraegy of eropy-based layer-wise usupervised pre-raiig followed by codiioal lielihood-based bac propagaio learig, we allow for a arbirary umber of saes i he iermediae layers. This learig scheme firs lears each iermediae layer separaely i a usupervised maer, ad he fie-ues all he parameers joily. 3 Layer-wise Supervised Learig If we resric he umber of saes a iermediae layers o be he same as ha i he fial layer ad rea each sae a iermediae layers he same as ha i he fial layer, we ca rai he iermediae layers layer-by-layer usig he same label used o rai he fial layer. Noe ha he oupu of he deep-srucured CRF model is a sae sequece; so he parameers i he fial layer are opimized by maximizig he regularized codiioal log-lielihood (3) a he sae-sequece level. I coras o he highes layer, all remaiig layers are raied by maximizig he frame-level margial log-lielihood of J Λ, X =, log p y x ; Λ σ (6) sice his margial probabiliy is he oly addiioal iformaio passed io he higher layers. This crierio, however, is equivale o he sae-sequece level crierio J Λ, X whe he zero-h-order CRF is used i he iermediae layers sice J Λ, X = log p y x ; Λ σ = log exp λ,i if i y, y, x, Z x ; Λ = λ i f i y, x, log Z x ; Λ,i = λ i f i y, x, log Z x ; Λ, = J Λ, X. i σ σ σ J Λ, X ca be opimized i a complexiy of O TY, where T is he umber of frames ad Y is he umber of saes. Sice he oupu of each frame i he zero-h-order CRF is idepede of each oher, he process ca be furher speeded up usig parallel compuig echiques. Noe ha he observaio feaures a each layer ca be cosruced differely, ad possibly across differe frames ha he previous layer also. This allows for he grea flexibiliy of he higher layers o icorporae loger-spa feaures from lower-layer decodig resuls. Allowig for log-spa feaures ca be helpful for speech recogiio [0] [][][3] ass. We ow describe some desirable heoreical properies of his raiig sraegy. Theorem : The objecive fucio J Λ, X o he raiig se will o decrease as more layers are added i he deep-srucure CRF. Proof: Le's cosider he exesio from a N-layer deep-srucured CRF o a N+ layer deep-srucured CRF. The parameers for he firs N- layers are he same for boh sysems. For he N-layer sysem, he observaio feaures a he fial layer are cosruced o x N ad he correspodig parameer se is Λ N. For he N+-layer sysem, he observaio feaures are cosruced o he observaios ha are augmeed by p y N x N a each frame, where we use p o idicae ha he probabiliy is esimaed usig he N-h layer i he N+-layer sysem. The correspodig parameer se a he fial layer i he N+-layer sysem is Λ N+ Λ N. Sice (7) 4

5 max J Λ N Λ N, X max J Λ N + Λ N+, X (8) ad he opimizaio problem is covex a each layer, he learig algorihm which achieves he global opimum eabled by he covexiy ca always fid a parameer se i he N+-layer sysem ha gives a higher value of J Λ, X. I direcly follows ha Corollary : The deep-srucured CRF performs o worse ha he sigle-layer CRF o he raiig se. Noe ha he codiioal log-lielihood icrease i he raiig se ca be carried over o he es se wih a properly chose regularizaio erm. However, as he umber of iermediae layers coiues o grow, he gai will eveually saurae. 4 Layer-wise Usupervised Learig wih Fie Tuig The layer-wise supervised raiig paradigm described i Secio 3 wors oly whe he umber of saes i he iermediae layers is he same as ha i he fial layer so ha he same supervisio ca be used o rai each layer. This requireme, however, sigificaly resrics he poeial of he deep-srucured CRF for exracig powerful, opimizaio-drive ieral represeaios auomaically from he origial daa sequece. I his secio, we relax his cosrai ad allow for compleely differe ieral represeaios wih vasly differe umber of saes i he iermediae layers. This relaxaio requires a differe raiig algorihm wih differe objecive fucio(s) as a iermediae sep. A cocepually simple approach o rai he deep-srucured CRF wih arbirarily cofigured iermediae layers is o rai all he model parameers joily. However, i has bee show [3][4][5][6] ha whe he umber of layers icreases, joi raiig ca be very iefficie ad leads o poor local opimum. Aleraively, oe ca rai he iermediae layers oe by oe i a usupervised maer, for example, i a geeraive way by opimizig he associaio bewee he ipu ad he oupu for each iermediae layer. I his paper, we propose a layer-wise usupervised learig sraegy wih a discrimiaive flavor where we cas he iermediae layer learig problem io a muli-objecive programmig (MOP) oe. More specifically, we miimize he average frame-level codiioal eropy ad maximize he sae occupaio eropy a he same ime. Miimizig he average frame-level codiioal eropy forces he iermediae layers o be sharp idicaors of subclasses (or clusers) for each ipu vecor, while maximizig he occupaio eropy guaraees ha he ipu vecors be represeed disicly by differe iermediae saes. The raiig of his MOP problem is carried ou i a similar way o ha described i [9]. Specifically, we sar from maximizig he sae occupaio eropy. We he updae he parameers by aleraig bewee miimizig he frame-level codiioal eropy ad maximizig he average sae occupaio eropy. A each epoch, we opimize oe objecive by allowig he oher oe o become slighly worse wihi a limied rage. This rage is gradually igheed epoch by epoch. The model parameers are he fie ued usig he codiioal lielihood-based bac propagaio we will describe shorly. 4. M a xi mize he sae occupaio ero py For simpliciy, le us deoe by x, h, ad Λ h = λ i he ipu, oupu, ad parameers of a iermediae layer, respecively. The iermediae layer sae occupaio eropy is defied as where H = p logp (9) 5

6 p = p K = x, Λ h. (0) The derivaive of H wih respec o λ i ca be calculaed as Sice H p = = logp + logp logp p p = K logp + p = x, Λ h. () p = x, Λ h = p x, Λ h p x, Λ h f i, x, () we obai he fial gradie H = K logp + p x, Λ h p x, Λ h f i, x,. (3) 4. M ii mize he fra me -le v el codiioal ero py The frame-level codiioal eropy a he iermediae layer ca be wrie as H x, Λ h = p x, Λ h logp x, Λ h. (4) Followig he similar procedure we compue he derivaive of H x, Λ h wih respec o λ i as H x, Λ h = logp x, Λ h + = logp x, Λ h + p x, Λ h p x, Λ h p x, Λ h f i, x, (5) 4.3 Fie uig wih codiioal lielihood-base d bac pro pagaio I he fie uig sep, we aim o opimize he sae sequece log-lielihood L Λ N, Λ h N,, Λ h = logp y x, Λ N, Λ h N,, Λ h = L Λ N, Λ h N,, Λ h. (6) joily for all parameers codiioed o all he layers, where Λ N is he parameer se for he fial layer, ad Λ h N,, Λ h are parameers for he N hidde layers. The observaio as he ipu o he fial layer is x f h f h f h N, =,, T (7) where he hidde layer's frame-level log-lielihood is f h = log p x, f h,, f h, Λ h if > (8) 6

7 ad f h = log p x, Λ h if =. (9) The derivaive of he objecive fucio over λ i is L Λ N, Λ h N,, Λ h = N j = L Λ N, Λ h N,, Λ h f h j f h j N = p y x, Λ N, Λ h N,, Λ h λ h j f h j j = (0) where f h j λ h j i ca be recursively calculaed as i (0) by oicig ha f has he same form as he L Λ N, Λ h N,, Λ h excep wih fewer layers. 5 Experimeal Resuls Table summarizes he recogiio accuracy (RA) o a seve laguage/dialec recogiio as usig he layer-wise usupervised learig wih fie uig approach, where he disribuio cosrai refers o he feaure expasio approach described i [8], CRF refers o he sigle-layer liear-chai CRF, ad DSCRF refers o he deep-srucured CRF described i his paper. Noe ha he DSCRF wih 8 hidde saes ad four-o disribuio cosrai has he same umber of parameers as he Gaussia mixure model (GMM) wih 56 mixures. Due o he page limi, readers are referred o [9] for he deailed experimeal seup. As a compariso, he bes cofiguraio of GMM usig he maximum muual iformaio (MMI) raiig coais 56 Gaussia mixures ad achieved 8.5% recogiio accuracy o his as. I is clear from Table ha he deep-srucured CRF sigificaly ouperforms he sigle-layer CRF wih recogiio accuracies of 83.6% vs. 44.6% ad 79.5% vs. 34.3% wih ad wihou he disribuio cosrai respecively before usig he adem feaures ad he fie uig. Whe he adem feaure is applied, he recogiio accuracy ca be improved o 85.% which was furher improved o 86.4% whe he fie uig is applied. Addiioal resuls o he layer-wise supervised raiig ad o oher ass such as aural laguage processig ca be foud i []. Table : Summary of he recogiio accuracy (RA) o he seve laguage/dialec recogiio as Model # Saes Disribuio /Mixures Cosrai Tadem RA(%) CRF - o CRF - yes o o 79.5 DSCRF+prerai 8 yes o yes yes 85. DSCRF+fieue 8 yes yes Summary I his paper, we have described a deep-srucured CRF model, i which muliple layers of CRFs are saced ogeher o achieve higher classificaio accuracy. We illusraed wo approaches o learig he model parameers i he deep-srucured CRF. I is u-shell, he deep-srucured CRF shares may ideas from he deep belief ewor (DBN) [][3]. However, i differeiaes iself from he DBN i ha he layer-wise pre-raiig is carried ou i a discrimiaive flavor ad ha he sequeial iformaio is 7

8 iegraed i he same way as used i he coveioal CRF. The laer corass he DBN, which requires addiioal emporal processig mechaisms o model he sequeial ipu daa. Ac owledg me s We wish o ha Dr. Chi-Hui Lee a Georgia Isiue of Techology ad Dr. Xiao Li a Microsof Research for helpful discussios. Refereces [] J. Laffery, A. McCallum, ad F. Pereira, Codiioal radom fields: Probabilisic models for segmeig ad labelig sequece daa, I Proceedigs of he Ieraioal Coferece o Machie Learig, pp. 8 89, 00. [] D. Yu, S. Wag, ad L. Deg, "Sequeial labelig usig deep-srucured codiioal radom fields, submied o IEEE Joural of Seleced Topics i Sigal Processig, 009 [3] T. T. Truye, "O codiioal radom fields: Applicaios, feaure selecio, parameer esimaio ad hierarchical modellig", Ph.D. disseraio, Curi Uiversiy of Techology, 008. [4] C. M. Bishop, Paer Recogiio ad Machie Learig, Spriger, ISBN , 006. [5] J. Darroch, ad D. Racliff, Geeralized ieraive scalig for log-liear models, A. Mah. Saisics, 43: , 97. [6] J. Nocedal, Updaig quasi-newo marices wih limied sorage, Mahemaics of Compuaio, vol. 35, pp , 980. [7] M. Riedmiller, ad H. Brau, A direc adapive mehod for faser bac-propagaio learig: The RPROP algorihm, i proc. of IEEE ICNN, vol., pp [8] D. Yu, L. Deg, ad A. Acero, "Usig coiuous feaures i he maximum eropy model", Paer Recogiio Leers. Vol. 30, Issue 8, Jue, 009. doi:0.06/j.parec [9] D. Yu, S. Wag, Z. Karam, L. Deg, "Laguage recogiio usig deep-srucured codiioal radom fields", submied o ICASSP 00. [0] D. Yu, L. Deg, Y. Gog, ad A. Acero, A ovel framewor ad raiig algorihm for variable-parameer hidde Marov models", IEEE ras. o Audio, Speech, ad Laguage Processig, vol. 7, o. 7, pp , IEEE, Sepember 009. [] D. Yu, ad L. Deg, "Solvig oliear esimaio problems usig splies", IEEE Sigal Processig Magazie, vol. 6, o. 4pp.86-90, July, 009. [] G. E. Hio ad R. Salahudiov, Reducig he dimesioaliy of daa wih eural ewors, Sciece, 33(5786), pp , 006. [3] G. E. Hio, S. Osidero, ad Y.-W. Teh, "A fas learig algorihm for deep belief es", Neural Compuaio, 006, 8 (7). pp [4] Y. Begio, P. Lambli, D. Popovici, ad H. Larochelle, "Greedy layer-wise raiig of deep ewors", NIPS 006. [5] M. A. Razao, C. Pouley, S. Chopra, ad Y. LeCu "Efficie learig of sparse represeaios wih a eergy-based model ", NIPS 006. [6] Y. Begio, "Learig deep archiecures for AI", Techical Repor 3, uiversiy of Moreal. [7] L. Ladicy, C. Russell, P. Kohli, P. H. S. Torr, Associaive hierarchical CRFs for objec class image segmeaio, i Proc. ICCV 009. [8] L. Liao, D. Fox, ad H. Kauz, "Hierarchical codiioal radom fields for GPS-based aciviy recogiio", i Proc. of Ieraioal Symposium of Robois Research (ISRR), 007. [9] H. Hermasy, D. P. W. Ellis, S. Sharma, "Tadem coeciois feaure exracio for coveioal HMM sysems", i Proc. ICASSP, vol.3, pp , 000. [0] L. Deg, D. Yu, ad A. Acero, "A Bidirecioal Targe-Filerig Model of Speech Coariculaio ad Reducio: Two-Sage Implemeaio for Phoeic Recogiio", IEEE Tras. Audio, Speech & Laguage Proc, vol. 4, No., pp 56-65, Ja 006. doi: 0.09/TSA [] L. Deg, D. Yu, ad A. Acero, "Srucured speech modelig", IEEE Tras. o Audio, Speech ad Laguage Processig. Vol. 4 No. 5, Sep 006. pp [] D. Yu, L. Deg, ad A. Acero, "Evaluaio of a Log-coexual-spa Hidde Trajecory Model ad Phoeic Recogizer Usig A* Laice Search", i Proc. of Ierspeech, 005, pp [3] D. Yu, L. Deg, A. Acero, "A Laice Search Techique for a Log-Coexual-Spa Hidde Trajecory Model of Speech", Speech Commuicaio, Elsevier. Volume: 48 Issue: 9, Sep 006. pp doi:0.06/j.specom [4] S. Yama ad C.-H. Lee, A flexible classifier desig framewor based o muli-objecive programmig, IEEE Tras. o Audio, Speech, ad Laguage Processig, vol. 6, o. 4, pp ,