SEMI-TIED FULL-COVARIANCE MATRICES FOR HMMS

Transcription

1 SEMI-TIED FULL-COVARIANCE MATRICES FOR HMMS M.J.F. Gales Deceber 9, 997 Cotets Bass. Block Dagoal Matrces : : : : : : : : : : : : : : : : : : : : : : : : : : : :. Cooly used atrx dervatve : : : : : : : : : : : : : : : : : : : : : : : :.3 Collares : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : ML Estato of Se-Ted Full Covaraces 3 3 Updatg the Mea 3 4 Updatg the Dagoal Covarace Matrx 3 5 Updatg the Full-Covarace Trasfor 5 5. Ipleetato : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 6 5. Ipleetato : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : Ipleetato 3 : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 7 6 The Updatg Sequece 6. Mea ad Dagoal Covarace Frst : : : : : : : : : : : : : : : : : : : : : : 6.. To update the ea ad the dagoal covarace wth A fxed : : : : : : 6.. To update the full-covarace trasfor assug the ea ad dagoal covarace are fxed : : : : : : : : : : : : : : : : : : : : : : : : : : : 9 6. Mea ad Full-Covarace Trasfor Frst : : : : : : : : : : : : : : : : : : : To update the ea ad the full-covarace trasfor, wth Σ () dag fxed : To update the dagoal covarace, wth the ea ad the trasfor fxed 0 7 Maxu Lkelhood Lear Regresso for Speaker Adaptato 7. Assug SI Dagoal-Covarace : : : : : : : : : : : : : : : : : : : : : : : 7.. Splfcatos of W : : : : : : : : : : : : : : : : : : : : : : : : : : 7. Assug SI Se-Ted Full-Covarace : : : : : : : : : : : : : : : : : : : : Assug SI Ay Full-Covarace wth Hyper-MLLR : : : : : : : : : : : : : : Assug Se-Ted Full-Covarace wth Hyper-MLLR : : : : : : : : : : : : 5

2 Bass. Block Dagoal Matrces If The Σ Σ Σ Σ 0 Σ 0 Σ Σ, Σ, 0 0 Σ, (x, ) T Σ, (x, ) (x, T ) Σ, (x, )+(x, T ) Σ, (x, ) Therefore we oly dscuss full-atrces here. Applcato to block dagoal atrces are easy sply treat each block depedetly as a full atrx.. Cooly used atrx dervatve.3 T A + A log (Σ T ), T For ay atrx A dd, defe cofactor of a, COF(a ), beg (,) + tes the deterat of the atrx resultg fro reovg the -th row ad the -th colu fro A.The. A d a COF (a ) d a COF (a ) for ay ad ay.. If A s vertble, A, A fcof(a)g Lear Algebra, p. Lear Algebra, p05.

3 ML Estato of Se-Ted Full Covaraces Let d represets the desoalty of the data x. Supposefor each Gaussa (), there exsts dd such that Σ () 4 full Σ () dag T where s shared by a set of Gaussas, M. does ot eed to be orthogoal or orthoral. We also defe A 4, The log N (x; () ; Σ () dag ;), d log + log Σ (), dag + log +[, (x, )] T Σ (), dag [, (x, )] d log + log Σ () dag,loga +[A(x, () )] T Σ (), dag [A(x, () )] log N (Ax; A () ; Σ () )+log A () dag So the crease coputato cost s oe atrx ultplcato (d d tes d ) per A per frae of data x. Ths kd of covarace s called se-ted full covarace because Σ () s ot dag ted; oly the trasfor A s ted; ad after the trasfor every Gaussa has a full covarace atrx. 3 Updatg the Mea Q( ˆ () ˆ (), Assug both  ad ˆΣ () are full-raked, dag Â(x, ˆ () )T ˆΣ (), dag Â(x, ˆ () ) o,ât ˆΣ (), dag  (x, ˆ() ) 0 d (x, ˆ () )0 d o o Therefore ˆ () x () 4 Updatg the Dagoal Covarace Matrx If Σ () were ot dagoal, defe dag C 4 Σ (), dag 3

4 The Q(Ĉ), Sce Σ () s syetrc, so s C. dag Ĉ, Ĉ, ˆΣ () dag, log Ĉ +[Â(x, ˆ() )] T Ĉ [Â(x, ˆ() )],(ĈT ), +[Â(x, ˆ() )] [Â(x, ˆ() )] T o 0 dd If we force Σ () to be dagoal, assue dag Σ () 4 dag 0 Â(x, ˆ () )Â(x, ˆ () T ) Â(x, ˆ () 0 ::: 0 0 ::: 0 0 ::: ::: d Q( vˆ ), ˆ A 4 Â(x, ˆ () ) T (3) v! 0 ::: 0 0 v ::: 0 0 ::: ::: v d log vˆ + Â(x, ˆ () ) ˆv,, Â(x, ˆ () ) Â(x vˆ, ˆ () ) o, vˆ o, vˆ 0 I other words, dagfforula(3)g ˆ,Âx,, Â ˆ () (4) If A s fxed whle Σ () s beg re-estated,the dag ˆ, Ax,, A ˆ () (5) 4

5 5 Updatg the Full-Covarace Trasfor Q(Â), M o Â(x,log +, ˆ () )T ˆΣ Â(x (),, ˆ () ) dag where M s the set of Gaussas that share ), + ˆΣ (), dag  (x, ˆ() )(x, ˆ () ) To 0 dd By Collary ad, the (; )-etry of the above equato s, Â,COF(â ) + ˆ where â ŷ + d k;k6! ) â () kŷ k ŷ ()  + âcof(â) e ŷ () 4 x, ˆ () Note ths s the ˆ () Forula () wthout applyg the trasfor A to data x. Σ (), dag Ay() y ()T A e Rearragg the above equato gves us where ĉ ĉ ĉ 3 ( d k 4 A,a COF(a ) a () ky k y  ĉâ + ĉ â + ĉ3 0 (6) ( ( ˆ ˆ COF(â ) ŷ " Âeŷ(),COF(â )+  e ˆ + COF(â ) k6 k6 â () kŷ k ŷ () ) #) â () kŷ k ŷ () Sce ˆΣ () Forula (4) depeds o  ad vce versa, t s dffcult to re-estate both dag sultaeously. So whle re-estatg A, let sfxσ () (ad vce versa). Moreover, f we dag use the curret estate of A for all the rght had sdes of the c-equatos, we ca estate  teratvely utl all the eleets coverge or Q(Â) coverges. c () COF(a ) ŷ ) 5

6 c c 3 ( ( " A eŷ,cof(a ) + A e + COF(a ) k6 k6 a () kŷ k ŷ () #) a () kŷ k ŷ () ) (7) ) â,c +, p c, 4c c 3 c There are two solutos, so there s the questo of whch root s to be chose. Dfferetatg Forula (6) Â, ĉâ + ĉ, pĉ +, 4ĉ ĉ 3 Â < 0 sce we are axzg Q. There s o strct costrat that A be postve. But practce ths s varablythe case becausea I tally. Therefore we choose practcally. â,c + p c, 4c c 3 c 5. Ipleetato Ipleetg Forula (7) drectly wth u space requred, each A has the followg forato: struct Afo { float **a; double **cof, det, c, c, c3; } A; where A ca be coputed dyacally by e det_ot_ A.det - A.a[][] * A.cof[][]; The we copute the rght had sdes of Forula (7) for each pece of data x ad accuulate the to A.c, A.c, ad A.c3 for each A. Ths approach has the advatage of usg the least aout of space. I partcular, f there are A s the syste, t requres oly 3 doubles per a etry ad thus 3 d bytes are requred f double s used. A typcal sze s 3**50*39*39.5 MB. However, ths approach eglects a lot of coo expressos Forula (7) ad thus s too coputatoally expesve. 6

7 5. Ipleetato Aother extree pleetato s to extract as ay coo expressos as possble. Let s see how uch ths would cost ters of eory. 9 < c COF(a ) () ŷ : () ; If we ca store < : c A e () ŷ < +COF(a ) a k : k6 c 3,COF(a ) +A a e k k6 ŷ () ŷ ()T < : 9 ; ŷ () k ŷ () k ŷ () ŷ () 9 ; (x, ˆ () )(x, ˆ () ) T whch s d etres per Gaussa, the at the ed of data processg we ca copute c s to solve Â. Ths approach s the best coputato-effcet way. However, t requres d N floats or doubles of space, where N s the uber of Gaussas the syste. A typcal uber wll be 39 * 39 * 0k * 4 bytes for floats, or.4606 GB for doubles!! 5.3 Ipleetato 3 Therefore a good coprose betwee the above two pleetatos s to defe The G () 4 c COF(a )G () c COF(a ) a kg () k k6 c 3,COF(a ) ŷ () ŷ ()T 9 ; (x, ˆ () )(x, ˆ () ) T () + A e G() +A e k6 a kg () k (9) 7

8 Ths saves a lot of coputato copared wth the frst pleetato. Space wse, t eeds d 3 etres per A. A typcal sze s 39*39*39*50*4 for floats, or MB for doubles. I addto, f block dagoals are used, the space requred wll be d3 per A, whered s the desoalty of each block. Ifthereareb equally szed blocks, space s saved fro d 3 etres per A to d 3 b etres per A. 6 The Updatg Sequece I ths secto, we assue Ipleetao 3 s used. Frst of all, f we wat to sultaeously update, Σ dag ad A, the because of the depedecy betwee  ad ˆΣ dag we would eed to do soethg as follows:. Update ˆ.. Update ˆΣ dag wth A. 3. Iteratvely estate  wth the curret A ad Σ dag ad ˆ. 4. Uless covergece, A Σ dag ˆΣdag go to step. Due to the eory requreet ad coplexty, we decde to gve up updatg all the paraeters sultaeously. The questo left s f( () ; Σ () dag )+Ag+ or f(() ;A) + Σ () dag g+? 3 It turs out that ether of the has about the sae cost, both coputato ad eory. 6. Mea ad Dagoal Covarace Frst 6.. To update the ea ad the dagoal covarace wth A fxed A I tally.. Read odel cout ad copute fa () ; Σ () dag ;Ag.. Covert each pece of data x to Ax order to copute Forula () to copute. 3. Accuulate x, x Ax, ad x (Ax) per Gaussa utl the ed of the trag corpus. Ths takes ( + d + d) N floats/doubles, where N s the total uber of Gaussas the syste. 4. Wrte out the above odel couts so that we ca copute Forula () ad (5) to get the updated odel. 5. Copy the put A to the output odel. 6. Wrte out HMM trasto couts. 3 The HMM trasto probablty ad the Gaussa xture weght ca always be updated ether at the frst phase or secod phase easly. Â

9 6.. To update the full-covarace trasfor assug the ea ad dagoal covarace are fxed. Read the odel couts output by Secto 6.. to copute fa ˆ () ; ˆΣ () dag ;Ag.. Restore the u-trasfored ea ˆ () A, (A ˆ () ) for coputg G (). 3. Covert each pece of data x to Ax order to copute Forula () to copute. The orgal u-trasfored data x has to be saved too for the followg step. 4. Accuulate G () Forula () for each A utl the ed of the trag corpus. Ths takes d 3 etres per A. I addto, we also eed to accuulate for Forula (9). Ths takes aother sgle etry per A. 5. Iteratvely estate  utl covergece: (a) Copute COF(a ) ada for each A wth ts curret value. e (b) For each row of each A, useg () to copute â. Thus we solve  row by row. (c) If Â, A <or or Q(Â) coverges, stop. Otherwse, set A  adgoto step (a). 6. Wrte out all  s. 7. Now read the odel couts x output by Secto 6.. aga. Use the utrasfored ew ea ˆ () (whch s already coputed at step ) to copute ad wrte out x  ˆ().. Slarly, use the ew dagoal varace (whch s already coputed at step ) ad the ea wrtte at step 7 to copute ad wrte out f ˆ () +[ ˆ() ] g x. 9. Copy HMM trasto couts ad the put x to the output. We eed to do step 7 because for the ext terato of trag or decodg, the ea should use the ew A, whle the odel couts wrtte at Secto 6.. used the old A. We eed to do step because ˆΣ () should be obtaed by usg the old A whch s beg dag overwrtte by the ew values. 6. Mea ad Full-Covarace Trasfor Frst 6.. To update the ea ad the full-covarace trasfor, wth Σ () dag fxed Expad Forula () to G () < :,, < : 9 xx T ; x x A ˆ ()T A ˆ ()T 9 ; T 9

10 + A ˆ () ˆ ()T. Read odel f () ; Σ () dag ;Ag.. Covert the put odel to fa () ; Σ () dag g. The u-trasfored ea () eeds ot to be saved. 3. Covert each pece of data x to Ax order to copute Forula () to copute. The orgal u-trasfored data x has to be saved too for the followg step. 4. Accuulate x, x x, ad xxt (0) for G () Forula (0) (ad ˆ () Forula ()) utl the ed of the trag corpus. Ths takes d 3 etres per A ad d + etres per Gaussa. All tes Forula (0) ad (9) ca be derved fro these statstcs. 5. Use Forula () to copute ˆ (), whch s ot oly oe of the output, but also wll be used ext. 6. Iteratvely estate  utl covergece: (a) Copute COF(a ) ada for each A wth ts curret value. e (b) For each row of each A, copute G () accordg to Forula (0). The use Forula (9) to copute â. (c) If Â, A <or Q(Â) coverges, stop. Otherwse, set A  adgotostep (a). 7. Wrte out all  s.. Wrte out ˆ () ad ts u-oralzed weght x for each Gaussa. 9. Wrte out HMM trasto probabltes. 0. Copy the put Σ () to the output odel. dag 6.. To update the dagoal covarace, wth the ea ad the trasfor fxed. Read odel f ˆ () ; Σ () dag ;Âg.. Covert the put odel to fâ ˆ() ; Σ () dag g. 3. Covert each pece of data x to Ax order to copute Forula () to copute. The orgal u-trasfored data x eeds ot to be saved. 4. Accuulate x,ad x (Âx,ˆ() ) per Gaussa utl the ed of the trag corpus. Ths takes ( + d) N floats/doubles, where N s the total uber of Gaussas the syste. 5. Use Forula (3) to copute ad wrte out ˆΣ () dag. 6. Copy the rest of paraeters the put odel to the output odel. 0

11 Notce at step 4, stead of usg Forula (4), we have to use Forula (3) drectly. Ths s because Â(x vˆ, ˆ ),Âx, + ˆ,,  ˆ () (  " x #) I the last ter, the ter wth the bracket [] s ot ecessarly equal to ˆ () coputed Secto 6.., because the data alget ad ca be dfferet fro what was coputed the. 7 Maxu Lkelhood Lear Regresso for Speaker Adaptato Gve a sall aout of speaker-depedet data, we wat to trasfor the speaker-depedet (SI) Gaussa eas to speaker-depedet (SD) wth a lear trasfor, assug the rest of SI paraeters are fxed: (SD) B dd (SI) + b d Use aother otato, W s dexed fro..d, 0..d. W d(d+) d, bb ) (SD) W 7. Assug SI Dagoal-Covarace Assue the SI covarace s dagoal. Furtherore defe W 4 fgaussas that share the sae Wg. Q(Ŵ), W (x, Ŵ () ) T Σ (), dag (x, Ŵ () ), Σ (), dag (x, Ŵ ()To () ) 0 d(d+)

12 Σ (), dag x ()T Σ (), Ŵ () dag ()T W W Both sdes are d (d + ) atrces. The left-had sde, Z, cabe accuulatedfro the data, ( ) z x () () W for ::d; 0::d wth () 0 forall() s. To copute the rght-had sze, Y, otce Therefore, where y Σ (), dag d k0 Ŵ () ()T ŵ k W < : (), G () (d+)(d+) () k 4 () W d k0 d k0 d k0 () Σ (),Ŵ, () ()T dag k k ( (), ŵ k (), () ŵ k)( k () ) () k 9 A ; () d k0 ŵ kg () k () () ()T (3) () Notce ths G s othg related to the G defed Secto 5.3 ad here G k both dces rage fro 0..d. Let W deote the -th row of W. Equato () ples Notce that G s syetrc, so Y Ŵ G () Z Z T G ()T Ŵ T G () Ŵ T So each row of Ŵ ca be solved depedetly by usg ay of the stadard Gaussa Elato Method. What we eed to accuulate whle scag the trag corpus s the x per Gaussa for coputg Z Forula (), ad per Gaussa for coputg G Forula (3). 7.. Splfcatos of W If B s block dagoal, sply treat each block depedetly as a full atrx. If B s dagoal, t s sply block dagoal wth block sze for each block. However, for the dagoal case, there s a ore effcet way of solvg the varables, see HTK Tr.

13 7. Assug SI Se-Ted Full-Covarace If the se-ted full covarace s used for the SI odel, Σ () full () Σ () dag ()T A (), Σ () dag the Q fucto becoes, Q(Ŵ), W, A (), T A (), x, Ŵ () T (), Σ dag A (), x, Ŵ () Notce that for ; W, t ay be true that A ( ) A ( ). The tyg sharg A ad sharg W eed ot to be the W W Because A ()T Σ (), dag A()o x ()T z (A ()T Σ (), dag A() )q W < : d p,a ()T Σ (), dag A() (x, Ŵ () ) ()To 0 h d W d p a pa pq (), p q apapq( xq) () p A ()T Σ (), dag A()o Ŵ () ()T 9 ; () (4) Whe A I, Equato (4) becoes (). Aga, to copute z, x eeds to be accuulated. I the worst case, ths s a lear equato syste wth d (d + ) varables ad equatos. Therefore t ca stll be solved usg the stadard Gaussa Elato Method. However wth the large desoalty, t s possble that the lear costats do ot have the full rak. Moreover solvg such a hgh desoal equato syste s very expesve. 7.3 Assug SI Ay Full-Covarace wth Hyper-MLLR Whe a arbtrary full covarace s used, we frst decopose the covarace atrx to ts egevalues ad ts orthooral 4 egevectors: Σ () full () Σ () dag ()T The we assue the SD ea ˆ () ad the SI ea () has the followg relatoshp: 4 Reeber ths eas ()T (),. A () ˆ () B, A () () + b 3

14 I other words, ˆ () 4 A (), BA () () + A (), b where A () 4 (), ad W (bb) ust lke what we defed at the begg of ths secto. where That s, A () ˆ () 4 b BA () : 6 4 [b B] W f A () fa () 4 ()... () d A () () A () () Oe should copare ths exteded ea wth that defed at the begg of ths secto. Also otce that f the SI covarace s dagoal (A I), ths trasforato degeerates to that case. We watto fdthe optalw to axze the trag probablty. Aga W cabe shared across a set of Gaussas W. Q(Ŵ),, W W, W W Σ (), dag (x, ˆ () ) Σ (), full (x, ˆ () ) A () (x, ˆ () ) T Σ (), dag A () x, Ŵ f A ()o T Σ (), dag, A () x f A ()T A () (x, ˆ () ) A () x, Ŵ A f ()o, Σ (), dag (A() x, Ŵ () A f ) A f ()To 0 d(d+) W () Σ (), ()T Ŵ A f A f dag Therefore, Forula () ad (3) ca be used aga to fd a closed for soluto for each row of W. ( ) z (A() x) (A () () ) W y d < () ŵ k : (), k0 W (A () () )k (A () () ) 9 A ; d k0 ŵ kg () k 4

15 G () (d+)(d+) 4 W () fa () f A ()T A few ssues eed to be addressed: For two Gaussas the sae W, the fal trasfor A ( ), b A ( ), BA ( ) 6 A ( ), b A ( ), BA () are stll dfferet f A ( ) 6 A ( ). Therefore I call t hyper-mllr. Both the feature data x ad the Gaussa ea have to be coverted by A () for each Gaussa. The oly Σ () s used; Σ() ca go away. dag full The above te exhbt o extra coputatoal cost actually. Ths s because whe full covaraces are used, we always do the coverso o the put ea ad data ayway (see Forula ()). Oce W s foud, stead of coputg ˆ (), A (), b A (), BA () () we sply apply W to f A () The regular decodg/trag algorths whle applyg A () to data x ca cotue as usual. Ths s because (x, ˆ () ) T Σ (), (x, ˆ full () ) A () (x, ˆ () ) T Σ (), dag A () (x, ˆ () ) A () x, W T A () () Σ (), dag A () x, W A () () 7.4 Assug Se-Ted Full-Covarace wth Hyper-MLLR Ths s exactly the sae as the above sub-secto, except that egevector decoposto s ot volved. Istead becoes the shared covarace trasfor defed Secto. Also otce that A () ca be shared across a set of Gaussas M, whch s ot ecessarly the sae as W. As a atter of fact, M W usually. M s usually the phoe class: all the Gaussas that correspod to the sae base phoe share the sae covarace trasforato A. W depeds o the aout of the SD data. Usually we do t have uch SD data. So a broad phoe class (such as asal, vowel, etc.) s usually used for W. That s, usually f two Gaussas share the sae covarace trasfor A, the they share the sae SD ea trasfor W. Butot vce versa. However, whe ore ad ore SD data are avalable, ths ay break. 5