Parameter Reestimation of HMM and HSMM (Draft) I. Introduction

Transcription

1 mee eesmon of HMM nd HSMM (Df Yng Wng Eml: Absc: hs echncl epo summzes he pmee eesmon fomule fo Hdden Mkov Model (HMM nd Hdden Sem-Mkov Model (HSMM ncludng hes defnons fowd nd bckwd vbles nd pmee eesmon fomule. cully he sep-by-sep devon s povded n ge del. hs epo s wen fo my pesonl use whch ncludes mel fom sevel ppes [??]. Lef poblems: pobbly nd pobbly densy e no clely dsngushed I. Inoducon he pmee eesmon of HMM nd HSMM e ue sml whch nvolves eve clculon of fowd/bckwd vbles occupon pobbly vbles nd eesmed model pmees. hese vbles nd pmees e nged n ble. he noons wll be explcly defned when hey ppeed fo he fs me n he mn body of hs epo. ble. Vbles nd pmees nvolved n pmee eesmon of HMM nd HSMM Cse Model ype A Sngle HMM HMM Seuence HMM HSMM ( α ( β ( ( ( μ Σ ˆ ˆ L ˆ cˆ sm sm sm sm α β ( α ( β ( ( ( μ Σ ˆ ˆ ˆ L ˆ cˆ ξ ˆ s sm sm sm sm ( ( ( ˆ L ˆ ˆ cˆ ( ( ( ( ( sm μ sm Σ sm sm α β ( ( ( ˆ L ˆ ˆ cˆ ˆ ξ ˆ s ( ( ( ( ( ( ( sm μ sm Σ sm sm. nng Copus Befoe we move o he HMM nd HSMM heoy we fsly defne nng copus (o nng se. A nng copus consss of mny ndependen obsevon seuences.e. { } O O ( whee s he numbe of obsevon seuence nd s he ndex of obsevon seuence n he whole nng copus. An obsevon seuence s seuence of obsevon veco ( O oo o ( whee s he lengh of -h obsevon seuence nd o s he -h fme of -h / 33

2 obsevon seuence. When we ke n obsevon seuence s exmple o dvnce ou devon nd do no ce bou he uppe scp uppe scp cn be dopped whou mbguy hus n obsevon seuence O s defned s ( O oo o (3 whee o s he -h obsevon veco of he seuence n ueson nd s he lengh of he seuence.. emnology II. mee eesmon of HMM A Hdden Mkov Model (HMM s sochsc pocess defned s ( b ( A. (4 A smple HMM s depced n Fg. whee gy ccles epesen non-emng ses whe ccles epesen emng ses deced cs epesen nsons nd specfclly dshed deced c epesen ee nson. lese see he followng fo deled explnon A. Se Fg.. A smple HMM he symbol n (4 s he ol numbe of ses he fs nd ls ses e ssumed o be non-emng ses (clled eny non-emng se nd ex non-emng se nd he second o ( -h ses e emng se whch ems obsevble veco when vsed evey me. he wo non-emng ses e epesened by gy ccles nd ohe whe ccles epesen emng ses. We ssume he Mooe model whch ems obsevons when vsng emng se ech me. All hese ses e no obsevble decly; nsed hey e pobblsclly eled o obsevon seuence hough oupu pobbly whch wll be descbed le n hs secon. B. nson obbly he symbol A n (4 s he se nson mx ( A (5 whee s he se nson pobbly nd s defned s / 33

3 ( ε ε < < ( ε ε < <. (6 ( ε ε < < ( ( ( ε ε Becuse of he exsence of non-emng se s nvolvng non-emng se should be defned cefully s bove nd hese s seem odd he fs sgh. We defne nd expln he noons ppeed n (6 n del hee. he symbol ε s hdden se seuence nd hdden se me whee s dscee me ndex ( ε ε. he nson pobbly ε s he < < (7 s he mos esy one o expln whch s he nson pobbly fom emng se o emng se nd such nson pobbly s elevn wh me ndex hus he fs ode HMM s vully ssumed. he uppe symbol o fo me mens he me mmedely befoe o fe me especvely whch mens vey sho me peod whch cnno ccommode n obsevon befoe o fe hs nson occus. In my undesndng HMM s only ched o he fs se of s only ched o he ls se of HMM nd hese wo odd me noons nd e used o ensue he escon of eneng HMM(s wh he fs fme of n obsevon seuence nd exng HMM(s wh he ls fme of n obsevon seuence especvely. he vgue explnon hee should ge cle when devng fowd nd bckwd vbles. ow le us come o he explnon of nson pobbly ( ε ε < < (8 whch s he nson pobbly fom he eny non-emng se o some emng se. he nson pobbly ( ε ε < < (9 mens he pobbly fom he emng se o he ex non-emng se. he nson pobbly ( ( ( ( ε ε (0 mens he pobbly fom he eny non-emng se o he ex non-emng se by skppng ll emng ses n he -h model n seuence of HMMs whee. hs defnon seem o be odd he fs sgh s s defned n he suon of seuence of HMMs bu wll be ge nul when devng fowd bckwd lgohms. Symmeclly ( cn be defned euvlenly s he symbol fo hdden se seuence s usully chosen s o s n mos leue o HK book bu s chosen s he seuence numbe of HMM n senence HMMs nd s s chosen s he sem ndex. o cuse he les conflcon wh he don of HMM leue we decde o choose he seldom used Geek symbol ε s se seuence. 3 / 33

4 ( ε ε ( ( (. ( h s o sy hee s les wo dffeen wys o defne ee nson whch e shown n Fg.. hs fgue s cle o mnfes he essence of ee nson ( whch s o skp he -h model n HMM seuence. Appenly he sngle HMM s poblemc. ( e e l ( ( ( - - s zeo n he suon of sngle HMM; ohewse ( ( ( ( - - e e l (--h model -h model (-h model Fg.. Dffeen wys o defne ee nson When > 0 he coespondng nson c s usully clled ee nson nd he HMM connng ee nson s clled ee model. ee model s useful o epesen oponl speech un n he feld of Auomc Speech ecognon (AS bu s seldom used n he feld of ex o Speech (S. eveheless we deve eesmon fomule n he mos genel cse whch pems he exsence of ee model. ee nson skps ll emng ses whch cuses model o mch nng seuence wh zeo lengh. As esul ee model s no vld n he cse of nng sngle HMM bu vld n he cse of nng seuence of HMMs. o ensue nson pobbly mx A o be vld he followng consns mus be ssfed 0 < <. ( < A nson cn only occu fom eny non-emng se o mddle emng se o emng se o ex non-emng se h mens > 0 s only possble when < <. (3 hs consn on mx A s vsulzed s 0 0 A (4 whee he blue block of elemens consss nson beween emng ses he geen block consss of nsons fom eny non-emng se he mgen block consss of nsons o he ex non-emng se nd he ed elemen epesens ee nson. hese fou blocks coespond o (7 (8 (9 nd (0 especvely. In HMM bsed Speech Synhess Sysem (HS lef-gh HMMs wh no skp e nomlly used. A smple of lef-gh HMM s shown n Fg / 33

5 3 4 5 Fg. 3. A smple lef-gh HMM he nson mx fo lef-gh HMM s he smple whch s shown n ( A. ( C. Oupu obbly he oupu pobbly ( b n (4 s only dependen on he se numbe < < nd vully could be ny vld pobbly mss funcon (MF o pobbly densy funcon (DF. In pcce b ( s usully defned s S ( ( ( s b o b o < < (6 s s s whee S s he ol numbe of ndependen sems γ o s s he sub-veco belongng o s -h sem bs ( s he pobbly densy funcon fo s -h sem γ s s he exponenl sem wegh wh γ s > 0 bu we usully hve s γ. he sem pobbly b ( s ny vld MF o DF bu Gussn mxue model (GMM s usully used.e. whee ( ; μ Σ M s ( o ( o b c b s s sm sm s m M s m c ( o ; μ Σ sm s sm sm s could be defned (7 s mulve Gussn pobbly densy funcon (DF wh men veco μ nd covnce mx Σ c sm s he mxue wegh of m -h Gussn of sem s of se nd M s s he mxue numbe of sem s of se. Gussn DF s ( o μ Σ ( o μ ( o; μ Σ e (8 n Σ ( π whee n s he dmenson of Gussn dsbuon. In he followng devon of fowd bckwd lgohms oupu pobbly ( b o s 5 / 33

6 usully wen euvlenly s ( ( ε b o o. (9 We lso noe h oupu pobbly ( b o s dependen solely on he cuen obsevon veco o nd s coespondng se numbe ε nd s no elevn o ohe hdden se numbe o obsevon veco me ohe hn. D. Mkov opey A sochsc pocess hs he Mkov popey f he condonl pobbly of fuue ses of he pocess condonl on he ps nd pesen vlues depends only upon he pesen ses no on he seuence of he evens h peceded pesen me. he hdden ses n HMM hs he Mkov popey e.g. ( ε ε ε k ( ε ε (0 whee s ny vld me fo n obsevon seuence O ncludng he odd me nd nd k s ny vld se numbe ncludng h of he non-emng ses. Fomlly he elevn blue em n (0 cn be dopped bsed on Mkov popey. he Mkov popey s mpon n he followng devon of fowd bckwd lgohms.. Geneve ocess of HMM ( Gven ppope vlues of A b ( he HMM cn be used s geneo o genee obsevon seuence O s follows []: Se. he emng se < < s chosen ccodng o he nson pobbly < <. 3 An obsevon o s chosen ccodng o he DF ( b. 4 If < he nex se < < s chosen ccodng o he nson pobbly ; ohewse ns o he ex non-emng se wh pobbly 5 Se nd eun o sep 3. Fom he geneve pocess we know h he se sys n he eny non-emng se befoe ny obsevon s obseved.e.. ( ε 0 ( whee ( s he pobbly of ndom even n he penheses. We lso know h he se mus sy n he ex non-emng se fe he whole obsevon seuence s obseved. h s hp://en.wkped.og/wk/mkov_popey 6 / 33

7 o sy he pobbly of n obsevon seuence O geneed by HMM s ( ε ε O. ( Comped wh he mos popul fom of ( O e.g. n HK book ( s somewh cumbesome bu moe ccue by expessng he consns of ε nd ε explcly. If full se seuence ( εε ε ε s gven o expln he obsevon seuence O he on pobbly of obsevon nd hdden ses s gven by ( O ε ( o ( o ( o b b b. (3 ε ε εε ε ε ε ε ε hen he geneon pobbly n ( cn be expessed s ( O ε ε ( ε ε O ε ε ε ( o ( o ( o b b b ε ε εε ε ε ε ε ε. (4 whee he sum s ove ll possble se seuence ε. 3. mee eesmon of Sngle Model Fowd bckwd lgohms (FBAs fo sho heefe e he bss of pmee eesmon hus we deve hese lgohms fs. he lgohms e slghly dffeen fo sngle HMM nd fo seuence of HMMs. he cse of sngle HMM s ddessed n hs secon nd he cse of seuence of HMMs s lmos pllelly pesened n he nex secon. A. Lnk ule of Condonl obbly Befoe we move o he mn body of FBAs s useful o pesen he lnk ule of condonl pobbly explcly. Assumng E E En e ndom evens whch do no necessly ndependen he followng condonl pobbly fomul holds ( ( ( E E E E E. (5 By ecusve ulzng (5 we hve he so clled lnk ule of condonl ndependence ( n ( n ( n n ( EE En ( En EE En ( En EE En EE E EE E E EE E ( ( ( 3 ( n n ( n n E E E E E E E E E E E E E E. (6 By popely combnng lnk ule of condonl pobbly hee nd foemenoned Mkov popey s no dffcul o deve FBAs fo HMM n del n he nex secon. B. Fowd Bckwd Algohm Fowd Algohm he fowd vble ( α fo n obsevon seuence (3 nd HMM (4 s defned s 7 / 33

8 ( ( oo o ( ( oo o ( ( oo o α ε ε α ε ε < <. (7 α ε ε Inl: ( ( ( ε α (by (. (8 ( ( o e e l (by defnon ( e l ( e e l ( o e e l ( lnk ule of condonl ndependence b( o ( o b < < ( ( o e e l ( o e e e l (by defnon ( full pobbly fomul poned by se me ( o ε ε ( ε o e ε ( lnk ule of condonl ndependence ( ( e e l ( by defnon nd dop elevn blue ems ecuson: ( 3 ( ( by defnon ( ( oo o e e l (by defnon 0 ( e nd e ( > cnno be ssfed smulneously. (9. (30. (3 8 / 33

9 ( ( oo o e e l (by defnon ( oo o e e e l ( full pobbly fomul poned by se me ( oo o ε ε ( ε oo o e ε l ( o oo o e e e l ( lnk ule of condonl ndependence ( ( ( o ( by defnon nd dop elevn blue ems ( b ( o ( by defnon ( b( o e e l e l < < ( ( oo o e e l (by defnon ( oo o e e e l ( full pobbly fomul poned by se me ( oo o e ε ( ε oo o e ε. (3. (33 lnk ule of condonl ndependence ( ( ( e e l ( by defnon nd dop elevn blue ems ( ( by defnon emnon: Obvously he geneon pobbly n ( s Bckwd Algohm ( ε ε α ( O. (34 he nely symmec bckwd vble ( β s defned s ( ( o o o ( ( o o o ( ( oo o β ε ε β ε ε < < β ε ε (35 Inl: ( ( ( β ε ε. (36 9 / 33

10 ( ( e e ( by defnon < <. (37 b b ( ( o e e l ( by defnon ( o e e e l ( full pobbly fomul poned by se me ( e e l ( o e e l ( lnk ule of condonl ndependence b ( ecuson: ( ( by defnon. (38 ( ( o o o e e l 0 ( e nd e ( < cnno be ssfed smulneously b ( ( o o o e e l b (by defnon ( o o o e e e l ( full pobbly fomul poned by se me ( e e l ( o e e l o o e e e o l ( ( lnk ule of condonl ndependence ( o e l ( o o e e l ( by defnon nd dop elevn blue ems ( o b ( ( b by defnon < < (39 (40 0 / 33

11 emnon: b ( ( oo o e e l ( oo o e e e l (by defnon ( full pobbly fomul poned by se me ( e e l ( o e e l ( o o e e e o l ( lnk ule of condonl ndependence ( o e l ( o o e e l ( by defnon nd dop elevn blue ems ( o b ( ( b by defnon Obvously he geneon pobbly n ( cn lso be wen s ( O e e l C. eesmon Fomul ( O e e l ( e l ( e e l ( ( e l ( he defnon of condonl pobbly ( ( by defnon (4 O. (4 b Wh gven nng copus { } O O defned n ( nd known HMM hype pmees ( SM s pmee eesmon of HMM s o solve he opmzon poblem nson Mx ˆ g mx ( O. (43 Wh clculed fowd bckwd vbles locl mxmum of ( wh EM lgohm [] s follows. he eesmon fomul fo nson mx s O cn be deved / 33

12 ˆ ε ε ε ε O ( ε ε ε O ( O ε ε ε ε ( ε ε ε O ( condon pobby fomu ( ( b ( o b ( < < ( b ( ( εuvεncε subsuon sεε bεow whee uppe scp s he ndex of obsevon seuence n nng copus nd nd nd O e e e l ( (44 O ε ε ( (45 o o o o e e ( by defnon e l < < ( ( o ( o o ε ε o ε o o e ε ( condonl pobbly fomul o o e e l dop elevn blue ems ( ( ( b ( ( by defnon ( (46 / 33

13 O e e e e ( o o o o o e e e e ( by defnon ( ( o o e e ( ε o o e ε ( o o o e e e o o ε o o o e e ε ( nk ue of condon ndependence ( ( ( e e ( o e o o e e ( by defnon nd dop eevn bue ems ( ( ( o b ( b < < hen ulzng (46 Smlly ulzng (46. (47 ˆ ε ε ε O ( ( O ε ε ε. (48 ( β ( ( < < ˆ ε ε ε O l ( ε ε ε l O ( O ε ε ε l ( ε ε ε l O. (49 ( condonl pobbly fomul ( ( b ( ( b ( ( < < Fnlly sngle HMM cnno be ee model hus ˆ 0. (50 Oupu pobbly o eesme model pmees eled o oupu pobbly we fsly suppose h 3 / 33

14 b S ( ( o bs ( os s γ s S M s c smbsm ( os s m γ s s S M s csm ( os; μ sm Σ sm < < s m γ. (5 he pobbly of occupyng he m -h mxue n sem s of se fo he -h fme of he -h obsevon seuence s Lsm ( ε O ε ε l s m e e ε ( O ( O e e e l ( ( ls m os e l O e e l ( condonl pobbly fomul nd dop elevn blue ems ( ( b ( csmbsm ( os bs ( os ( euvlence subsuon nd by defnon U c b b * ( ( o b ( ( o sm sm s s (5 whee ls m mens h o s belongs o he m -h mxue n sem s nd ( s s ε l l m o s he mxue occupncy pobbly n sem s of se nd ( o ε l ( o ( os c b c b l m s M s bs b U ( S ( o b ( o * s k k k k s ( o sm sm s sm sm s ( o hus he eesmon fomul fo pmees of oupu pobbly n HMM s 4 / 33 k ( 3 c b sk sk s O ε ε ( Specfclly fo HMM wh sngle Gussn fom of oupu pobbly (53. (54 L sm ( ε O ε ε (. (55 α ( β (

15 Σˆ sm ( Lsm os μ ˆ (56 L sm ( L ( ( ˆ ( ˆ sm os μ sm os μ sm sm Lsm ( (57 whee sm ( Lsm c ˆ (58 L ( M s ( ( L L. (59 m sm o speed up he clculon of Σ ˆ sm n (57 HK uses ck n sscs ( ( ˆ ( ( ( ( ( ˆ D X E X E X X µ E X µ. (60 whee X s ndom vble E( X s he expecon of X D( X s he vnce of X ˆµ s ny consn bu usully s ough guess of E( X. Bsed on (60 (56 nd (57 cn be wen s Lsm s sm Lsm ( ( ( o μ μˆ μ (6 sm sm Lsm ( ( os μ sm ( os μ sm ˆ Σ ( ˆ ( ˆ sm μ sm μ sm μ sm μ sm Lsm ( 4. mee eesmon of Seuence of HMMs. (6 We ls he mny noons used deve he FBAs fo HMM seuence nd pesen he pmee eesmon fomule n hs secon. A. oons nd Convenons he followng noons e used n hs secon some of whch e ledy defned some of whch e smple exenson by ddng n ppen model seuence numbe n HMM seuence. ble. oons n he cse of nng HMM seuence 5 / 33

16 he se of ll pmees defnng HMM se { M} M umbe of models n HMM se M A model seuence ( umbe of HMMs n model seuence Model ndex n model seuence S s umbe of sems Sem ndex umbe of ses n he -h HMM n he model seuence Se ndex ( nson pobbly fom se o se n model ( s M he mxue numbe n model se sem s ( sm c Wegh of mxue componen m n model se sem s ( sm μ Men veco of mxue componen m n model se sem s ( sm Σ Covnce mx of mxue componen m n model se sem s O A nng obsevon seuence Lengh of nng obsevon seuence me ndex of obsevon seuence o o s he obsevon me he obsevon veco fo sem s me he numbe of obsevon seuence n nng copus he ndex of obsevon seuence n he nng copus O ( ( he -h nng obsevon seuence b he oupu pobbly of model se specfclly < < he nson pobbly fom HMM o s mmedely followng HMM n model seuence bsed on me xs s. A pece of HMM seuence s shown n Fg. 4 o move ou nuon n he followng devon (--h model -h model (-h model Fg. 4. A smple HMM seuence 6 / 33

17 In he cse of HMM seuence he se sys n he eny non-emng se of he fs model befoe ny obsevon s obseved.e. ( ( e. (63 0 ohewse he pobbly of n obsevon seuence O geneed by seuence of HMMs n model se s ( ( ( ε ε O. (64 In he cse of nng seuence of HMMs ee models e llowed o ppe n HMM seuence wh wo escons he fs nd ls HMMs could no be ee models nd; hee e no successve ee models. I do no fully undesnd why hee e hese wo consns when I we hs pgph. he fs escon mens ( ( 0. (65 B. Fowd Bckwd Algohm Fowd Algohm he fowd vble ( ( α fo n obsevon seuence (3 nd seuence of HMMs s defned s ( ( ( ( ( oo o ( ( ( ( ( oo o α ε ε α ε ε < <. (66 ( ( ( ( ( ( oo o α ε ε ε he summon n ( ( α s fom o. Obvously hee s no nson fom ( ε o ( ε h s no nson fom ex se o ex se n model so he uppe lm fo he summon n ( ( α s no bu. On he ohe hnd hee s nson fom he eny se o he ex se.e. ee nson fom se o se bu he summon excludes hs nson by se he lowe lm of he summon o nsed of. he summon s vsulzed n Fg. 5 coveed by blue ellpse. 7 / 33

18 h model ( ( α Fg. 5. he summon n α ( ( he movon of excludng ee nson n he summon n ( ( α should be hus cle: ( ( α s used o ggege pl obsevon pobbly fom nne emng ses of HMM bu does no encompss pl obsevon pobbly whch skps he HMM n ueson. Inl: ( When When > o summze ( ( α s ( ( ( ( ( ( by defnon α e e. (67 ( ( ( ( ( ( ( ( ( ( α ε ε ε ε ε. (68 ( ( α s ( ( ( ( ( ( ( ( e e e l ( e e l ( dd he vl consn of blue em ( ( ( ( ( ( ( e e e l e e l. (69 ( lnk ule of condonl ndependence ( ( ( ( ( e ( by defnon nd dop elevn blue em ( ( ( ( by defnon e l ( 3. (70 ( ( ( ( 8 / 33

19 ( ( ( ( ( ( ( o e e l ( ( ( ( o e e e l ( dd he vl consn of blue em ( ( ( ( ( o e e e l ( ( ( ( ( ( e e e l e e l ( lnk ule of condonl ndependence ( ( ( ( ( ( e e o ( l e l ( by defnon nd dop elevn blue ems ( ( ( ( b ( o ( by defnon < < ( ( ( ( o e e e l ( ( ( ( ( ( ( o o e ε ε ε ε ecuson: ( 3 ( ( ( lnk ule of condonl ndependence ( ( ( ( ( e e l ( by defnon nd dop elevn blue ems ( ( ( ( ( ( oo o e e l ( ( ( ( oo o e e e l ( ( ( ( oo o e e e l full pobbly fomul poned by nsons. (7. (7 mmedely befoe me n model ( ( ( ( ( oo o ( ε ε e ( ( ( ( ( ( oo o oo o e ( ( ε ε ε ε he second euon sgn n (73 deved by full pobbly fomul s vsulzed n Fg / 33 ( ( eplce e wh euvlen e lnk ule of condonl ndependence ( ( ( ( ( ( ( ( oo o α ε ε ε ε ( by defnon nd dop elevn blue ems ( ( ( ( ( ( by defnon. (73

20 ( ( α ( ( ( α ( ( α ( ( ( α ( ( α α α ( ( α ( h model h model ( ( Fg. 6. he full pobbly fomul n α ( ( ( ( ( oo o e e l ( ( ( ( ( oo o ε ε o oo o e ε ( lnk ule of condonl ndependence ( ( ( ( oo o e e e l ( ( ( ( oo o e e e l ( l ( ( o e l full pobbly fomul poned by nsons n model mmedely befoe me nd dop elevn blue ems ε ε ( ε ε ( ( ( ( ( ( oo o ε ε ( ε oo o e ε ( ( ( ( ( ( e oo o oo o b ( ( o ( lnk ule of condonl ndependence by defnon ( ( ( ( ( ( ( ( ( l b ( ( ( o e e l e e ( by defnon nd dop elevn blue ems ( ( ( ( ( ( ( b ( o ( by defnon. (74 he hd euon sgn n (74 deved by full pobbly fomul s vsulzed n Fg. 7. ( ( ( ( ( ( ( ( ( b ( o h model Fg. 7. he full pobbly fomul n α 0 / 33 ( (

21 ( ( ( ( ( ( oo o e e e l ( ( ( ( ( ( ( oo o oo o e ε ε ε ε ( lnk ule of condonl ndependence ( ( ( ( ( e e l ( by defnon nd dop elevn blue ems ( ( ( by defnon (75 he fs euon sgn n (75 by defnon s vsulzed n Fg. 5. emnon: he geneon pobbly n (64 s ( O e e l ( ( ( ( e e e l ( ( ( ( oo o e e e l ( ( oo o full pobbly fomul poned by se mmedely befoe me ( ( ( oo o e e l ( by defnon nd lnk ule of condonl ndependence ( ( ( ( ( ( ( e α α ε ( by defnon nd dop elevn blue ems ( ( ( ( ( ( by defnon ( ( ( ( 0 Bckwd Algohm ( ( ( ( ε oo o e ε. (76 he bckwd vble ( ( β fo n obsevon seuence (3 nd seuence of HMMs s defned s ( ( ( ( ( o o o ( ( ( ( ( o o o ( ( ( ( ( ( oo o β ε ε β ε ε < <. (77 β ε ε ε he summon n β s fom o. Obvously hee s no nson fom ( ( ( ε o ( ε h s no nson fom eny se o eny se n model so he lowe lm fo / 33

22 he summon n β s no bu. On he ohe hnd hee s nson fom he eny se ( ( o he ex se.e. ee nson fom se o se bu he summon excludes hs nson by se he uppe lm of he summon o nsed of vsulzed n Fg. 8 coveed by blue ellpse.. he summon s β ( ( Fg. 8. he summon n β ( ( he movon of excludng ee nson n he summon n β should be hus cle: ( ( β s used o ggege pl obsevon pobbly fom nne emng ses of HMM ( ( bu does no encompss pl obsevon pobbly whch skps he HMM n ueson. Inl: ( ( ( ( ( ( ( by defnon b e e. (78 When When < ( ( β s ( ( β s ( ( ( ( ( β ε ε. (79 o summze ( ( ( ( ( ( ( ( ( ε e ε b e e l ( dd he vl consn of blue em ( ( ( ( ( ( e e l e ( lnk ule of condonl ndependence ( ( ( e e ( e e l ( l ( by defnon nd dop elevn blue em ( ( b ( ( by defnon. (80 / 33

23 b ( ( β. (8 ( ( ( ( β ( ( ( ( e e l ( ( ( ( ε e ε ( dd he vl consn of blue em ( ( ( ( ( ( e e l e e e ( l ( lnk ule of condonl ndependence. (8 ( ( ( ( e e l ( by defnon nd dop elevn blue ems ( ( b ( < < ( by defnon ( ( ( ( b ( ( oo o e e e l ( ( ( ( ( e e l ( o e e l ( ( ( ( ε e o ε lnk ule of condonl ndependence. (83 ( ( ( ( ( l ( ( o e l e e ecuson: ( When ( ( β s ( by defno n nd dop elevn blue ems ( ( ( b o b ( ( ( by defnon ( ( ( ( ( o o o β ε ε. (84 ( ( ( ( ( o o o ( ( 0 ( e nd e ( < cnno be ssfed smulneously β ε ε ( ( When < β s. (85 3 / 33

24 ( ( ( ( ( o o o ( ( ( ( o o o e e e l ( ( ( ( o o o e e e l β ε ε o summze full pobbly fomul poned by nsons ( ( ( ( ( o o o e e e l ( ( ( ( e e l ( ( ( ( o o o e e e l ( mmedely fe me n model ( ( eplce e wh euvlen e lnk ule of condonl ndependence ( ( ( o o o ( ( ( β ε ε ( by defnon nd dop elevn blue ems ( ( ( ( b ( ( by defnon b β 0 ( ( ( β ( ( ( β (. (86 (87 he second euon sgn n (86 deved by full pobbly fomul s vsulzed n Fg. 9. β ( ( ( ( β ( ( ( β ( ( β ( ( β ( ( ( β -h model (-h model Fg. 9. he full pobbly fomul n β ( ( 4 / 33

25 ( ( ( ( ( o o o ( ( ( ( o o o e e e l ( ( ( ( o o o e e e l b e e l full pobbly fomul poned by nsons n model mmedely fe me ( ( ( ( ( o e e e l ( e l ( ( ( ( o o 3 o e e o e l ( ( ( ( ( ( e e l ( o e e o o e l ( lnk ule of condonl ndependence ( ( ( l ( e e l ( ( ( ( ( e e l ( o e e l o o ( ( o e o o 3 o ( by defnon nd dop elevn blue ems ( ( ( o ( by defnon ( ( ( ( b ( b b < <. (88 he second euon sgn n (88 deved by full pobbly fomul s vsulzed n Fg. 0. b ( ( ( ( ( o b ( ( b ( ( b ( h model Fg. 0. he full pobbly fomul n β ( ( 5 / 33

26 β ( ( ( ( ( ( oo o e e e l ( ( ( ( ( o e e e l ( e l ( ( ( ( o o o ε e o ε ( lnk ule of condonl ndependence ( ( ( o e l o o o e ( ( e ( l ( βy defnon nd dop elevn βlue ems ( ( ( β o β ( ( ( βy defnon he fs euon sgn n (75 by defnon s vsulzed n Fg. 5. emnon: he geneon pobbly n (64 s ( ( ( O e e l ( ( ( ( oo o e e e l ( ( ( ( oo o e e e l b ( ( full pobbly fomul poned by se mmedely fe me ( ( ( ( ( e l e e l oo o ( ( ( ( e e e l ( by defnon nd lnk ule of condonl ndependence ( ( ( oo o ( ( ( β ε ε ( by defnon nd dop elevn blue ems ( ( ( ( b ( ( by defnon b b ( ( ( ( 0 C. eesmon Fomul Wh gven nng copus { }. (89. (90 O O defned n ( nd known HMM hype pmees ( SM s n HMM se pmee eesmon of HMM s o solve he opmzon poblem ˆ g mx ( O. (9 Mos of he followng eesmon fomule e sml o he cse of sngle HMM wh n ppen penheszed uppe scp epesenng model ndex n HMM seuence. nson Mx 6 / 33

27 Wh clculed fowd bckwd vbles locl mxmum of ( wh EM lgohm [] s follows. he eesmon fomul fo nson mx s ˆ ( ( ( ( ( ε ε ε ε O ( ( ( ( ε ε ε O ( ( ( ( ( O ε ε ε ε ( ( ( ( O ε ε ε ( condon pobby fomu ( ( ( ( ( ( b ( o b ( < < ( ( ( b ( ( εuvεncε subsuon sεε bεow O cn be deved (9 whee uppe scp s he ndex of obsevon seuence n nng copus nd ( ( O ε ε ( (93 nd nd ( ( ( O e e e ( l ( ( ( o o o o e e by defnon e l ( ( ( ( ( ( ( ( o o e e l o o e o o e e ( l (94 condonl pobbly fomul ( ( ( ( ( o o e dop elevn blue ems e l ( ( ( ( ( ( b by defnon < < ( 7 / 33

28 ( ( ( ( O e e e e ( ( ( ( ( o o o o o e e e e ( by defnon ( ( ( ( ( ( o o o e e e ( ( ( ( ( ( o o e e ε o o e ε ( ( ( ( o o ε o o o e e ε ( nk ue of condon ndependence ( ( ( ( ( ( ( ( ( ( ( e e o e o o e e by defnon nd dop eevn bue ems ( ( ( ( ( b ( o b ( ( < <. (95 hen whee ˆ ( ( ( ( ( ε ε ε ε O ( ( ( ( ε ε ε O ( ( ( ( ( O ε ε ε ε ( ( ( ( O ε ε ε ( condon pobby fomu ( ( ( ( ( ( b ( o b ( ( ( ( ( ( ( ( b ( ( b ( < < ( εuvεncε subsuon sεε bεow (96 8 / 33

29 nd ( ( ( ( O e e e e ( ( ( ( ( o o o o o e e e by defnon e ( e ( e ( ( ( ( o o o e e e ( ( ( ( ( ( o o e e o o e ( ( ( ( o o ε o o e e o ε ( nk ue of condon ndependence ( ( ( ( ( ( ( ( ( ( ( ( e e o e e o o e by defnon nd dop eevn bue ems ( ( ( ( ( ( o ( ( ( b b by defnon < < ( ( ( O e e e ( ( ( ( ( o o o o e e e e l ( ( ( ( e e l o o l ( ( ( ( o o o o e e e e l ( full pobbly fomul ( poned by nsons ( < mmedely fe me ( ( ( ( o o ε ε o o e ε ( ( ( ( ( ( ( o o e e l e o ( o e e l ( ( ( ( o o ε o o e e ε ( lnk ule of condonl ndependence ( ( ( ( ( ( o o ( ( ( ( ( ( o o e e l α ε ε ε ( e e l ( ( o o e e l ( by defnon nd dop elevn blue ems ( ( ( ( ( ( ( ( ( ( by defnon ( b b < < Smlly ulzng (94 (97. (98 9 / 33

30 whee ˆ ( ( ( ( ( ε ε ε ε O ( ( ( ( ε ε ε O ( ( ( ( ( O ε ε ε ε ( ( ( ( O ε ε ε ( ( condon pobby fomu (99 ( ( ( ( b ( < < ( ( ( b ( ( εuvεncε subsuon sεε bεow O e ( ( ( ( e e e ( ( ( ( o o o o o e e e e ( by defnon ( ( ( ( ( ( ( o o o o e ( ( e e ε ε ( ( ( ( ( o o e o o o e e ( e. (00 nk ue of condon ndependence ( ( ( ( ( ( ( ( ( e e o o e e ( by defnon nd dop eevn bue ems ( ( ( b ( ( by defnon < < Fnlly by ulzng he second em nd whole p of (98 ee nson s eesmed s ˆ ( ( ( ( ( ε ε ε ε O ( ( ( ( ε ε ε O ( ( ( ( ( O ε ε ε ε ( ( ( ( O ε ε ε ( condon pobby fomu ( ( ( ( ( b ( ( ( ( ( ( ( ( b ( ( b ( < < (0 o eesme pmees of oupu pobbly we mus defne some uxly vbles fs 30 / 33

31 of ll. he uxly fowd vble s U ( ( ( ( ( ( ( ( ( ( (. (0 3 Fg. 7 s helpful fo us o undesnd he physcl menng of hs uxly vble. hen pl oupu pobbly fo pllel sems nd pobbly of whole obsevon seuence b S ( * ( ( s o bk ( ok k O e k s (03 ( ( O ε ε ( he pobbly of occupyng he m -h mxue n sem s of se of model fo he -h fme of he -h obsevon seuence s ( sm L ( ( ε O ε ε l ( ( ( ( ls m O ε ε ε l ( ( O ε ε ε l ( ( ( ( ls m os ε l O ε ε l ( condonl pobbly fomul nd dop εlεvn bluε εms (04 ( ( csmbsm ( os ( ( b bs ( os ( εuvlεncε subsuon nd by dεfnon ( ( ( ( U c b ( b ( ( ( ( * sm sm os b s o whee ( ls m mens h o s belongs o he m -h mxue n sem s of model nd ( s s ε l l m o s he mxue occupncy pobbly n sem s of se ( ( ( ( ( ( c ( ( smbsm os csmbsm o s ( s os ε l ( M s bs ( os cskbsk os k l m ( ( ( Specfclly fo HMM wh sngle Gussn fom of oupu pobbly (05 ( ( ( ( L ( sm ε O ε ε (. (06 ( ( α ( β ( he eesmon fomul fo oupu pobbly fo HMM seuence s sml o h n he cse of sngle HMM excep he dded ppen uppe scp h s 3 / 33

32 Σˆ ( sm ( sm ( L os μ ˆ (07 ( sm L ( ( ( ( ( ( L ( ˆ ˆ sm os μ sm os μ sm ( sm ( Lsm ( (08 whee ( sm ( sm ( ( L c ˆ (09 L ( ( ( M s ( sm m ( L L. (0 o speed up he clculon of ˆ ( sm Σ he sme ck n (60 cn be used. hus (07 nd (08 s ewen s ( sm ( ( ( Lsm ( os μ sm ( Lsm ( ( sm μˆ μ ( ( ( ( ( ( Lsm ( os μ sm os μ sm ˆ ( ( ( ( ( Σ ( ˆ ˆ sm μ sm μ sm μ sm μ sm ( Lsm ( (. ( III. mee eesmon of HSMM. emnology (o complee. Geneve ocess of HSMM 3. Fowd Bckwd Algohm fo Sngle HSMM 4. Fowd Bckwd Algohm fo HSMM Seuence A. Bckwd lgohm B. Fowd lgohm 3 / 33

33 5. mee eesmon of HSMM A. Sngle Model Cse B. Model Seuence Cse efeences. bne L. nd B.-h. Jung Fundmenls of speech ecognon 993: ence Hll.. Dempse A...M. Ld nd D.B. ubn Mxmum lkelhood fom ncomplee d v he EM lgohm. Jounl of he oyl Sscl Socey. Sees B (Mehodologcl 977: p / 33