Hidden Markov Models

Transcription

1 Hdden Marv Mdel Berln Chen 004 Reference:. Rabner and Juang. Fundamenal f Speech Recgnn. Chaper 6. Huang e. al. Spen Language rceng. Chaper Vaegh. Advanced Dgal Sgnal rceng and e Reducn. Chaper 5 4. Rabner. A ural n Hdden Marv Mdel and Seleced Applcan n Speech Recgnn. rceedng f he IEEE vl. 77. February 989

2 Inrducn Hdden Marv Mdel (HMM) Hry ublhed n paper f Baum n lae 960 and early 970 Inrduced peech prceng by Baer (CMU) and Jelne (IBM) n he 970 Aumpn Speech gnal can be characerzed a a paramerc randm prce arameer can be emaed n a prece well-defned manner hree fundamenal prblem Evaluan f prbably (lelhd) f a equence f bervan gven a pecfc HMM Deermnan f a be equence f mdel ae Adumen f mdel parameer a be accun fr berved gnal AI Berln Chen

3 Obervable Marv Mdel Obervable Marv Mdel (Marv Chan) Fr-rder Marv chan f ae a rple (SAπ) S a e f ae A he marx f rann prbable beween ae ( - - )( - ) A π he vecr f nal ae prbably π ( ) he upu f he prce he e f ae a each nan f me when each ae crrepnd an bervable even he upu n any gven ae n randm (deermnc!) mple decrbe he peech gnal characerc Fr-rder and me-nvaran aumpn AI Berln Chen 3

4 Obervable Marv Mdel (cn.) Example : A 3-ae Marv Chan Sae generae ymbl A nly Sae generae ymbl B nly and Sae 3 generae ymbl C nly 0.6 A π [ ] B C Gven a equence f berved ymbl O{CABBCABC} he nly ne crrepndng ae equence {S 3 S S S S 3 S S S 3 } and he crrepndng prbably A 0.7 (O ) (S 3 )(S S 3 )(S S )(S S )(S 3 S )(S S 3 )(S S )(S 3 S ) AI Berln Chen 4

5 Obervable Marv Mdel (cn.) Example : A hree-ae Marv chan fr he Dw Jne Indural average he prbably f 5 cnecuve up day ( 5 cnecuve up day) a a a 0.5 ( 0.6) π a π ( π ) AI Berln Chen 5

6 Obervable Marv Mdel (cn.) Example 3: Gven a Marv mdel wha he mean ccupancy duran f each ae p d ( d) d d ( a ) ( a ) dp prb.deny funcn f Expecednumber f ( a ) d ( d) d( a ) ( a ) ( a ) ( a ) a d a a duran d duran n a ae n ae a d d AI Berln Chen 6

7 Hdden Marv Mdel AI Berln Chen 7

8 Hdden Marv Mdel (cn.) HMM an exended vern f Obervable Marv Mdel he bervan urned be a prbablc funcn (dcree r cnnuu) f a ae nead f an ne--ne crrepndence f a ae he mdel a dubly embedded chac prce wh an underlyng chac prce ha n drecly bervable (hdden) Wha hdden? he Sae Sequence! Accrdng he bervan equence we are n ure whch ae equence generae! Elemen f an HMM (he Sae-Oupu HMM) {SABπ} S a e f ae A he marx f rann prbable beween ae B a e f prbably funcn each decrbng he bervan prbably wh repec a ae π he vecr f nal ae prbably AI Berln Chen 8

9 Hdden Marv Mdel (cn.) w mar aumpn Fr rder (Marv) aumpn he ae rann depend nly n he rgn and denan me-nvaran Oupu-ndependen aumpn All bervan are dependen n he ae ha generaed hem n n neghbrng bervan AI Berln Chen 9

10 Hdden Marv Mdel (cn.) w mar ype f HMM accrdng he bervan Dcree and fne bervan: he bervan ha all dnc ae generae are fne n number V{v v v 3 v M } v R L In h cae he e f bervan prbably drbun B{b (v )} defned a b (v )( v ) M : bervan a me : ae a me fr ae b (v ) cn f nly M prbably value A lef--rgh HMM AI Berln Chen 0

11 Hdden Marv Mdel (cn.) w mar ype f HMM accrdng he bervan Cnnuu and nfne bervan: b he bervan ha all dnc ae generae are nfne and cnnuu ha V{v v R L } In h cae he e f bervan prbably drbun B{b (v)} defned a b (v)f O S ( v ) b (v) a cnnuu prbably deny funcn (pdf) and fen a mxure f Mulvarae Gauan (rmal) Drbun ( v) M w ( π) L Σ Cvarance Marx exp v µ Σ ( v µ ) Mean Vecr Obervan Vecr AI Berln Chen

12 Hdden Marv Mdel (cn.) Mulvarae Gauan Drbun When X(X X X L ) a L-dmennal randm vecr he mulvarae Gauan pdf ha he frm: f ( X x µ Σ ) ( x; µ Σ ) exp L x µ Σ x π Σ and he - h he he dermnan elevmen σ f f Σ σ where u he L - dmennal mean vecr Σ he cverance marx Σ Σ E [ ] [ ] x µ x µ E xx Σ E ( µ ) [( x )] [ ] µ x µ E x x µ µ If X X X L are ndependen he cvarance marx reduced dagnal cvarance he drbun a L ndependen calar Gauan drbun Mdel cmplexy reduced µµ AI Berln Chen

13 Hdden Marv Mdel (cn.) Mulvarae Gauan Drbun AI Berln Chen 3

14 Hdden Marv Mdel (cn.) Cvarance marx f he crrelaed feaure vecr (Mel Frequency fler ban upu) Cvarance marx f he parally decrrelaed feaure vecr (MFCC ceprum whu C 0 ) AI Berln Chen 4

15 Hdden Marv Mdel (cn.) Mulvarae Mxure Gauan Drbun (cn.) Mre cmplex drbun wh mulple lcal maxma can be apprxmaed by Gauan (a unmdal drbun) mxure f M M ( x) w ( x; µ Σ ) w Gauan mxure wh enugh mxure cmpnen can apprxmae any drbun AI Berln Chen 5

16 Hdden Marv Mdel (cn.) Example 4: a 3-ae dcree HMM E 0.6 A b b b3 π ( A) 0.3 b ( B) 0. b ( C) ( A) 0.7 b ( B) 0. b ( C) ( A) 0.3 b3 ( B) 0.6 b3 ( C) [ ] Gven a equence f bervan O{ABC} here are 7 pble crrepndng ae equence and herefre he crrepndng prbably {A:.7B:.C:.} 7 7 ( O ) ( O S ) ( O S ) ( S ) S : ae. g. when S { 3} ( O S ) ( A ) ( B ) ( C 3 ) ( S ) ( ) ( ) ( ) 0.5*0.7 * equence {A:.3B:.C:.5} {A:.3B:.6C:.} 0.7 *0.* AI Berln Chen 6

17 Hdden Marv Mdel (cn.) an : O{ 3 }: he bervan (feaure) equence S { 3 } : he ae equence : mdel fr HMM {ABπ} (O ) : 用 mdel 計算 O 的機率值 (O S) : 在 O 是 ae equence S 所產生的前提下 用 mdel 計算 O 的機率值 (OS ) : 用 mdel 計算 [OS] 兩者同時成立的機率值 (S O) : 在已知 O 的前提下 用 mdel 計算 S 的機率值 Ueful frmula Bayean Rule : ( AB) ( B) ( A B) ( B A) ( A) ( B) ( A B) ( B A) ( A) ( A B) ( B) ( A B ) ( AB ) ( B ) ( B A ) ( A ) ( B ) : mdel decrbng he prbably chan rule AI Berln Chen 7

18 Hdden Marv Mdel (cn.) Ueful frmula (Cn.): margnal prbably ( A) all B B f ( A B) ( A B) ( B) all B ( A B) db f ( A B) f ( B) B f db B dree and dn f B cnnuu f E x x...x n ( x x...x ) ( x ) ( x )...( x ) z ( q( z) ) n are ndependen ( z ) q( ) ( z) q( z) dz f z z n z : dcree z : cnnuu B B 3 A B B 5 B 4 Expecan AI Berln Chen 8

19 hree Bac rblem fr HMM Gven an bervan equence O(.. ) and an HMM (SABπ) rblem : Hw effcenly cmpue (O )? Evaluan prblem rblem : Hw che an pmal ae equence S( )? Decdng rblem rblem 3: Hw adu he mdel parameer (ABπ) maxmze (O )? Learnng / ranng rblem AI Berln Chen 9

20 Bac rblem f HMM (cn.) Gven O and fnd (O ) rb[bervng O gven ] Drec Evaluan Evaluang all pble ae equence f lengh ha generang bervan equence O ( O ) ( O S ) ( O S ) ( S ) all S S : he prbably f each pah S By Marv aumpn (Fr-rder HMM) S π all ( ) ( ) a a... 3 a S By chan rule By Marv aumpn AI Berln Chen 0

21 AI Berln Chen Bac rblem f HMM (cn.) Drec Evaluan (cn.) : he n upu prbably alng he pah S By upu-ndependen aumpn he prbably ha a parcular bervan ymbl/vecr emed a me depend nly n he ae and cndnally ndependen f he pa bervan S O b S O By upu-ndependen aumpn

22 Bac rblem f HMM (cn.) Drec Evaluan (Cn.) ( O ) ( S ) ( O S ) all all S ([ π a a... a ][ b ( ) b ( )... b ( )]).. Huge Cmpuan Requremen: O( ) Expnenal cmpuanal cmplexy Cmplexy π ( ) a b ( )... a b ( ) A mre effcen algrhm can be ued evaluae Frward/Bacward rcedure/algrhm b ( - ) 3 : MUL - ( ) b ( ) ADD ( O ) AI Berln Chen

23 Bac rblem f HMM (cn.) Drec Evaluan (Cn.) 3 Sae-me rell Dagram Sae me O O O 3 O - O mean b ( ) ha been cmpued a mean a ha been cmpued AI Berln Chen 3

24 Bac rblem f HMM - he Frward rcedure Bae n he HMM aumpn he calculan f ( ) and ( ) nvlve nly and pble cmpue he lelhd wh recurn n Frward varable : α (... ) he prbably ha he HMM n ae a me havng generang paral bervan AI Berln Chen 4

25 Algrhm. Inalzan. Inducn α 3.ermnan Bac rblem f HMM - he Frward rcedure (cn.) α + Cmplexy: O( ) MUL ADD : : π b ( ) α a b ( ) ( O ) α () Baed n he lace (rell) rucure Cmpued n a me-ynchrnu fahn frm lef--rgh where each cell fr me cmpleely cmpued befre prceedng me + All ae equence regardle hw lng prevuly merge nde (ae) a each me nance + ( + )( - ) + ( - ) ( - ) + ( - ) - AI Berln Chen 5

26 AI Berln Chen 6 Bac rblem f HMM - he Frward rcedure (cn.) () b a b b b b α α fr-rder Marv aumpn B A A B A b B all B A A A A B B A upu ndependen aumpn

27 Bac rblem f HMM - he Frward rcedure (cn.) α 3 (3)( ) [α ()*a 3 + α ()*a 3 +α (3)*a 33 ]b 3 ( 3 ) Sae me O O O 3 O - O mean b ( ) ha been cmpued a mean a ha been cmpued AI Berln Chen 7

28 Bac rblem f HMM - he Frward rcedure (cn.) A hree-ae Hdden Marv Mdel fr he Dw Jne Indural average (0.6* * *0.009)* AI Berln Chen 8

29 Bac rblem f HMM - he Bacward rcedure Bacward varable : β ()( ). Inalzan : β. Inducn: β () a b ( ) β ermnan : Cmplexy MUL: ( O ) π b ( ) β ( -) + ; ADD: ( -) ( -) + AI Berln Chen 9

30 AI Berln Chen 30 Bac rblem f HMM - Bacward rcedure (cn.) Why? β α O β α O () () β α O O

31 Bac rblem f HMM - he Bacward rcedure (cn.) β (3)( 3 4 3) a 3 * b ( 3 )*β 3 () +a 3 * b ( 3 )*β 3 ()+a 33 * b ( 3 )*β 3 (3) 3 Sae me O O O 3 O - O AI Berln Chen 3

32 Bac rblem f HMM Hw che an pmal ae equence S( )? he fr pmal crern: Che he ae are ndvdually m lely a each me Defne a perr prbably varable γ () ( ) O ( O ) m ( ) O ( m O ) α α γ () ( O ) m β ( m) β ( m) Slun : * arg max [γ ()] rblem: maxmzng he prbably a each me ndvdually S* * * * may n be a vald equence (e.g. a * + * 0) AI Berln Chen 3

33 Bac rblem f HMM (cn.) ( 3 3 O )α 3 (3)*β 3 (3) 3 Sae α 3 (3) β 3 (3) me O O O 3 O - O AI Berln Chen 33

34 Bac rblem f HMM - he Verb Algrhm he ecnd pmal crern: he Verb algrhm can be regarded a he dynamc prgrammng algrhm appled he HMM r a a mdfed frward algrhm Inead f ummng up prbable frm dfferen pah cmng he ame denan ae he Verb algrhm pc and remember he be pah Fnd a ngle pmal ae equence S( ) Hw fnd he ecnd hrd ec. pmal ae equence (dffcul?) he Verb algrhm al can be lluraed n a rell framewr mlar he ne fr he frward algrhm Sae-me rell dagram AI Berln Chen 34

35 Algrhm Fnd a Defne a By nducn δ Bac rblem f HMM - he Verb Algrhm (cn.) be ae equence bervan O We can bacrace frm S (.. ) (.. )? new varable δ () max [.... ] he be cre alng a ngle pah a me whch accun [ ] b ( ) maxδ a arg maxδ * a arg max δ fr he fr bervan and end n ae ψ () fr a + gven... Fr bacracng Cmplexy: O( ) AI Berln Chen 35

36 Bac rblem f HMM - he Verb Algrhm (cn.) 3 Sae δ 3 (3) me O O O 3 O - O AI Berln Chen 36

37 Bac rblem f HMM - he Verb Algrhm (cn.) A hree-ae Hdden Marv Mdel fr he Dw Jne Indural average (0.6*0.35)* AI Berln Chen 37

38 Bac rblem f HMM - he Verb Algrhm (cn.) Algrhm n he lgarhmc frm Fnd a Defne a By nducn δ We can bacracefrm S (.. ) (.. )? be ae equence bervan O () max lg [.... ] he be crealng a ngle pah a me whch accun fr hefr bervan and end n ae + + new varable δ ψ.. [ ] lga + lgb ( + ) ( + lga ) max δ + arg max δ * arg maxδ () fr a gven...fr bacracng AI Berln Chen 38

39 Hmewr- A hree-ae Hdden Marv Mdel fr he Dw Jne Indural average Fnd he prbably: (up up unchanged dwn unchanged dwn up ) Fnd he pmal ae equence f he mdel whch generae he bervan equence: (up up unchanged dwn unchanged dwn up) AI Berln Chen 39

40 rbably Addn n F-B Algrhm In Frward-bacward algrhm peran uually mplemened n lgarhmc dman lg + Aume ha we wan add and lg lg( + ) f ele lg lg b b ( lg ) b lg b + lg + lg + b ( lg ) b lg b + lg + lg + b b b x he value f lg can be b + b aved n n a able peedup he peran AI Berln Chen 40

41 rbably Addn n F-B Algrhm (cn.) An example cde #defne LZERO (-.0E0) // ~lg(0) #defne LSMALL (-0.5E0) // lg value < LSMALL are e LZERO #defne mnlgexp -lg(-lzero) duble LgAdd(duble x duble y) { duble empdffz; f (x<y) { emp x; x y; y emp; } dff y-x; //nce ha dff < 0 f (dff<mnlgexp) // f y far maller han x reurn (x<lsmall)? LZERO:x; ele { z exp(dff); reurn x+lg(.0+z); } } AI Berln Chen 4

42 Bac rblem 3 f HMM Inuve Vew Hw adu (re-emae) he mdel parameer (ABπ) maxmze (O )? he m dffcul f he hree prblem becaue here n nwn analycal mehd ha maxmze he n prbably f he ranng daa n a cle frm he daa ncmplee becaue f he hdden ae equence Well-lved by he Baum-Welch (nwn a frward-bacward) algrhm and EM (Expecan-Maxmzan) algrhm Ierave updae and mprvemen AI Berln Chen 4

43 AI Berln Chen 43 Bac rblem 3 f HMM Inuve Vew (cn.) Relan beween he frward and bacward varable () b a... α α a b β β () () β α β α O O

44 Defne a new varable: ξ Bac rblem 3 f HMM Inuve Vew (cn.) ( ) ( O ) + + rbably beng a ae a me and a ae a me + ξ ( ) α ( + O ) ( O ) () a b ( ) β + ( O ) α m n () a b ( ) β Recall he perr prbably varable: ( ) ( O ) γ + + ξ < α + + ( m) a b ( ) β ( n) () ( ) ( ) (fr ) γ Ο e : γ () mn n + + al can be repreene d a p ( A B ) m α α β ( A B ) ( B ) p ( m) β ( m) AI Berln Chen 44

45 Bac rblem 3 f HMM Inuve Vew (cn.) ( O )α 3 (3)*a 3 *b ( 4 )*β (4) 3 Sae me O O O 3 O - O AI Berln Chen 45

46 Bac rblem 3 f HMM Inuve Vew (cn.) ( O ) ξ + ξ ( ) expeced number f rann frm ae ae n O ( ) γ O γ () ξ ( ) expeced number f rann frm ae n A e f reanable re-eman frmula fr {Aπ} O π γ expeced () freqency (number f me) n ae expeced number f rann frm ae ae expeced number f rann frm ae a me a - Frmulae fr Sngle ranng Uerance - ξ γ () AI Berln Chen 46

47 Bac rblem 3 f HMM Inuve Vew (cn.) A e f reanable re-eman frmula fr {B} Fr dcree and fne bervan b (v )( v ) b ( v ) ( v ) expeced number f me n ae and bervng ymbl v expeced number f me n ae uch ha v γ γ Fr cnnuu and nfne bervan b (v)f O S ( v ) b M ( v ) c ( v µ Σ ) M ; c exp µ ( π ) ( v µ ) Σ ( v ) L / Σ Mdeled a a mxure f mulvarae Gauan drbun AI Berln Chen 47

48 AI Berln Chen 48 Bac rblem 3 f HMM Inuve Vew (cn.) Fr cnnuu and nfne bervan (Cn.) Defne a new varable he prbably f beng n ae a me wh he -h mxure cmpnen accunng fr () () M m m m m c c m m m m m m m m ; ; appled) aumpn - ndependen n (berva... Σ µ Σ µ O O O O O O O O β α β α γ γ γ γ γ c c c 3 3 Drbun fr Sae γ γ M m m : e γ γ B B A p B A p

49 Bac rblem 3 f HMM Inuve Vew (cn.) Fr cnnuu and nfne bervan (Cn.) c expeced number f me n ae and mxure expeced number f me n ae M γ m γ ( ) ( m) µ weghed average (mean) f bervan a ae and mxure γ ( ) ( ) γ Σ weghed cvarance f berva n a ae ( ) ( µ )( ) γ µ γ ( ) and mxure Frmulae fr Sngle ranng Uerance AI Berln Chen 49

50 Bac rblem 3 f HMM Inuve Vew (cn.) Fr cnnuu and nfne bervan (Cn.) π a c µ Σ expeced number f me n ae and mxure expeced number f me n ae L expeced weghed average (mean) f ( ) ( µ )( ) γ µ l l freqency L γ l l ( ) (number f me) expeced number f rann frm ae ae expeced number f rann frm ae n ae a me L - l ξ l L - l γ l bervan a ae weghed cvarance f berva n a ae and mxure ( ) L l γ l L M l γ l m ( ) and mxure Frmulae fr Mulple (L) ranng Uerance () L L l γ l () ( m ) L l l L γ γ l ( ) l ( ) AI Berln Chen 50

51 Bac rblem 3 f HMM Inuve Vew (cn.) Fr dcree and fne bervan (cn.) π expeced freqency (number f me) n ae a me ( ) L L l γ l () a expeced number f ran n frm ae ae expeced number f ran n frm ae L - l ξ l L - l γ l () b ( v ) ( v ) expeced number f me n ae and bervng ymbl v expeced number f me n ae L l γ l uch ha v L l γ l Frmulae fr Mulple (L) ranng Uerance AI Berln Chen 5

52 Semcnnuu HMM he HMM ae mxure deny funcn are ed geher acr all he mdel frm a e f hared ernel he emcnnuu r ed-mxure HMM b ae upu rbably f ae M M b f v b ( ) ( µ Σ ) -h mxure wegh -h mxure deny funcn r -h cdewrd f ae (hared acr HMM M very large) (dcree mdel-dependen) A cmbnan f he dcree HMM and he cnnuu HMM A cmbnan f dcree mdel-dependen wegh ceffcen and cnnuu mdel-ndependen cdeb prbably deny funcn Becaue M large we can mply ue he L m gnfcan value f ( v ) Experence hwed ha L ~3% f M adequae aral yng f f fr dfferen phnec cla v AI Berln Chen 5

53 Semcnnuu HMM (cn.) µ Σ b.. b ( ).. b ( M) b.. b.. b ( ) ( M) b 3.. b 3.. b 3 ( ) ( M) µ Σ ( µ Σ ) 3 3 ( µ Σ ) M M AI Berln Chen 53

54 Inalzan f HMM A gd nalzan f HMM ranng : Segmenal K-Mean Segmenan n Sae Aume ha we have a ranng e f bervan and an nal emae f all mdel parameer Sep : he e f ranng bervan equence egmened n ae baed n he nal mdel (fndng he pmal ae equence by Verb Algrhm) Sep : Fr dcree deny HMM (ung M-cdewrd cdeb) b ( ) he number f vecr wh cdeb ndex he number f vecr n ae Fr cnnuu deny HMM (M Gauan mxure per ae) cluer he bervan vecr w µ Σ m m m whn each ae number f vecr clafed n cluer m f ae dvded by he number f vecr n ae Sep 3: Evaluae he mdel cre If he dfference beween he prevu and curren mdel cre greaer han a hrehld g bac Sep herwe p he nal mdel generaed n ae n a e f M ample mean f he vecr clafed n cluer m f ae cluer ample cvarance marx f he vecr clafed n cluer m f ae AI Berln Chen 54

55 Inalzan f HMM (cn.) ranng Daa Mdel Reeman Emae parameer f Obervan va Segmenal K-mean SaeSequence Segmeman Inal Mdel Mdel Cnvergence? O YES Mdel arameer AI Berln Chen 55

56 Inalzan f HMM (cn.) An example fr dcree HMM 3 ae and cdewrd 3 Sae O O O 3 O 4 O 5 O 6 O 7 O 8 O 9 O 0 b (v )3/4 b (v )/4 b (v )/3 b (v )/3 b 3 (v )/3 b 3 (v )/3 v v AI Berln Chen 56

57 Inalzan f HMM (cn.) An example fr Cnnuu HMM 3 ae and 4 Gauan mxure per ae 3 Sae O O O K-mean {µ Σ ω } {µ Σ ω } Glbal mean Cluer mean Cluer mean {µ 3 Σ 3 ω 3 } {µ 4 Σ 4 ω 4 } AI Berln Chen 57

58 HMM plgy Speech me-evlvng nn-anary gnal Each HMM ae ha he ably capure me qua-anary egmen n he nn-anary peech gnal A lef--rgh plgy a naural canddae mdel he peech gnal I general repreen a phne ung 3~5 ae (Englh) and a yllable ung 6~8 ae (Mandarn Chnee) AI Berln Chen 58

59 Knwn Lman f HMM he aumpn f cnvennal HMM n Speech rceng he ae duran fllw an expnenal drbun Dn prvde adequae repreenan f he empral rucure f peech d a a Fr-rder (Marv) aumpn: he ae rann depend nly n he rgn and denan Oupu-ndependen aumpn: all bervan frame are dependen n he ae ha generaed hem n n neghbrng bervan frame Reearcher have prped a number f echnque addre hee lman albe hee lun have n gnfcanly mprved peech recgnn accuracy fr praccal applcan. AI Berln Chen 59

60 Knwn Lman f HMM (cn.) Duran mdelng gemerc/ expnenal drbun emprcal drbun Gamma drbun Gauan drbun AI Berln Chen 60

61 HMM Lman (cn.) he HMM parameer raned by he Baum-Welch algrhm and EM algrhm were nly lcally pmzed Lelhd Curren Mdel Cnfguran Mdel Cnfguran Space AI Berln Chen 6

62 Hmewr-A 0.34 {A:.34B:.33C:.33} {A:.33B:.34C:.33} {A:.33B:.33C:.34} ranse :. ABBCABCAABC. ABCABC 3. ABCA ABC 4. BBABCAB 5. BCAABCCAB 6. CACCABCA 7. CABCABCA 8. CABCA 9. CABCA ranse :. BBBCCBC. CCBABB 3. AACCBBB 4. BBABBAC 5. CCA ABBAB 6. BBBCCBAA 7. ABBBBABA 8. CCCCC 9. BBAAA AI Berln Chen 6

63 Hmewr-A (cn.). leae pecfy he mdel parameer afer he fr and 50h eran f Baum-Welch ranng. leae hw he recgnn reul by ung he abve ranng equence a he eng daa (he -called nde eng). *Yu have perfrm he recgnn a wh he HMM raned frm he fr and 50h eran f Baum-Welch ranng repecvely 3. Whch cla d he fllwng eng equence belng? ABCABCCAB AABABCCCCBBB 4. Wha are he reul f Obervable Marv Mdel were nead ued n and 3? AI Berln Chen 63

64 Ilaed Wrd Recgnn Wrd Mdel M ( X ) M Lelhd f M Wrd Mdel M Speech Sgnal Feaure Exracn Feaure Sequence X ( X M ) Wrd Mdel M V ( X ) M V Lelhd f M Lelhd f M V M Le Wrd Selecr Label ( X ) arg max ( X M ) M ML Verb Apprxman Label ( X ) arg max max ( X S M ) S Wrd Mdel M Sl ( X M ) Sl Lelhd f M Sl AI Berln Chen 64

65 he EM Algrhm {A:.3B:.C:.5} {A:.7B:.C:.} {A:.3B:.6C:.} p(o ) p(o )> p(o ) A B Oberved daa : O : ball equence Laen daa : S : ble equence arameer be emaed maxmze lg(o ) {(A)(B)(B A)(A B)(R A)(G A)(R B)(G B)} AI Berln Chen 65

66 he EM Algrhm (cn.) Inrducn f EM (Expecan Maxmzan): Why EM? Smple pmzan algrhm fr lelhd funcn rele n he nermedae varable called laen ( 隱藏的 )daa In ur cae here he ae equence he laen daa Drec acce he daa neceary emae he parameer mpble r dffcul In ur cae here alm mpble emae {AB π} whu cnderan f he ae equence w Mar Sep : E : expecan wh repec he laen daa ung he curren emae f he parameer and cndned n he E [ ] S O bervan M: prvde a new eman f he parameer accrdng Maxmum lelhd (ML) r Maxmum A err (MA) Crera AI Berln Chen 66

67 ML and MA he EM Algrhm (cn.) Eman prncple baed n bervan: ( x x ) x... he Maxmum Lelhd (ML) rncple fnd he mdel parameer Φ ha he lelhd p( x Φ ) maxmum fr example f Φ { µ Σ} he parameer f a mulvarae nrmal drbun and X..d. (ndependen dencally drbued) hen he ML emae f Φ µ Σ µ ML n x x n Σ ML n n n ( x µ )( x µ ) he Maxmum A err (MA) rncple fnd he mdel parameer Φ ha he lelhd p( Φ x ) maxmum X { X X... } ML X n { } ML AI Berln Chen 67

68 he EM Algrhm (cn.) he EM Algrhm mpran HMM and her learnng echnque Dcver new mdel parameer maxmze he lg-lelhd f ncmplee daa lg ( O ) by eravely maxmzng he expecan f lg-lelhd frm cmplee daalg O S Ung calar randm varable nrduce he EM algrhm he bervable ranng daa O We wan maxmze ( O ) a parameer vecr he hdden (unbervable) daa S E.g. he cmpnen dene f bervable daa O r he underlyng ae equence n HMM AI Berln Chen 68

69 he EM Algrhm (cn.) Aume we have and emae he prbably ha each S ccurred n he generan f O reend we had n fac berved a cmplee daa par ( O S ) wh frequency prprnal he prbably ( O S ) cmpued a new he maxmum lelhd emae f De he prce cnverge? Algrhm ( O S ) ( S O ) ( O ) Lg-lelhd expren and expecan aen ver S lg lg Baye rule cmplee daa lelhd ncmplee daa lelhd S unnwn mdel eng ( O ) lg ( O S ) lg ( S O ) ae expecan ver S ( O ) [ ( S O ) lg ( O )] [ ( S O ) lg ( O S )] [ ( S O ) lg ( S O )] S AI Berln Chen 69

70 he EM Algrhm (cn.) Algrhm (Cn.) We can hu expre a fllw lg We wan ( O ) [ ( S O ) lg ( O S ) ] [ ( S O ) lg ( S O ) ] S Q( ) H ( ) where Q H ( ) [ ( S O ) lg ( O S ) ] S ( ) [ ( S O ) lg ( S O ) ] lg S lg O lg ( O ) lg ( O ) ( O ) lg ( O ) [ Q( ) H ( )] Q( ) H ( ) Q( ) Q( ) H ( ) + H ( ) S [ ] AI Berln Chen 70

71 he EM Algrhm (cn.) ha he fllwng prpery H S S 0 ( ) + H ( ) S ( S O ) ( S O ) lg ( S O ) ( S O ) ( S O ) ( S O ) [ ( S O ) ( S O ) ] 0 ( Qlg x x ) H + H herefre fr maxmzng lg ( O ) we nly need maxmze he Q-funcn (auxlary funcn) Q ( ) H( ) H + ( ) [ ( S O ) lg( O S ) ] S Kullbuac-Lebler (KL) dance Jenen nequaly Expecan f he cmplee daa lg lelhd wh repec he laen ae equence AI Berln Chen 7

72 AI Berln Chen 7 EM Appled Dcree HMM ranng Apply EM algrhm eravely refne he HMM parameer vecr By maxmzng he auxlary funcn Where and can be expreed a ) ( π B A [ ] S S S O O S O S O S O lg lg Q b a b a b a lg lg lg lg lg lg lg lg S O S O S O π π π S O S O

73 AI Berln Chen 73 EM Appled Dcree HMM ranng (cn.) Rewre he auxlary funcn a v b b Q a a Q Q Q Q Q Q all all all lg lg lg lg lg lg O O O S O b O O O S O a O O O S O b a S b S a S π b a π π π π π w y???

74 EM Appled Dcree HMM ranng (cn.) he auxlary funcn cnan hree ndependen erm a and ( ) π Can be maxmzed ndvdually All f he ame frm b F F ( y) g( y y... y ) ( y) ha maxmum value when : w lg y y where w w y and y 0 AI Berln Chen 74

75 AI Berln Chen 75 EM Appled Dcree HMM ranng (cn.) rf: Apply Lagrange Mulpler + + w w y w w y y w 0 y w y F y y lg w y lg w F l l l l l l ha Suppe Mulpler By applyng Lagrange Cnran

76 AI Berln Chen 76 EM Appled Dcree HMM ranng (cn.) he new mdel parameer e can be expreed a: B A π () () () () + v v b a.... γ γ γ ξ γ π O O O O O O

77 EM Appled Cnnuu HMM ranng (cn.) b Cnnuu HMM: he ae bervan de n cme frm a fne e bu frm a cnnuu pace he dfference beween he dcree and cnnuu HMM le n a dfferen frm f ae upu prbably Dcree HMM requre he quanzan prcedure map bervan vecr frm he cnnuu pace he dcree pace Cnnuu Mxure HMM c b M M M c he ae bervan drbun f HMM mdeled by mulvarae Gauan mxure deny funcn (M mxure) c M ; c exp L µ / ( π ) Σ ( µ Σ ) ( µ ) Σ ( ) w w w 3 3 Drbun fr Sae AI Berln Chen 77

78 EM Appled Cnnuu HMM ranng (cn.) b Expre wh repec each ngle mxure cmpnen b e: M ( a ) K ( O S ) π a b + π a + M M... M [ ] ( O S K ) π ( ) a + c b : ne alng f he pble wh he ae mxure equence cmpnen S ( a + a a )( a + a a )...( a + a a ) M M M... [ c b ( )] equence M a M M ( O ) ( O S K ) S K AI Berln Chen 78

79 EM Appled Cnnuu HMM ranng (cn.) herefre an auxlary funcn fr he EM algrhm can be wren a: Q ( ) [ ( S K O ) lg ( O S K )] S S K K ( O S K ) ( O ) lg ( O S K ) ( O S K ) lgπ + + lga lgb ( ) lg + + lgc ( ) Q ( π ) + Q ( a ) + Q ( b ) Q ( c ) Q c π a b + nal prbable ae rann prbable Gauan deny funcn mxure cmpnen AI Berln Chen 79

80 EM Appled Cnnuu HMM ranng (cn.) he nly dfference we have when cmpared wh Dcree HMM ranng Q b M ( b ) ( Ο ) lg b ( ) γ ( ) Q c ( c ) ( Ο ) lg c ( ) M AI Berln Chen 80

81 EM Appled Cnnuu HMM ranng (cn.) Le b γ M ( ) ( Ο ) ( ) ( ; µ Σ ) ( π ) L Σ exp ( µ ) Σ ( µ ) lg b lg b µ Q b µ L lg ( π ) + lg Σ ( µ ) Σ ( µ ) ( b ) µ ( ) ( µ ) { γ ( ) Σ ( µ )} Σ [ γ ( ) ] γ ( ) M γ ( ) lg b ( ) µ 0 ( C + C ) d ( x Cx) x dx and ymmerc here Σ AI Berln Chen 8

82 EM Appled Cnnuu HMM ranng (cn.) lg Q b lg L lg ( π ) lg Σ ( µ ) Σ ( µ ) ( Σ ) b ( b ) ( Σ ) γ ( ) [ Σ Σ ( µ )( µ ) Σ ] ( ) lg b ( ) ( Σ ) [ Σ ] Σ Σ ( µ )( µ ) ( ) Σ γ ( ) Σ ( µ )( µ ) ( µ )( µ ) ( ) Σ Σ Σ γ ( ) Σ Σ ( µ )( µ ) b Σ γ γ [ γ ( ) ( µ )( µ ) ] Σ M γ γ Σ ( ) Σ Σ Σ 0 Σ Σ Σ d ( a X b) X ab X dx d[ de( X )] de( X ) X dx and ymmerc here Σ AI Berln Chen 8

83 EM Appled Cnnuu HMM ranng (cn.) he new mdel parameer e fr each mxure cmpnen and mxure wegh can be expreed a: µ Σ c ( O ) ( O ) ( O ) ( O ) ( O ) ( )( ) µ µ ( O ) ( O ) ( O ) γ ( ) M γ ( ) [ γ ] γ ( ) [ γ ( )( µ )( µ )] γ ( ) AI Berln Chen 83

84 Meaure f ASR erfrmance Evaluang he perfrmance f aumac peech recgnn (ASR) yem crcal and he Wrd Recgnn Errr Rae (WER) ne f he m mpran meaure here are ypcally hree ype f wrd recgnn errr Subun An ncrrec wrd wa ubued fr he crrec wrd Delen A crrec wrd wa med n he recgnzed enence Inern An exra wrd wa added n he recgnzed enence Hw deermne he mnmum errr rae? AI Berln Chen 84

85 Meaure f ASR erfrmance (cn.) Calculae he WER by algnng he crrec wrd rng agan he recgnzed wrd rng A maxmum ubrng machng prblem Can be handled by dynamc prgrammng Example: deleed Crrec : he effec clear Recgnzed: effec n clear WER+ WAR 00% Errr analy: ne delen and ne nern Meaure: wrd errr rae (WER) wrd crrecn rae (WCR) wrd accuracy rae (WAR) Wrd Wrd Wrd mached nered mached Mgh be hgher han 00% Sub. + Del. + In. wrd Errr Rae 00% 50%. f wrd n he crrec enence 4 Mached wrd 3 Crrecn Rae 00% 75%. f wrd n he crrec enence 4 Mached - In. wrd 3 Accuracy Rae 00% 50%. f wrd n he crrec enence 4 Mgh be negave AI Berln Chen 85

86 Meaure f ASR erfrmance (cn.) A Dynamc rgrammng Algrhm (exb) //dene fr he wrd lengh f he crrec/reference enence //dene fr he wrd lengh f he recgnzed/e enence mnmum wrd errr algnmen a he a grd [] /h nd f algnmen /h e Ref AI Berln Chen 86

87 Meaure f ASR erfrmance (cn.) Algrhm (by Berln Chen) Sep : Inalzan : Ref G[0][0] 0; fr... n { //e G[][0] G[ -][0] + ; B[][0] ; //Inern e } (Hrznal Drecn) fr... m { //reference G[0][] G[0][ -] + ; B[0][] ; // Delen } (Vercal Drecn) Sep : Ieran : fr...n { //e fr...m { //reference G[ -][] + (Inern) G[][-] (Delecn) G[][] mn + G[ -][-] + (f LR[]! L[] Subun) G[ -][-] (f LR[] L[] Mach) ; //Inern (Hrznal Drecn) ; //Delen (Vercal Drecn) B[][] 3;//Subun (Dagnal Drecn) 4; //mach (Dagnal Drecn) } //fr reference } //fr e Sep 3 : Meaure G[n][m] Wrd Errr Rae 00% m Wrd Accuracy Rae 00 % Wrd Opmal f ele f ele and Bacrace bacrace B[][] B[][] : pah (B[n][m]... B[0][0]) prn " prn " LR[] prn " LR[] Errr Rae L[]" ; //Iner "; //Delen LR[] "; //H/Mac e: he penale fr ubun delen and nern errr are all e be here n hen g lef hen g dwn h r Subu n hen g dwn dagnally AI Berln Chen 87

88 Crrec/Reference Wrd Sequence Meaure f ASR erfrmance (cn.) A Dynamc rgrammng Algrhm Inalzan fr (;<m;++) { //reference grd[0][] grd[0][-]; grd[0][].dr VER; grd[0][].cre + Delen; grd[0][].del ++; } HK m m Del. 3 Del. Del. 0 grd[0][0].cre grd[0][0].n grd[0][0].del 0; grd[0][0].ub grd[0][0].h 0; grd[0][0].dr IL; 0 Del. (-) (--) In. () Del. (-) In. (nm) n- n In. In. 3In. Recgnzed/e Wrd fr (;<n;++) { // e Sequence grd[][0] grd[-][0]; grd[][0].dr HOR; grd[][0].cre +Inen; grd[][0].n ++; } AI Berln Chen 88 Del.

89 Meaure f ASR erfrmance (cn.) HK rgram fr (;<n;++) //e { grd grd[]; grd grd[-]; fr (;<m;++) //reference { h grd[].cre +nen; d grd[-].cre; f (lref[]! le[]) d + uben; v grd[-].cre + delen; f (d<h && d<v) { /* DIAG h r ub */ grd[] grd[-]; grd[].cre d; grd[].dr DIAG; f (lref[] le[]) ++grd[].h; ele ++grd[].ub; } ele f (h<v) { /* HOR n */ grd[] grd[]; grd[].cre h; grd[].dr HOR; ++ grd[].n; } ele { /* VER del */ grd[] grd[-]; grd[].cre v; grd[].dr VER; ++grd[].del; } } /* fr */ } /* fr */ //rucure agnmen //rucure agnmen //rucure agnmen Example Crrec (0500) (0400) (0300) (000) (000) Crrec: e: C C B C A (040) (030) (00) (00) (000) (InDelSubH) (03) (0) (0) r (00) (0) r(000) (00) (0) r (30) H A e 0 0 In B B A B C (0000) (000) (000) (3000) (4000) In B (0) (0) H B (0) Del C (00) A C B C C B A B C H A Del C H B H C Del C (03) (03) H C (0) r (0) (00) (300) Delee C Algnmen : WER 60% Sll have an Oher pmal algnmen! AI Berln Chen 89

90 Meaure f ASR erfrmance (cn.) Example e: he penale fr ubun delen and nern errr are all e be here (InDelSubH) Algnmen : WER 80% Crrec: e: Crrec: e: In B (0500) (0400) (0300) (000) (000) C C B C A (0000) 0 0 A C B C C B A A C H A Del C Sub B H C A C B C C B A A C Sub A Sub C Sub B H C Crrec Del C Del C (040) (030) (00) (00) (03) (0) (0) r (00) (0) r(000) (0) r (30) Del C () () Sub B (0) () () (0) r (0) (00) Delee C H C (000) (00) (00) (300) H A e In B B A A C (000) (000) (3000) (4000) Algnmen 3: WER80% Algnmen : WER80% Crrec: e: In B A C B C C B A A C H A Sub C Del B H C Del C AI Berln Chen 90

91 Meaure f ASR erfrmance (cn.) w cmmn eng f dfferen penale fr ubun delen and nern errr /* HK errr penale */ uben 0; delen 7; nen 7; /* IS errr penale*/ ubenis 4; delenis 3; nenis 3; AI Berln Chen 9

92 Hmewr-B Meaure f ASR erfrmance Reference 桃 芝 颱 風 重 創 花 蓮 光 復 鄉 大 興 村 死 傷 慘 重 感 觸 最 多 ASR Oupu 桃 芝 颱 風 重 創 花 蓮 光 復 鄉 打 新 村 次 傷 殘 周 感 觸 最 多. AI Berln Chen 9

93 Hmewr-B (cun.) 506 B re f ASR upu Repr he CER (characer errr rae) f he fr ne and 506 re he reul huld hw he number f ubun delen and nern errr Overall Reul SE: %Crrec0.00 [H0 S ] WORD: %Crr8.5 Acc8.5 [H75 D4 S3 I0 9] Overall Reul SE: %Crrec0.00 [H0 S00 00] WORD: %Crr87.66 Acc86.83 [H083 D77 S348 I0 357] Overall Reul SE: %Crrec0.00 [H0 S00 00] WORD: %Crr87.9 Acc87.8 [H657 D93 S84 I ] Overall Reul SE: %Crrec0.00 [H0 S ] WORD: %Crr86.83 Acc86.06 [H5744 D89 S7839 I ] AI Berln Chen 93

94 Symbl fr Mahemacal Operan AI Berln Chen 94