Speech Recognition. Berlin Chen Department of Computer Science & Information Engineering National Taiwan Normal University

Transcription

1 Reference: Hdden Marv Mdel fr Speech Recgnn Berln Chen Deparmen f Cmpuer Scence & Infrman Engneerng anal awan rmal Unvery. Rabner and Juang. Fundamenal f Speech Recgnn. Chaper 6. Huang e. al. Spen Language rceng. Chaper Rabner. A ural n Hdden Marv Mdel and Seleced Applcan n Speech Recgnn. rceedng f he IEEE vl. 77. February Gale and Yung. he Applcan f Hdden Marv Mdel n Speech Recgnn Chaper Yung. HMM and Relaed Speech Recgnn echnlge. Chaper 7 Sprnger Handb f Speech rceng Sprnger J.A. Blme A Genle ural f he EM Algrhm and Applcan arameer Eman fr Gauan Mxure and Hdden Marv Mdel U.C. Bereley R-97-0

2 Hry Hdden Marv Mdel (HMM: A Bref Overvew ublhed n paper f Baum n lae 960 and early 970 Inrduced peech prceng by Baer (CMU and Jelne (IBM n he 970 (dcree HMM hen exended cnnuu HMM by Bell Lab Aumpn Speech gnal can be characerzed a a paramerc randm (chac prce arameer can be emaed n a prece well-defned d manner hree fundamenal prblem Evaluan f prbably (lelhd f a equence f bervan gven a pecfc HMM Adumen f mdel parameer a be accun fr berved gnal Deermnan f a be equence f mdel ae S - Berln Chen

3 Schac rce A chac prce a mahemacal mdel f a prbablc expermen ha evlve n me and generae a equence f numerc value Each numerc value n he equence mdeled by a randm varable A chac prce u a (fne/nfne equence f randm varable Example (a he equence f recrded value f a peech uerance (b he equence f daly prce f a c (c he equence f hurly raffc lad a a nde f a cmmuncan newr (d he equence f radar meauremen f he pn f an arplane S - Berln Chen 3

4 Obervable Marv Mdel Obervable Marv Mdel (Marv Chan Fr-rder Marv chan f ae a rple (SAπ S a e f ae A he marx f rann prbable beween ae ( - - ( - A π he vecr f nal ae prbable π ( he upu f he prce he e f ae a each nan f me when each ae crrepnd an bervable even he upu n any gven ae n randm (deermnc! mple decrbe he peech gnal characerc Fr-rder and me-nvaran aumpn S - Berln Chen 4

5 ( S S S ( S S S Obervable Marv Mdel (cn. ( S S ( S S ( S S S S S S (rev. Sae Cur. Sae S S ( S SS S S ( S SS ( S S S ( S SS ( S S S ( S S S S S S S Fr-rder Marv chan f ae Secnd-rder Marv chan f ae S - Berln Chen 5

6 Obervable Marv Mdel (cn. Example : A 3-ae Marv Chan Sae generae ymbl A nly Sae generae ymbl B nly and Sae 3 generae ymbl C nly 0.6 A π A [ 0.5 ] B C B Gven a equence f berved ymbl O{CABBCABC} he nly ne crrepndng ae equence {S 3 S S S S 3 S S S 3 } and he crrepndng prbably (O (S 3 (S S 3 (S S (S S (S 3 S (S S 3 (S S (S 3 S S - Berln Chen 6

7 Obervable Marv Mdel (cn. Example : A hree-ae Marv chan fr he Dw Jne Indural average he prbably f 5 cnecuve up day ( 5 cnecuve up day π a a a 0.5 ( a π ( π S - Berln Chen 7

8 Obervable Marv Mdel (cn. Example 3: Gven a Marv mdel wha he mean ccupancy duran f each ae ( d prbably ma funcn f d ( a ( a Expeced number f d duran n a ae duran d d d a d ( a d n ae d ( d d( a ( a ( a ( a a a a a rbably d me (Duran S - Berln Chen 8

9 Hdden Marv Mdel S - Berln Chen 9

10 Hdden Marv Mdel (cn. HMM an exended vern f Obervable Marv Mdel he bervan urned be a prbablc funcn (dcree r cnnuu f a ae nead f an ne--ne crrepndence f a ae he mdel a dubly embedded chac prce wh an underlyng chac prce ha n drecly bervable (hdden Wha hdden? he Sae Sequence! Accrdng he bervan equence we are n ure whch ae equence generae! Elemen f an HMM (he Sae-Oupu HMM {SABπ} S a e f ae A he marx f rann prbable beween ae B a e f prbably funcn each decrbng he bervan prbably wh repec a ae π he vecr f nal ae prbable S - Berln Chen 0

11 Hdden Marv Mdel (cn. w mar aumpn Fr rder (Marv aumpn he ae rann depend nly n he rgn and denan me-nvaran ( ( τ τ ( A Oupu-ndependen aumpn All bervan are dependen n he ae ha generaed hem n n neghbrng bervan ( K ( K + + S - Berln Chen

12 Hdden Marv Mdel (cn. w mar ype f HMM accrdng he bervan Dcree and fne bervan: he bervan ha all dnc ae generae are fne n number V{v v v 3 v M } v R L In h cae he e f bervan prbably drbun B{b (v } defned d a b (v ( v M M : bervan a me : ae a me fr ae b (v cn f nly M prbably value A lef--rgh HMM S - Berln Chen

13 Hdden Marv Mdel (cn. w mar ype f HMM accrdng he bervan Cnnuu and nfne bervan: b he bervan ha all dnc ae generae are nfne and cnnuu ha V{v { v R d } In h cae he e f bervan prbably drbun B{b{ (v} defned a b ( (vf O S( ( v b (v a cnnuu prbably deny funcn (pdf and fen a mxure f Mulvarae Gauan (rmal Drbun π Σ M ( v w ( ( d exp v μ Σ v μ Mxure Wegh Cvarance Marx Mean Vecr Obervan Vecr S - Berln Chen 3

14 Hdden Marv Mdel (cn. Mulvarae Gauan Drbun When X(X X X d a d-dmennal randm vecr he mulvarae Gauan pdf ha he frm: f ( X x μ Σ ( x; μ Σ d exp ( x μ Σ ( x μ Σ where μ he L - dmennal Σ he cverance marx Σ and he - Σ he he deermnan f h elevmen σ f Σ σ ( π mean vecr μ E[ x] E[ ( x μ( x μ ] E[ xx ] μμ Σ E [( x μ ( x μ ] E[ xx ] μ μ If X X X d are ndependen he cvarance marx reduced dagnal cvarance he drbun a d ndependen calar Gauan drbun Mdel cmplexy gnfcanly reduced S - Berln Chen 4

15 Hdden Marv Mdel (cn. Mulvarae Gauan Drbun S - Berln Chen 5

16 Hdden Marv Mdel (cn. Cvarance marx f he crrelaed feaure vecr (Mel-frequency fler ban upu Cvarance marx f he parally de-crrelaed feaure vecr (MFCC whu C 0 MFCC: Mel-frequency eque cepral ceffcen S - Berln Chen 6

17 Hdden Marv Mdel (cn. Mulvarae Mxure Gauan Drbun (cn. Mre cmplex drbun wh mulple lcal maxma can be apprxmaed by Gauan (a unmdal drbun mxure f M M ( x w ( x; μ Σ w Gauan mxure wh enugh mxure cmpnen can apprxmae any drbun S - Berln Chen 7

18 Hdden Marv Mdel (cn. Example 4: a 3-ae dcree HMM 0.6 Ergdc HMM E A b A 0.3 b B 0. b b A 0.7 b B 0. b b π 3 ( C 0.5 ( C 0. ( A 0.3 b3 ( B 0.6 b3 ( C 0. [ ] {A:.7B:.C:.} {A:.3B:.C:.5} 0.5 {A:.3B:.6C:.} Gven a equence f bervan O{ABC} here are 7 pble crrepndng ae equence and herefre he crrepndng prbably 7 7 ( O ( O S ( O S ( S S : ae equence. g. when S { 3} ( O S ( A ( B ( C 3 ( S ( ( ( 0.5*0.7 * *0.* S - Berln Chen 8

19 an : Hdden Marv Mdel (cn. O{ 3 }: he bervan (feaure equence S { 3 } : he ae equence : mdel fr HMM {ABπ} Bπ} (O : he prbably f bervng O gven he mdel (O S : he prbably f bervng O gven and a ae equence S f (OS : he prbably f bervng O and S gven (S O : he prbably f bervng S gven O and Ueful frmula Baye Rule : ( A B ( A ( AB B A ( B ( B ( A B ( B A ( A ( A B ( B ( A B ( AB B ( A ( B B A : mdel dldecrbng he prbably bl chan rule S - Berln Chen 9

20 Hdden Marv Mdel (cn. Ueful frmula (Cn.: al rbably herem margnal prbably ( A all B B f ( A B ( A B ( B all B ( A B db f ( A B f ( B B f db B dree and dn f B cnnuu B B 3 f x x...xn are ndependen A ( x x... xn ( x ( x... ( x n B B 5 B 4 E z [ q( z ] ( z q( ( z q ( z dz f z z z z : dcree : cnnuu Venn Dagram Expecan S - Berln Chen 0

21 hree Bac rblem fr HMM Gven an bervan equence O(.. and an HMM (SABπ rblem : Hw effcenly cmpue (O? Evaluan prblem rblem : Hw che an pmal ae equence S(? Decdng rblem rblem 3: Hw adu he mdel parameer (ABπ maxmze (O? Learnng / ranng rblem S - Berln Chen

22 Bac rblem f HMM (cn. Gven O and fnd (O rb[bervng O gven ] Drec Evaluan Evaluang all pble ae equence f lengh ha generang bervan equence O ( O ( O S ( O S ( S all S S : he prbably f each pah S By Marv aumpn (Fr-rder HMM S π all ( ( a a a 3... S By chan rule By Marv aumpn S - Berln Chen

23 Bac rblem f HMM (cn. Drec Evaluan (cn. O S : he n upu prbably alng he pah S By upu-ndependen aumpn he prbably ha a parcular bervan ymbl/vecr emed a me depend nly n he ae and cndnally ndependen f he pa bervan O S b ( By upu-ndependen aumpn ( S - Berln Chen 3

24 Bac rblem f HMM (cn. Drec Evaluan (Cn. ( b ( ( O ( S ( O S all S all ( [ π a a... a ] [ b ( b (... b ( ] ( a b (... a b ( π b..... Huge Cmpuan Requremen: O( 3 Expnenal cmpuanal cmplexy Cmplexy ( - : MUL - ADD A mre effcen algrhm can be ued evaluae Frward/Bacward rcedure/algrhm ( O S - Berln Chen 4

25 Bac rblem f HMM (cn. Drec Evaluan (Cn. 3 Sae Sae-me rell Dagram me O O O 3 O - O mean b ( ha been cmpued a mean a ha been cmpued S - Berln Chen 5

26 Bac rblem f HMM - he Frward rcedure Bae n he HMM aumpn he calculan f ( and ( nvlve nly and pble cmpue he lelhd wh recurn n Frward varable : α (... he prbably ha he HMM n ae a me havng generang paral bervan S - Berln Chen 6

27 Algrhm Bac rblem f HMM - he Frward rcedure (cn.. Inalzan α ( π b (. Inducn α 3.ermnan + α a b ( + - ( O α ( Cmplexy: O( MUL ADD : : ( + ( - + ( - ( - + ( - Baed n he lace (rell rucure Cmpued n a me-ynchrnu fahn frm lef--rgh where each cell fr me cmpleely cmpued befre prceedng me + All ae equence regardle hw lng prevuly merge nde (ae a each me nance S - Berln Chen 7

28 Bac rblem f HMM he Frward rcedure (cn - he Frward rcedure (cn. A A B B A upu α upu ndependen aumpn B A A B A b b b b B all B A A b b b... fr rder S - Berln Chen 8 ( b a α fr-rder Marv aumpn

29 Bac rblem f HMM - he Frward rcedure (cn. α 3 (3( [α (*a 3 + α (*a 3 +α (3*a 33 ]b 3 ( 3 3 Sae me O O O 3 O - O mean b ( ha been cmpued a mean a ha been cmpued S - Berln Chen 9

30 Bac rblem f HMM - he Frward rcedure (cn. A hree-ae Hdden Marv Mdel fr he Dw Jne Indural average (0.6* * *0.009* S - Berln Chen 30

31 Bac rblem f HMM - he Bacward rcedure Bacward varable : β (( Inalzan : β ( a b -. Inducn: β + β 3. ermnan : Cmplexy MUL: ADD: ( O π b ( β + ( - + ( - ( - + ; S - Berln Chen 3

32 Bac rblem f HMM Bacward rcedure (cn - Bacward rcedure (cn. Why? β α O Why? β α β α O O ( ( β α O O S - Berln Chen 3

33 Bac rblem f HMM - he Bacward rcedure (cn. β (3( 3 ( 3 4 a 3 * b ( 3 *β 3 ( +a 3 * b ( 3 *β 3 (+a 33 * b ( 3 *β 3 (3 3 Sae me O O O 3 O - O S - Berln Chen 33

34 Bac rblem f HMM Hw che an pmal ae equence S((? he fr pmal crern: Che he ae are ndvdually m lely a each me Defne a perr prbably varable γ ( ( O γ ( ( O O α β ( O ( m O α ( m β ( m m m ae ccupan prbably (cun a f algnmen f HMM ae he bervan (feaure Slun : * arg max [γ (] rblem: maxmzng he prbably a each me ndvdually S* * * * may n be a vald equence (e.g. a * + * 0 S - Berln Chen 34

35 Bac rblem f HMM (cn. ( 3 3O αα 3 (3*ββ 3 (3 3 Sae 3 3 α 3 (3 3 β 3 (3 3 3 a me O O O 3 O - O S - Berln Chen 35

36 Bac rblem f HMM - he Verb Algrhm he ecnd pmal crern: he Verb algrhm can be regarded a he dynamc prgrammng algrhm appled he HMM r a a mdfed frward algrhm Inead f ummng up prbable frm dfferen pah cmng he ame denan ae he Verb algrhm pc and remember he be pah Fnd a ngle pmal ae equence S( Hw fnd he ecnd hrd ec. pmal ae equence (dffcul? he Verb algrhm al can be lluraed n a rell framewr mlar he ne fr he frward algrhm Sae-me rell dagram S - Berln Chen 36

37 Algrhm Fnd a bervan O Defne a Bac rblem f HMM - he Verb Algrhm (cn. be ae equence S (.. (..? new varable δ By nducn fr a gven ( max [.... ].. We can bacrace frm he be cre alng a ngle pah a me whch accun fr he fr bervan and end n ae [ ] b ( maxδ a arg maxδ δ + + ψ + * a arg max δ... Fr bacracng Cmplexy: O( S - Berln Chen 37

38 Bac rblem f HMM - he Verb Algrhm (cn. 3 Sae δ 3 ( me O O O 3 O - O S - Berln Chen 38

39 Bac rblem f HMM - he Verb Algrhm (cn. A hree-ae Hdden Marv Mdel fr he Dw Jne Indural average (0.6*0.35* S - Berln Chen 39

40 Bac rblem f HMM - he Verb Algrhm (cn. Algrhm n he lgarhmc frm Fnd a bervan O Defne a S (.. (..? be ae equence new varable δ By nducn fr a gven ( max lg [.... ].. he be crealng a ngle pah a me whch accun fr hefr bervan and end n ae [ ( + lga ] gb ( + arg max( δ + lga ( max δ ( δ + + lg + ψ + We can bacracefrm * arg max δ (...Fr bacracng S - Berln Chen 40

41 Hmewr- A hree-ae Hdden Marv Mdel fr he Dw Jne Indural average Fnd he prbably: (up up unchanged dwn unchanged dwn up Fnd he pmal ae equence f he mdel whch generae he bervan equence: (up up unchanged dwn unchanged dwn up S - Berln Chen 4

42 rbably Addn n F-B Algrhm In Frward-bacward algrhm peran uually mplemened n lgarhmc dman lg + Aume ha we wan add and lg lg( + f lg ele lg b b ( lg b lg b + lg + lg + b lg b lg b ( + lg + lg ( + b b b x he value f lg can be b + b aved n n a able peedup he peran S - Berln Chen 4

43 rbably Addn n F-B Algrhm (cn. An example cde #defne LZERO (-.0E0 // ~lg(0 #defne LSMALL (-0.5E0 // lg value < LSMALL are e LZERO #defne mnlgexp -lg(-lzero // ~-3 duble LgAdd(duble x duble y { duble empdffz; f (x<y { emp x; x y; y emp; } dff y-x; //nce ha dff < 0 f (dff<mnlgexp // f y far maller han x reurn (x<lsmall? LZERO:x; ele { z exp(dff; reurn x+lg(.0+z; } } S - Berln Chen 43

44 Bac rblem 3 f HMM Inuve Vew Hw adu (re-emae he mdel parameer (ABπ maxmze (O OL r lg(o OL? Belngng a ypcal prblem f nferenal ac he m dffcul f he hree prblem becaue here n nwn analycal mehd ha maxmze he n prbably f he ranng daa n a cle frm L ( O O... OL lg ( Ol lg O L lg R ( Ol lg( S ( Ol S l l l all S - Suppe ha we have L ranng uerance fr he HMM -S :a pble ae equence f he HMM he lg f um frm dffcul deal wh he daa ncmplee becaue f he hdden ae equence Well-lved by he Baum-Welch (nwn a frward-bacward algrhm ag and EM (Expecan-Maxmzan a a a ag algrhm Ierave updae and mprvemen Baed n Maxmum Lelhd (ML crern S - Berln Chen 44

45 Maxmum Lelhd (ML Eman: A Schemac Depcn (/ Hard Agnmen Gven he daa fllw a mulnmal drbun (B S /40.5 Sae S (W S /40.5 S - Berln Chen 45

46 Maxmum Lelhd (ML Eman: A Schemac Depcn (/ Sf Agnmen Gven he daa fllw a mulnmal drbun Maxmze he lelhd f he daa gven he algnmen γ Sae S O γ ( ( O Sae S γ ( + γ ( (B S ( / ( (B S (0.3+0./.6/ ( /.50.7 (W S ( / ( / (W S ( / 0.5/ ( / S - Berln Chen 46

47 Bac rblem 3 f HMM Inuve Vew (cn Inuve Vew (cn. Relanhp beween he frward and bacward varable ( b a... α α a b β β + + β β α O ( ( β α β α O O S - Berln Chen 47

48 Defne a new varable: Bac rblem 3 f HMM Inuve Vew (cn. ( ( O ξ + + rbably beng a ae a me and a ae a me + ξ ( α ( + ( O O p ( A B ( a b ( β α ( a b ( β + + Recall he perr prbably varable: ( ( O γ + + ( O α ( m a b ( ( n m n e + ξ < ( ( ( (fr γ Ο : γ ( mn n + β+ al can be repreene d a α m α ( A B ( B p β ( m β ( m S - Berln Chen 48

49 Bac rblem 3 f HMM Inuve Vew (cn. ( O αα 3 (3*a a 3 *bb ( 4 *ββ (4 3 Sae me O O O 3 O - O S - Berln Chen 49

50 Bac rblem 3 f HMM Inuve Vew (cn. ξ ( ( O + ξ ( expeced number f rann frm ae ae n O γ ( ( O γ ( ξ ( expeced number f rann frm ae n A e f reanable re-eman eman frmula fr {Aπ} O π expeced freqency (number f me n ae a me γ ( - ξ a - expeced number f rann frm ae ae expeced number f rann frm ae Frmulae fr Sngle ranng Uerance γ S - Berln Chen 50

51 Bac rblem 3 f HMM Inuve Vew (cn. A e f reanable re-eman frmula fr {B} Fr dcree and fne bervan b (v ( v b ( v ( v expeced number f me n ae and bervng ymbl v expeced number f me n ae γ uch ha v γ ( Fr cnnuu and nfne bervan b (vf O S ( v b M ; c exp v μ Σ v μ L / π Σ M ( v c ( v μ Σ Mdeled a a mxure f mulvarae Gauan drbun S - Berln Chen 5

52 Bac rblem 3 f HMM Inuve Vew (cn Inuve Vew (cn. Fr cnnuu and nfne bervan (Cn. B B A p B A p Defne a new varable he prbably f beng n ae a me wh he h mxure cmpnen accunng fr γ γ wh he -h mxure cmpnen accunng fr m m O O O γ c p m p m m O O O O O γ γ c c 3 3 p m p m p aumpn appled (bervan - ndependen O O O γ Drbun fr Sae M p m p m (bervan - ndependen aumpn appled... γ M m m : e γ γ S - Berln Chen 5 ( ( M m m m m c c ; ; Σ μ Σ μ β α β α

53 Bac rblem 3 f HMM Inuve Vew (cn. Fr cnnuu and nfne bervan (Cn. expeced number f me n ae and mxure expeced number f me n ae c M γ m γ ( ( m μ weghed average (mean f bervan a ae and mxure γ ( ( γ Σ weghed cvarance f berva n a ae γ ( ( μ ( μ γ ( and mxure Frmulae fr Sngle ranng Uerance S - Berln Chen 53

54 Bac rblem 3 f HMM Inuve Vew (cn. Mulple ranng Uerance F/B F/B F/B 台師大 3 S - Berln Chen 54

55 Bac rblem 3 f HMM Inuve Vew (cn. Fr cnnuu and nfne bervan (Cn. π expeced freqency (number f me n ae a me ( L L l γ l a expeced number f rann frm ae ae expeced number f rann frm ae L l - l ξ l L l - l γ l ( c expeced number f me n ae and mxure expeced number f me n ae L l l γ l L l M l γ l m ( ( m μ weghed average (mean f bervan a ae and mxure L l l γ l L l l l γ ( ( Σ weghed cvarance f berva n a ae L γ l l l ( ( μ ( μ L l γ l l ( and mxure Frmulae fr Mulple (L ranng Uerance S - Berln Chen 55

56 Bac rblem 3 f HMM Inuve Vew (cn. Fr dcree and fne bervan (cn. π expeced freqency (number f me n ae a me ( L L l γ l ( a expeced dnumber f ran n frm ae ae expeced number f ran n frm ae L - l l ξ l L - l l γ l ( ( b ( v ( v expeced number f me n ae and bervng ymbl v expeced number f me n ae L l l γ l uch ha v L l l γ l Frmulae fr Mulple (L ranng Uerance S - Berln Chen 56

57 Semcnnuu HMM he HMM ae mxure deny funcn are ed geher acr all he mdel frm a e f hared ernel he emcnnuu r ed-mxure HMM b ae upu rbably f ae M M b f v b ( ( μ Σ -h mxure wegh -h mxure deny funcn r -h cdewrd f ae (hared acr HMM M very large (dcree mdel-dependen dependen A cmbnan f he dcree HMM and he cnnuu HMM A cmbnan f dcree mdel-dependen wegh ceffcen and cnnuu mdel-ndependen cdeb prbably deny funcn Becaue M large we can mply ue he L m gnfcan value f ( v Experence hwed ha L ~3% f M adequae aral yng f f fr dfferen phnec cla v S - Berln Chen 57

58 Semcnnuu HMM (cn. μ Σ 3 b.. b (.. b ( M b b.. b (.. b ( M b 3.. b 3 (.. b ( M 3 μ Σ Σ b ( μ 3 ( μ Σ M M S - Berln Chen 58

59 HMM plgy Speech me-evlvng nn-anary gnal Each HMM ae ha he ably capure me qua-anary egmen n he nn-anary peech gnal A lef--rgh plgy a naural canddae mdel he peech gnal (al called he bead-n-a-rng mdel I general repreen a phne ung 3~5 ae (Englh and a yllable ung 6~8 ae (Mandarn Chnee S - Berln Chen 59

60 Inalzan f HMM A gd nalzan f HMM ranng : Segmenal K-Mean Segmenan n Sae 3 Aume ha we have a ranng e f bervan and an nal emae f all mdel parameer Sep : he e f ranng bervan equence egmened n ae baed n he nal mdel (fndng he pmal ae equence by Verb Algrhm Sep : Fr dcree deny HMM (ung M-cdewrd cdeb he number f vecr wh cdeb ndex n ae b ( he number f vecr n ae Fr cnnuu deny HMM (M Gauan mxure per ae cluer he bervan vecr w μ m m whn each ae number f vecr clafed n cluer m f ae dvded by he number f vecr n ae n a e f ample mean f he vecr clafed n cluer m f ae M cluer Σ m ample cvarance marx f he vecr clafed n cluer m f ae Sep 3: Evaluae he mdel cre If he dfference beween he prevu and curren mdel cre greaer han a hrehld g bac Sep herwe p he nal mdel generaed S - Berln Chen 60

61 Inalzan f HMM (cn. ranng Daa Mdel Reeman Emae parameer f Obervan va Segmenal K-mean SaeSequence Segmeman Inal Mdel Mdel Cnvergence? O YES Mdel arameer S - Berln Chen 6

62 Inalzan f HMM (cn. An example fr dcree HMM 3 ae and cdewrd 3 Sae O O O 3 O 4 O 5 O 6 O 7 O 8 O 9 O 0 b (v 3/4 b (v /4 b (v /3 b (v /3 b 3 (v /3 b 3 (v /3 v v S - Berln Chen 6

63 Inalzan f HMM (cn. An example fr Cnnuu HMM 3 ae and 4 Gauan mxure per ae 3 Sae O O O K-mean {μ Σ ω } {μ ΣΣ ωω } Glbal mean Cluer mean Cluer mean {μ 3 Σ 3 ω 3 } {μ 4 Σ 4 ω 4 } S - Berln Chen 63

64 Knwn Lman f HMM (/3 he aumpn f cnvennal HMM n Speech rceng he ae duran fllw an expnenal drbun Dn prvde adequae repreenan f he empral rucure f peech d ( a ( a Fr-rder (Marv aumpn: he ae rann depend nly n he rgn and denan Oupu-ndependen aumpn: all bervan frame are dependen n he ae ha generaed hem n n neghbrng bervan frame Reearcher have prped a number f echnque addre hee lman albe hee lun have n gnfcanly mprved peech recgnn accuracy fr praccal applcan. S - Berln Chen 64

65 Knwn Lman f HMM (/3 Duran mdelng gemerc/ expnenal drbun emprcal drbun Gamma drbun Gauan drbun S - Berln Chen 65

66 Knwn Lman f HMM (3/3 he HMM parameer raned by he Baum-Welch algrhm and EM algrhm were nly lcally pmzed Lelhd Curren Mdel Cnfguran Mdel Cnfguran Space S - Berln Chen 66

67 Hmewr- (/ 0.34 {A:.34B:.33C:.33} {A:.33B:.34C:.33} {A:.33B:.33C:.34} ranse :. ABBCABCAABC. ABCABC 3. ABCA ABC 4. BBABCAB 5. BCAABCCAB 6. CACCABCA 7. CABCABCA 8. CABCA 9. CABCA ranse :. BBBCCBC. CCBABB 3. AACCBBB 4. BBABBAC 5. CCA ABBAB 6. BBBCCBAA 7. ABBBBABA 8. CCCCC 9. BBAAA S - Berln Chen 67

68 Hmewr- (/. leae pecfy he mdel parameer afer he fr and 50h eran f Baum-Welch ranng. leae hw he recgnn reul by ung he abve ranng equence a he eng daa (he -called nde eng. *Yu have perfrm he recgnn a wh he HMM raned frm he fr and 50h eran f Baum-Welch ranng repecvely 3. Whch cla d he fllwng eng equence belng? ABCABCCAB AABABCCCCBBB 4. Wha are he reul f Obervable Marv Mdel were nead ued n and 3? S - Berln Chen 68

69 Ilaed Wrd Recgnn Wrd Mdel M p ( X M Lelhd f M Wrd Mdel M Speech Sgnal Feaure Exracn Feaure Sequence X ( M p X Wrd Mdel M V p ( X M V Lelhd Selecr f M Label ( X arg max p ( X M Lelhd f M V M Le Wrd M ML Verb Apprxman Label ( X arg max max p( X S M [ ] S Wrd Mdel M Sl ( M p X Sl Lelhd f M Sl S - Berln Chen 69

70 Meaure f ASR erfrmance (/8 Evaluang he perfrmance f aumac peech recgnn (ASR yem crcal and he Wrd Recgnn Errr Rae (WER ne f he m mpran meaure here are ypcally hree ype f wrd recgnn errr Subun An ncrrec wrd wa ubued fr he crrec wrd Delen A crrec wrd wa med n he recgnzed enence Inern An exra wrd wa added n he recgnzed enence Hw deermne he mnmum errr rae? S - Berln Chen 70

71 Meaure f ASR erfrmance (/8 Calculae he WER by algnng he crrec wrd rng agan he recgnzed wrd rng A maxmum m ubrng machng prblem Can be handled by dynamc prgrammng deleed Example: Crrec : he effec clear Recgnzed: effec n clear WER+ WAR 00% mached nered mached Errr analy: ne delen and ne nern Meaure: wrd errr rae (WER wrd crrecn rae (WCR wrd accuracy rae (WAR Mgh be hgher han 00% Sub. + Del. + In. wrd Wrd Errr Rae 00% 50%. f wrd n he crrec enence 4 Mached wrd 3 Wrd Crrecn Rae 00% 75%. f wrd n he crrec enence 4 Mached - In. wrd 3 Wrd Accuracy Rae 00% 50%. f wrd n he crrec enence 4 Mgh be negave S - Berln Chen 7

72 Meaure f ASR erfrmance (3/8 A Dynamc rgrammng Algrhm (exb //dene fr he wrd lengh f he crrec/reference enence //dene fr he wrd lengh f he recgnzed/e enence mnmum wrd errr algnmen a he a grd [] /h nd f algnmen /h e Ref S - Berln Chen 7

73 Meaure f ASR erfrmance (4/8 Algrhm (by Berln Chen Sep : Ieran : Sep : Inalzan : Ref fr...n { //e G[0][0] 0; fr... n { //e } G[][0] G[ -][0] + ; B[][0] ; //Inern (Hrznal Drecn fr... m { //reference } G[0][] G[0][ -] + ; B[0][] ; // Delen (Vercal Drecn e fr...m { //reference G[ -][] + (Inern G[][-] (Delecn G[][] mn + G[ -][-] + (f LR[]! L[] Subun G[ -][-] (f LR[] L[] Mach ; //Inern (Hrznal Drecn ; //Delen (Vercal Drecn B[][] 3;//Subun (Dagnal Drecn 4; //mach (Dagnal Drecn } //fr reference } //fr e Sep 3 : Meaure and Bacrace : G[n][m] Wrd Errr Rae 00% m Wrd Accuracy Rae 00 % Wrd Errr Rae Opmal bacrace pah (B[n][m]... B[0][0] f B[][] prn " L[]" ; //Iner n ele f B[][] prn " LR[] "; //Delen ele prn " LR[] LR[] "; //H/Mac e: he penale fr ubun delen and nern errr are all e be here hen hen g lf lef g dwn h r Subu n hen g dwn dagnally S - Berln Chen 73

74 Meaure f ASR erfrmance (5/8 A Dynamc rgrammng Algrhm Inalzan Crrec/Reference Wrd Sequence m m-. fr (;<m;++ { //reference grd[0][] grd[0][-]; grd[0][].dr VER; grd[0][].cre. + Delen;. grd[0][].del ++;. 4 } 3Del. 3 Del. Del. HK grd[0][0].cre ] grd[0][0].n ] grd[0][0].del 0; grd[0][0].ub grd[0][0].h 0; grd[0][0].dr IL; 0 0 Del. (- (-- In. ( Del. (- In. (nm n- n In. In. 3In. Recgnzed/e Wrd fr (;<n;++ { // e Sequence grd[][0] grd[-][0]; grd[][0].dr HOR; grd[][0].cre +Inen; grd[][0].n ++; } S - Berln Chen 74 Del.

75 HK rgram Meaure f ASR erfrmance (6/8 Example Crrec (InDelSubH fr (;<n;++ //e (0500 (0500 { grd grd[]; grd grd[-]; C (040 fr (;<m;++ //reference (03 (0 Delee C r (30 (03 { h grd[].cre +nen; d grd[-].cre; (0400 C! (030 (0 f (lref[] le[] (0 (03 H C d + uben; v grd[-].cre + delen; (0300 B f (d<h && d<v { /* DIAG h r ub */ (00 (0 (0 (0 grd[] grd[-]; //rucure agnmen r (00 H B r (0 grd[].cre d; grd[].dr DIAG; (000 C (00 (0 f (lref[] le[] ++grd[].h; (0 (00 r(000 ele ++grd[].ub; Del C } ele f (h<v { /* HOR n */ (000 (000 A (000 (00 (00 (300 grd[] grd[]; //rucure agnmen grd[].cre h; H A e grd[].dr HOR; 0 ++ grd[].n; 0 In B B A B C } (0000 (000 (000 (3000 (4000 ele { /* VER del */ grd[] grd[-]; //rucure agnmen Algnmen : WER 60% grd[].cre v; grd[].dr dr VER; ++grd[].del; } } /* fr */ } /* fr */ Crrec: A C B C C Sll have an e: B A B C Oher pmal In B H A Del C H B H C Del C algnmen! S - Berln Chen 75

76 Meaure f ASR erfrmance (7/8 Example e: he penale fr ubun delen and nern errr are all e be here (InDelSubH Algnmen : WER 80% Crrec: e: Crrec: e: In B Crrec (0500 (0400 C C (0300 (0300 B (000 (000 C A 0 0 (0000 A C B C C B A A C H A Del C Sub B A C B C C B A A C Sub A Sub C Sub B H C H C Del C Del C (040 (03 (030 (0 (00 (00 (0 r (00 (0 r (30 ( ( ( Sub B (0 (0 (0 r(000 Del C ( (0 r (0 (00 Delee C H C (000 (00 (00 (300 H A e In B B A A C (000 (000 (3000 (4000 Algnmen 3: WER80% Algnmen : WER80% Crrec: e: A C B C C B A A C In B H A Sub C Del B H C Del C S - Berln Chen 76

77 Meaure f ASR erfrmance (8/8 w cmmn eng f dfferen penale fr ubun delen and nern errr /* HK errr penale */ uben 0; delen 7; nen 7; /* IS errr penale*/ ubenis 4; delenis 3; nenis 3; S - Berln Chen 77

78 Self-Exerce (/ Meaure f ASR erfrmance Reference 桃 芝 颱 風 重 創 花 蓮 光 復 鄉 大 興 村 死 傷 慘 重 感 觸 最 多 ASR Oupu 桃 芝 颱 風 重 創 花 蓮 光 復 鄉 打 新 村 次 傷 殘 周 感 觸 最 多. S - Berln Chen 78

79 Self-Exerce (/ 506 B re f ASR upu Repr he CER (characer errr rae f he fr ne and 506 re he reul huld hw he number f ubun delen and nern errr Overall Reul SE: %Crrec [H0 S ] WORD: %Crr8.5 Acc8.5 [H75 D4 S3 I0 9] Overall Reul SE: %Crrec0.00 [H0 S00 00] WORD: %Crr87.66 Acc86.83 [H083 D77 S348 I0 357] Overall Reul SE: %Crrec0.00 [H0 S00 00] WORD: %Crr87.9 Acc87.8 [H657 D93 S84 I ] Overall Reul SE: %Crrec0.00 [H0 S ] WORD: %Crr86.83 Acc86.06 [H5744 D89 S7839 I ] S - Berln Chen 79

80 Symbl fr Mahemacal Operan S - Berln Chen 80

81 he EM Algrhm (/ {A:.3B:.C:.5} {A:.7B:.C:.} {A:.3B:.6C:.} p(o p(o > p(o A B Oberved daa : O : ball equence Laen daa : S : ble equence arameer be emaed maxmze lg(o {(A(B(B A(A B(R A(G A(R B(G B} S - Berln Chen 8

82 he EM Algrhm (/7 Inrducn f EM (Expecan Maxmzan: Why EM? Smple pmzan algrhm fr lelhd funcn rele n he nermedae varable called laen daa In ur cae here he ae equence he laen daa Drec acce he daa neceary emae he parameer mpble r dffcul In ur cae here alm mpble emae {AB π} whu cnderan f he ae equence w Mar Sep : E : expecan wh repec he laen daa ung he curren emae f he parameer and cndned n he E [ ] S O bervan M: prvde a new eman f he parameer accrdng Maxmum lelhd (ML r Maxmum A err (MA Crera S - Berln Chen 8

83 he EM Algrhm (3/7 ML and MA Eman prncple baed n bervan: ( x x x... x n X { X X... } X n he Maxmum Lelhd (ML rncple fnd he mdel parameer Φ ha he lelhd p ( x Φ maxmum fr example f Φ { μ Σ} he parameer f a mulvarae nrmal ldrbun b andxx d..d. (ndependen d d dencally drbued hen he ML emae f Φ μ Σ μ ML n x Σ ML n n { } n ( x μ ( x μ he Maxmum A err (MA rncple fnd he mdel parameer Φ ha he lelhd p( Φ x maxmum ML ML S - Berln Chen 83

84 he EM Algrhm (4/7 he EM Algrhm mpran HMM and her learnng echnque Dcver new mdel parameer maxmze he lg-lelhd f ncmplee daa lg ( O by eravely maxmzng he lg O S expecan f lg-lelhd frm cmplee daa Frly ung calar (dcree randm varable nrduce he EM algrhm he bervable ranng daa O We wan maxmze ( O a parameer vecr he hdden (unbervable daa S E.g. he cmpnen prbable (r dene f bervable daa r he underlyng ae equence n HMM O S - Berln Chen 84

85 he EM Algrhm (5/7 Aume we have and emae he prbably ha each S ccurred n he generan f O reend we had n fac berved a cmplee daa par ( O S wh frequency prprnal he prbably ( O S cmpued a new he maxmum lelhd emae f De he prce cnverge? Algrhm unnwn mdel eng ( O S ( S O ( O Baye rule cmplee daa lelhd ncmplee daa lelhd Lg-lelhd expren and expecan aen ver S lg lg S ( O lg ( O S lg ( S O ae expecan ver S ( O [ ( S O lg ( O ] [ ( S O lg ( O S ] [ ( S O lg ( S O ] S S - Berln Chen 85

86 he EM Algrhm (6/7 Algrhm (Cn. ( lg O We can hu expre a fllw lg ( O [ ( S O lg ( O S ] [ ( S O lg ( S O ] S Q( H ( where Q H ( [ ( S O lg ( O S ] S ( [ ( S O lg ( S O ] We wan lg S lg ( O lg ( O ( O lg ( O [ Q ( H ( ] Q ( H ( Q( Q( H ( + H ( S [ ] S - Berln Chen 86

87 he EM Algrhm (7/7 ( H( H + ha he fllwng prpery H ( + H ( S S S 0 ( S O lg ( S O ( S O ( S O Kullbuac-Lebler (KL dance ( S O ( S O ( Q lg x x Jenen nequaly S O S O [ ] ( + H ( 0 H herefre fr maxmzng lg ( O we nly need maxmze he Q-funcn (auxlary funcn Q ( [ ( S O lg( O S ] S Expecan f he cmplee daa lg lelhd wh repec he laen ae equence S - Berln Chen 87

88 EM Appled Dcree HMM ranng (/5 Apply EM algrhm eravely refne he HMM pp y g y parameer vecr By maxmzng he auxlary funcn ( π B A [ ] S S O S O S O lg Q S S O O S O lg Where and can be expreed a b a S O π S O S O b a b a lg lg lg lg S O S O π π S - Berln Chen b a lg lg lg lg S O π

89 EM Appled Dcree HMM ranng (/5 Rewre he auxlary funcn a Q Q Q Q ( Q π ( π + Q a ( a + Q b ( b ( O S? ( O π π lg π lg π all S ( O ( O O S? ( O + a ( a lg a + all S O ( O O S? ( O b b lg b lg b all S O v ( O w y lg a ( S - Berln Chen 89

90 EM Appled Dcree HMM ranng (3/5 he auxlary funcn cnan hree ndependen erm a and ( π Can be maxmzed ndvdually All f he ame frm g( y y... y F y F y b where w lg y y and y 0 ha maxmum value when : y w w S - Berln Chen 90

91 EM Appled Dcree HMM ranng (4/5 rf: Apply Lagrange Mulpler l Mulpler By applyng Lagrange + w w F y y lg w y lg w F l l l ha Suppe Cnran + y 0 y y F l l Cnran w w w y l l w y S - Berln Chen 9 Lagrange Mulpler: hp://

92 EM Appled Dcree HMM ranng (5/5 he new mdel parameer e can be B A π he new mdel parameer e can be expreed a: B A π O γ π O O + a γ ξ O O γ γ O O v v b.... γ γ O O S - Berln Chen 9 γ O

93 EM Appled Cnnuu HMM ranng (/7 b Cnnuu HMM: he ae bervan de n cme frm a fne e bu frm a cnnuu pace he dfference beween he dcree and cnnuu HMM le n a dfferen frm f ae upu prbably Dcree HMM requre he quanzan prcedure map bervan vecr frm he cnnuu pace he dcree pace Cnnuu Mxure HMM M c b he ae bervan drbun f HMM mdeled by mulvarae Gauan mxure deny funcn (M mxure M M c ( ; μ Σ c exp μ Σ μ L / π Σ M c w w w w 3 3 Drbun fr Sae S - Berln Chen 93

94 EM Appled Cnnuu HMM ranng (/7 b cmpnen b ( Expre wh repec each ngle mxure p ( O S π a b + p e: π a... + M ( a M M + ( a + a a ( a + a a...( a + a +... a... M M M a [ c b ] M M M [ ] ( O S K π ( a + c b K : ne f p he pble mxure alng wh he ae equence S ( O p( O S K S K cmpnen equence M S - Berln Chen 94

95 EM Appled Cnnuu HMM ranng (3/7 herefre an auxlary funcn fr he EM algrhm can be wren a: Q ( [ ( S K O lg p ( O S K ] S K S K p ( O S K p( O lg p ( O S K ( O S K lgπ + + lga lgb ( lg p + + lgc ( Q ( π + Q ( a + Q ( b Q ( c Q c + π a b nal prbable ae rann prbable Gauan deny funcn mxure cmpnen S - Berln Chen 95

96 EM Appled Cnnuu HMM ranng (4/7 he nly dfference we have when cmpared wh Dcree HMM ranng Q b γ ( M ( b ( Ο lg b ( Q c M ( c ( Ο lg c ( S - Berln Chen 96

97 EM Appled Cnnuu HMM ranng (5/7 Le γ b lg b M ( ( Ο ( ( ; μ Σ lg b μ b b μ ( π ( μ Σ ( μ exp L Σ L lg ( π + lg Σ ( μ Σ ( μ ( b Σ ( μ M Q μ γ ( lg b ( μ { γ ( Σ ( μ } [ γ ( ] γ ( 0 ( C + C d ( x Cx x dx and Σ ymmerc here Σ S - Berln Chen 97

98 EM Appled Cnnuu HMM ranng (6/7 lg b lg b L lg ( π lg Σ ( μ Σ ( μ ( Σ ( ( μ ( μ Σ Σ Σ Σ Σ [ Σ Σ ( μ ( μ Σ ] γ b b ( Σ ( Σ Q γ M lg b ( Σ Σ ( μ ( μ γ [ Σ ] ( Σ γ ( Σ ( μ ( μ γ ( Σ Σ Σ γ ( Σ Σ ( μ ( μ Σ [ γ ( ( μ ( μ ] γ ( Σ d ( a X b X ab X dx d[ de( X ] de( X X dx and ymmerc here Σ 0 Σ Σ S - Berln Chen 98

99 EM Appled Cnnuu HMM ranng (7/7 he new mdel parameer e fr each mxure p cmpnen and mxure wegh can be expreed a: p O [ ] p p p γ O O O μ p p γ O O O [ ] p p p γ μ μ O μ μ O O Σ p p γ O O γ S - Berln Chen 99 M c γ γ