peech NL ad he Web uhpak Bhaacharyya CE Dep. IIT Bombay Lecure 38: Uuperved learg HMM CFG; Baum Welch lecure 37 wa o cogve NL by Abh Mhra Baum Welch uhpak Bhaacharyya
roblem HMM arg emac ar of peech Taggg NL Try Morph Aaly Marah Frech CRF HMM MEMM Hd Eglh Laguage Algorhm Baum Welch uhpak Bhaacharyya 2
Clac problem wh repec o HMM.Gve he obervao equece fd he poble ae equece- Verb 2.Gve he obervao equece fd probably- forward/backward algorhm 3.Gve he obervao equece fd he HMM prameer.- Baum-Welch algorhm Baum Welch uhpak Bhaacharyya 3
robablc FM a :0.3 a :0. a 2 :0.4 a :0.3 a 2 :0.2 2 a :0.2 a 2 :0.2 a 2 :0.3 The queo here : wha he mo lkely ae equece gve he oupu equece ee Baum Welch uhpak Bhaacharyya 4
Developg he ree ar.0 0.0 2 0. 0.3 0.2 0.3.. *0.=0. 0.3 2 0.0 2 0.0 a 0.2 0.4 0.3 0.2.. 2 2 a 2 0.*0.2=0.02 0.*0.4=0.04 0.3*0.3=0.09 0.3*0.2=0.06 Chooe he wg equece per ae per erao Baum Welch uhpak Bhaacharyya 5
Tree rucure cod 0.09 0.06 2 0. 0.3 0.2 0.3.. 0.09*0.=0.009 0.027 2 0.02 2 0.08 a 0.3 0.2 0.2 0.4 a 2. 2 2 0.008 0.0054 0.0024 0.0048 The problem beg addreed by h ree * arg max a a2 a a2 a-a2-a-a2 he oupu equece ad µ he model or he mache Baum Welch uhpak Bhaacharyya 6
ah foud: workg backward 2 2 a a 2 a a 2 roblem aeme: Fd he be poble equece * arg max O where ae eq O Oupu eq Model or Mache { 0 A T} Model or Mache ar ymbol ae colleco Alphabe e T defed a a k k Trao Baum Welch uhpak Bhaacharyya 7
How o compue o 0 o o 2 o 3 o m O O Margalzao Coder he obervao equece O O O 0 0 2 2... Om 3.. m m Where repree he ae equece. Baum Welch uhpak Bhaacharyya 8
Compug o 0 o o 2 o 3 o m ]. ]...[. [................ 0 0 0 0 0 0 2 0 0 2 0 2 0 m m m m m m m m m m O O O O O O O O O O O Baum Welch uhpak Bhaacharyya 9
Forward ad Backward robably Calculao Baum Welch uhpak Bhaacharyya 0
Forward probably Fk Defe Fk= robably of beg ae havg ee o 0 o o 2 o k Fk=o 0 o o 2 o k Wh m a he legh of he oberved equece There are N ae oberved equece=o 0 o o 2..o m =Σ p=0n o 0 o o 2..o m p =Σ p=0n Fm p Baum Welch uhpak Bhaacharyya
Forward probably cod. Fk q = o 0 o o 2..o k q = o 0 o o 2..o k q = o 0 o o 2..o k- o k q = Σ p=0n o 0 o o 2..o k- p o k q = Σ p=0n o 0 o o 2..o k- p. o k q o 0 o o 2..o k- p = Σ p=0n Fk-p. o k q p o k = Σ p=0n Fk-p. p q O 0 O O 2 O 3 O k O k+ O m- O m 0 2 3 p q m fal Baum Welch uhpak Bhaacharyya 2
Backward probably Bk Defe Bk= robably of eeg o k o k+ o k+2 o m gve ha he ae wa Bk=o k o k+ o k+2 o m \ Wh m a he legh of he whole oberved equece oberved equece=o 0 o o 2..o m = o 0 o o 2..o m 0 =B00 Baum Welch uhpak Bhaacharyya 3
Bk p Backward probably cod. = o k o k+ o k+2 o m \ p = o k+ o k+2 o m o k p = Σ q=0n o k+ o k+2 o m o k q p = Σ q=0n o k q p o k+ o k+2 o m o k q p = Σ q=0n o k+ o k+2 o m q. o k q p o k = Σ q=0n Bk+q. p q O 0 O O 2 O 3 O k O k+ O m- O m 0 2 3 p q m fal Baum Welch uhpak Bhaacharyya 4
HMM Trag Baum Welch or Forward Backward Algorhm Baum Welch uhpak Bhaacharyya 5
Key Iuo a a b q b a r a b b Gve: Ialzao: Compue: Approach: Trag equece robably value r ae eq rag eq ge expeced cou of rao compue rule probable Ialze he probable ad recompue hem EM lke approach Baum Welch uhpak Bhaacharyya 6
Baum-Welch algorhm: cou ab q a b r ab ab rg = abb aaa bbb aaa equece of ae wh repec o pu ymbol o/p eq ae eq a q r b b a q q r a a b b b a a a q r q q q r q r Baum Welch uhpak Bhaacharyya 7
Calculag probable from able a q r b q b T=#ae w A=#alphabe ymbol k 5/ 8 3/ 8 T c A l m c w k w m Table of cou rc De O/ Cou q r a 5 q q b 3 r q a 3 r q b 2 Now f we have a o-deermc rao he mulple ae eq poble for he gve o/p eq ref. o prevou lde feaure. Our am o fd expeced cou hrough h. l Baum Welch uhpak Bhaacharyya 8
Ierplay Bewee Two Equao Wk T c A l0 m0 Wk c Wm l C 0 Wk 0 W 0 Wk 0 w 0 No. of me he rao occur he rg w k Baum Welch uhpak Bhaacharyya 9
Illurao b:0.7 a:0.6 q a:0.67 b:.0 r Acual Dered HMM b:0.48 q a:0.04 r a:0.48 b:.0 Ial gue Baum Welch uhpak Bhaacharyya 20
a Oe ru of Baum-Welch algorhm: rg ababb a b b a a b b b b pah a q r b r q a q q q r q r q q 0.00077 0.0054 0.0054 0 0.0007 7 q r q q q q 0.00442 0.00442 0.00442 0.0044 2 q q q r q q 0.00442 0.00442 0.00442 0.0044 2 q q q q q q 0.02548 0.0 0.000 0.0509 6 b q q 0.0088 4 0.0088 4 0.0764 4 Rouded Toal 0.035 0.0 0.0 0.06 0.095 New robable 0.06 =0.0/0. ae equece 0+0.06+ 0.095.0 0.36 0.58 * ε codered a arg ad edg ymbol of he pu equece rg. Through mulple erao he probably value wll coverge. Baum Welch uhpak Bhaacharyya 2
Relaed example: Word algme Eglh hree rabb a b Frech ro lap w x 2 rabb of Greoble b c d 2 lap de Greoble x y z Baum Welch uhpak Bhaacharyya 22
Ial robable: each cell deoe a w a x ec. a b c d w /4 /4 /4 /4 x /4 /4 /4 /4 y /4 /4 /4 /4 z /4 /4 /4 /4
cou a b a b c d b c d a b c d w x w /2 /2 0 0 x y z w 0 0 0 0 x /2 /2 0 0 x 0 /3 /3 /3 y 0 0 0 0 y 0 /3 /3 /3 z 0 0 0 0 z 0 /3 /3 /3 Baum Welch uhpak Bhaacharyya 24
Reved probable able a b c d w /2 /4 0 0 x /2 5/2 /3 /3 y 0 /6 /3 /3 z 0 /6 /3 /3
a b reved cou a b c d b c d a b c d w x w /2 3/8 0 0 x y z w 0 0 0 0 x /2 5/8 0 0 x 0 5/9 /3 /3 y 0 0 0 0 y 0 2/9 /3 /3 z 0 0 0 0 z 0 2/9 /3 /3 Baum Welch uhpak Bhaacharyya 26
Re-Reved probable able a b c d w /2 3/6 0 0 x /2 85/44 /3 /3 y 0 /9 /3 /3 z 0 /9 /3 /3 Coue ul covergece; oce ha bx bdg ge progrevely roger; b=rabb x=lap
Compuaoal par /2 k k W W W W w W W W w W W W W W W W C k k k 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ] [ ] [ ] [ ] [ 0 0 0 w 0 w w 2 w k w - w 0 - + Baum Welch uhpak Bhaacharyya 28
Compuaoal par 2/2 0 0 0 0 0 0 0 0 0 0 B w F B w W F B w W F W W w W W W w W W W w W k k k k k k w 0 w w 2 w k w - w 0 - + Baum Welch uhpak Bhaacharyya 29
Dcuo. ymmery breakg: Example: ymmery breakg lead o o chage al value b:.0 b:0.5 a:0.5 a:.0 Dered a:0.5 b:0.25 a:0.25 b:0.5 a:0.5 a:0.25 b:0.5 b:0.5 Ialzed 2 ruck Local maxma 3. Label ba problem robable have o um o. Value ca re a he co of fall of value for oher. Baum Welch uhpak Bhaacharyya 30
HMM CFG O oberved equece w m eece X ae equece pare ree model G grammar Three fudameal queo Baum Welch uhpak Bhaacharyya 3
HMM CFG How lkely a cera obervao gve he model? How lkely a eece gve he grammar? O w G How o chooe a ae equece whch be expla he obervao? How o chooe a pare whch be uppor he eece? m arg max X O arg max w m G X Baum Welch uhpak Bhaacharyya 32
HMM CFG How o chooe he model parameer ha be expla he oberved daa? How o chooe rule probable whch maxmze he probable of he oberved eece? arg max O w m G arg max G Baum Welch uhpak Bhaacharyya 33
Iereg robable N Wha he probably of havg a N a h poo uch ha wll derve he buldg? - N 45 N Ide robable The guma prayed he buldg wh bulle 2 3 4 5 6 7 Oude robable Wha he probably of arg from N ad dervg The guma prayed a N ad wh bulle? - N 45 Baum Welch uhpak Bhaacharyya 34
Iereg robable Radom varable o be codered The o-ermal beg expaded. E.g. N The word-pa covered by he o-ermal. E.g. 45 refer o word he buldg Whle calculag probable coder: The rule o be ued for expao : E.g. N DT NN The probable aocaed wh he RH oermal : E.g. DT ubree de/oude probable & NN ubree de/oude probable Baum Welch uhpak Bhaacharyya 35
Oude robably pq : The probably of begg wh N & geerag he o-ermal N pq ad all word oude w p..w q p pq q m p q w N w G N N w w p- w p w q w q+ w m Baum Welch uhpak Bhaacharyya 36
Ide robable pq : The probably of geerag he word w p..w q arg wh he o-ermal N pq. p q w N G pq pq N N w w p- w p w q w q+ w m Baum Welch uhpak Bhaacharyya 37
N Oude & Ide robable: example 45 for "he buldg" The guma prayed N wh bulle G N N45 G 45 for "he buldg" he buldg N 45 N The guma prayed he buldg wh bulle 2 3 4 5 6 7 Baum Welch uhpak Bhaacharyya 38
CFG Trag Baum Welch uhpak Bhaacharyya 39
EM Algorhm for rag Baum Welch uhpak Bhaacharyya 40