Speech, NLP and the Web

Transcription

1 peech NL ad he Web uhpak Bhaacharyya CE Dep. IIT Bombay Lecure 38: Uuperved learg HMM CFG; Baum Welch lecure 37 wa o cogve NL by Abh Mhra Baum Welch uhpak Bhaacharyya

2 roblem HMM arg emac ar of peech Taggg NL Try Morph Aaly Marah Frech CRF HMM MEMM Hd Eglh Laguage Algorhm Baum Welch uhpak Bhaacharyya 2

3 Clac problem wh repec o HMM.Gve he obervao equece fd he poble ae equece- Verb 2.Gve he obervao equece fd probably- forward/backward algorhm 3.Gve he obervao equece fd he HMM prameer.- Baum-Welch algorhm Baum Welch uhpak Bhaacharyya 3

4 robablc FM a :0.3 a :0. a 2 :0.4 a :0.3 a 2 :0.2 2 a :0.2 a 2 :0.2 a 2 :0.3 The queo here : wha he mo lkely ae equece gve he oupu equece ee Baum Welch uhpak Bhaacharyya 4

5 Developg he ree ar *0.= a a 2 0.*0.2= *0.4= *0.3= *0.2=0.06 Chooe he wg equece per ae per erao Baum Welch uhpak Bhaacharyya 5

6 Tree rucure cod *0.= a a The problem beg addreed by h ree * arg max a a2 a a2 a-a2-a-a2 he oupu equece ad µ he model or he mache Baum Welch uhpak Bhaacharyya 6

7 ah foud: workg backward 2 2 a a 2 a a 2 roblem aeme: Fd he be poble equece * arg max O where ae eq O Oupu eq Model or Mache { 0 A T} Model or Mache ar ymbol ae colleco Alphabe e T defed a a k k Trao Baum Welch uhpak Bhaacharyya 7

8 How o compue o 0 o o 2 o 3 o m O O Margalzao Coder he obervao equece O O O Om 3.. m m Where repree he ae equece. Baum Welch uhpak Bhaacharyya 8

9 Compug o 0 o o 2 o 3 o m ]. ]...[. [ m m m m m m m m m m O O O O O O O O O O O Baum Welch uhpak Bhaacharyya 9

10 Forward ad Backward robably Calculao Baum Welch uhpak Bhaacharyya 0

11 Forward probably Fk Defe Fk= robably of beg ae havg ee o 0 o o 2 o k Fk=o 0 o o 2 o k Wh m a he legh of he oberved equece There are N ae oberved equece=o 0 o o 2..o m =Σ p=0n o 0 o o 2..o m p =Σ p=0n Fm p Baum Welch uhpak Bhaacharyya

12 Forward probably cod. Fk q = o 0 o o 2..o k q = o 0 o o 2..o k q = o 0 o o 2..o k- o k q = Σ p=0n o 0 o o 2..o k- p o k q = Σ p=0n o 0 o o 2..o k- p. o k q o 0 o o 2..o k- p = Σ p=0n Fk-p. o k q p o k = Σ p=0n Fk-p. p q O 0 O O 2 O 3 O k O k+ O m- O m p q m fal Baum Welch uhpak Bhaacharyya 2

13 Backward probably Bk Defe Bk= robably of eeg o k o k+ o k+2 o m gve ha he ae wa Bk=o k o k+ o k+2 o m \ Wh m a he legh of he whole oberved equece oberved equece=o 0 o o 2..o m = o 0 o o 2..o m 0 =B00 Baum Welch uhpak Bhaacharyya 3

14 Bk p Backward probably cod. = o k o k+ o k+2 o m \ p = o k+ o k+2 o m o k p = Σ q=0n o k+ o k+2 o m o k q p = Σ q=0n o k q p o k+ o k+2 o m o k q p = Σ q=0n o k+ o k+2 o m q. o k q p o k = Σ q=0n Bk+q. p q O 0 O O 2 O 3 O k O k+ O m- O m p q m fal Baum Welch uhpak Bhaacharyya 4

15 HMM Trag Baum Welch or Forward Backward Algorhm Baum Welch uhpak Bhaacharyya 5

16 Key Iuo a a b q b a r a b b Gve: Ialzao: Compue: Approach: Trag equece robably value r ae eq rag eq ge expeced cou of rao compue rule probable Ialze he probable ad recompue hem EM lke approach Baum Welch uhpak Bhaacharyya 6

17 Baum-Welch algorhm: cou ab q a b r ab ab rg = abb aaa bbb aaa equece of ae wh repec o pu ymbol o/p eq ae eq a q r b b a q q r a a b b b a a a q r q q q r q r Baum Welch uhpak Bhaacharyya 7

18 Calculag probable from able a q r b q b T=#ae w A=#alphabe ymbol k 5/ 8 3/ 8 T c A l m c w k w m Table of cou rc De O/ Cou q r a 5 q q b 3 r q a 3 r q b 2 Now f we have a o-deermc rao he mulple ae eq poble for he gve o/p eq ref. o prevou lde feaure. Our am o fd expeced cou hrough h. l Baum Welch uhpak Bhaacharyya 8

19 Ierplay Bewee Two Equao Wk T c A l0 m0 Wk c Wm l C 0 Wk 0 W 0 Wk 0 w 0 No. of me he rao occur he rg w k Baum Welch uhpak Bhaacharyya 9

20 Illurao b:0.7 a:0.6 q a:0.67 b:.0 r Acual Dered HMM b:0.48 q a:0.04 r a:0.48 b:.0 Ial gue Baum Welch uhpak Bhaacharyya 20

21 a Oe ru of Baum-Welch algorhm: rg ababb a b b a a b b b b pah a q r b r q a q q q r q r q q q r q q q q q q q r q q q q q q q q b q q Rouded Toal New robable 0.06 =0.0/0. ae equece * ε codered a arg ad edg ymbol of he pu equece rg. Through mulple erao he probably value wll coverge. Baum Welch uhpak Bhaacharyya 2

22 Relaed example: Word algme Eglh hree rabb a b Frech ro lap w x 2 rabb of Greoble b c d 2 lap de Greoble x y z Baum Welch uhpak Bhaacharyya 22

23 Ial robable: each cell deoe a w a x ec. a b c d w /4 /4 /4 /4 x /4 /4 /4 /4 y /4 /4 /4 /4 z /4 /4 /4 /4

24 cou a b a b c d b c d a b c d w x w /2 /2 0 0 x y z w x /2 /2 0 0 x 0 /3 /3 /3 y y 0 /3 /3 /3 z z 0 /3 /3 /3 Baum Welch uhpak Bhaacharyya 24

25 Reved probable able a b c d w /2 /4 0 0 x /2 5/2 /3 /3 y 0 /6 /3 /3 z 0 /6 /3 /3

26 a b reved cou a b c d b c d a b c d w x w /2 3/8 0 0 x y z w x /2 5/8 0 0 x 0 5/9 /3 /3 y y 0 2/9 /3 /3 z z 0 2/9 /3 /3 Baum Welch uhpak Bhaacharyya 26

27 Re-Reved probable able a b c d w /2 3/6 0 0 x /2 85/44 /3 /3 y 0 /9 /3 /3 z 0 /9 /3 /3 Coue ul covergece; oce ha bx bdg ge progrevely roger; b=rabb x=lap

28 Compuaoal par /2 k k W W W W w W W W w W W W W W W W C k k k ] [ ] [ ] [ ] [ w 0 w w 2 w k w - w Baum Welch uhpak Bhaacharyya 28

29 Compuaoal par 2/ B w F B w W F B w W F W W w W W W w W W W w W k k k k k k w 0 w w 2 w k w - w Baum Welch uhpak Bhaacharyya 29

30 Dcuo. ymmery breakg: Example: ymmery breakg lead o o chage al value b:.0 b:0.5 a:0.5 a:.0 Dered a:0.5 b:0.25 a:0.25 b:0.5 a:0.5 a:0.25 b:0.5 b:0.5 Ialzed 2 ruck Local maxma 3. Label ba problem robable have o um o. Value ca re a he co of fall of value for oher. Baum Welch uhpak Bhaacharyya 30

31 HMM CFG O oberved equece w m eece X ae equece pare ree model G grammar Three fudameal queo Baum Welch uhpak Bhaacharyya 3

32 HMM CFG How lkely a cera obervao gve he model? How lkely a eece gve he grammar? O w G How o chooe a ae equece whch be expla he obervao? How o chooe a pare whch be uppor he eece? m arg max X O arg max w m G X Baum Welch uhpak Bhaacharyya 32

33 HMM CFG How o chooe he model parameer ha be expla he oberved daa? How o chooe rule probable whch maxmze he probable of he oberved eece? arg max O w m G arg max G Baum Welch uhpak Bhaacharyya 33

34 Iereg robable N Wha he probably of havg a N a h poo uch ha wll derve he buldg? - N 45 N Ide robable The guma prayed he buldg wh bulle Oude robable Wha he probably of arg from N ad dervg The guma prayed a N ad wh bulle? - N 45 Baum Welch uhpak Bhaacharyya 34

35 Iereg robable Radom varable o be codered The o-ermal beg expaded. E.g. N The word-pa covered by he o-ermal. E.g. 45 refer o word he buldg Whle calculag probable coder: The rule o be ued for expao : E.g. N DT NN The probable aocaed wh he RH oermal : E.g. DT ubree de/oude probable & NN ubree de/oude probable Baum Welch uhpak Bhaacharyya 35

36 Oude robably pq : The probably of begg wh N & geerag he o-ermal N pq ad all word oude w p..w q p pq q m p q w N w G N N w w p- w p w q w q+ w m Baum Welch uhpak Bhaacharyya 36

37 Ide robable pq : The probably of geerag he word w p..w q arg wh he o-ermal N pq. p q w N G pq pq N N w w p- w p w q w q+ w m Baum Welch uhpak Bhaacharyya 37

38 N Oude & Ide robable: example 45 for "he buldg" The guma prayed N wh bulle G N N45 G 45 for "he buldg" he buldg N 45 N The guma prayed he buldg wh bulle Baum Welch uhpak Bhaacharyya 38

39 CFG Trag Baum Welch uhpak Bhaacharyya 39

40 EM Algorhm for rag Baum Welch uhpak Bhaacharyya 40