Natural Language Processing NLP Hidden Markov Models. Razvan C. Bunescu School of Electrical Engineering and Computer Science

Transcription

1 Naural Language rcessng NL 6840 Hdden Markv Mdels Razvan C. Bunescu Schl f Elecrcal Engneerng and Cmpuer Scence bunescu@h.edu

2 Srucured Daa Fr many applcans he..d. assumpn des n hld: pels n mages f real becs. hyperlnked web pages. crss-cans n scenfc papers. enes n scal newrks. sequences f wrds/leers n e. successve me frames n speech. sequences f base par n DNA. muscal nes n a nal meldy. daly values f a parcular sck. Lecure 08 2

3 Srucured Daa Fr many applcans he..d. assumpn des n hld: pels n mages f real becs. hyperlnked web pages. crss-cans n scenfc papers. enes n scal newrks. sequences f wrds/leers n e. successve me frames n speech. sequences f base par n DNA. muscal nes n a nal meldy. daly values f a parcular sck. Sequenal Daa Lecure 08 3

4 rbablsc Graphcal Mdels GMs use a graph fr cmpacly:. Encdng a cmple dsrbun ver a mul-dmensnal space. 2. Represenng a se f ndependences ha hld n he dsrbun. rperes and 2 are n a deep sense equvalen. rbablsc Graphcal Mdels: Dreced:.e. Bayesan Newrks.e. Belef Newrks. Undreced:.e. Markv Randm Felds Lecure 08 4

5 rbablsc Graphcal Mdels Dreced GMs: Bayesan Newrks: Dynamc Bayesan Newrks: Sae Observan Mdels:» Hdden Markv Mdels.» Lnear Dynamcal Sysems Kalman flers. Undreced GMs: Markv Randm Felds MRF. Cndnal Randm Felds CRF. Sequenal CRFs. Lecure 08 5

6 Bayesan Newrks A Bayesan Newrk srucure G s a dreced acyclc graph whse ndes X X 2 X n represen randm varables and edges crrespnd drec nfluences beween ndes: Le ax dene he parens f X n G; Le NnDescendX dene he varables n he graph ha are n descendans f X. Then G encdes he fllwng se f cndnal ndependence assumpns called he lcal ndependences: Fr each X n G: X NnDescendX ax Lecure 08 6

7 Bayesan Newrks. Because X NnDescendX ax fllws ha: n X X 2 X n X ax 2. Mre generally d-separan:. Tw ses f ndes X and Y are cndnally ndependen gven a se f ndes E X Y E f X and Y are d-separaed by E. Lecure 08 7

8 Sequenal Daa Q: Hw can we mdel sequenal daa? Ignre sequenal aspecs and rea he bservans as..d. T - 2 Rela he..d. assumpn by usng a Markv mdel. T - Lecure 08 8

9 Markv Mdels X T s a sequence f randm varables. S {s s N } s a sae space.e. akes values frm S. Lmed Hrzn: sk s 2 Sanary: k sk 2 sk X s sad be a Markv chan. Lecure 08 9

10 Markv Mdels: arameers S {s s N } are he vsble saes. Π {π } are he nal sae prbables. π s A {a } are he sae ransn prbables. a s s Π A A A A A A T - Lecure 08 0

11 Markv Mdels as DBNs A Markv Mdel s a Dynamc Bayesan Newrk:. B 0 Π s he nal dsrbun ver saes. Π 2. B A s he 2-me-slce Bayesan Newrk 2-TBN. A The unrlled DBN Markv mdel ver T me seps: Π A A A A A A T - Lecure 08

12 Markv Mdels: Inference Π A A A A A A T - p X p T T p T π a Eercse: cmpue pap Lecure 08 2

13 m h Order Markv Mdels Frs rder Markv mdel: Secnd rder Markv mdel: m h rder Markv mdel: 3 Lecure 08 T p X p 2 2 T p p X p 2 T m m m m p p p X p

14 Markv Mdels Vsble Markv Mdels: Develped by Andre A. Markv [Markv 93] mdelng he leer sequences n ushkn s Eugene Onyegn. Hdden Markv Mdels: The saes are hdden laen varables. The saes prbablscally generae surface evens r bservans. Effcen ranng usng Epecan Mamzan EM Mamum Lkelhd ML when agged daa s avalable. Effcen nference usng he Verb algrhm. Lecure 08 4

15 Hdden Markv Mdels HMMs rbablsc dreced graphcal mdels: Hdden saes shwn n brwn. Vsble bservans shwn n lavender. Arrws mdel prbablsc ndependences. T - - T Lecure 08 5

16 HMMs: arameers T - - T S {s s N } s he se f saes. K {k k M } { M} s he bservans alphabe. X T s a sequence f saes. O T s a sequence f bservans. Lecure 08 6

17 HMMs: arameers Π A A A A A A T - B B B B B - T Π {π } S are he nal sae prbables. A {a } } S are he sae ransn prbables. B {b k } S k K are he symbl emsn prbables. b k k s Lecure 08 7

18 Hdden Markv Mdels as DBNs A Hdden Markv Mdel s a Dynamc Bayesan Newrk:. B 0 Π s he nal dsrbun ver saes. Π 2. B A s he 2-me-slce Bayesan Newrk 2-TBN. A B The unrlled DBN Markv mdel ver T me seps prev. slde. Lecure 08 8

19 A hyslgcal Mdel f Bld Glucse Dynamcs as a DBN [Bunescu e al. ICMLA 3] The sae ransn BN capures dependences amng physlgcal varables a cnsecuve me seps: 9

20 HMMs: Inference and Tranng Three fundamenal quesns: Gven a mdel µ A B Π cmpue he prbably f a gven bservan sequence.e. poµ Frward-Backward. 2 Gven a mdel µ and an bservan sequence O cmpue he ms lkely hdden sae sequence Verb. X ˆ arg ma X O µ X 3 Gven an bservan sequence O fnd he mdel µ A B Π ha bes eplans he bserved daa EM. Gven bservan and sae sequence O X fnd µ ML. Lecure 08 20

21 HMMs: Decdng T - - T Gven a mdel µ A B Π cmpue he prbably f a gven bservan sequence O T.e. poµ Lecure 08 2

22 HMMs: Decdng T - - T b O X µ b b 2 2 T T Lecure 08 22

23 HMMs: Decdng T - - T b O X µ b b 2 2 T T X µ π a a a T T Lecure 08 23

24 HMMs: Decdng T - - T b O X µ b b 2 2 T T X µ π a a a T T O X µ O X µ X µ Lecure 08 24

25 HMMs: Decdng T - - T b O X µ b b 2 2 T T X µ π a a a T T O X µ O X µ X µ O µ O X µ X µ X Lecure 08 25

26 HMMs: Decdng T - - T p O µ π b a b { T } T Π Tme cmpley? Lecure 08 26

27 HMMs: Frward rcedure T - - T Defne: α µ Then slun s: N p O µ α T Lecure 08 27

28 HMMs: Decdng 28 Lecure 08 T - - T α

29 HMMs: Decdng 29 Lecure 08 T - - T α

30 HMMs: Decdng 30 Lecure 08 T - - T α

31 HMMs: Decdng 3 Lecure 08 T - - T α

32 HMMs: Decdng 32 Lecure 08 T - - T N N N N b a α α

33 HMMs: Decdng 33 Lecure 08 T - - T N N N N b a α α

34 HMMs: Decdng 34 Lecure 08 T - - T N N N N b a α α

35 HMMs: Decdng 35 Lecure 08 T - - T N N N N b a α α

36 The Frward rcedure. Inalzan α π b N 2. Recursn: α α a N b N < T 3. Termnan: N p O µ α T Lecure 08 36

37 The Frward rcedure: Trells Cmpuan s α Σ s 2 α 2 a b s 3 α 3... a 2 b a 3 b a N b s α s N α N Lecure 08 37

38 HMMs: Backward rcedure T - - T Defne: β µ T Then slun s: N p O µ π β b Lecure 08 38

39 The Backward rcedure. Inalzan β T N 2. Recursn: β ab β N < N T 3. Termnan: N p O µ π β b Lecure 08 39

40 HMMs: Decdng T - - T Frward rcedure: Backward rcedure: Cmbnan: p O N p O µ α T N p O µ π β N µ α β b Lecure 08 40

41 HMMs: Inference and Tranng Three fundamenal quesns: Gven a mdel µ A B Π cmpue he prbably f a gven bservan sequence.e. poµ Frward-Backward. 2 Gven a mdel µ and an bservan sequence O cmpue he ms lkely hdden sae sequence Verb. X ˆ arg ma X O µ X 3 Gven an bservan sequence O fnd he mdel µ A B Π ha bes eplans he bserved daa EM. Gven bservan and sae sequence O X fnd µ ML. Lecure 08 4

42 Bes Sae Sequence wh Verb Algrhm T - - T X ˆ arg ma p X O µ X arg ma p X O µ X Tme cmpley? arg ma p T T T µ Lecure 08 42

43 The Verb Algrhm T - - T Xˆ p arg ma p T T Xˆ ma p T T The prbably f he ms prbable pah ha leads : δ ma p p Xˆ maδ T N T T Lecure 08 µ µ 43

44 The Verb Algrhm T - - T The prbably f he ms prbable pah ha leads : δ ma p I can be shwn ha: δ maδ ab N Lecure 08 Cmpare wh: α α a N b 44

45 The Verb Algrhm: Trells Cmpuan s δ ma s 2 δ 2 a b s 3 δ 3... s N δ N a 2 b a 3 b a N b s δ Lecure 08 45

46 The Verb Algrhm. Inalzan δ π b ψ 2. Recursn δ ψ 3. Termnan p Xˆ maδ T ˆ T 0 4. Sae sequence backrackng ψ ˆ maδ ab N argmaδ ab N N arg ma δ T ˆ N Lecure 08 Tme cmpley? 46

47 HMMs: Inference and Tranng Three fundamenal quesns: Gven a mdel µ A B Π cmpue he prbably f a gven bservan sequence.e. poµ Frward-Backward. 2 Gven a mdel µ and an bservan sequence O cmpue he ms lkely hdden sae sequence Verb. 3 Gven an bservan sequence O fnd he mdel µ A B Π ha bes eplans he bserved daa EM. Gven bservan and sae sequence O X fnd µ ML. Lecure 08 47

48 arameer Esman wh Mamum Lkelhd Gven bservan and sae sequences O X fnd µ ABΠ. 48 Lecure 08 s s p a ˆ s C s s C a k s k p b ˆ k s C s k C b arg ma ˆ µ µ µ X O p s p π X s C ˆ π Eercse: Rewre use Laplace smhng.

49 arameer Esman wh Epecan Mamzan Gven bservan sequences O fnd µ ABΠ. ˆ µ arg ma p O µ µ There s n knwn analyc mehd fnd slun. Lcally mamze poµ usng erave hll-clmbng: he Baum-Welch r Frward-Backward algrhm: - Gven a mdel µ and bservan sequence updae he mdel parameers µˆ beer f he bservans. - A specal case f he Epecan Mamzan mehd. Lecure 08 48

50 The Baum-Welch Algrhm EM [E] Assume µ s knwn cmpue hdden parameers ξ γ : ξ he prbably f beng n sae s a me and sae s a me. ξ T α a ξ m N b α β m β m epeced number f ransns frm s s 2 γ he prbably f beng n sae s a me. T γ ξ γ N epeced number f Lecure 08 ransns frm s 49

51 The Baum-Welch Algrhm [M] Re-esmae µ usng epecans f ξ γ : µˆ ˆ π γ aˆ bˆ k T { : T ξ γ k} T γ γ Baum has prven ha p O ˆ µ p O µ Lecure 08 5

52 The Baum-Welch Algrhm. Sar wh sme randm mdel µ ABΠ. 2. [E sep] Cmpue ξ γ and her epecans. 3. [M sep] Cmpue ML esmae µˆ. 4. Se µ ˆ µ and repea frm 2. unl cnvergence. Lecure 08 52

53 HMMs Three fundamenal quesns: Gven a mdel µ A B Π cmpue he prbably f a gven bservan sequence.e. poµ Frward/Backward. 2 Gven a mdel µ and an bservan sequence O cmpue he ms lkely hdden sae sequence Verb. 3 Gven an bservan sequence O fnd he mdel µ A B Π ha bes eplans he bserved daa Baum-Welch r EM. Gven bservan and sae sequence O X fnd µ ML. Lecure 08 53