Markov Models. Machine Learning & Data Mining. Prof. Alexander Ihler. Some slides adapted from Andrew Moore s lectures

Transcription

1 Markov Models Machine Learning & Data Mining Prof. Alexander Ihler Some slides adapted from Andrew Moore s lectures

2 Markov system System has d states, s s d Discrete Eme intervals, t=0,,,t At Eme t, system is in state x t At each t, system transieons to another state according to d = 3 t = 0 x 0 = 2 State transieon diagram S Current state Bayes Net on states x over Eme: a Markov chain (think of this as a d d matrix P ) x 0 x x 2 Each condieonal probability distribueon is idenecal ( homogeneous )

3 Markov system Another view: lavce of states State sequence = path in lavce State transieon diagram S

4 Ex: Wumpus World Person & Wumpus in cave Wander randomly Cave is dark; assume layout known State = (locaeon of person, locaeon of wumpus) # states? (6) x (6) = 256; IniEal state distribueon, p(x 0 )? Dynamics: Person & Wumpus wander randomly to adjacent square at each Eme Some possible queseons: What s the expected Eme unel the Wumpus eats us? What s the probability we find the gold first? What s the probability the Wumpus will eat us at the next step?

5 Ex: Wumpus World Person & Wumpus in cave Wander randomly Cave is dark; assume layout known Example: Given that it s Eme t and we re OK, what s the probability the wumpus eats us at Eme t+? If We are omnipotent (see enere cave) easy to compute from dynamics If we re blind (no informaeon at all) Markov model If we have some indirect informaeon hidden Markov model

6 CompuEng probabiliees S How to compute the state distribueon at Eme t? Simple answer: enumerate over all paths. Ex t=2, x 0 =2: p([2,,]) =.33*0 p([2,,2]) =.33*0 p([2,,3]) =.33* p([2,2,]) =.66*.33 p([2,2,2]) =.66*.66 Problem: number of paths of length t? O(d t ) How can we use the structure of the problem? (e.g., lavce) Use induceon ( dynamic programming )

7 CompuEng probabiliees S We can compute the state distribueon at Eme t: T = T

8 CompuEng probabiliees S We can compute the state distribueon at Eme t: T = T ComputaEon? O( t d 2 ) What s the state occupancy distribueon in the far future? Does it depend on x 0? In general: p(x t ) = p 0 P P = p 0 (P) t

9 CompuEng probabiliees S We can compute the state distribueon at Eme t: Notes: StaEonary distribueon: s(x) exists & is unique, so that p(x t ) becomes independent of p(x 0 ), if: (a) p(..) is irreducible: (b) p(..) is acyclic: Ex: if not (a): (Long-term prob will depend on inieal state dist) Ex: if not (b):

10 Dynamic programming S Observe, say, x 4 = 2 What s the (value of the) most likely state sequence? ???

11 Dynamic programming S Observe, say, x 4 = 2 What s the (value of the) most likely state sequence????

12 Dynamic programming S Observe, say, x 4 = 2 What s the (value of the) most likely state sequence????

13 Dynamic programming S Observe, say, x 4 = 2 What s the (value of the) most likely state sequence????

14 Dynamic programming S Observe, say, x 4 = 2 What s the (value of the) most likely state sequence? =

15 Dynamic programming S Observe, say, x 4 = 2 What s the (value of the) most likely state sequence? =

16 Dynamic programming S Observe, say, x 4 = 2 What s the (value of the) most likely state sequence? x 0 x x 2 x 3 x =

17 Dynamic programming S Observe, say, x 4 = 2 Similar algorithm for compueng marginals: x 0 x x 2 x 3 x 4

18 Hidden Markov Model S In addieon to the Markov state variables x t We also have emission variables, o t Model is specified by Bayes Net on states x t and observaeons o t over Eme t x 0 x x 2 o 0 o o 2 Typically, we ll observe the values of the o s (shaded) Induces a model over the x s, and use this to answer queries about x s

19 Ex: Wumpus World Person & Wumpus in cave Wander randomly Cave is dark; assume layout known Observe a bit about state Walls (tell us something about our locaeon) Breeze (tell us we are next to one of the pits) Smell (tell us something about Wumpus locaeon) SEll can t observe the complete state, but more informaeon now

20 Ex: Hidden Markov model IniEal state distribueon S A,B B,C State transieon probabiliees d = 3 t = 0 x 0 = 2 A,C ObservaEon probabiliites (think of this as a d d matrix P ) (think of this as a k d matrix Q )

21 A,B Ex: state esemaeon S IniEal distribueon: Observe A: * * * = ) Observe: [A,A,B,A, ] What state are we in? Depends on both P, Q posterior A,C B,C = ) T = Observe B: T * * * 0 = = )

22 State esemaeon: filtering EsEmate state distribueon at Eme t given observaeons up to t Forward messages: Z t is the scalar that normalizes f t (x t ): ObservaEon likelihood: x 0 x x 2 x 3 x 4 o 0 o o 2 o 3 o 4

23 State esemaeon: smoothing EsEmate state distribueon at Eme t given future observaeons Forward messages: Reverse messages: Z t is the scalar that normalizes f t (x t ): ObservaEon likelihood: x 0 x x 2 x 3 x 4 o 0 o o 2 o 3 o 4 Marginal probabiliees:

24 Example HMM applicaeons Robot state estimation (animation: Deiter Fox, UW) AcEvity recognieon (from [Garcia-Ceja et al. 204]) Speech recognieon (image from Dan Ellis webpage)

25 Summary Markov models, hidden Markov models Dynamic programming For state distribueon at Eme t ( forward-backward ) For most probable sequence of states ( Viterbi ) Learning HMMs ExpectaEon-MaximizaEon (also Baum-Welch )