Today (Ch 14) Markov chains and hidden Markov models Graphical representation Transition probability matrix Propagating state distributions The stationary distribution Next lecture (Ch 14) Markov chains and hidden Markov models Hidden Markov models Viterbi algorithm
Complete the sentence I had a glass of red wine with my grilled
Describing a sequence with a Markov chain Let XX 0, XX 1, be a sequence of discrete finite-valued random variables The sequence is a Markov chain if the probability of the current value XX tt only depends on the previous value XX tt 1 PP XX tt XX 1,, XX tt 1 = PP(XX tt XX tt 1 ) Also assume that the transition probabilities do not change with time PP XX tt XX tt 1 = PP XX tt 1 XX tt 2 = = PP(XX 1 XX 0 )
Markov chain: coin example Toss a fair coin until you see two heads in a row and then stop What is the probability of stopping after exactly 2 flips? What is the probability of stopping after exactly 3 flips? What is the probability of stopping after exactly 4 flips?
Markov chain as a graph: coin example
Forming a recurrence relation: coin example Let pp nn be the probability of stopping after exactly nn flips We already know that pp 1 = 0, pp 2 = 1 4, pp 3 = 1 8, pp 4 = If nn > 2, there are two ways the sequence can start Toss T and then end in exactly nn 1 tosses Toss HT and then end in exactly nn 2 tosses So we derive a recurrence relation 1 8 pp nn = 1 2 pp nn 1 + 1 4 pp nn 2
Transition probability matrix: weather example Let s model daily weather as one of three states (sunny, rainy, snowy) with transition probabilities shown in the diagram below These probabilities can be represented in a transition probability matrix
Transition probability matrix properties The transition probability matrix PP is a square matrix with entries pp iiii Since pp iiii = PP XX tt = jj XX tt 1 = ii pp iiii 0 and pp iiii = 1 jj
Probability distributions over states Let ππ be a row vector containing the probability distribution over states at tt = 0 ππ ii = PP(XX 0 = ii) Example: suppose that it is rainy today; then ππ = 0 1 0 Let pp (tt) be a row vector containing the probability distribution over states at tt pp ii (tt) = PP(XXtt = ii)
Propagating the probability distribution Propagating from tt = 0 to tt = 1, (1) pp jj = PP XX1 = jj = ii PP XX 1 = jj, XX 0 = ii = ii PP XX 1 = jj, XX 0 = ii PP(XX 0 = ii) = ii pp iiii ππ ii In matrix notation, pp (1) = ππpp
Probability distributions: weather example Suppose that it is rainy today, so we know that ππ = 0 1 0 What are the probability distributions for the weather tomorrow and the next day? 0.7 0.2 0.1 pp (1) = ππpp = 0 1 0 0.2 0.6 0.2 = 0.2 0.6 0.2 0.4 0.1 0.5 pp (2) = pp (1) PP = 0.2 0.6 0.2 0.7 0.2 0.1 0.2 0.6 0.2 0.4 0.1 0.5 = 0.28 0.40 0.32
Propagating to tt = We have just seen that pp (2) = pp (1) PP = ππpp PP = ππpp 2 In general pp (tt) = ππpp tt If you can reach any state from any other state, the Markov chain is called irreducible and has the following property lim tt ππpptt = ss where is ss called the stationary distribution of the Markov chain
Stationary distribution The stationary distribution ss has the property that sspp = ss In other words, ss is a row eigenvector of PP with eigenvalue of 1 Example: regardless of the initial probability distribution ππ, the stationary distribution for our daily weather model is ss = lim tt ππ 0.7 0.2 0.1 0.2 0.6 0.2 0.4 0.1 0.5 tt = 18 37 11 37 8 37
The billion dollar eigenvector
Randomly surfing a network of webpages
Initialize the distribution uniformly
Update the distribution iteratively...
until the stationary distribution is reached
Sometimes the surfer gets trapped
The teleportation fix The PageRank algorithm avoids the problem of getting trapped in a subnetwork of pages by allowing the web surfer to teleport from any webpage to any other with small probability Teleportation corresponds to entering a URL directly into the browser