18.175: Lecture 30 Markov chains

Size: px

Start display at page:

Download "18.175: Lecture 30 Markov chains"

Stanley Day
5 years ago
Views:

1 18.175: Lecture 30 Markov chains Scott Sheffield MIT

2 Outline Review what you know about finite state Markov chains Finite state ergodicity and stationarity More general setup

3 Outline Review what you know about finite state Markov chains Finite state ergodicity and stationarity More general setup

4 Markov chains Consider a sequence of random variables X 0, X 1, X 2,... each taking values in the same state space, which for now we take to be a finite set that we label by {0, 1,..., M}.

5 Markov chains Consider a sequence of random variables X 0, X 1, X 2,... each taking values in the same state space, which for now we take to be a finite set that we label by {0, 1,..., M}. Interpret X n as state of the system at time n.

6 Markov chains Consider a sequence of random variables X 0, X 1, X 2,... each taking values in the same state space, which for now we take to be a finite set that we label by {0, 1,..., M}. Interpret X n as state of the system at time n. Sequence is called a Markov chain if we have a fixed collection of numbers P ij (one for each pair i, j {0, 1,..., M}) such that whenever the system is in state i, there is probability P ij that system will next be in state j.

7 Markov chains Consider a sequence of random variables X 0, X 1, X 2,... each taking values in the same state space, which for now we take to be a finite set that we label by {0, 1,..., M}. Interpret X n as state of the system at time n. Sequence is called a Markov chain if we have a fixed collection of numbers P ij (one for each pair i, j {0, 1,..., M}) such that whenever the system is in state i, there is probability P ij that system will next be in state j. Precisely, P{X n+1 = j X n = i, X n 1 = i n 1,..., X 1 = i 1, X 0 = i 0 } = P ij.

8 Markov chains Consider a sequence of random variables X 0, X 1, X 2,... each taking values in the same state space, which for now we take to be a finite set that we label by {0, 1,..., M}. Interpret X n as state of the system at time n. Sequence is called a Markov chain if we have a fixed collection of numbers P ij (one for each pair i, j {0, 1,..., M}) such that whenever the system is in state i, there is probability P ij that system will next be in state j. Precisely, P{X n+1 = j X n = i, X n 1 = i n 1,..., X 1 = i 1, X 0 = i 0 } = P ij. Kind of an almost memoryless property. Probability distribution for next state depends only on the current state (and not on the rest of the state history).

9 Simple example For example, imagine a simple weather model with two states: rainy and sunny.

10 Simple example For example, imagine a simple weather model with two states: rainy and sunny. If it s rainy one day, there s a.5 chance it will be rainy the next day, a.5 chance it will be sunny.

11 Simple example For example, imagine a simple weather model with two states: rainy and sunny. If it s rainy one day, there s a.5 chance it will be rainy the next day, a.5 chance it will be sunny. If it s sunny one day, there s a.8 chance it will be sunny the next day, a.2 chance it will be rainy.

12 Simple example For example, imagine a simple weather model with two states: rainy and sunny. If it s rainy one day, there s a.5 chance it will be rainy the next day, a.5 chance it will be sunny. If it s sunny one day, there s a.8 chance it will be sunny the next day, a.2 chance it will be rainy. In this climate, sun tends to last longer than rain.

13 Simple example For example, imagine a simple weather model with two states: rainy and sunny. If it s rainy one day, there s a.5 chance it will be rainy the next day, a.5 chance it will be sunny. If it s sunny one day, there s a.8 chance it will be sunny the next day, a.2 chance it will be rainy. In this climate, sun tends to last longer than rain. Given that it is rainy today, how many days to I expect to have to wait to see a sunny day?

14 Simple example For example, imagine a simple weather model with two states: rainy and sunny. If it s rainy one day, there s a.5 chance it will be rainy the next day, a.5 chance it will be sunny. If it s sunny one day, there s a.8 chance it will be sunny the next day, a.2 chance it will be rainy. In this climate, sun tends to last longer than rain. Given that it is rainy today, how many days to I expect to have to wait to see a sunny day? Given that it is sunny today, how many days to I expect to have to wait to see a rainy day?

15 Simple example For example, imagine a simple weather model with two states: rainy and sunny. If it s rainy one day, there s a.5 chance it will be rainy the next day, a.5 chance it will be sunny. If it s sunny one day, there s a.8 chance it will be sunny the next day, a.2 chance it will be rainy. In this climate, sun tends to last longer than rain. Given that it is rainy today, how many days to I expect to have to wait to see a sunny day? Given that it is sunny today, how many days to I expect to have to wait to see a rainy day? Over the long haul, what fraction of days are sunny?

16 Matrix representation To describe a Markov chain, we need to define P ij for any i, j {0, 1,..., M}.

17 Matrix representation To describe a Markov chain, we need to define P ij for any i, j {0, 1,..., M}. It is convenient to represent the collection of transition probabilities P ij as a matrix: A = P 00 P P 0M P 10 P P 1M P M0 P M1... P MM

18 Matrix representation To describe a Markov chain, we need to define P ij for any i, j {0, 1,..., M}. It is convenient to represent the collection of transition probabilities P ij as a matrix: A = P 00 P P 0M P 10 P P 1M P M0 P M1... P MM For this to make sense, we require P ij 0 for all i, j and M j=0 P ij = 1 for each i. That is, the rows sum to one.

19 Transitions via matrices Suppose that p i is the probability that system is in state i at time zero.

20 Transitions via matrices Suppose that p i is the probability that system is in state i at time zero. What does the following product represent? P 00 P P 0M P 10 P P 1M ( ) p0 p 1... p M P M0 P M1... P MM

21 Transitions via matrices Suppose that p i is the probability that system is in state i at time zero. What does the following product represent? P 00 P P 0M P 10 P P 1M ( ) p0 p 1... p M P M0 P M1... P MM Answer: the probability distribution at time one.

22 Transitions via matrices Suppose that p i is the probability that system is in state i at time zero. What does the following product represent? P 00 P P 0M P 10 P P 1M ( ) p0 p 1... p M P M0 P M1... P MM Answer: the probability distribution at time one. How about the following product? ( p0 p 1... p M ) A n

23 Transitions via matrices Suppose that p i is the probability that system is in state i at time zero. What does the following product represent? P 00 P P 0M P 10 P P 1M ( ) p0 p 1... p M P M0 P M1... P MM Answer: the probability distribution at time one. How about the following product? ( p0 p 1... p M ) A n Answer: the probability distribution at time n.

24 Powers of transition matrix We write P (n) ij over n steps. for the probability to go from state i to state j

25 Powers of transition matrix We write P (n) ij for the probability to go from state i to state j over n steps. From the matrix point of view P (n) 00 P (n) P (n) 0M P (n) 10 P (n) P (n) P (n) M0 1M P (n) M1... P (n) MM = P 00 P P 0M P 10 P P 1M P M0 P M1... P MM n

26 Powers of transition matrix We write P (n) ij for the probability to go from state i to state j over n steps. From the matrix point of view P (n) 00 P (n) P (n) 0M P (n) 10 P (n) P (n) P (n) M0 1M P (n) M1... P (n) MM = P 00 P P 0M P 10 P P 1M P M0 P M1... P MM If A is the one-step transition matrix, then A n is the n-step transition matrix. n

27 Questions What does it mean if all of the rows are identical?

28 Questions What does it mean if all of the rows are identical? Answer: state sequence X i consists of i.i.d. random variables.

29 Questions What does it mean if all of the rows are identical? Answer: state sequence X i consists of i.i.d. random variables. What if matrix is the identity?

30 Questions What does it mean if all of the rows are identical? Answer: state sequence X i consists of i.i.d. random variables. What if matrix is the identity? Answer: states never change.

31 Questions What does it mean if all of the rows are identical? Answer: state sequence X i consists of i.i.d. random variables. What if matrix is the identity? Answer: states never change. What if each P ij is either one or zero?

32 Questions What does it mean if all of the rows are identical? Answer: state sequence X i consists of i.i.d. random variables. What if matrix is the identity? Answer: states never change. What if each P ij is either one or zero? Answer: state evolution is deterministic.

33 Simple example Consider the simple weather example: If it s rainy one day, there s a.5 chance it will be rainy the next day, a.5 chance it will be sunny. If it s sunny one day, there s a.8 chance it will be sunny the next day, a.2 chance it will be rainy.

34 Simple example Consider the simple weather example: If it s rainy one day, there s a.5 chance it will be rainy the next day, a.5 chance it will be sunny. If it s sunny one day, there s a.8 chance it will be sunny the next day, a.2 chance it will be rainy. Let rainy be state zero, sunny state one, and write the transition matrix by ( ).5.5 A =.2.8

35 Simple example Consider the simple weather example: If it s rainy one day, there s a.5 chance it will be rainy the next day, a.5 chance it will be sunny. If it s sunny one day, there s a.8 chance it will be sunny the next day, a.2 chance it will be rainy. Let rainy be state zero, sunny state one, and write the transition matrix by ( ).5.5 A =.2.8 Note that A 2 = ( )

36 Simple example Consider the simple weather example: If it s rainy one day, there s a.5 chance it will be rainy the next day, a.5 chance it will be sunny. If it s sunny one day, there s a.8 chance it will be sunny the next day, a.2 chance it will be rainy. Let rainy be state zero, sunny state one, and write the transition matrix by ( ).5.5 A =.2.8 Note that ( ) A = ( ) Can compute A 10 =

37 Does relationship status have the Markov property? In a relationship Single It s complicated Married Engaged

38 Does relationship status have the Markov property? In a relationship Single It s complicated Married Engaged Can we assign a probability to each arrow?

39 Does relationship status have the Markov property? In a relationship Single It s complicated Married Engaged Can we assign a probability to each arrow? Markov model implies time spent in any state (e.g., a marriage) before leaving is a geometric random variable.

40 Does relationship status have the Markov property? In a relationship Single It s complicated Married Engaged Can we assign a probability to each arrow? Markov model implies time spent in any state (e.g., a marriage) before leaving is a geometric random variable. Not true... Can we make a better model with more states?

41 Outline Review what you know about finite state Markov chains Finite state ergodicity and stationarity More general setup

42 Outline Review what you know about finite state Markov chains Finite state ergodicity and stationarity More general setup

43 Ergodic Markov chains Say Markov chain is ergodic if some power of the transition matrix has all non-zero entries.

44 Ergodic Markov chains Say Markov chain is ergodic if some power of the transition matrix has all non-zero entries. Turns out that if chain has this property, then π j := lim n P (n) ij exists and the π j are the unique non-negative solutions of π j = M k=0 π kp kj that sum to one.

45 Ergodic Markov chains Say Markov chain is ergodic if some power of the transition matrix has all non-zero entries. Turns out that if chain has this property, then π j := lim n P (n) ij exists and the π j are the unique non-negative solutions of π j = M k=0 π kp kj that sum to one. This means that the row vector π = ( ) π 0 π 1... π M is a left eigenvector of A with eigenvalue 1, i.e., πa = π.

46 Ergodic Markov chains Say Markov chain is ergodic if some power of the transition matrix has all non-zero entries. Turns out that if chain has this property, then π j := lim n P (n) ij exists and the π j are the unique non-negative solutions of π j = M k=0 π kp kj that sum to one. This means that the row vector π = ( ) π 0 π 1... π M is a left eigenvector of A with eigenvalue 1, i.e., πa = π. We call π the stationary distribution of the Markov chain.

47 Ergodic Markov chains Say Markov chain is ergodic if some power of the transition matrix has all non-zero entries. Turns out that if chain has this property, then π j := lim n P (n) ij exists and the π j are the unique non-negative solutions of π j = M k=0 π kp kj that sum to one. This means that the row vector π = ( ) π 0 π 1... π M is a left eigenvector of A with eigenvalue 1, i.e., πa = π. We call π the stationary distribution of the Markov chain. One can solve the system of linear equations π j = M k=0 π kp kj to compute the values π j. Equivalent to considering A fixed and solving πa = π. Or solving (A I )π = 0. This determines π up to a multiplicative constant, and fact that π j = 1 determines the constant.

48 Simple example If A = ( ), then we know πa = ( π 0 π 1 ) ( ) = ( ) π 0 π 1 = π.

49 Simple example If A = ( ), then we know πa = ( π 0 π 1 ) ( ) = ( ) π 0 π 1 = π. This means that.5π 0 +.2π 1 = π 0 and.5π 0 +.8π 1 = π 1 and we also know that π 1 + π 2 = 1. Solving these equations gives π 0 = 2/7 and π 1 = 5/7, so π = ( 2/7 5/7 ).

50 Simple example If A = ( ), then we know πa = ( π 0 π 1 ) ( ) = ( ) π 0 π 1 = π. This means that.5π 0 +.2π 1 = π 0 and.5π 0 +.8π 1 = π 1 and we also know that π 1 + π 2 = 1. Solving these equations gives π 0 = 2/7 and π 1 = 5/7, so π = ( 2/7 5/7 ). Indeed, πa = ( 2/7 5/7 ) ( ) = ( 2/7 5/7 ) = π.

51 Simple example If A = ( ), then we know πa = ( π 0 π 1 ) ( ) = ( ) π 0 π 1 = π. This means that.5π 0 +.2π 1 = π 0 and.5π 0 +.8π 1 = π 1 and we also know that π 1 + π 2 = 1. Solving these equations gives π 0 = 2/7 and π 1 = 5/7, so π = ( 2/7 5/7 ). Indeed, πa = ( 2/7 5/7 ) ( Recall ( that ) A 10 = ) = ( 2/7 5/7 ) = π. ( 2/7 5/7 2/7 5/7 ) = ( π π )

52 Outline Review what you know about finite state Markov chains Finite state ergodicity and stationarity More general setup

53 Outline Review what you know about finite state Markov chains Finite state ergodicity and stationarity More general setup

54 Markov chains: general definition Consider a measurable space (S, S).

55 Markov chains: general definition Consider a measurable space (S, S). A function p : S S R is a transition probability if

56 Markov chains: general definition Consider a measurable space (S, S). A function p : S S R is a transition probability if For each x S, A p(x, A) is a probability measure on S, S).

57 Markov chains: general definition Consider a measurable space (S, S). A function p : S S R is a transition probability if For each x S, A p(x, A) is a probability measure on S, S). For each A S, the map x p(x, A) is a measurable function.

58 Markov chains: general definition Consider a measurable space (S, S). A function p : S S R is a transition probability if For each x S, A p(x, A) is a probability measure on S, S). For each A S, the map x p(x, A) is a measurable function. Say that X n is a Markov chain w.r.t. F n with transition probability p if P(X n+1 B F n ) = p(x n, B).

59 Markov chains: general definition Consider a measurable space (S, S). A function p : S S R is a transition probability if For each x S, A p(x, A) is a probability measure on S, S). For each A S, the map x p(x, A) is a measurable function. Say that X n is a Markov chain w.r.t. F n with transition probability p if P(X n+1 B F n ) = p(x n, B). How do we construct an infinite Markov chain? Choose p and initial distribution µ on (S, S). For each n < write P(X j B j, 0 j n) = µ(dx 0 ) B 0 p(x 0, dx 1 ) B 1 B n p(x n 1, dx n ). Extend to n = by Kolmogorov s extension theorem.

60 Markov chains Definition, again: Say X n is a Markov chain w.r.t. F n with transition probability p if P(X n+1 B F n ) = p(x n, B).

61 Markov chains Definition, again: Say X n is a Markov chain w.r.t. F n with transition probability p if P(X n+1 B F n ) = p(x n, B). Construction, again: Fix initial distribution µ on (S, S). For each n < write P(X j B j, 0 j n) = µ(dx 0 ) B 0 p(x 0, dx 1 ) B 1 B n p(x n 1, dx n ). Extend to n = by Kolmogorov s extension theorem.

62 Markov chains Definition, again: Say X n is a Markov chain w.r.t. F n with transition probability p if P(X n+1 B F n ) = p(x n, B). Construction, again: Fix initial distribution µ on (S, S). For each n < write P(X j B j, 0 j n) = µ(dx 0 ) B 0 p(x 0, dx 1 ) B 1 B n p(x n 1, dx n ). Extend to n = by Kolmogorov s extension theorem. Notation: Extension produces probability measure P µ on sequence space (S 0,1,..., S 0,1,... ).

63 Markov chains Definition, again: Say X n is a Markov chain w.r.t. F n with transition probability p if P(X n+1 B F n ) = p(x n, B). Construction, again: Fix initial distribution µ on (S, S). For each n < write P(X j B j, 0 j n) = µ(dx 0 ) B 0 p(x 0, dx 1 ) B 1 B n p(x n 1, dx n ). Extend to n = by Kolmogorov s extension theorem. Notation: Extension produces probability measure P µ on sequence space (S 0,1,..., S 0,1,... ). Theorem: (X 0, X 1,...) chosen from P µ is Markov chain.

64 Markov chains Definition, again: Say X n is a Markov chain w.r.t. F n with transition probability p if P(X n+1 B F n ) = p(x n, B). Construction, again: Fix initial distribution µ on (S, S). For each n < write P(X j B j, 0 j n) = µ(dx 0 ) B 0 p(x 0, dx 1 ) B 1 B n p(x n 1, dx n ). Extend to n = by Kolmogorov s extension theorem. Notation: Extension produces probability measure P µ on sequence space (S 0,1,..., S 0,1,... ). Theorem: (X 0, X 1,...) chosen from P µ is Markov chain. Theorem: If X n is any Markov chain with initial distribution µ and transition p, then finite dim. probabilities are as above.

65 Examples Random walks on R d.

66 Examples Random walks on R d. Branching processes: p(i, j) = P ( im=1 ξ m = j ) where ξ i are i.i.d. non-negative integer-valued random variables.

67 Examples Random walks on R d. Branching processes: p(i, j) = P ( im=1 ξ m = j ) where ξ i are i.i.d. non-negative integer-valued random variables. Renewal chain.

68 Examples Random walks on R d. Branching processes: p(i, j) = P ( im=1 ξ m = j ) where ξ i are i.i.d. non-negative integer-valued random variables. Renewal chain. Card shuffling.

69 Examples Random walks on R d. Branching processes: p(i, j) = P ( im=1 ξ m = j ) where ξ i are i.i.d. non-negative integer-valued random variables. Renewal chain. Card shuffling. Ehrenfest chain.

18.600: Lecture 32 Markov Chains

18.600: Lecture 32 Markov Chains Scott Sheffield MIT Outline Markov chains Examples Ergodicity and stationarity Outline Markov chains Examples Ergodicity and stationarity Markov chains Consider a sequence