1 if the i-th toss is a Head (with probability p) ξ i = 0 if the i-th toss is a Tail (with probability 1 p)

Size: px

Start display at page:

Download "1 if the i-th toss is a Head (with probability p) ξ i = 0 if the i-th toss is a Tail (with probability 1 p)"

Elisabeth Gallagher
5 years ago
Views:

1 6 Chapter 3. Markov Chain: Introduction Whatever happened in the past, be it glory or misery, be Markov! 3.1. Examples Example 3.1. (Coin Tossing.) Let ξ 0 0 and, for i 1, 1 if the i-th toss is a Head (with probability p) ξ i 0 if the i-th toss is a Tail (with probability 1 p) Set X n n i0 ξ i, n 0. X n is the random number which counts the number of Heads up to the n-th toss. Then, P(X n+1 j X 0 0, X 1 i 1,..., X n 1 i n 1, X n i) P(ξn+1 1 X 0 0, X 1 i 1,..., X n 1 i n 1, X n i) if j i + 1 P(ξ n+1 0 X 0 0, X 1 i 1,..., X n 1 i n 1, X n i) if j i P(ξn+1 1) if j i + 1 P(ξ n+1 0) if j i p if j i p if j i P(ξn+1 1 X n i) if j i + 1 P(ξ n+1 0 X n i) if j i P(X n+1 j X n i) It implies, once the present X n is fixed, the past history X 0,..., X n 1, shall not affect the future distribution of X n+1. Here, we are using time n as present, time before n is past, and time beyond n is future. The above statement is the same as saying that the future X k : k n + 1} and the past X k : k n 1} are conditionally independent given the present X n (taking any fixed value.) Remark. Not only n + 1 but, all future distribution X n+1, X n+2,... shall depend on the past and now only through now. Example 3.2. (Mickey in Maze) Mickey mouse travels in a maze with nine 3 3 cells. The cells are numbered as 0, 1,..., 8 from left to right and top down. Each step Mickey travels from where it is to one of the surrounding connected cells with equal chance

2 7 Let X n denote the cell number of Mickey at step n. (X 0 4). Then, P(X n+1 j X 0 0, X 1 i 1,..., X n 1 i n 1, X n i) P(X n+1 j X n i). Suppose currently Mickey is in cell 5, for example, the future movement or path of Mickey is irrelevant with the past movement or path of Mickey. In other words, how Mickey has got to cell 5 in the past has nothing to do with how Mickey would move around in the future. The process X n : n 0, 1, 2,...} is a Markov chain. Example for fun ( First Blood. ) John Rambo only obeys the order from Colonel Samuel Trautman, who supposedly only obeys the order from the Pentagon. Then Pentagon Trautman Rambo forms a MC Definitions/Descriptions 1. Stochastic Process: a family of random variables X t } index by t. 2. State space: the set of values of the stochastic processes, which in general does not have to be real numbers. 3. Markov Process: A stochastic process X t } indexed by time t such that, at each time t, the future of the process X s : s > t} is conditionally independent of the past of the process X s : s < t} given the present of the process X t taking any fixed value. Another interpretation is: at each time t, the future of the process X s : s > t} depends on the the past of the process X s : s < t} only through the present X t. Caution: X s : s > t is in general not independent of X s : s < t. 4. Markov chain (M C): Markov process with discrete state space. Discrete state space is usually denoted by numbers 0, 1, 2, Discrete/continuous time Markov chain: Discrete time M C: the time domain is 0, 1, 2,...}. Continuous time MC: the time domain is [0, ). Examples 3.1 and 3.2 are discrete time Markov chains. Poisson process is continuous time M C. Brownian motion is a continuous time Markov process (but not MC). For a discrete time MC: X n : n 0, 1,...}: the defining equation is, for any n 0, P(X n+1 j X 0 0, X 1 i 1,..., X n 1 i n 1, X n i) P(X n+1 j X n i) Transition probabilities. One-step transition probability: P(X n+1 j X n i). If one-step transition probability is irrelevant with n, i.e., it s the same for all n, we call the MC X t } a MC with stationary transition probabilities. Throughout the course, we only consider MC with stationary transition probabilities. Let P P(X n+1 j X n i). The one-step transition probability matrix, or transition matrix in brief, is P P 00 P 01 P P 10 P 11 P P 20 P 21 P

3 8 For the Mickey in Maze (Example 3.2) example: P Denote the k-step transition probability as P (k) probability matrix as ( 1/4 1/4 1/4 1/4 P (k) P(X n+k j X n i) and the k-step transition P (k) For notational convenience, we always let P (1) P, P (0) 1 if i j 0 if i j. Then, Theorem 3.1. For all 0 m n and n 0, Proof. Write P (n) Observe that P (n) P (n) P (n) P n. ) P (m) il P (n m) lj P(X n j X 0 i) P(X n j, X 0 i)/p(x 0 i) P(X n j, X m l, X 0 i)/p(x 0 i) P(X n j X m l, X 0 i)p(x m l, X 0 i)/p(x 0 i) P(X n j X m l, X 0 i)p(x m l X 0 i) P(X n j X m l)p(x m l, X 0 i)/p(x 0 i) P (n m) lj P (m) il. is the (i + 1, j + 1)-th entry of the matrix P (n), P (m) il, l 0, 1, 2,... are the, l 0, 1, 2,... are the (j +1)-th column of the matrix (i+1)-th row of the matrix P (m), and P (n m) lj P (n m). Hence, P (n) P (m) P (n m)

4 9 for all 0 m n. Since P (1) P, we have P (2) PP P 2, P (3) PP (2) P 3,... P (n) P n by induction. Example 3.3 (Michey in Maze, Example 3.2 continued) Compute P (3) 48 Solution. Write P (3) P (3) k0 P 4 k P (2) k 8 k1,3,5,7 P 4 k P (2) k 8 1/4 k1,3,5,7 P (2) k 8 0; 1/3 1/2 1/3 + 1/3 1/4 1/3 + 1/3 1/4 1/3 0. and P (3) More Examples. Example 3.4. (An Inventory Model) Let X n be the number of TV sets at a store in the end of day n with X 0 2. Let ξ n be the sales of the number of TVs on day n. Assume ξ 1, ξ 2,... are iid (independent, identically distributed) such that 0.5 i 0 P(ξ n i) 0.4 i i 2 At the end of any day n, if X n 0 or 1, two TVs will be sent to the store overnight. Moreover, in the case of X n 1, another TV will be sent directly to the customer s house. If X n 1 or 2, nothing happens. With this inventory policy, Xn ξ X n+1 n+1 if X n 1, 2 2 ξ n+1 if X n 1, 0. Then, X 0, X 1, X 2,..., is a MC with state space 1, 0, 1, 2} and with one-step transition probability P Example 3.5. (The Ehrenfest Model) There are 2N particles in a jar separated by a membrane into two chambers A and B. Let Y n be the number of particles in A after n crossings. Each crossing is a particle from A to B or from B to A. Assume when any once crossing happens, it happens to any one of the 2N particles with each equal chance 1/2N. A B

5 10 Then Y n, n 0 is a MC with state space 0, 1, 2,..., 2N} and with one step transition probability: i/(2n) if j i 1 P P(Y n+1 j Y n i) 1 i/(2n) if j i else; for 0 i j 2N.

6 11 More DIY Exercises: Exercise 3.1 Bunny rabbit has three dens A, B and C. It likes A better than B and C. If it s in B or C on any night, it will always take chance 0.9 to go to A and chance 0.1 to go to the other den for the following night. Once it reaches A, it will stay there for two nights and the third night will be in B or C with equal chance 1/2. Let X n be the den Bunny stays for night n. What is the state space of the X n }? Is X n } a MC? Den C Den A Den B Exercise 3.2 For three events A, B and C, the following three statements are equivalent: (i) P(A B C) P(A C)P(B C); (ii). P(A C B) P(A C); and (iii) P(B A C) P(B C). (Assume all quantities here are well defined.) Notice that statement (i) says that A and B are conditionally independent given C. Exercise 3.3 Suppose X n, n 0, 1, 2,... is a discrete time MC. Let 0 n 0 < n 1 < n 2 <... be a subsequence of the nonnegative integers and Y k X nk. Is Y k : k 0, 1, 2,..., } a MC? Exercise 3.4 Suppose X n : n 2, 1, 0, 1, 2, } is a discrete time MC with the time being from all integers from to. Let Y n X n for all integers n. Is Y n : n 2, 1, 0, 1, 2, } a MC?

Math 180B Problem Set 3

Math 180B Problem Set 3 Problem 1. (Exercise 3.1.2) Solution. By the definition of conditional probabilities we have Pr{X 2 = 1, X 3 = 1 X 1 = 0} = Pr{X 3 = 1 X 2 = 1, X 1 = 0} Pr{X 2 = 1 X 1 = 0} = P