Solution: (Course X071570: Stochastic Processes)

Transcription

1 Solution I (Course X071570: Stochastic Processes) October 24, 2013 Exercise 1.1: Find all functions f from the integers to the real numbers satisfying f(n) = 1 2 f(n + 1) + 1 f(n 1) 1. 2 A secial solution to the equation is f(n) = n 2. The corresonding homogeneous difference equation has solution f 1 (n) = 1, f 2 (n) = n. Hence, a general solution is f(n) = n 2 + c 1 + c 2 n, c 1, c 2 R. Exercise 1.2: Let the one-ste transition robability matrix of a two-state Markov chain be given by ( ) 1 P =. 1 Show by induction that the n ste transition robability matrix is given by ( P n (2 1) n (2 1) = n ) (2 1) n (2 1) n. Omitted. Exercise 1.3: Suose four Markov chains have the following one-ste transition robability matrices resectively. Secify the communication classes of the Markov chains, and determine whether they are transient, recurrent and/or absorbent: P 1 = , P 2 = P 3 = , P = /3 2/

2 P 1. One communication class, it is ositive recurrent because there is a unique invariant robability vector π = (1/3, 1/3, 1/3). P 2. One communication class, it is ositive recurrent because there is a unique invariant robability vector π = (1/6, 1/6, 1/3, 1/3). P 3. Denote the labels for the columns by A, B,C,D,E sequentially. There are three communication classes: (1){A, C}, recurrent (irreducible, aeriodic); (2){D, E},recurrent (irreducible, aeriodic); (3){B}, transient. P 4. Denote the labels for the columns by A, B,C,D,E sequentially. There are four communication classes: (1){A, B}, recurrent (irreducible, aeriodic); (2){C},recurrent (absorbing); (3){D}, transient;(4){e}, transient. Exercise 1.4: Let X n be an irreducible Markov chain on the state sace {1, 2,, N}. Show that there exists C < + and ρ < 1 such that for any states i, j, P (X m j, 0 m n X 0 = i) Cρ n. Show that this imlies that E(T ) < +, where T is the first time that the Markov chain reaches the state j if the Markov chain started in state i. For fixed i, j, denote S = {1, 2,, N}\{j} and i = Z 0, i j imlies that there exists Z 1, Z 2,, Z m S such that P (Z 0, Z 1 )P (Z 1, Z 2 ) P (Z m, j) > 0. If m N 1, then there are at least two Zs are identical, without loss of generality, assume Z k1 = Z k2, k 1 < k 2. Note that P (Z 0, Z 1 )P (Z 1, Z 2 ) P (Z m, j) P (Z 0, Z 1 )P (Z 1, Z 2 ) P (k 2 k 1 ) (Z k1, Z k2 ) P (Z m, j), consider the new sequence (Z 1, Z 2,, Z k1, Z k2 +1,, Z m ),then we have found a shorter ath for the Markov Chain to reach j when started in i. It can be easily seen that the shortest ath aears only when m < N 1. That is, there exists a δ i,j > 0 such that, P (X m = j for some 0 m N X 0 = i) δ i,j. Let δ = min{δ i,j : i, j S}, since S is finite, we have δ > 0 and for all i, j, P (X m = j for some 0 m N X 0 = i) δ. 2

3 Write n = N + q where, q are nonnegative integers and q < N, when q 1, note that = P (X m j, 0 m n X 0 = i) P (i, x 1 ) P (X m = x m, (k 1)N + 1 m kn X (k 1)N = x (k 1)N ) (x 1,x 2,...,x n):x k S P (X m = x m, N + 1 m n X N = x N ) P (i, x 1 ) x 1 :x 1 S P (X m = x m, (k 1)N + 1 m kn X (k 1)N = x (k 1)N ) x m S,(k 1)N+1 m kn x m S,( 1)N+1 m n P (X m = x m, ( 1)N + 1 m n X ( 1)N = x ( 1)N ) max{p (X m j, (k 1)N + 1 m kn X (k 1)N = i) : i S } max{p (X m j, kn + 1 m n X kn = i) : i S } (1 δ) n/n. The above inequality also holds when q = 0. Let C = 1, ρ = (1 δ) 1/N, we get the first art of the desired results. The second art follows naturally. Exercise 1.5: Consider three urns, a red one, a white one and a blue one. The red urn contains 1 red and 4 blue balls; the white urn contains 3 white balls, 2 red balls, and 2 blue balls; the blue urn contains 4 white balls, 3 red balls and 2 blue balls. At the initial ste a ball is randomly icked from the red urn. At every subsequent stage, a ball is randomly selected from the urn whose colour is the same as that of the ball reviously icked and is then returned to that urn. Argue that the sequence of ball colours is a Markov chain by finding the transition matrix. What if the balls are not returned to the resective urn? If returned, the transition matrix is 1/5 0 4/5 P = 2/7 3/7 2/7. 3/9 4/9 2/9 If not returned, it is not a Markov Chain, for instance, P (X 3 = red X 2 = blue, X 1 = blue, X 0 = red) = 3/8 P (X 3 = red X 2 = blue, X 1 = red, X 0 = red) = 3/9. Exercise 1.6: M balls are initially distributed among m urns. At each stage one of the balls is selected at random, taken from whichever urn it is in, and then laced, at random, in one 3

4 of the other m 1 urns. Consider the Markov chain whose state at any time is the vector (n 1,, n m ) where n i denotes the number of balls in urn i. Guess the limiting robabilities for this Markov chain and then verify your guess. Guess: the limiting distribution is the Multinomial distribution π((n 1, n 2,, n m )) = M! n 1!n 2! n m!m M, where n i 0, n 1 +n 2 + +n m = M. Now for two vectors (n 1, n 2,, n m ) and (n 1, n 2,, n i + 1,, n j 1,, n m ), if i j, n j > 0, then the transition robability is Therefore, we have P ((n 1, n 2,, n i + 1,, n j 1,, n m ), (n 1, n 2,, n m )) = P (a ball is icked from the i th urn and laced in the j th urn) = n i + 1 M 1 m 1. 1 i j m = = π((n 1, n 2,, n i + 1,, n j 1,, n m )) P ((n 1, n 2,, n i + 1,, n j 1,, n m ), (n 1, n 2,, n m )) M! n i + 1 n 1!n 2! (n i + 1)! (n j 1)! n m!m M M 1 m 1 1 i j m M! n 1!n 2! n m!m M = π((n 1, n 2,, n m )). Or, you can also check that 1 i j m n j M(m 1) π((n 1, n 2,, n i + 1,, n j 1,, n m )) P ((n 1, n 2,, n i + 1,, n j 1,, n m ), (n 1, n 2,, n m )) = π((n 1, n 2,, n m )) P ((n 1, n 2,, n m ), (n 1, n 2,, n i + 1,, n j 1,, n m )). Exercise 1.7: Consider simle random walk on the grah below. 4

5 (a). In the long run, about what fraction of time is sent in vertex A? (b). Suose a walker starts in vertex A. walker returns to A? What is the exected number of stes until the (c). Suose a walker starts in vertex C. What is the exected number of visits to B before the walker reaches A? (d). Suose the walker starts in vertex B. What is the robability that the walker reaches A before the walker reaches C? (e). Again assume the walker starts in vertex C. What is the exected number of stes until the walker reaches A? Firstly, the one-ste transition robability matrix can be easily obtained: 0 1/3 1/3 1/3 0 1/3 0 1/3 0 1/3 P = 1/2 1/ / /2 0 1/2 0 1/2 0 we can further claim that this is an irreducible, aeriodic, discrete-time Markov Chain on a finite state sace. The invariant distribution π = (1/4, 1/4, 1/6, 1/6, 1/6) by solving πp = π. Or, you can also obtain π by exlaining clearly that the unique invariant distribution is roortional to the degree of each vertex. Now define (a). 1/4; (b). 4; τ x = inf{n 1 : X n = x}, x {A, B, C, D, E} (c). Define function f(x) = P (τ B < τ A X 0 = x), by adoting the one-ste analysis strategy, we have f(c) = P (τ B < τ A, X 1 = A X 0 = C) + P (τ B < τ A, X 1 = B X 0 = C) = 1/2. For other values of f, similarly, we can obtain the following system of equations: f(a) = 1/3 + 1/3f(C) + 1/3f(D) f(b) = 1/3f(C) + 1/3f(E) f(e) = 1/2 + 1/2f(D) f(d) = 1/2f(E). Therefore, f(b) = 7/18, f(d) = 1/3, f(e) = 2/3. Now, let N be the number of visits to B before reaching A if started in C. Then, we can use the Markov roerty to show that for any k 1, P (N = k) = f(c)f k 1 (B)(1 f(b)) and P (N = 0) = 1 f(c) = 1/2.Thus, EN = f(c) + k f k 1 (B)(1 f(b)) = 1/2 1/(1 f(b)) = 9/11. (Geometric distribution with arameter 1 f(b)) 5

6 (d). Define function g(x) = P (τ A < τ C X 0 = x), by adoting the one-ste analysis strategy, we can obtain the following system of equations: g(b) = 1/3 + 1/3g(E) g(e) = 1/2g(B) + 1/2g(D) g(d) = 1/2 + 1/2g(E). Therefore, g(b) = 4/7, g(d) = 6/7, g(e) = 5/7. The solution is 4/7. (e). Define T = inf{k : X k = A} and function h(x) = E(T X 0 = x), by adoting the one-ste analysis strategy, we can obtain the following system of equations: h(c) = 1/2 + 1/2(1 + h(b)) h(b) = 1/3 + 1/3(1 + h(c)) + 1/3(1 + h(e)) h(e) = 1/2(1 + h(b)) + 1/2(1 + h(d)) h(d) = 1/2 + 1/2(1 + h(e)). Therefore, h(b) = 36/11, h(c) = 29/11, h(d) = 34/11, h(e) = 46/11. The solution is 29/11. Exercise 1.8: Let X 1, X 2, be the successive values from indeendent rolls of a standard sixsided die. Let S n = n X i and i=1 T 1 = min{n 1 : S n is divisible by 8}, T 2 = min{n 1 : S n 1 is divisible by 8}, Find E(T 1 ) and E(T 2 ). Define X n as the remainder after division of S n by 8, X 0 = 0, then X n is a Markov Chain with transition robability matrix 0 1/6 1/6 1/6 1/6 1/6 1/ /6 1/6 1/6 1/6 1/6 1/6 1/ /6 1/6 1/6 1/6 1/6 P = 1/6 1/ /6 1/6 1/6 1/6 1/6 1/6 1/ /6 1/6 1/6 1/6 1/6 1/6 1/ /6 1/6 1/6 1/6 1/6 1/ /6 1/6 1/6 1/6 1/6 1/6 1/6 0 0 The invariant robability distribution is π = (1/8, 1/8, 1/8, 1/8, 1/8, 1/8, 1/8, 1/8). Define τ x = inf{n 1 : X n = x}, then E(T 1 ) = E(τ 0 X 0 = 0) = 8, E(T 2 ) = E(τ 1 X 0 = 0). Let 6

7 h(x) = E(τ 1 X 0 = x), the one-ste analysis suggests that h(1) = 8 h(0) = 1 + 1/6(h(2) + h(3) + h(4) + h(5) + h(6)) h(2) = 1 + 1/6(h(0) + h(3) + h(4) + h(5) + h(6) + h(7)) h(3) = 1 + 1/6(h(0) + h(4) + h(5) + h(6) + h(7)) h(4) = 1 + 1/6(h(0) + h(2) + h(5) + h(6) + h(7)) h(5) = 1 + 1/6(h(0) + h(2) + h(3) + h(6) + h(7)) h(6) = 1 + 1/6(h(0) + h(2) + h(3) + h(4) + h(7)) h(7) = 1 + 1/6(h(0) + h(2) + h(3) + h(4) + h(5)). Therefore, h(0) = , h(2) = , h(3) = , h(4) = , h(5) = , h(6) = , h(7) = That is, E(T 2 ) = Exercise 1.9: Let X n be the Markov chain with state sace Z and transition robability where > 1/2. Assume X 0 = 0. P (n, n + 1) =, P (n, n 1) = 1, (a). Let Y = min{x 0, X 1, }. What is the distribution of Y? (b). For ositive integer k, let T k = min{n : X n = k} and let e(k) = E(T k ). Exlain why e(k) = ke(1). (c). Find e(1). (d). Use (c) to give another roof that e(1) = if = 1/2. (a). Define τ i = inf{k : X k = i}. For a fixed sufficiently large ositive integer N and any fixed ositive integer k, define f(x) = P (τ k 1 > τ N X 0 = x) for k 1 x N, then we find the recursive formula f(x) = (1 )f(x 1) + f(x + 1), which imlies f(x) = c 1 + c 2 ( 1 )x, combined with the boundary conditions f( k 1) = 0, f(n) = 1, we have Therefore, c 1 = ( 1 ) k 1 ( 1 )N ( 1, c ) k 1 2 = f(0) = 1 ( 1 )N ( 1. ) k 1 1 ( 1 ) k 1 ( 1 )N ( 1. ) k 1 Note that f(0) is a decreasing sequence in N and bounded, by continuity, P (Y k) = lim f(0) = 1 (1 N + )k+1. That is, P ( Y = k) = P (Y k) P (Y k + 1) = (1 1 )( 1 )k, k 0. 7

8 (b). Write X n = n ξ k, where ξ 1, ξ 2,, ξ n are i.i.d random variables from ξ, where P (ξ = 1) =, P (ξ = 1) = 1. Note that T 1 = min{m : ξ ξ m = 1}, for general i, T k T k 1 = min{m : ξ Tk ξ Tk ξ Tk 1 +m = 1}, therefore, T 1, T 2 T 1,, T k T k 1 have the same distribution, which imlies E(T k ) = ke(1). (c). Write X n = n ξ k, where ξ 1, ξ 2,, ξ n are i.i.d random variables from ξ, where P (ξ = 1) =, P (ξ = 1) = 1. By definition, T 1 E(X T1 ) = E ξ k = E + ξ k I(T 1 k) Note that {T 1 k} = { m ξ j 1 : m k 1}, hence ξ k and I(T 1 k) are indeendent. Therefore, j=1 E(X T1 ) = Eξ + P (T 1 k) = EξE(T 1 ). Note that X T1 1, Eξ = 2 1, therefore, e(1) = 1/(2 1). (d). Omitted. Exercise 1.10: Suose J 1, J 2, are indeendent random variables with P (J i = 1) = 1 P (J i = 0) =. Let k be a ositive integer and let T k be the first time that k consecutive 1s have aeared. In other words, T k = n if J i = 1 for all n (k 1) i n and there is no m < n such that J i = 1 for all m (k 1) i m. Let X 0 = 0 and for n > 0, let X n be the number of consecutive 1s in the last run, i.e. X n = k if J n k = 0 and J i = 1 for n (k 1) i n. (a). Exlain why X n is a Markov chain with state sace {0, 1, 2, } and give the transition robabilities. (b). Show that the chain is irreducible and ositive recurrent and give the invariant robability π. (c). Find E(T k ) by writing an equation for E(T k ) in terms of E(T k 1 ) and then solving the recursive equation. (d). Find E(T k ) in a different way. Suose the chain starts in state k and let ˆT k be the time until returning to state k and ˆT 0 the time until the chain reaches state 0. Exlain why E( ˆT k ) = E( ˆT 0 ) + E(T k ), find E( ˆT 0 ) and use art (b) to determine E( ˆT k ). (a). k 0, P (k, k + 1) =, P (k, 0) = 1. 8

9 (b). For any two i, j, without loss of generality, assume i < j, we have P j i (i, j) P (i, i + 1)P (i + 1, i + 2) P (j 1, j) = j i > 0. Therefore, it is irreducible. Now solve the equations π(k)p (k, m) = π(m), k 0 use (a), we have the following recursive formula: m 1, π(m 1) = π(m); π(k)(1 ) = π(0). k 0 Under the constraint that k 0 π(k) = 1, we have π(m) = m (1 ), m 0. (c). Denote event E i as the i th aearance of (1, 1,, 1, 0) after a 0, i.e, k consecutive 1s followed by a 0 if J 0 is defined to be 0. Note that E 1, E 2, are indeendent and if we let Y n denote the value after the nth k consecutive 1s, then the first aearance of k + 1 consecutive occurs right after E 1, E 2,, E m if and only if Y 1 = Y 2 = = Y m = 0, Y m+1 = 1. That is, the first 1 aears at time m + 1 for Y n. Since Y n actually is a subsequence of J n, this m + 1 now is T 1 by definition. Therefore, E(T k+1 ) = (E(T k ) + 1) E(T 1 ) Note that T 1 follows a geometric distribution with arameter, i.e, P (T 1 = k) = (1 ) k 1, k 1, thus E(T 1 ) = 1/. The equation becomes E(T k+1 ) 1/( 1) = 1/(E(T k ) 1/( 1)). Solve the formula recursively, we obtain E(T k ) = (1 k )/( k (1 )). (d). Use (b), we know E( ˆT k ) = 1/( k (1 )). Since ˆT 0 follows a geometric distribution with arameter 1 (obtained from (a)), we have E( ˆT 0 ) = 1/(1 ). Then we get the formula for E(T k ). 9