Stochastic Processes 8.445 MIT, fall 20 Mid Term Exam Solutions October 27, 20 Your Name: Alberto De Sole Exercise Max Grade Grade 5 5 2 5 5 3 5 5 4 5 5 5 5 5 6 5 5 Total 30 30
Problem :. True / False questions. For each of the following statements just circle the letter T, if you think the statement is True, or the letter F, if you think the statement is False. T or F : If X, X 2, X 3,... is an irreducible Markov chain on a finite state space S = {,..., N}, then there is an equilibrium probability distribution π such that lim n P[X n = j X 0 = i] = π j for every i. T or F: If X, X 2, X 3,... is a Markov chain on a finite state space S = {,..., N}, then there is an invariant probability distribution π such that πp = π. T or F: Suppose that P is a finite stochastic matrix such that is a simple eigenvalue, and all other eigenvalues λ have λ <. Then lim n P[X n = j X 0 = i] exists, and it is the same for all i. T or F: Let X t, t [0, ), be a continuous time Markov chain with generating matrix A. If A ij > 0 for all i j, then ker(a) has dimension. T or F : The Markov property for a continuous time Markov chain can be equivalently formulated by saying that, for all s < t, we have P[X t = j X s = i] = P[X t X s = j i]. T or F: Let X t be the Poisson process of rate λ. Then, for all s < t, we have P[X t = j X s = i] = P[X t X s = j i]. 2
Problem 2:. Let X, X 2, X 3,... be a Markov chain on S = {, 2, 3, 4, 5, 6} with transition matrix 0 0 0.5 0.5 0 0 0 0 0 0 0 P = 0 0 0 0 0.5 0.5 0 0 0 0 0. 0.5 0.5 0 0 0 0 0 0 0 0 0 Compute (approximately) the following probabilities: (a) P[X 3000000000 = 2 X 0 = ], (b) P[X 300000000 = 2 X 0 = ], (c) P[X 3000000002 = 2 X 0 = ]. The chain is periodic of period d = 3. Indeed, its matrix has the block form: P = where A = [ 0.5 0.5 0 ]. Therefore P 3 = A3 0 0 0 A 3 0 0 0 A 3 blocks A 3 (which are irreducible aperiodic stochastic matrices). The subclasses accessible in one step are as follows: {, 2} {3, 4} {5, 6}. 0 A 0 0 0 A A 0 0 is block diagonal with diagonal Therefore, P[X n = 2 X 0 = ] is non zero only for n divisible by 3. In particular, P[X 300000000 = 2 X 0 = ] = P[X 300000000 = 2 X 0 = ] = 0, answering (b) and (c). As for (a), we have P[X 3000000000 = 2 X 0 = ] lim n P[X 3n = 2 X 0 = ] = π 2, where π is the (unique) equilibrium distribution for the irreducible aperiodic chain with transition matrix A 3, i.e. the unique solution of the equation πa 3 = π, or, equivalently, πa = π. This equation gives π 2 = 2 π, which has solution π = ( 2 3, 3 ). In conclusion, P[X 3000000000 = 2 X 0 = ] 3., (a) P[X 3000000000 = 2 X 0 = ] = 3 (b) P[X 300000000 = 2 X 0 = ] = 0 (c) P[X 3000000002 = 2 X 0 = ] = 0 3
Problem 3:. Let X, X 2, X 3,... be a Markov chain on S = {, 2, 3, 4, 5, 6} with transition matrix 0.8 0. 0 0. 0 0 0 0 0 0 0 P = 0 0 0 0 0 0 0 0 0.5 0.5 0. 0 0 0 0 0 0 0 0 0 0 (a) Describe the communicating classes, specifying whether they are transient or recurrent classes. (b) Compute, in the limit n, the following probability: P[X n = 6 X 0 = ]. The matrix P has the form Q P = 0 S P 0 0 P 2 where Q = [0.8] if the substochastic matrix of transitions among transient states, [ S = ] [0. 0 0. 0 0] 0 gives the transition probabilities from transient to recurrent states, P = is the transition matrix of the recurrent class {2, 3}, which is clearly periodic of period 2, and P 2 = 0 0.5 0.5 0 0 0 is the transition matrix of the recurrent class {4, 5, 6}, which is irreducible 0 0 aperiodic. We have lim n P[X n = 6 X 0 = ] = α {4,5,6} () π {4,5,6} 6, where α R () is the probability that the chain, starting at X 0 =, eventually ends up in the recurrent class R, while π R is the equilibrium distribution for the recurrent class R. In our case, for obvious symmetry reasons, we have α {2,3} () = α {4,5,6} () = 2. Alternatively, we can use the general formula: α {4,5,6} () = ( j=4,5,6 ( Q) S ) =,j ( 0.8) (0.+0+0) = 0. 0.2 = 2. Moreover, π {4,5,6} is solution of π {4,5,6} P 2 = π {4,5,6}. This equation gives π {4,5,6} 6 = π {4,5,6} 5 = 2 π{4,5,6} 4, which has solution π {4,5,6} = ( 2, 4, 4 ). In conclusion, lim n P[X n = 6 X 0 = ] = 2, 4 = 8. (a) Communicating classes: Transient class(es): {}, Recurrent class(es) {2, 3}, {4, 5, 6} (b) lim n P[X n = 6 X 0 = ] = 8 4
Problem 4:. Let X, X 2, X 3,... be a Markov chain on S = {, 2, 3} with transition matrix /3 /3 /3 P = /3 /3 /3. 0 0 Let T = inf{n 0 X n = 3} be the time of absorption at 3. Compute τ = E[T X 0 = ]. In this chain and 2 are transient states, and 3 is an absorbing state. Let τ = E[T X 0 = ], and τ 2 = E[T X 0 = 2]. For obvious symmetry reasons, τ = τ 2. Moreover, by first step analysis, τ = E[T X 0 = ] = E[T X = ]P[X = X 0 = ] + E[T X = 2]P[X = 2 X 0 = ] +E[T X = 3]P[X = 3 X 0 = ] = ( + E[T X 0 = ]) 3 + ( + E[T X 0 = 2]) 3 + 3 = ( + τ ) 3 + ( + τ 2) 3 + 3 = + 2 3 τ. Hence, 3 τ =, i.e. τ = 3. τ = 3 5
Problem 5:. Let X t be a Poisson process of rate λ. Let W, W 2,... be the waiting times (i.e. W n is the time of n-th jump). Compute P[W, W 2, W 3 5 X 7 = 3]. As proved in class, conditioned on X 7 = 3, the random variables W, W 2, W 3 are uniformly distributed in the region 0 w < w 2 < w 3 7, i.e. they have constant density f W,W 2,W 3 X 7=3 = V ol(0 w = <w 2<w 3 7) 7 3 /3!. Hence, P[W, W 2, W 3 5 X 7 = 3] = V ol(0 w < w 2 < w 3 5) V ol(0 w < w 2 < w 3 7) = 53 /3! 7 3 /3! = 53 7 3. Alternatively, if we let S, S 2, S 3 be i.i.d uniform random variable in [0, 7], we have, conditioned on X 7 = 3, that W = min(s, S 2, S 3 ), W 2 = 2nd min(s, S 2, S 3 ), W = max(s, S 2, S 3 ). Hence, P[W, W 2, W 3 5 X 7 = 3] = P[S, S 2, S 3 5] = P[S 5] 3 = ( 5) 3. 7 P[W, W 2, W 3 5 X 7 = 3] = ( 5 7) 3 6
Problem 6:. A radioactive source emits particles according to a Poisson process of rate 2 particles per minute. (a) Compute the probability p a that the first particle appears some time after 3 minutes and before 5 minutes. (b) Compute the probability p b that exactly one particle is emitted in the time interval from 3 to 5 minutes. (a) Recall that the time intervals T, T 2,... for the jumps of the Poisson process are independent identically distributed exponential random variables of rate λ = 2. To say that the first particle appears some time after 3 minutes and before 5 minutes is the same as to say that 3 < T < 5. Hence p a = P[3 < T < 5] = 5 3 2e 2t dt = e 2t 5 3 = e 6 e 0. (b) For be, we ask also that there are no other particles arriving in the interval [3, 5], i.e. that T + T 2 > 5. Hence p b = P[3 < T < 5, T + T 2 > 5] = 5 3 f T (t)p[t 2 > 5 t]dt = 5 3 2e 2t e 2(5 t) dt = 5 3 2e 0 dt = 4e 0. (a) p a = e 6 e 0 (a) p b = 4e 0 7
. (page intentionally left blank) 8
. (page intentionally left blank) 9