Homework set 3 - Solutions Math 495 Renato Feres Problems 1. (Text, Exercise 1.13, page 38.) Consider the Markov chain described in Exercise 1.1: The Smiths receive the paper every morning and place it on a pile after reading it. Each afternoon, with probability 1/3, someone takes all the papers in the pile and puts them in the recycling bin. Also, if ever there are at least five papers in the pile, Mr. Smith (with probability 1) takes the papers to the bin. Consider the number of papers in the pile in the evening. (a) Model this by a Markov chain by drawing its transition diagram and writing the transition matrix. (b) After a long time, what would be the expected number of papers in the pile? (c) Assume the pile starts with 0 papers. What is the expected time until the pile will again have 0 papers? (d) Do a stochastic simulation to obtain the expected number of papers in the pile using one of the Markov chain programs of the previous assignment. Does it agree to reasonable precision with the value you obtain analytically in part (b)? Solution. (a) The transition diagram is shown below. It represents my interpretation of the situation described by the problem. The transition matrix is: P = 1/3 2/3 0 0 0 1/3 0 2/3 0 0 1/3 0 0 2/3 0 1/3 0 0 0 2/3 1 0 0 0 0.
(b) The chain is irreducible and aperiodic. So it has a unique stationary distribution π. By solving the linear system πp = π we obtain π = 1 211 (81,54,36,24,16). In the long run, the number X of papers in the pile has the stationary distribution π. Therefore, E(X ) = 4 j =0 j π j = 1 (0 81 + 1 54 + 2 36 + 3 24 + 4 16) = 1.242. 211 (c) The expected return time to a given state is the reciprocal of the stationary probability of that state. Therefore, E[T 0 X 0 = 0] = 1/π 0 = 211/81 = 2.605. (d) For this problem we may use the Markov chain program of H.W. 2: #It is assumed that the states are {1,2,...,s} #where s is the length of pi0, and P is s-by-s. #pi0 is the probability distribution of the initial step #and P is the transition probabilities matrix. Markov=function(N,pi0,P) { #Number of states s=length(pi0) X=matrix(0,1,N) a=sample(c(1:s),1,replace=true,pi0) X[1]=a for (i in 2:N) { a=sample(c(1:s),1,replace=true,p[a,]) X[i]=a } X } The initial probability vector may be taken to be pi0 = c(1,0,0,0,0) which assumes initial state 0 with probability 1. For this problem, any other initial condition would work similarly well because we are interested in the system s long run, stationary regime. The transition probability matrix is a = 1/3 b = 2/3 P[1,]=c(a, b, 0, 0, 0) P[2,]=c(a, 0, b, 0, 0) P[3,]=c(a, 0, 0, b, 0) P[4,]=c(a, 0, 0, 0, b) P[5,]=c(1, 0, 0, 0, 0) The average number of newspapers on the pile can now be estimated by running 2
N=100000 X=Markov(N,pi0,P) mean(x)-1 One run of this script gave me the value > mean(x)-1 [1] 1.24089 Which seems to confirm the result 1.242 obtained analytically. Note that the sample standard deviation (which the problem does not ask you to find) would be > sd(x-1)/sqrt(n) [1] 0.00404502 This suggests to me that it is reasonable to think that the first two digits at least (mean = 1.2) are correct. 2. (Text, Exercise 1.18, page 40.) Consider a deck of cards numbered 1,...,n. At each time step we shuffle the cards by drawing a card at random and placing it at the top of the deck. This can be thought of as a Markov chain on S n, the set of all permutations of n elements. If λ denotes any permutation (a one-to-one correspondence of {1,...,n} with itself), and ν j denotes the permutation corresponding to moving the j th card to the top of the deck, then the transition probabilities for this chain are given by p(λ,λν j ) = 1 n, j = 1,...,n. (Note: the notation λν j should be understood as the composition of two permutation maps.) This chain is irreducible and aperiodic. It is easy to verify that the unique invariant probability is the uniform measure on S n the measure that assigns probability 1/n! to each permutation. Therefore, if we start with any ordering of the cards, after enough moves of this kind the deck will be well shuffled. Suppose we take a standard deck of cards with 52 cards and do one move every second. What is the expected amount of time in years until the deck returns to the original order? Solution. If we accept the claim that the stationary distribution is the uniform distribution on S n, which is a set of n! elements, then π j = 1/n! for every permutation j S n, and the mean return time to any permutation is 1/π j = n! in seconds. If n = 52 and σ is the original permutation of the deck, then E[T σ X 0 = σ] = 52! seconds. To get a better sense of the number, note that 52! = 10 ln52!/ln10 = 10 1 52 ln10 j =1 ln j 10 67.9 seconds 10 60 years. For comparison, the current measurement of the age of the universe is approximately 1.4 10 10 years. 3. (Text, Exercise 2.1, page 57.) Consider the queueing model (Example 3 of Section 2.1). [Edited question: You need not do part (d); it will be explained in class. Also see below the edited part (a).] 3
(a) For which values of p, q is the chain null recurrent, positive recurrent, transient? Edited part (a): For which values of p, q is the chain positive recurrent? [The rest of the problem will be explained in class. Note: The necessary and sufficient condition on p, q for positive recurrent will follow naturally once you solve part (b), which you may do first.] (b) For the positive recurrent case give the limiting probability distribution π. (c) For the positive recurrent case, what is the average length of the queue in equilibrium? (d) For the transient case, give α(x) = the probability starting at x of ever reaching state 0. Solution. (a) Let z = 0 and for each x 1 define the probability α(x) that X n = z for some n 0 given that X 0 = x. It is shown on page 49 of the textbook that α(x) = y S p(x, y)α(y), x > 0. It is also stated there that the chain is transient if there exists a solution α to this equation such that α(0) = 1 and the infimum of α(x) is 0. Using the transition probabilities p(0,0) = 1 p, p(0,1) = p, p(n,n + 1) = p(1 q), p(n,n) = pq + (1 p)(1 q), p(n,n 1) = q(1 p), for n 1 we obtain the following recursive relation for α: 0 = aα(n 1) bα(n) + cα(n + 1) where a = q(1 p),b = p + q 2pq,c = p(1 q). Note that a + c = b. Equivalently, α(n + 1) = a c α(n 1) + b c α(n). The recursive relation may be written in the matrix form ( ) ( α(n + 1) b/c a/c = α(n) 1 0 )( α(n) α(n 1) ). The eigenvalues of the coefficients matrix are the roots of λ 2 b c λ + a c = 0. Using the fact that b c a c = 1 we can easily solve for the roots and find λ = 1 and a c = q(1 p) p(1 q). From our class discussion of difference equations, we know that [ ] q(1 p) x α(x) = c 0 + c 1. p(1 q) 4
From α(0) = 1 it follows that c 0 + c 1 = 1, so [ ] q(1 p) x α(x) = (1 c 1 ) + c 1. p(1 q) It is easily checked that any such expression satisfies the recurrence relation. We may now use the criterion of transience given on page 50 of the textbook (conditions (2.2) - (2.4).) Those conditions are satisfied by setting c 1 = 1 and q(1 p) p(1 q) < 1. The latter is equivalent to q < p. Thus we conclude: The chain is transient if and only if q < p. When q p, we need to decide when the chain is null or positive recurrent. We know that the chain is positively recurrent if and only if it admits a stationary probability measure π. Such a measure is characterized by the equation π(x)p(x, y) = π(y) for all y S. In the present example, this equation amounts to the recursive equation x q n+1 π(n + 1) (p n + q n )π(n) + p n 1 π(n 1) = 0 where p n = p(1 q), q n = q(1 q) for n 1, p 0 = p, q 0 = 0. Note that when n = 0, q 1 π(1) p 0 π(0) = 0. From these relations we derive q n+1 π(n + 1) p n π(n) = q n π(n) p n 1 π(n 1) = = q 1 π(1) p 0 π(0) = 0 so that Replacing the values of p n, q n gives π(n + 1) = p n q n+1 π(n) = = p np n 1 p 0 q n+1 q n q 1 π(0). ( ) p p(1 q) n 1 π(n) = π(0) for n 1. q(1 p) q(1 p) It is now clear that for an invariant probability measure to exist it is necessary and sufficient that the geometric series 1+u+u 2 + converges, where u = p(1 q)/q(1 p). This happens if and only if p < q. Thus we conclude: the chain is transient if q < p the chain is null recurrent if q = p the chain is positive recurrent if q > p. (b) To obtain π(n) we first need to find the normalization constant so that n 0 π(n) = 1: { 1 = π(0) + π(1) + = π(0) 1 + p q(1 p) ( 1 + u + u 2 + )} { = π(0) 1 + p 1 q(1 p) 1 u } = q q p π(0) 5
where u = p(1 q)/q(1 p). Thus π(0) = (q p)/q. Therefore, π(0) = q p p(q p), π(n) = q q 2 (1 p) ( ) p(1 q) n 1 for n 1. q(1 p) (c) The average length of the queue at equilibrium is p(q p) p(q p) nπ(n) = n q 2 (1 p) un 1 = q 2 nu n 1. (1 p) The infinite series can be obtained by noting that, within its radius of convergence, nx n 1 = d dx x n = d 1 dx 1 x = 1 (1 x) 2. Evaluating at u = p(1 q)/q(1 p) gives the value nu n 1 = (q(1 p)/(q p)) 2. Therefore, the mean queue length is Expected queue length = p(1 p) q p. Note that if q is only slightly larger than p, then the expected length is very large. (d) From the expression for α(x) given above and the fact that α(x) must approach 0 as x goes to we conclude that [ ] q(1 p) x α(x) =. p(1 q) 4. (Text, Exercise 2.2, page 57.) Consider the following Markov chain with state space S = {0,1,2,...}. A sequence of positive numbers p 1, p 2,... is given with i=1 p i = 1. Whenever the chain reaches state 0 it chooses a new state according to the probability p i. Whenever the chain is at a state other than 0 it proceeds deterministically, one step at a time, toward 0. In other words, the chain has transition probability p(x, x 1) = 1, p(0, x) = p x, for x > 0. This is a recurrent chain since the chain keeps returning to 0. Under what conditions on the p x is the chain positive recurrent? In this case, what is the limiting probability distribution π? [Hint: it may be easier to compute E(T ) directly where T is the time of first return to 0 starting at 0.] Solution. The (recurrent) chain is positively recurrent if and only if the expected time of recurrence to any state if finite. Consider the return time T 0 to state 0. The condition for positively recurrent chain is E[T 0 ] = (n + 1)p n = 1 + np n <. The recursive equation for the invariant probability is π(x) = y S π(y)p(y, x). Only the transition probabilities to 0 given by p(0, x) = p x and p(x + 1, x) = 1 are non-zero. Thus, for n 1, π(n) = π(n + 1) + p n π(0). 6
Iterating this relation gives π(1) = π(0) and π(n) = π(0) The normalization constant is obtained from 1 = [ 1 n 1 j =1 [ π(n) = π(0) 1 + ] ( p j = p j )π(0). j =n ] np n from which we obtain π(0) = 1 j =n p j, π(n) = (n + 1)p for n 1. n (n + 1)p n 7