Homework Solution 4 for APPM4/556 Markov Processes 9.Reflecting random walk on the line. Consider the oints,,, 4 to be marked on a straight line. Let X n be a Markov chain that moves to the right with robability and to the left with robability, but subject this time to the rule that if X n tries to go to the left from or to the right from 4 it stays ut. Find (a) the transition robability for the chain, and the (b) the limiting amount of time the chain sends at each site. (a) (b) The limiting distribution of time sent in each state is 5 (,, 4, 8). 4. 4 In words, if he last shaved k days ago, he will not shave with robability k+. However, when has not shaved for 4 days his wife orders him to shave, and he does so with robability. (a) What is the long-run fraction of time Mickey shaves? (b) Does the stationary distribution for this chain satisfy the detailed balance condition? (a) The limiting distribution of time sent in each state is 4 (4,, 4, ). It follows that the long run fraction of days that Mickey shaves is 4 4. (b) o. In articular = π()(, ) π()(, ) = 8. 9.6 Let X n be the number of days since Miceky Markov last shaved, calculated at 7:AM when he is trying to decide if he wants to shave today. Suose that X n is a Markov chain with transition matrix 9.6(b) The Markov chain associated with a manufacturing rocess may be described as follows: A art to be manufactured will begin the rocess by entering ste. After ste, % of the arts must be reworked, i.e., returned to ste, % of the arts are thrown away, and 7% roceed to ste. After ste, 5% of the arts must be returned to ste, % to ste, 5% are scraed, and 8% emerge to be sold for a rofit. (a) Formulate a four-state Markov chain with states,,, and 4 where = a art that was scraed and 4 = a art that was sold for a rofit. (b) Comute the robability a art is scraed in the roduction rocess. (b) There are several ways to do this roblem. The following way is more intuitive.
The transition robability of the Markov chain is..7..5..5.8 By simle calculation we know π = π = while π and π 4 are undetermined. From the roblem we know a art will always start from ste and end at either ste or 4. In other words we only need to know the value of n (, ) for n. Since π = π =, we have n (i, ) = n (i, ) = for i =,,, 4, and n. By the roerty of Markov chain n (k, ) = 4 (k, j) n (j, ) j= Let a ij = lim n n (i, j) as a brief notation. If a ij exist and unique, then for n, the above equation can be written as a = a = 4 (, j)a j =.a +.7a +.a + ()a 4 j= 4 (, j)a j =.5a +.a +.5a +.8a 4 j= It is obvious that a = a 44 =, a 4 = a 4 =. So we have a =.a +.7a +. a =.5a +.a +.5. By solving the system of equations a =.85, a =.657 and the robability a art is scraed is.85. 9.9(b) Six children(dick, Helen, Joni, Mark, Sam, and Tony) lay catch. If Dick has the ball he is equally likely to throw it to Helen, Mark, Sam, and Tony. If Helen has the ball she is equally likely to throw it to Dick, Joni, Sam, and Tony. If Sam has the ball he is equally likely to throw it to Dick, Helen, Mark, and Tony. If either Joni or Tony gets the ball, they kee throwing it to each other. If Mark gets the ball he runs away with it. (a) Find the transition robability and classify the states of the chain. (b) Suose Dick has the ball at the beginning of the game. What is the robability Mark will end u with it? (a) The transition robability of the Markov chain is D H J M S T D /4 /4 /4 /4 H /4 /4 /4 /4 J M S /4 /4 /4 /4 T and we need to find the value of a, where a i = P i (Mark ends u with the ball). Conditioning on the first ste we obtain: a k = 6 (k, j)a j for k =,, 6. j=
After simlification we have a = 4 a + 4 a 5 + 4 a = 4 a + 4 a 5 a 5 = 4 a + 4 a + 4 and a =.4, a =., a 5 =.4. So the robability Mark will end u with the ball is.4. 9.4 Couon collector s roblem. We are interested now in the time it takes to collect a set of baseball cards. Let T k be the number of cards we have to buy before we have k that are distinct. Clearly, T =. A little more thought reveals that if each time we get a card chosen at random from all ossibilities, then for k, T k+ T k has a geometric distribution with success robability ( k)/. Use this to show that the mean time to collect a set of baseball cards is log, while the variance is k= /k. Since T k+ T k has a geometrix distribution with robability ( k)/, ( ) k E(T k+ T k ) = = k. So E(T k+ ) = E(T k ) + E(T k+ T k ) = E(T k ) + E(T ) = E(T ) + = E(T ) + log. j= k, ( ) = E(T ) + j = + j= j ( ) + ( ) = Since {T k } are indeendent to each other, V ar(t k+ ) = V ar(t k+ T k ) + V ar(t k ). So V ar(t k+ T k ) = k ) = V ar(t k+ ) = V ar(t k ) + ( k V ar(t ) = V ar(t ) + k ( k) k ( k). ( ) ( ( )) = j = V ar(t ) + ( j) = j= j= ( j) j = k= k j= ( j ) j
9.8 A criminal named Xavier and a oliceman named Yacov move between three ossible hideouts according to Markov chains X n and Y n with transition robabilities:.6...5.5 Xavier =..6. and Yakov =.5.5...6.5.5 At time T = min{n : X n = Y n } the game is over and the criminal is caught. (a) Suose X = i and Y = j i. Find the exect value of T. (b) Suose that the two layers generalize their strategies to Xavier = q q q Yakov = q q q q q q If Yakov uses q =.5 as he did in art (a), what values of should Xavier choose to maximize the exected time to get caught? Answer the last question again for q = /. (a) Observe that the terms in the transition matrices for X and Y are each constant along the diagonal as well off-diagonal. Let,, be the three states where X and Y stay. Without loss of generality, suose X is at and Y is at at time k. At time k +, the robability of getting caught or not is shown as caught not caught X=, Y=. X=, Y=. X=, Y=. X=, Y=. X=, Y=. X=, Y=. We can generalize a new Markov chain model with states caught and not caught. The transition robability is caught not caught caught not caught.4.6. The exect caught time is therefore ET = P ( k) = (.6) k =.6 =.5 k= k= (b) Since the states are still symmetric, we can generalize the table (suose X = and Y = at time k): 4
X=, Y= X=, Y= X=, Y= X=, Y= X=, Y= X=, Y= X=, Y= X=, Y= X=, Y= caught (-)q (-q) q not caught (-)(-q) (-)q q q q (-q) Counting the robability: ( )q + ( q) + q = + q q ( )( q) + ( )q + q + q + q + ( q) = ( + q q). caught not caught The transition matrix is caught. not caught ( +q-q -(+q-q) ) When q =.5, the transion matrix is, ( )/ ( + )/ The exect caught time would be ET = ( + )/ ( + )/ = +. To maximize the time to be caught, we look at the first derivative of ET : ( ) + (ET ) ( ) + ( + ) = = ( ) = ( ) >. The maximun value can have is.5, so max{et } =.5/.5 =. When q = /, ( + q q) = /, ET = / / =. 9.4 A warehouse has a caacity to hold four items. If the warehouse is neither full nor emty, the number of items in the warehouse changes whenever a new item is roduced or an item is sold. Suose that (no matter when we look) the robability that the next event is a new item is roduced is and that the new event is a sale is. If there is currently one item in the warehouse, what is the robability that the warehouse will become full before it becomes emty. We create a Markov chain with absorbing states at emty and full. That is, emty full emty. full 5
Let P (i) be the robability that at state i the warehouse becomes full before becoming emty. Then using that P () =, and P (4) =, it follows that P () = P (), P () = P () + P (), and P () = P () +. Solving this linear system gives that AP = b where A =, P = P () P () b =. P (), Solving the system gives that P = 7 (4, 6, 7). In articular P () = 4 7. 9.46 General birth and death chains. The state sace is {,,,...} and the transition robability has (x, x + ) = x (x, x ) = q x for x > (x, x) = r x for x while the other (x, y) =. Let V y = min{n : X n = y} be the time of the first visit to y and let h (x) = P x (V < V ). By considering what haens on the first ste, we can write h (x) = x h (x + ) + r x h (x) + q x h (x ). Set h () = c and solve this equation to conclude that is recurrent if and only if y y= x= q x/ x = where by convention x= =. First, h () = P (V < V ) = from the definition. Second, we find the solution of h (x). From the one-ste equation, We will rove h (x + ) = x [( r x )h (x) q x h (x )] = x [( x + q x )h (x) q x h (x )] = h (x) + q x x (h (x) h (x )) h (x) = c x i i= j= j by induction. The equality holds when x =. Suose it holds when x = t, then when x = t +, h (t + ) = h (t) + q t t (h (t) h (t )) t i = c i= j= t i = c i= j= t+ i = c i= j= + q t t c j t j. j + t j= j j= j 6
x i q Thus h (x) = c j i= j= j is the solution. Third, we find the form of c. Since h () = P (V < V ) = by definition, h () = c i i= j= i c = i= j= j j = c = h () and is the robability that we start at and visit before visiting. Define c = lim c i = i= j= j to be the robability that we start at and visit,,,... before visiting. Fourth, we make the roof. If i i= j= j =, c =. Then it is certain we will visit. And since we can only visit from or, we will visit infinitely often, so is recurrent. If i i= j= j >, c >. We will visit every state before visiting, which means we never visit so is transient. 9.48 Consider the Markov chain with state sace {,,,...} and transition robability (m, m + ) = ( ) for m m + (m, m ) = ( + ) for m m + and (, ) = (, ) = /4. Find the stationary distribution π. Since this is a birth-death chain, we exloit the detailed-balanced condition: Iterating the recursion leads to v m+ = (m, m + ) (m +, m) v (m + )(m + ) m = (m + )(m + 4) v m.. and so on, which simlifies to v = v 4 ; v = v 4 4 5 ; v = v 4 4 5 5 4 6 ; v = v 4 ; v = v 5 ; v = v 4 6 ; 7
from which the following attern is aarent (you can easily check that this solves the detailed-balanced condition): v v m = (m + ) (m + ). ow to determine the value of v by imosing that n= π n = : = n= v (n + )(n + ) = v = v ( n + ) n + n= ( n + n + + n + n + n= ) = v ( + ) = 9 4 v, so v = 4/9 and π n =, for all n. 4(n + )(n + ) 9.5 Consider a branching rocess as defined in Examle 7., in which each family has a number of children that follows a shifted geometric distribution: k = ( ) k for k, which counts the number of failures before the first success when success has robability. Comute the robability that starting from one individual the chain will be absorbed at. Let e be the extinction robability starting at one individual. It follows that e = ( ) k ( e ) k k= ( ) = (( )( e )) k k= ( ) = e ( ). It follows that e =, and hence the robability that the chain will be absorbed at, starting with one individual is. 7. Suose that the time to reair a machine is exonentially distributed random variable with mean. (a) What is the robability the reair takes more than hours. (b) What is the robability that the reair takes more than 5 hours given that it takes more than hours. (a) Since the mean is, the rate is /. P (T > ) = e / = e. (b) P (T > 5 T > ) = P (T > ) = e. 7. The lifetime of radio is exonentially distributed with mean 5 years. If Ted buys a 7 year-old radio, what is the robability it will be working years later? The rate is /5. P (T > T > 7) = P (T > ) = e /5. 8