MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Recitation 13 10/31/2008. Markov Chains

Transcription

1 MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/5.085J Fall 008 Recitation 3 0/3/008 Marko Chains Problem 9 from Poisson process exercises in [BT] Fact: If there is a single recurrent class, the frequency with which the edge i j is traersed is π i p ij. This fact allows us to sole some problems easily. Consider a birth-death Marko chain. State space,..., m. If you are at state i, you go to i + with probability d i and i with probability b i. The numbers b i, d i are gien. You stay at i with probability b i d i. See problem in Marko chain chapter of [BT] for a picture. The edge i i+ is traersed in the same proportion as the edge i+ i, so π i b i = π i+ d i+, or b i π i+ = π i. d i+ The aboe recursion allows one to write all the stationary probabilities in terms of π as b π = π, d and finally Together with the equation b b b π 3 = π = π, d 3 d d 3 b m b b b m π m = π m = π. d m d d 3 d m π i =, i

2 this completely determines π i. For example, suppose b i = /3, d i = /3 for all i. Then, π i+ = π i, and so π ( ) =, m and lets assume m is ery large, so that and and so on. π, π, 4 π 3, 8 Random walk on a graph. A particle performs a random walk on the ertex set of a connected undirected graph G, which for simplicity we assume to hae neither loops nor multiple edges. At each stage it moes to a neighbor of its current position, each such neighbor being chosen with equal probability. If G has η < edges, show that the stationary distribution is gien by π = d /(η), where d is the degree of each ertex. One way to do this problem is to simply check that the proposed solution satisfies the defining equations: πp = π, and π = (we can see immediately that we hae nonnegatiity). We hae: d π = η = d η =, since the sum of the degrees is twice the number of edges (each edge increases the sum of the degrees by exactly ). Similarly, we can show that πp = π. Let us define δ u to be if ertices u and are adjacent,

3 and 0 otherwise. Then, we hae: ( ) π P u = d δ u η d = δ u. η But δ u is the number of edges incident to node u, that is, δ u = d u. Therefore we hae: This is what we wanted to show. d u π P u = d u = = π u. η η 3

4 [Note: HW-07; from [GS]] Exercise 33. A particle performs a random walk on a bow tie ABCDE drawn beneath, where C is the knot. From any ertex, its next step is equally likely to be to any neighbouring ertex. Initially it is at A. Find the expected alue of: (a) The time of first return to A. (b) The number of isits to D before returning to A. (c) The number of isits to C before returning to A. (d) The time of first return to A, gien that there were no isits to E before the return to A. (e) The number of isits to D before returning to A, gien that there were no isits to E before the return to A. Figure : A simple example of the set operation we describe. Solution: First, we can compute that the steady state distribution is π A = π B = π D = π E = /6, and π C = /3. We can do this either by soling a system of linear equations (as usual) or just use our result from the first problem aboe. (a) By the result from class, and on the handout, we hae: t A = /π A = 6. Alternatiely, we can sole the following system of equations (obsere than t A appears in only one equation): t A = (t B + ) + (t C + ) t B = + (t C + ) t C = + (t B + ) + (t D + ) + (t E + ) t D = (t C + ) + (t E + ) t E = (t C + ) + (t D + ). 65

5 (b) By the result from the handout on Marko Chains, we know that E[# transitions to D in a cycle that starts and ends at A] π D =, E[# transitions in a cycle that starts and ends at A] from which we find that the quantity we wish to compute is 6π D =. (c) Using the same method as in part (b), we find the answer to be 6π C =. (d) We let P i ( ) = P( X 0 = i), and let T j be the time of the first passage to state j, and let ν i = P i (T A < T E ). Then, as we obtained the equations aboe, that is, by conditioning on the first step, we hae ν A = ν B + ν C ν B = + ν C ν C = + ν B + ν D ν D = ν C. Soling these, we find: ν A = 5/8, ν B = 3/4, ν C = /, ν D = /4. Now we can compute the conditional transition probabilities, which we call τ ij. We hae: τ AB = P A (X = B T A < T E ) P A (X = B)P B (T A < T E ) = P A (T A < T E ) ν B 3 = =. ν A 5 Similarly, we find: τ AC = /5, τ BA = /3, τ BC = /3, τ CA = /, τ CB = 3/8, τ CD = /8, τ DC =. Now we hae essentially reduced to a problem like part (a). We can compute the conditional expectation by soling a system of linear equations using the new transition probabilities: t A t B = + 3 t B + t C 5 5 = + () + t C 3 3 t C = + () + 3 t B + t D 8 8 t D = + t C. Soling these equations, yields t A = 4/5. 66

6 (e) We can use the conditional transition probabilities aboe, to reduce to a problem essentially like that in part (b). Let N be the number of isits to D. Then, denoting by η i the expected alue of N gien that we start at i, and that T A < T E, we hae the equations: Soling, we obtain: η A = /0. 3 η A = η B + η B 5 5 η B = 0 + η C 3 3 η C = 0 + η B + ( + η D ) 8 8 η D = η C. 67

7 MIT OpenCourseWare J / 5.085J Fundamentals of Probability Fall 008 For information about citing these materials or our Terms of Use, isit: