Topics in Probability Theory and Stochastic Processes Steven R. Dunbar. Waiting Time to Absorption

Transcription

1 Steven R. Dunbar Department of Mathematics 203 Avery Hall University of Nebraska-Lincoln Lincoln, NE Voice: Fax: Topics in Probability Theory and Stochastic Processes Steven R. Dunbar Waiting Time to Absorption Rating Mathematically Mature: proofs. may contain mathematics beyond calculus with

2 Section Starter Question For a Markov chain with an absorbing state, describe the random variable for the time until the chain gets absorbed. Key Concepts. For a Markov chain with r transient states and N + r absorbing states reorder the states so the absorbing states come last and nonabsorbing, i.e. transient, states come first. Then the transition matrix has the canonical form: ( ) Q R. Here Q is a r r submatrix of transition probabilities among the transient states, R is the r (N + r) transition probabilities from the transient states to the absorbing states, 0 is a (N + r) r matrix of 0s representing the transition probabilities from absorbing states to transient states, and I is an (N + r) (N + r) identity matrix. 2. The matrix N = (I Q) is the fundamental matrix for the absorbing Markov chain. The entries N ij of this matrix have a probabilistic interpretation. The entries N ij are the expected number of times that the chain started from state i will be in state j before ultimate absorption. 3. A compact expression in vector-matrix form for the waiting time w to absorption is (I Q)w = so w = (I Q). 2

3 Vocabulary. Let {X n } be a finite-state Markov chain with states labeled 0,, 2, 3,..., N. Suppose that states 0,, 2,..., r are transient in that P (n) ij 0 as n for 0 i, j r, while states r,..., N are absorbing, P ii = for r n N. Define the random absorption time T = min {n 0 : X n r}. 2. The absorption probability matrix B is the probability of starting at state i and ending at a given absorbing state j. 3. For a Markov chain with r transient states and N + r absorbing states. If necessary, reorder the states so the absorbing states come last and non-absorbing, i.e. transient, states come first. Then the transition matrix has the canonical form: ( ) Q R. The matrix N = (I Q) is the fundamental matrix for the for the absorbing Markov chain. 4. The Nth harmonic number H N is H N = N. 3

4 Mathematical Ideas Theory Let {X n } be a finite-state Markov chain with states labeled 0,, 2, 3,..., N. Let the transition probability matrix be P. Reorder the states so the absorbing states come last and non-absorbing, i.e. transient, states come first. The states 0,, 2,..., r are transient in that P (n) ij 0 as n for 0 i, j r, while states r,..., N are absorbing, P ii = for r n N. Then the transition probability matrix has the canonical form ( ) Q R () Here Q is a r r submatrix of transition probabilities among the transient states, R is the r (N + r) transition probabilities from the transient states to the absorbing states, 0 is a (N + r) r matrix of 0s representing the transition probabilities from absorbing states to transient states, and I is an (N + r) (N + r) identity matrix. Starting at one of the transient states X 0 = i where 0 i r such a process will remain in the transient states for some random duration. Ultimately, the process gets trapped in one of the absorbing states i = r,... N. Of special interest is the mean time until absorption. Also of interest is the probability distribution of the states into which absorption takes place. Define the random absorption time T = min {n 0 : X n r}. The expected value of this random time w i = E [T X 0 = i], i = 0,..., r. is an important first measure of the random variable T. First-step analysis tells us that the absorption time from state i is the first step to another transient state, plus a weighted average according to the transition probabilities over the transient states, of the absorption times from the other transient states. In symbols, first-step analysis says r w i = + P ij w j. 4 j=0

5 Let j (X n ) be the indicator function for state j of the Markov chain X n. Recall that this means j (X n ) = if X n = j and j (X n ) = 0 otherwise. For a Markov chain with r transient states and N + r absorbing states, reorder the states so the absorbing states come last and non-absorbing, i.e. transient, states come first. Then the transition matrix has the canonical form: ( Q R Here Q is a r r submatrix of transition probabilities among the transient states, R is the r (N + r) transition probabilities from the transient states, 0 is a (N + r) r matrix of 0s representing the transition probabilities from absorbing states to transient states, and I is an (N + r) (N + r) identity matrix. Expressing the first-step analysis compactly in vector-matrix form as (I Q)w = then w = (I Q). The matrix N = (I Q) is the fundamental matrix for the for the absorbing Markov chain. The entries N ij of this matrix have a probabilistic interpretation. The entries N ij are the expected number of times that the chain started from state i will be in state j before ultimate absorption. To see why this is true, think of how (I Q) would look when written as a geometric series. The absorption probabilities B are the probability of starting at state i and ending up at a given absorbing state j. The absorption probabilities come from N by the matrix product B = NR = (I Q) R. That is, add up all the probabilities of going to state j, weighted by the number of times we expect to be in the transient states. For an absorbing Markov chain, the nth power P n of the transition probability matrix will approach a limit matrix of P of the form Examples ( Q B Example. Its your 30th birthday, and your friends bought you a cake with 30 candles on it. You make a wish and try to blow them out. Every time you blow, you blow out a random number of candles between one and the ). ).

6 number that remain, including one and that other number. How many times on average do you blow before you extinguish all the candles? Let the states be the number of candles blown out, so the N = 3 states are 0,, 2, 3,..., 30. Instead of solving this at once, try a smaller problem first, with the number of candles N =. Then the transition probability matrix is Then the equations for the waiting time, that is the number of attempts needed to blow out the candles are w 0 = + w + w 2 + w 3 + w 4 w = + 4 w w w 4 w 2 = + 3 w w 4 w 3 = + 2 w 4 w 4 =. Solving this recursively, w 4 =, w 3 = +, w 2 2 = + ( + ) +, so w 2 = + +. Continuing to solve recursively 2 3 w = w 0 = The inductive pattern is clear. The waiting time with N candles is the Nth harmonic number H N w 0 = H N = N. For the original problem with 30 candles the transient states are, 2, 3,..., 30 and the absorbing state is 0. Then the transition probability matrix as in () 6

7 is The quantity of interest is the expected waiting time until absorption into the state 0. Expressing the first-step analysis compactly in vector-matrix form as (I Q)v = Substituting in the values from the transition matrix and solving with Octave, this is w 0 = H Sources This section is adapted from An Introduction to Stochastic Modeling by Taylor and Karlin and Random Walks and Electrical Networks by Doyle and Snell. The example is adapted from the January 3, 207 Riddler problem from the webpage fivethirtyeight.com Algorithms, Scripts, Simulations Algorithm Scripts Commands are Q = diag(./ [30:-:] ) * ( triu( ones(30,30) ) - eye(30) ); wait = inverse( eye(30) - Q) * ones(30,); wait() HN = sum(./ [30:-:] ) 7

8 Problems to Work for Understanding Reading Suggestion: References [] Peter G. Doyle and J. Laurie Snell. Random Walks and Electric Networks. The Mathematical Association of America, 984. [2] H. M. Taylor and Samuel Karlin. An Introduction to Stochastic Modeling. Academic Press, third edition,

9 Outside Readings and Links: I check all the information on each page for correctness and typographical errors. Nevertheless, some errors may occur and I would be grateful if you would alert me to such errors. I make every reasonable effort to present current and accurate information for public use, however I do not guarantee the accuracy or timeliness of information on this website. Your use of the information from this website is strictly voluntary and at your risk. I have checked the links to external sites for usefulness. Links to external websites are provided as a convenience. I do not endorse, control, monitor, or guarantee the information contained in any external website. I don t guarantee that the links are active at all times. Use the links here with the same caution as you would all information on the Internet. This website reflects the thoughts, interests and opinions of its author. They do not explicitly represent official positions or policies of my employer. Information on this website is subject to change without notice. Steve Dunbar s Home Page, to Steve Dunbar, sdunbar at unl dot edu Last modified: Processed from L A TEX source on January 30, 207 9