Continuous Time Markov Chain Examples

Transcription

1 Continuous Markov Chain Examples Example Consider a continuous time Markov chain on S {,, } The Markov chain is a model that describes the current status of a match between two particular contestants: X(t) signifies that contestant A is at a disadvantage at time t in the match, X(t) signifies that neither contestant has an advantage at time t, and X(t) signifies that contestant A is at an advantage at time t The transitions between states are governed by a Markov chain with transition probabilities Q This matrix of transition probabilities tells us where we are likely to transition when we transition The holding times for states, and follow exponential distributions with means /, / and /, respectively This means that if we are currently in state we will wait, on average, / time units until the next transition; if we are currently in state we will wait, on average, / time units until the next transition; if we are currently in state we will wait, on average, / time units until the next transition These holding times tell us, given our current state, when we are likely to transition We need to find the infinitesimal generator A in order to analyze the chain The rates at which the chain transitions away from each state, α(i), are given by α(), α() and α() These are the reciprocals of the means of the exponential holding times The rates at which we transition from one state to another, α(i, j), i j, are given by α(i, j) Q ij α(i) The infinitesimal generating matrix is therefore Diagonalizing A, we have H A, D, H This means that the transition probabilities can be easily computed as P (t) He Dt H H e t H e t What about a limiting distribution? The chain is irreducible, and so lim P (t) t, which is the first row of H because the first column of H was chosen to correspond to the zero eigenvalue and was scaled so that it contained all ones Convergence to this limiting distribution can be seen in Figure, where the elements of P (t) are plotted as a function of time

2 P(X(t) X() i) P(X(t) X() i) i i i i i i P(X(t) X() i) i i i Figure : The function P ij (t) P (X(t) j X() i) in Example for all i and j Take note of the fast convergence to the limiting distribution

3 First Passage s Say that when the match starts neither contestant is at an advantage (in other words, X() ) What is the expected time until contestant A is at a disadvantage (ie, what is the expected time that the chain reaches state for the first time)? We need to find the distribution of first passage times : T (j) inf{t : X(t) j}, In this example we are interested in E(T (j) X() i) with i Because the chain starts in state i, E(T (i) X() i) For the other states we can use a first step analysis Let S be the first time at which the chain changes states; recall that S Exp(α(i)) Then E(T (j) X() i) E(S X() i) + E(T (j) X() k)p (X(S) k X() i) by the Markov property; the sum does not include i because the chain cannot make its first transition to i and does not include j because E(T (j) X() j) We know and so E(S X() i) α(i), P (X(S) k X() i) α(i, k)/α(i), k i, E(T (j) X() i) α(i) + α(i) α(i, k)e(t (j) X() k) Using the notation v(k) E(T (j) X() k), we have or α(i)v(i) + α(i, k)v(k), α(i)v(i) + α(i, k)v(k) Write the matrix obtained by deleting the row and column corresponding to state j as A(j) Then for all possible starting values i j, the equation above can be written in matrix form as + A(j)v yielding v [ A(j)] (Think on your own about why A(j) is guaranteed to be invertible) In our Example, we are interested in the expected first passage time for state : ( ) A(), and [ A()] which means that E(T () X() ) and E(T () X() ) 7 So the expected time at which contestant A is first at a disadvantage is time units,

4 Example Our continuous time Markov chain model for the competition between two contestants/teams in Example allowed the contest to go on indefinitely A more realistic model would have two additional states, say states and, that correspond to contestant/team A winning and losing, respectively We can adjust the matrix Q governing transitions to accommodate these new states: Q Under this new model it is impossible for A to transition directly from being at a disadvantage to winning, and it is impossible for A to transition directly from being at an advantage to losing This is application specific Because states and are absorbing, their holding times are both infinite, so that α() α() Assuming the other holding times are distributed as in Example, the infinitesimal generator matrix A for the newly-defined process is therefore A The matrix of transition probabilities P (t) can still be computed via diagonalization of A; Figure plots these probabilities as a function of t Under our new model, states, and are transient Note that, as t, the probability of ending up in these states goes to zero Computing the matrix of transition probabilities is somewhat more complicated now Clearly, P (t) P (t) for all t It is also clear that P ij (t) for all j {,, } and i {, } This leaves us with lim P (t) Π t The remaining elements must be found via first step analysis Define the time at which we absorb into either state or as T inf{t : X(t) {, }} Say we are interested in the probability of absorbing into state : u(i) P (X(T ) X() i), i {,, } A first-step analysis conditions on S, the first time the chain changes states: u(i) P (X(T ) X() i) P (X(T ) X(S) j, X() i)p (X(S) j X() i) j S\{i}

5 P(X(t) X() i) P(X(t) X() i) i i i i i i P(X(t) X() i) P(X(t) X() i) i i i i i i P(X(t) X() i) i i i Figure : The function P ij (t) P (X(t) j X() i) in Example The starting values i {, } are not shown because those are absorbing states Notice that the limiting probabilities for the transient states are zero

6 j S\{i} α(i, ) α(i) α(i, j) P (X(T ) X(S) j) α(i) + j S\{i,} α(i, j) P (X(T ) X() j) α(i) In this example, we have three equations with three unknowns: α()u() α(, ) + α(, )u() + α(, )u(), α()u() α(, ) + α(, )u() + α(, )u(), α()u() α(, ) + α(, )u() + α(, )u(), or writing this in terms of A, A i + j S\ A ij u(j) for i,, Letting à be the matrix obtained by removing all rows and columns corresponding to the absorbing states and, and letting R be the vector containing the rows of A corresponding to states, and in the column corresponding to state, we have Ãu + R u [ Ã] R 7 7 The column of Π corresponding to state can now be filled in: lim P (t) Π 7 t

7 By the law of total probability we must have lim P (t) Π t 7 We also could have created a matrix U and a matrix R so that U [ Ã] R [ Ã] 7 7