Continuous-Time Markov Chain Consider the process {X(t),t 0} with state space {0, 1, 2,...}. The process {X(t),t 0} is a continuous-time Markov chain if for all s, t 0 and nonnegative integers i, j, x(u), 0 u < s: P {X(t + s) = j X(s) = i,x(u) = x(u), 0 u < s} = P(X(t + s) = j X(s) = i) Stationary or homogeneous transition probabilities: P(X(t + s) = j X(s) = i) is independent of s, All CTMC are assumed stationary in this chapter. T i : the amount of time that the process stays in state i before making a transition into a different state. 1. P(T i > s + t T i > s) = P(T i > t): memoryless property. 2. T i has to be exponentially distributed. 3. E(T i ) = 1/v i. P i,j : when the process leaves state i, the probability of entering another state j: P i,i = 0, P i,j = 1, all i j 1
A Shoe Shine Shop: Consider a shoe shine establishment consisting of two chairs chair 1 and chair 2. A customer upon arrival goes initially to chair 1 where his shoes are cleaned and polish is applied. After this is done the customer moves on to chair 2 where the polish is buffed. The service times at the two chairs are assumed to be independent random variables that are exponentially distributed with respective rates µ 1 and µ 2. Suppose that potential customers arrive in accordance with a Poisson process having rate λ, and that a potential customer will enter the system only if both chairs are empty. State Interpretation 0 system is empty 1 a customer is in chair 1 2 a customer is in chair 2 v 0 = λ, v 1 = µ 1, v 2 = µ 2 P 0,1 = P 1,2 = P 2,0 = 1 2
Birth and Death Process When there are n individuals in the system: New arrivals enter the system at an exponential rate λ n. People leave the system at an exponential rate µ n. The time until next arrival is independent of the time until the next departure. {λ n } 0 : arrival (birth) rate. {µ n } 0 : departure (death) rate. For birth and death process: v 0 = λ 0 v i = λ i + µ i, i > 0 P 0,1 = 1 P i,i+1 = P i,i 1 = λ i, i > 0 λ i + µ i µ i, i > 0 λ i + µ i 3
Example Birth and Death Processes Poisson process: µ n = 0, for all n 0, λ n = λ, for all n 0. It s a pure birth process. Birth process with linear birth rate (Yule Process): Each member acts independently of the others and takes an exponentially distributed amount of time with mean 1/λ to give birth. µ n = 0, for all n 0, λ n = nλ, n 0. Linear Growth Model with Immigration: Each individual gives birth at an exponential rate λ; there is an exponential rate of increase θ of the population due to an external source such as immigration; deaths are assumed to occur at an exponential rate µ for each member of the population. µ n = nµ, n 1. λ n = nλ + θ, n 0. Let M(t) = E(X(t)). Assume M(0) = i. M (t) = (λ µ)m(t) + θ { θ M(t) = λ µ [e(λ µ)t 1] + ie (λ µ)t λ µ θt + i λ = µ 4
Example Birth and Death Processes The Queueing System M/M/1: Customers arrive at a single-server service station according to a Poisson process having rate λ. Upon arrival, each customer goes directly into service if the server is free; if not, then the customer joins the queue. When the server finishes serving a customer, the customer leaves the system and the next customer in line, if there are any waiting, enters the service. The service times are independent exponential random variables with mean 1/µ. µ n = µ, n 1 λ n = λ, n 0 A Multiserver Exponential Queueing System: Consider an exponential queueing system in which there are s servers available, each serving at rate µ. An entering customer first waits in line and then goes to the first free server. { nµ, 1 n s µ n = sµ, n > s λ n = λ, n 0 5
A general birth and death process with birth rates {λ n } and death rates {µ n }, where µ 0 = 0. T i : the time starting from state i to enter state i + 1. E(T 0 ) = 1 λ 0. E(T i ) = 1 + µ i E(T i 1 ), i 1. λ i λ i { 1, if the first transition from i is to i + 1 I i = 0, if the first transition from i is to i 1 P(I i = 1) = E(T i I i = 1) = E(T i I i = 0) = E(T i ) = λ i λ i + µ i 1 λ i + µ i 1 + E(T i 1 ) + E(T i ) λ i + µ i 1 + µ i (E(T i 1 ) + E(T i )) λ i + µ i λ i + µ i E(T i ) = 1 λ i + µ i λ i E(T i 1 ), i 1 The expected time to go from state i to state j where i < j is: E(T i ) + E(T i+1 ) + + E(T j 1 ) 6
Consider V ar(t i ): V ar(t i ) = 1 λ i (λ i + µ i ) + µ i V ar(t i 1 ) + λ i µ i (E(T i 1 ) + E(T i )) 2 µ i + λ i Time to go from state k to j, k < j: V ar(time to go from k to j) = j 1 i=k V ar(t i ) 7
Transition Probability Function P i,j (t) Transition probabilities: P i,j (t) = P {X(t + s) = j X(s) = i} Proposition 6.1: For a pure birth process having λ i λ j when i j P i,j (t) = j k=i j 1 k=i P i,i = e λ it e λ kt j r k,r=i j 1 e λ kt r k,r=i λ r λ r λ k λ r λ r λ k, i < j Yule Process: λ n = nλ, n 1. Let X(0) = i. Then the population size at time t has a negative binomial distribution with parameter i and e λt (the sum of i independent and identically distributed geometric random variables with parameter e λt ). ( ) j 1 P i,j (t) = e iλt (1 e λt ) j i, j i 1 i 1 8
Instantaneous transition rates: q i,j = v i P i,j. v i = j v i P i,j = j q i,j Lemma 6.2: P i,j = q i,j v i = lim h 0 1 P i,i (h) h q i,j j q i,j = v i P i,j (h) lim = q i,j i j h 0 h Lemma 6.3 (Chapman-Kolmogorov Equations): For all s 0, t 0, P i,j (t + s) = P i,k (t)p k,j (s) k=0 9
Theorem 6.1 (Kolmogorov s Backward Equations): For all states i, j, and time t 0, P i,j(t) = k i q i,k P k,j (t) v i P i,j (t) Theorem 6.2 (Kolmogorov s Forward Equations): Under suitable regularity conditions, P i,j(t) = k i q k,j P i,k (t) v j P i,j (t) Proposition 6.4: For a pure birth process, P i,i (t) = e λ it, i 0 P i,j (t) = λ j 1 e λ jt t 0 eλ js P i,j 1 (s)ds, j i + 1 10
Limiting Probabilities Assume the following limit exists and is independent of the initial state i: P j lim t P i,j (t) A sufficient condition for P j to exist: All states of the MC communicate in the sense that starting in state i there is a positive probability of ever being in state j for all i, j and the MC is positive recurrent in the sense that, starting in any state, the mean time to return to that state is finite. Equations for P j : v j P j = k j q k,j P k P j = 1 j 11
Limiting Probabilities (Examples) Birth and death process: State Rate at which leave = rate at which enter 0 λ 0 P 0 = µ 1 P 1 1 (λ 1 + µ 1 )P 1 = µ 2 P 2 + λ 0 P 0 2 (λ 2 + µ 2 )P 1 = µ 3 P 3 + λ 1 P 1 n, n 1 (λ n + µ n )P n = µ n+1 P n+1 + λ n 1 P n 1 Assume n=1 P 1 = λ 0 µ 1 P 0 P 2 = λ 1λ 0 µ 2 µ 1 P 0 P 3 = λ 2λ 1 λ 0 µ 3 µ 2 µ 1 P 0. P n = λ n 1λ n 2 λ 1 λ 0 µ n µ n 1 µ 2 µ 1 P 0 λ 0 λ 1 λ n 1 µ 1 µ 2 µ n < P 0 = 1 1 + n=1 λ 0 λ 1 λ n 1 µ 1 µ 2 µ n P n = λ 0 λ 1 λ n 1 µ 1 µ 2 µ n (1 + λ 0 λ 1 λ n 1 n=1 µ 1 µ 2 µ n ), n 1 12
For the multiserver exponential queueing system, the condition reduces to λ sµ < 1. For the linear growth model with immigration, the condition reduces to λ < µ. In the M/M/1 queue λ n = λ, µ n = µ. If λ < µ, P n = (λ/µ) n 1 + n=1 (λ/µ)n = (λ/µ)n (1 λ/µ), n 0 Consider the shoe shine shop example. State Rate at which leave = rate at which enter 0 λp 0 = µ 2 P 2 1 µ 1 P 1 = λp 0 2 µ 2 P 2 = µ 1 P 1 Also P 0 + P 1 + P 2 = 1. µ 1 µ 2 P 0 = µ 1 µ 2 + λ(µ 1 + µ 2 ) λµ 2 P 1 = µ 1 µ 2 + λ(µ 1 + µ 2 ) λµ 1 P 2 = µ 1 µ 2 + λ(µ 1 + µ 2 ) 13