Irreducibility Irreducible every state can be reached from every other state For any i,j, exist an m 0, such that i,j are communicate, if the above condition is valid Irreducible: all states are communicate with each other Absorbing state: p jj =1 Reducible chain, i is an absorbing state p ( m0 ) ij 0 1
Recurrence ( ) f n : the probability that first returning to j jj occurs n steps after leaving j ( n) f : the probability that the chain ever jj : f jj n1 returns to j j is recurrent, if f jj =1 j is transient, if f jj <1 If j is recurrent, mean recurrence time: If m jj <, j is positive recurrent If m jj =, j is null recurrent E.g., random walk return to origin 0, ½, ¼, 1/8, 1/16, n1 2 m jj nf ( n) jj
Periodicity Period of a state i Greatest common divisor (gcd) Aperiodic Markov chain A DTMC (discrete time Markov chain) is aperiodic iff all states have period 1 Example period is 2, for DTMC with P 0 1 1 0 Limiting of P n does not exist, P 2n does. ( n) gcd n: pii 0 3
Ergodicity Ergodic Markov chain positive recurrent, irreducible, aperiodic finite, irreducible, aperiodic Physical meaning If ergodic, time average = ensemble average For some moments, process is ergodic Fully ergodic: for all moments, process is ergodic See page 35 of Gross book 4
Limiting distribution If a finite DTMC is aperiodic and irreducible, ( ) then lim p n ij exists and is independent of i n 3 types of probabilities Limiting prob. Time-average prob. Stationary prob. ( ) l : lim n j pij n t 1 p j : lim 1( X ( n) j) t t n1, s.t. P j 5
Limiting probability Limiting probability of i The limiting situation to enter i, may not exist (consider periodic case) Time-average probability of i % of time staying at state i Stationary probability Given the initial state is chosen according to π, all the future states follows the same distribution 6
Example, binary communication Limiting prob. Time-average prob. Stationary prob. channel Solve the equation and Conclusion Limiting prob. may not exist, otherwise they are all identical. π and p are always the same DNE, if a b 1 l j b a (, ) a b a b p j P b a (, ) a b a b e 1 7
Limiting probability Relations of 3 probabilities If the limiting probability exists, then these 3 probabilities match! If ergodic, limiting prob. exists and 3 prob. match How to solve? P, e1 Pe e π is the eigen vector of matrix P with eigen value 1 balance equation 8
Example, discrete time birth-death process Model the dynamic of population Parameters Increase by 1 with a birth prob. b Decrease by 1 with a death prob. d Stay the same state with prob. 1-b-d Chalk write Write the transition probability matrix P Solve the equation for stationary prob. Understand the balance equation 9
Part II: continuous time Markov chain (CTMC) Continuous time discrete state Markov process Definition (Markovian property) X(t) is a CTMC, if for any n and any sequence t 1 <t 2 <,,<t n, we have P[ X ( t ) j X ( t ) i, X ( t ) i,..., X ( t ) i ] n n1 n1 n2 n2 1 1 P[ X ( t ) j X ( t ) i ] n n1 n1 10
Exponentially distributed sojourn time of CTMC Sojourn time at state i, τ i, i=1,2, S, is exponentially distributed Proved with Markovian property P[ s t s] P[ t] i i i P[ s t] P[ s] P[ t] i i i P[ s t, s] P[ s] P[ t] i i i i dp[ i s t] f ( s) P[ ] i i t ds dp[ i t] f (0) ds i P[ t] i i f ( t) f (0) e i s0 i ln P[ t] f (0) t f i (0) t i f i P[ t] e i (0) t Sojourn time is exponentially distributed! f i (0) is transition rate of state i 11
Basic elements of CTMC State space: S={1,2,,S}, assume it is discrete and finite for simplicity Transition prob. In matrix form If homogeneous, denote p ( s, t) : P[ X ( t) j X ( s) i], s t P(t) is independent of s (start time) Chapman Kolmogorov equation P( s, t) P( s, u) P( u, t), s u t ij P( s, t) : pij ( s, t) P( t) : P( s, s t) 12
Infinitesimal generator (transition rate matrix) Differentiation of C-K equation, let ut P( s, t) t P( s, t) Q( t), s t t 0 Q(t) is called the infinitesimal generator of transition matrix function P(s,t) since Its elements are Qt ( ) lim P( t, t t) I t P( s, t) Q() t t P( s, t) t (u) (, ) Q s P s t e u pii ( t, t t) 1 pij ( t, t t) qii ( t) lim 0 qij ( t) lim 0, i j t 0 t 0 t t S qij ( t) 0, for all i Q( t) e 0 j1 relation between transition rate q ij and probability p ij -q ii is the rate flow out of i, q ij is the rate flow into i 13
Infinitesimal generator If homogeneous, denote Q=Q(t) Derivative of transition probability matrix Pt () t P() t Q With initial condition P(0)=I, we have P() t e Qt where Qt 1 1 e I Qt Q t Q t 2! 3! 2 2 3 3 :... Infinitesimal generator (transition rate matrix) Q can generate the transition probability matrix P(t) 14
State probability () t is the state probability at time t We have ( t) (0) P( t) (0) e Qt Derivative operation d () t (0) e Qt Q ( t) Q dt Decompose into element di() t dt ( t) q ( t) q i ii j ji ji 15
Steady state probability For ergodic Markov chain, we have lim ( t) lim p ( t) t Balance equation di () t dt In matrix j j ij t i ( t) qii j ( t) q ji 0 ji Q 0 e 1 Qe 0 Comparing this with discrete case 16
Birth-death process A special Markov chain Continuous time: state k transits only to its neighbors, k-1 and k+1, k=1,2,3, model for population dynamics basis of fundamental queues state space, S={0,1,2, } at state k (population size) Birth rate, λ k, k=0,1,2, Death rate, μ k, k=1,2,3, 17
Infinitesimal generator Element of Q q k,k+1 = λ k, k=0,1,2, q k,k-1 = μ k, k=1,2,3, q k,j = 0, k-j >1 q k,k-1 = - λ k - μ k Chalk writing: tri-diagonal matrix Q 18
State transition probability analysis In Δt period, transition probability of state k kk+1: p k,k+1 (Δt)= λ k Δt+o(Δt) kk-1: p k,k-1 (Δt)= μ k Δt+o(Δt) kk: p k,k (Δt)= [1-λ k Δt-o(Δt)][1- μ k Δt-o(Δt)] = 1- λ k Δt-μ k Δt+o(Δt) kothers: p k,j (Δt)= o(δt), k-j >1 19
State transition equation State transition equation ( t t) ( t) p ( t) ( t) p ( t) ( t) p ( t) o( t) k k k, k k1 k1, k k1 k1, k k-1 k=1,2, Case of k=0 is omitted, for homework k State k k+1 j k-j >1 at time t at time t+ Δt 20
State transition equation State transition equation: ( t t) ( t) ( ) t ( t) t ( t) t ( t) o( t) k k k k k k 1 k 1 k 1 k 1 With derivative form: d k () t dt d 0() t dt ( ) ( t) ( t) ( t), k 1,2,... k k k k1 k1 k1 k1 ( t) ( t), k 0 0 0 1 1 This is the equation of dπ = πq Study the transient behavior of system states ( t) (0) e Qt 21
Easier way to analyze birth-death process State transition rate diagram 0 1 2 k 1 k k 1 0 1 2 k-1 k k+1 1 2 3 k 1 k k 1 This is transition rate, not probability If use probability, multiply dt and add self-loop transition Contain the same information as Q matrix 22
State transition rate diagram Look at state k based on state transition rate diagram Flow rate into state k I ( t) ( t) k k 1 k 1 k 1 k 1 Flow rate out of state k O ( ) ( t) k k k k Derivative of state probability d k () t dt I O ( t) ( t) ( ) ( t) k k k 1 k 1 k 1 k 1 k k k 23
State transition rate diagram Look at a state, or a set of neighboring states Change rate of state probability = incoming flow rate outgoing flow rate Balance equation For steady state, i.e., the change rate of state probability is 0, then we have balance equation incoming flow rate = outgoing flow rate, to solve the steady state probability This is commonly used to analyze queuing systems, Markov systems 24
Pure birth process Pure birth process Death rate is μ k =0 Birth rates are identical λ k = λ, for simplicity We have the derivative equation of state prob. d k () t k1( t) k( t), k 1,2,... dt d 0() t 0( t), k 0 dt assume initial state is at k=0, i.e., π 0 (0)=1, we have () t 0 t e 25
Pure birth process For state k=1 d1() t dt we have () t Generally, we have 1 t t The same as Poisson distribution! e () t 1 te k ( t) t k ( t) e, t 0, k 0,1,2,... k! 26
Pure birth process and Poisson process Pure birth process is exactly a Poisson process Exponential inter-arrival time with mean 1/λ k, number of arrivals during [0,t] k ( t) t Pk e, k 0,1,2,... k! Z-transform of random variable k To calculate the mean and variance of k Laplace transform of inter-arrival time Show it or exercise on class To calculate the mean and variance of inter-arrival time 27
Arrival time of Poisson process X is the time that exactly having k arrivals X = t 1 + t 2 + + t k t k is the inter-arrival time of the kth arrival X is also called the k-erlang distributed r.v. The probability density function of X Use convolution of Laplace transform k ( ) ( ) s * * X s A s k Look at table of Laplace transform k1 ( x) f X ( x) e ( k 1)! arrival rate k-1 arrival during (0,x) x 28