TCOM 5: Networkig Theory & Fudametals Lecture 3 Jauary 29, 23 Prof. Yais A. Korilis
3-2 Topics Markov Chais Discrete-Time Markov Chais Calculatig Statioary Distributio Global Balace Equatios Detailed Balace Equatios Birth-Death Process Geeralized Markov Chais Cotiuous-Time Markov Chais
3-3 Markov Chai Stochastic process that takes values i a coutable set Example: {,,2,,m}, or {,,2, } Elemets represet possible states Chai jumps from state to state Memoryless (Markov) Property: Give the preset state, future jumps of the chai are idepedet of past history Markov Chais: discrete- or cotiuous- time
3-4 Discrete-Time Markov Chai Discrete-time stochastic process {X : =,,2, } Takes values i {,,2, } Memoryless property: P{ X = j X = i, X = i,..., X = i } = P{ X = j X = i} + + P = P{ X = j X = i} ij + Trasitio probabilities P ij, P = j= Trasitio probability matrix P=[P ij ] P ij ij
3-5 Chapma-Kolmogorov Equatios step trasitio probabilities P = P{ X = j X = i},, m, i, j ij + m m Chapma-Kolmogorov equatios P ij + m m ij ik kj k= P = P P,, m, i, j is elemet (i, j) i matrix P Recursive computatio of state probabilities
3-6 State Probabilities Statioary Distributio State probabilities (time-depedet) π = PX { = j}, π = (π,π,...) j { = } = { = } { = = } π j = πi ij i= i= PX j PX ipx j X i P I matrix form: π = π P = π P =... = π P 2 2 If time-depedet distributio coverges to a limit π lim π π is called the statioary distributio = π = πp Existece depeds o the structure of Markov chai
3-7 Classificatio of Markov Chais Irreducible: States i ad j commuicate: Irreducible Markov chai: all states commuicate Aperiodic: State i is periodic: m m, : P >, P > d > : P > =αd ij ji ii Aperiodic Markov chai: oe of the states is periodic 2 2 3 4 3 4
3-8 Limit Theorems Theorem : Irreducible aperiodic Markov chai For every state j, the followig limit π = lim PX { = j X = i}, i=,,2,... j exists ad is idepedet of iitial state i N j (k): umber of visits to state j up to time k N j( k) P π j = lim X = i = k k π j : frequecy the process visits state j
3-9 Existece of Statioary Distributio Theorem 2: Irreducible aperiodic Markov chai. There are two possibilities for scalars: π = lim PX { = j X = i} = lim P j ij. π j =, for all states j No statioary distributio 2. π j >, for all states j π is the uique statioary distributio Remark: If the umber of states is fiite, case 2 is the oly possibility
3- Ergodic Markov Chais Markov chai with a statioary distributio π >, j =,,2,... j States are positive recurret: The process returs to state j ifiitely ofte A positive recurret ad aperiodic Markov chai is called ergodic Ergodic chais have a uique statioary distributio π = lim P j ij Ergodicity Time Averages = Stochastic Averages
3- Calculatio of Statioary Distributio A. Fiite umber of states Solve explicitly the system of equatios m π = π P, j =,,..., m j i ij i= m j ipij πi = i= i= πi = i= Numerically from P which coverges to a matrix with rows equal to π Suitable for a small umber of states B. Ifiite umber of states Caot apply previous methods to problem of ifiite dimesio Guess a solutio to recurrece: π = π, j =,,...,
3-2 Example: Fiite Markov Chai Abset-mided professor uses two umbrellas whe commutig betwee home ad office. If it rais ad a umbrella is available at her locatio, she takes it. If it does ot rai, she always forgets to take a umbrella. Let p be the probability of rai each time she commutes. What is the probability that she gets wet o ay give day? p 2 p p Markov chai formulatio i is the umber of umbrellas available at her curret locatio p Trasitio matrix P = p p p p
3-3 Example: Fiite Markov Chai 2 p p p p P = p p p p π = ( p)π2 π = πp π = ( p)π + pπ p π, π, π π = p 3 3 p 3 p 2 = = 2 = π2 π π i i = + p π + π+ π2 = p P{gets wet} = π p = p 3 p
3-4 Example: Fiite Markov Chai Takig p =.: p π =,, =.3,.345,.345 3 p 3 p 3 p P =.9..9. ( ) Numerically determie limit of P.3.345.345 lim P =.3.345.345 ( 5).3.345.345 Effectiveess depeds o structure of P
3-5 Global Balace Equatios Markov chai with ifiite umber of states Global Balace Equatios (GBE) π P = π P π P = π P, j j ji i ij j ji i ij i= i= i j i j P is the frequecy of trasitios from j to i π j ji Frequecy of Frequecy of trasitios out of j = trasitios ito j Ituitio: j visited ifiitely ofte; for each trasitio out of j there must be a subsequet trasitio ito j with probability
3-6 Global Balace Equatios Alterative Form of GBE If a probability distributio satisfies the GBE, the it is the uique statioary distributio of the Markov chai Fidig the statioary distributio: Guess distributio from properties of the system Verify that it satisfies the GBE { } π P = π P, S,,2,... j ji i ij j S i S i S j S Special structure of the Markov chai simplifies task
3-7 Global Balace Equatios Proof π = π P ad P = j i ij ji i= i= π P = π P π P = π P j ji i ij j ji i ij i= i= i j i j π P = π P π P = π P j ji i ij j ji i ij i= i= j S i= j S i= π j Pji + Pji = πipij + πipij j S i S i S j S i S i S π j Pji = πi Pij j S i S i S j S
3-8 Birth-Death Process S S c P P, P, + 2 + P P P, P, P +, Oe-dimesioal Markov chai with trasitios oly betwee eighborig states: P ij =, if i-j > Detailed Balace Equatios (DBE) π P = π P =,,..., + + +, Proof: GBE with S ={,,,} give: π P = π P π P = π P j ji i ij, + + +, j= i= + j= i= +
3-9 Example: Discrete-Time Queue I a time-slot, oe arrival with probability p or zero arrivals with probability -p I a time-slot, the customer i service departs with probability q or stays with probability -q Idepedet arrivals ad service times State: umber of customers i system p p( q) p p( q) 2 + ( p) q( p) q( p) ( p)( q) + pq q( p) ( p)( q) + pq
3-2 Example: Discrete-Time Queue p p( q) p( q) p( q) 2 + ( p) q( p) q( p) ( p)( q) + pq q( p) p/ q πp= π q( p) π = π p p( q) π p( q) = π + q( p) π+ = π, q( p) p( q) Defie: ρ p/ q, α q( p) ρ π = π ρ p π = α π, p π+ = απ, ( p)( q) + pq
3-2 Example: Discrete-Time Queue Have determied the distributio as a fuctio of π How do we calculate the ormalizatio costat π? Probability coservatio law: ρ ρ π π = = α = + = + = p ( p) ( α) Notig that ρ p π = α π, q( p) p( q) q( p) q p q ( p)( α ) = ( p) = = ρ π = ρ π = ρ( α) α,
3-22 Detailed Balace Equatios Geeral case: Imply the GBE Need ot hold for a give Markov chai Greatly simplify the calculatio of statioary distributio Methodology: π P = π P i, j =,,... j ji i ij Assume DBE hold have to guess their form Solve the system defied by DBE ad Σ i π i = If system is icosistet, the DBE do ot hold If system has a solutio {π i : i=,, }, the this is the uique statioary distributio
3-23 Geeralized Markov Chais Markov chai o a set of states {,, }, that wheever eters state i The ext state that will be etered is j with probability P ij Give that the ext state etered will be j, the time it speds at state i util the trasitio occurs is a RV with distributio F ij {Z(t): t } describig the state the chai is i at time t: Geeralized Markov chai, or Semi-Markov process It does ot have the Markov property: future depeds o The preset state, ad The legth of time the process has spet i this state
3-24 Geeralized Markov Chais T i : time process speds at state i, before makig a trasitio holdig time Probability distributio fuctio of T i H () t = P{ T t} = P{ T t ext state j} P = F () t P i i i ij ij ij j= j= ET [ i] = tdhi( t) T ii : time betwee successive trasitios to i X is the th state visited. {X : =,, } Is a Markov chai: embedded Markov chai Has trasitio probabilities P ij Semi-Markov process irreducible: if its embedded Markov chai is irreducible
3-25 Limit Theorems Theorem 3: Irreducible semi-markov process, E[T ii ] < For ay state j, the followig limit p = lim P{ Z( t) = j Z() = i}, i =,,2,... j t exists ad is idepedet of the iitial state. p j = ET [ ] ET [ ] T j (t): time spet at state j up to time t Tj() t P pj = lim Z() = i = t t j jj p j is equal to the proportio of time spet at state j
3-26 Occupacy Distributio Theorem 4: Irreducible semi-markov process; E[T ii ] <. Embedded Markov chai ergodic; statioary distributio π π = π P, j ; π = j i ij i i= i= Occupacy distributio of the semi-markov process p j π j proportio of trasitios ito state j E[T j ] mea time spet at j Probability of beig at j is proportioal to π j E[T j ] π jet [ j] =, j =,,... π ET [ ] i i i
3-27 Cotiuous-Time Markov Chais Cotiuous-time process {X(t): t } takig values i {,,2, }. Wheever it eters state i Time it speds at state i is expoetially distributed with parameter ν i Whe it leaves state i, it eters state j with probability P ij, where Σ j i P ij = Cotiuous-time Markov chai is a semi-markov process with ν i F ( t) = e t, i, j =,,... ij Expoetial holdig times: a cotiuous-time Markov chai has the Markov property
3-28 Cotiuous-Time Markov Chais Whe at state i, the process makes trasitios to state j i with rate: q ν P ij i ij Total rate of trasitios out of state i j i q = ν P = ν ij i ij i j i Average time spet at state i before makig a trasitio: ET [ ] = / ν i i
3-29 Occupacy Probability Irreducible ad regular cotiuous-time Markov chai Embedded Markov chai is irreducible Number of trasitios i a fiite time iterval is fiite with probability From Theorem 3: for ay state j, the limit p = lim P{ X( t) = j X() = i}, i =,,2,... j t exists ad is idepedet of the iitial state p j is the steady-state occupacy probability of state j p j is equal to the proportio of time spet at state j [Why?]
3-3 Global Balace Equatios Two possibilities for the occupacy probabilities: p j =, for all j p j >, for all j, ad Σ j p j = Global Balace Equatios p q = pq, j =,,... j ji i ij i j i j Rate of trasitios out of j = rate of trasitios ito j If a distributio {p j : j =,, } satisfies GBE, the it is the uique occupacy distributio of the Markov chai Alterative form of GBE: p q = p q, S {,,...} j ji i ij j S i S i S j S
3-3 Detailed Balace Equatios Detailed Balace Equatios pq = pq, i, j=,,... j ji i ij Simplify the calculatio of the statioary distributio Need ot hold for ay give Markov chai Examples: birth-death processes, ad reversible Markov chais
3-32 Birth-Death Process S S c λ λ λ λ 2 + µ µ 2 µ µ + Trasitios oly betwee eighborig states q = λ, q = µ, q =, i j > ii, + i ii, i ij Detailed Balace Equatios λ,,,... p = µ + p + = Proof: GBE with S ={,,,} give: pq = pq λ p = µ p j ji i ij + + j= i= + j= i= +
3-33 Birth-Death Process µ p = λ p λ λ λ 2 λ λ 2 λ λi = = 2 =... = = µ µ µ µ µ µ i= µ i+ p p p p p λ i λ i λi p p p, if = = i= µ i+ = i= µ i+ = i= µ i+ = + = = + < Use DBE to determie state probabilities as a fuctio of p Use the probability coservatio law to fid p Usig DBE i problems: Prove that DBE hold, or Justify validity (e.g. reversible process), or Assume they hold have to guess their form ad solve system
3-34 M/M/ Queue Arrival process: Poisso with rate λ Service times: iid, expoetial with parameter µ Service times ad iterarrival times: idepedet Sigle server Ifiite waitig room N(t): Number of customers i system at time t (state) λ λ λ λ 2 + µ µ µ µ
3-35 M/M/ Queue λ λ λ λ 2 + µ Birth-death process DBE Normalizatio costat Statioary distributio µ µ p = λ p λ p = p = ρp =... = ρ p µ = + ρ = = ρ, if ρ < p p p = = p = ρ ( ρ), =,,... µ µ
3-36 The M/M/ Queue Average umber of customers ( ) ( ) = = = N = p = ρ ρ = ρ ρ ρ ρ λ N = ρ( ρ) = = ( ) 2 ρ ρ µ λ Applyig Little s Theorem, we have N T = = λ λ = λ µ λ µ λ Similarly, the average waitig time ad umber of customers i the queue is give by W = T 2 ρ ρ = ad NQ = λw = µ µ λ ρ