Markov Chains. October 5, Stoch. Systems Analysis Markov chains 1

Transcription

1 Markov Chains Alejandro Ribeiro Dept. of Electrical and Systems Engineering University of Pennsylvania October 5, 2011 Stoch. Systems Analysis Markov chains 1

2 Markov chains. Definition and examples Markov chains. Definition and examples Chapman Kolmogorov equations Gambler s ruin problem Queues in communication networks: Transition probabilities Classes of States Limiting distributions Ergodicity Queues in communication networks: Limit probabilities Stoch. Systems Analysis Markov chains 2

3 Markov chains Consider time index n = 0, 1, 2,... & time dependent random state X n State X n takes values on a countable number of states In general denotes states as i = 0, 1, 2,... Might change with problem Denote the history of the process X n = [X n, X n 1,..., X 0 ] T Denote stochastic process as X N The stochastic process X N is a Markov chain (MC) if P [ X n+1 = j Xn = i, X n 1 ] = P [ Xn+1 = j Xn = i ] = P ij Future depends only on current state X n Stoch. Systems Analysis Markov chains 3

4 Observations Process s history X n 1 irrelevant for future evolution of the process Probabilities P ij are constant for all times (time invariant) From the definition we have that for arbitrary m P [ X n+m Xn, X n 1 ] = P [ Xn+m Xn ] X n+m depends only on X n+m 1, which depends only onx n+m 2,... which depends only on X n Since P ij s are probabilities they re positive and sum up to 1 P ij 0 P ij = 1 j=1 Stoch. Systems Analysis Markov chains 4

5 Matrix representation Group transition probabilities P ij in a matrix P P 00 P 01 P P 10 P 11 P P :=.... P i0 P i1 P i Not really a matrix if number of states is infinite Stoch. Systems Analysis Markov chains 5

6 Graph representation A graph representation is also used P i 1,i 1 P ii P i+1,i+1 P i 2,i 1 P i 1,i P i,i+1 P i+1,i+2 i 1 i i +1 P i 1,i 2 P i,i 1 P i+1,i P i+2,i+1 Useful when number of states is infinite Stoch. Systems Analysis Markov chains 6

7 Example: Happy - Sad I can be happy (X n = 0) or sad (X n = 1). Happiness tomorrow affected by happiness today only Model as Markov chain with transition probabilities P := ( ) 0.8 H S Inertia happy or sad today, likely to stay happy or sad tomorrow (P 00 = 0.8, P 11 = 0.7) 0.3 But when sad, a little less likely so (P 00 > P 11 ) Stoch. Systems Analysis Markov chains 7

8 Example: Happy - Sad, version 2 Happiness tomorrow affected by today and yesterday Define double states HH (happy-happy), HS (happy-sad), SH, SS Only some transitions are possible HH and SH can only become HH or HS HS and SS can only become SH or SS 0.1 P := HH HS SH SS More time happy or sad increases likelihood of staying happy or sad 0.3 State augmentation Capture longer time memory Stoch. Systems Analysis Markov chains 8

9 Random (drunkard s) walk Step to the right with probability p, to the left with prob. (1-p) p p p p i 1 i i +1 1 p 1 p 1 p 1 p States are 0, ±1, ±2,..., number of states is infinite Transition probabilities are P i,i+1 = p, P i,i 1 = 1 p, P ij = 0 for all other transitions Stoch. Systems Analysis Markov chains 9

10 Random (drunkard s) walk - continued Random walks behave differently if p < 1/2, p = 1/2 or p > 1/2 100 p = 0.45 p = 0.50 p = position (in steps) position (in steps) position (in steps) time time time With p > 1/2 diverges to the right (grows unbounded almost surely) With p < 1/2 diverges to the left With p = 1/2 always come back to visit origin (almost surely) Because number of states is infinite we can have all states transient They are not revisited after some time (more later) Stoch. Systems Analysis Markov chains 10

11 Two dimensional random walk Take a step in random direction East, West, South or North E, W, S, N chosen with equal probability States are pairs of coordinates (x, y) x = 0, ±1, ±2,... and y = 0, ±1, ±2,... Transiton probabilities are not zero only for points adjacent in the grid Latitude (North South) Longitude (East West) P [ x(t +1) = i +1, y(t + 1) = j x(t) = i, y(t) = j ] = P [ x(t +1) = i 1, y(t + 1) = j x(t) = i, y(t) = j ] = 1 4 P [ x(t +1) = i, y(t + 1) = j +1 x(t) = i, y(t) = j ] = 1 4 P [ x(t +1) = i, y(t + 1) = j 1 x(t) = i, y(t) = j ] = 1 4 Latitude (North South) Longitude (East West) Stoch. Systems Analysis Markov chains 11

12 More about random walks Some random facts of life for equiprobable random walks In one and two dimensions probability of returning to origin is 1 Will almost surely return home In more than two dimensions, probability of returning to origin is less than 1 In three dimensions probability of returning to origin is 0.34 Then 0.19, 0.14, 0.10, 0.08,... Stoch. Systems Analysis Markov chains 12

13 Random walk with borders (gambling) As a random walk, but stop moving when i = 0 or i = J Models a gambler that stops playing when ruined, Xn = 0 Or when reaches target gains Xn = J 1 p p 0... i 1 i i p 1 p 1 J States are 0, 1,..., J. Finite number of states (J). Transition probs. P i,i+1 = p, P i,i 1 = 1 p, P 00 = 1, P JJ = 1 P ij = 0 for all other transitions States 0 and J are called absorbing. Once there stay there forever The rest are transient states. Visits stop almost surely Stoch. Systems Analysis Markov chains 13

14 Chapman Kolmogorov equations Markov chains. Definition and examples Chapman Kolmogorov equations Gambler s ruin problem Queues in communication networks: Transition probabilities Classes of States Limiting distributions Ergodicity Queues in communication networks: Limit probabilities Stoch. Systems Analysis Markov chains 14

15 Multiple step transition probabilities What can be said about multiple transitions? Transition probabilities between two time slots P 2 ij := P [ X m+2 = j Xm = i ] Probabilities of X m+n given X n n-step transition probabilities Pij n := P [ X m+n = j Xm = i ] Relation between n-step, m-step and (m + n)-step transition probs. Write P m+n ij in terms of P m ij and P n ij All questions answered by Chapman-Kolmogorov s equations Stoch. Systems Analysis Markov chains 15

16 2-step transition probabilities Start considering transition probs. between two time slots P 2 ij = P [ X n+2 = j Xn = i ] Using the theorem of total probability Pij 2 = P [ X n+2 = j Xn+1 = k, X n = i ] P [ X n+1 = k Xn = i ] k=1 In the first probability, conditioning on X n = i is unnecessary. Thus P 2 ij = P [ X n+2 = j X n+1 = k ] P [ X n+1 = k X n = i ] k=1 Which by definition yields Pij 2 = P kj P ik k=1 Stoch. Systems Analysis Markov chains 16

17 n-step, m-step and (m + n)-step Identical argument can be made (condition on X 0 to simplify notation, possible because of time invariance) P m+n ij = P [ X n+m = j X 0 = i ] Use theorem of total probability, remove unnecessary conditioning and use definitions of n-step and m-step transition probabilities P m+n ij = P m+n ij = P m+n ij = P [ X m+n = j X m = k, X 0 = i ] P [ X m = k X 0 = i ] k=1 P [ X m+n = j Xm = k ] P [ X m = k X0 = i ] k=1 k=1 P n kjp m ik Stoch. Systems Analysis Markov chains 17

18 Interpretation Chapman Kolmogorov is intuitive. Recall P m+n ij = k=1 P n kjp m ik Between times 0 and m + n time m occurred At time m, the chain is in some state X m = k Pik m is the probability of going from X 0 = i to X m = k Pkj n is the probability of going from X m = k to X m+n = j Product P m ik Pn kj is then the probability of going from X 0 = i to X m+n = j passing through X m = k at time m Since any k might have occurred sum over all k Stoch. Systems Analysis Markov chains 18

19 Matrix form Define matrices P (m) with elements P m ij, P(n) with elements P n ij and P (m+n) with elements P m+n ij k=1 Pn kj Pm ik is the (i, j)-th element of matrix product P(m) P (n) Chapman Kolmogorov in matrix form P (m+n) = P (m) P (n) Matrix of (n + m)-step transitions is product of n-step and m-step Stoch. Systems Analysis Markov chains 19

20 n-step transition probabilities For m = n = 1 (2-step transition probabilities) matrix form is P (2) = PP = P 2 Proceed recursively backwards from n Have proved the following P (n) = P (n 1) P = P (n 2) PP =... = P n Theorem The matrix of n-step transition probabilities P (n) is given by the n-th power of the transition probability matrix P. i.e., Henceforth we write P n P (n) = P n Stoch. Systems Analysis Markov chains 20

21 Example: Happy-Sad Happiness transitions in one day (not the same as earlier example) P := ( ) 0.8 H S 0.3 Transition probabilities between today and the day after tomorrow? P 2 := ( ) H 0.30 S 0.45 Stoch. Systems Analysis Markov chains 21

22 Example: Happy-Sad (continued)... After a week and after a month ( ) P := P 30 := ( ) Matrices P 7 and P 30 almost identical lim n P n exists Note that this is a regular limit After a month transition from H to H with prob. 0.6 and from S to H also 0.6 State becomes independent of initial condition Rationale: 1-step memory initial condition eventually forgotten Stoch. Systems Analysis Markov chains 22

23 Unconditional probabilities All probabilities so far are conditional, i.e., P [ X n = j X0 = i ] Want unconditional probabilities p j (n) := P [X n = j] Requires specification of initial conditions p i (0) := P [X 0 = i] Using theorem of total probability and definitions of Pij n and p j (n) p j (n) := P [X n = j] = P [ X n = j X0 = i ] P [X 0 = i] i=1 = Pij n p i (0) Or in matrix form (define vector p(n) := [p 1 (n), p 2 (n),...] T ) p(n) = P nt p(0) i=1 Stoch. Systems Analysis Markov chains 23

24 Example: Happy-Sad Transition probability matrix P := ( p(0) = [1, 0] p(0) = [0, 1] ) P(Happy) P(Sad) P(Happy) P(Sad) Probabilities Probabilities Time (days) Time (days) For large n probabilities p(t) are independent of initial state p(0) Stoch. Systems Analysis Markov chains 24

25 Gambler s ruin problem Markov chains. Definition and examples Chapman Kolmogorov equations Gambler s ruin problem Queues in communication networks: Transition probabilities Classes of States Limiting distributions Ergodicity Queues in communication networks: Limit probabilities Stoch. Systems Analysis Markov chains 25

26 Gambler s ruin problem You place $b bets, (a) With probability p you gain $b and (b) With probability q = (1 p) you loose your $b bet Start with an initial wealth of $ib Define bias factor α := q/p If α > 1 more likely to loose than win (biased against gambler) α < 1 favors gambler (more likely to win than loose ) α = 1/2 game is fair You keep playing until (a) You go broke (loose all your money) (b) You reach a wealth of $bn Prob. S i of reaching $bn before going broke for initial wealth $ib? S stands for success Stoch. Systems Analysis Markov chains 26

27 Gambler s Markov chain Model as Markov chain. In fact, have met before (normalize by $b) Transition probabilities are P i,i+1 = p, P i,i 1 = q, P 00 = P NN = 1 Sates 0 and N said absorbing. Eventually end up in one of them Remaining states said transient 1 0 p p... i 1 i i q q 1 N Initial state = initial wealth, say $ib Stoch. Systems Analysis Markov chains 27

28 Recursive relations Prob. S i of successful betting run depends on current state i only We can relate probabilities of SBR from adjacent states S i = S i+1 P i,i+1 + S i 1 P i,i 1 = S i+1 p + S i 1 q Recall p + q = 1. Reorder terms p(s i+1 S i ) = q(s i S i 1 ) Recall definition of bias α = q/p S i+1 S i = α(s i S i 1 ) Stoch. Systems Analysis Markov chains 28

29 Recursive relations (continued) If current state is 0 then S i = S 0 = 0. Can write S 2 S 1 = α(s 1 S 0 ) = αs 1 Substitute this in the expression for S 3 S 2 S 3 S 2 = α(s 2 S 1 ) = α 2 S 1 Apply recursively backwards from S i S i 1 S i S i 1 = α(s i 1 S i 2 ) =... = α i 1 S 1 Sum up all of the former to obtain S i S 1 = S 1 ( α + α α i 1) The latter can be written as a geometric series S i = S 1 ( 1 + α + α α i 1) Stoch. Systems Analysis Markov chains 29

30 Geometric series Geometric series can be summed. Assuming α 1 S i = 1 αi 1 α S 1 Write for i = 1. When in state N, S N = 1 1 = S N = 1 αn 1 α S 1 Compute S 1 from latter and substitute into expression for S i S i = 1 αi 1 α N For α = 1 S i = is 1, 1 = S N = NS 1, S i = (i/n) Stoch. Systems Analysis Markov chains 30

31 For large N Consider exit bound N arbitrarily large. For α 1, S i (α i 1)/α N 0 If win prob. does not exceed loose probability will almost surely loose all money For α < 1, P i 1 α i If win prob. exceeds loose probability might win If initial wealth i sufficiently high, will most likely win Stoch. Systems Analysis Markov chains 31

32 Queues in communication systems Markov chains. Definition and examples Chapman Kolmogorov equations Gambler s ruin problem Queues in communication networks: Transition probabilities Classes of States Limiting distributions Ergodicity Queues in communication networks: Limit probabilities Stoch. Systems Analysis Markov chains 32

33 Queues in communication systems Communication systems goal Move packets from generating sources to intended destinations Between arrival and departure we hold packets in a memory buffer Want to design buffers appropriately Stoch. Systems Analysis Markov chains 33

34 Non concurrent queue Time slotted in intervals of duration t n-th slot between times n t and (n + 1) t During each time slot a packet arrives with probability λ Packet arrival rate is expected number of arrivals per unit of time Expected number of arrivals is λ, then arrival rate is λ/ t A packet is transmitted (departs) with probability µ Departure rate (expected departures per unit time) is µ/ t Assume no simultaneous arrival and departure (no concurrence) Reasonable for small t (µ and λ are likely small) Stoch. Systems Analysis Markov chains 34

35 Queue evolution equations q n denotes number of packets in queue in n-th time slot A n = nr. of packet arrivals, D n = nr. of departures (during n-th slot) If there are no packets in queue q n = 0 then there are no departures Queue length at time n + 1 can be written as q n+1 = q n + A n, if q n = 0 If q n > 0, departures and arrivals may happen q n+1 = [ ] +, q n + A n D n if qn > 0 A n {0, 1}, D n {0, 1} and either A n = 1 or D n = 1 but not both Arrival and departure probabilities are P [A n = 1] = λ, P [D n = 1] = µ Stoch. Systems Analysis Markov chains 35

36 Queue evolution probabilities Future queue lengths depend on current length only Probability of queue length increasing P [ q n+1 = i + 1 q n = i ] = P [A n = 1] = λ, for all i Queue length might decrease only if q n > 0. Probability is P [ q n+1 = i 1 qn = i ] = P [D n = 1] = µ, for all i > 0 Queue length stays the same if it neither increases nor decreases P [ q n+1 = i qn = i ] = 1 λ µ, for all i > 0 P [ q n+1 = 0 qn = 0 ] = 1 λ No departures when q n = 0 explain second equation Stoch. Systems Analysis Markov chains 36

37 Queue as a Markov chain MC with states 0, 1, 2,.... Identify states with queue lengths Transition probabilities for i 0 are P i,i 1 = λ, P i,i = 1 λ µ, P i,i+1 = µ For i = 0 P 0,0 = 1 λ and P 01 = λ 1 λ 0 1 λ µ 1 λ ν 1 λ µ λ λ λ λ... i 1 i i µ µ µ Stoch. Systems Analysis Markov chains 37

38 Numerical example: Probability propagation Build matrix P truncating at maximum queue length L = 100 Arrival rate λ = 0.3. Departure rate µ = 0.33 Initial probability distribution p(0) = [1, 0, 0,...] T (queue empty) queue length 0 queue length 10 queue length 20 Propagate probabilities with product P n p(0) Probabilities obtained are Probabilities P [ q n = i q0 = 0 ] = p i (n) Time A few i s (0, 10, 20) shown Probability of empty queue 0.1. Occupancy decrease with index Stoch. Systems Analysis Markov chains 38

39 Classes of states Markov chains. Definition and examples Chapman Kolmogorov equations Gambler s ruin problem Queues in communication networks: Transition probabilities Classes of States Limiting distributions Ergodicity Queues in communication networks: Limit probabilities Stoch. Systems Analysis Markov chains 39

40 Transient and recurrent states States of a MC can be recurrent or transient Transient states might be visited at the beginning but eventually visits stop Almost surely, X n i for n sufficiently large (qualifications needed) Visits to recurrent states keep happening forever Fix arbitrary m Almost surely, X n = i for some n m (qualifications needed) T R R T R Stoch. Systems Analysis Markov chains 40

41 Definitions Let f i be the probability that starting at i, MC ever reenters state i [ f i := P X n = i ] [ X0 = i = P X n = i ] Xm = i n=1 n=m+1 State i is recurrent if f i = 1 Process reenters i again and again (almost surely). Infinitely often State i is transient if f i < 1 Positive probability (1 f i ) of never coming back to i Stoch. Systems Analysis Markov chains 41

42 Recurrent states example State R 3 is recurrent because P [ X 1 = R 3 X0 = R 3 ] = 1 State R 1 is recurrent because P [ X 1 = R 1 X0 = R 1 ] = 0.3 P [ X 2 = R 1, X 1 R 1 X0 = R 1 ] = (0.7)(0.6) P [ X 3 = R 1, X 2 R 1, X 1 R 1 X 0 = R 1 ] = (0.7)(0.4)(0.6). P [ X n = R 1, X n 1 R 1,..., X 1 R 1 X0 = R 1 ] = (0.7)(0.4) n 1 (0.6) 1 T 1 R T R 1 R Sum up: f i = P [ ] X n = R 1, X n 1 R 1,..., X 1 R 1 X 0 = R 1 n=1 = ( n=1 ) ( ) 0.4 n = = Stoch. Systems Analysis Markov chains 42

43 Transient state example States T 1 and T 2 are transient Probability of returning to T 1 is f T1 = (0.6) 2 = 0.36 Might come back to T 1 only if it goes to T 2 (with prob. 0.6) Will come back only if it moves back from T 2 to T 1 (with prob. 0.6) T R R T R Likewise, f T2 = (0.6) 2 = 0.36 Stoch. Systems Analysis Markov chains 43

44 Accessibility State j is accessible from state i if P n ij > 0 for some n 0 It is possible to enter j if MC initialized at X 0 = i Since P 0 ii = P [ X 0 = 1 X0 = i ] = 1, state i is accessible from itself T R R T R All states accessible from T 1 and T 2 Only R 1 and R 2 accessible from R 1 or R 2 None other than itself accessible from R 3 Stoch. Systems Analysis Markov chains 44

45 Communication States i and j are said to communicate (i j) if i is accessible from j, P n ij > 0 for some n; and j is accessible from i, P m ji > 0 for some m Communication is an equivalence relation Reflexivity: i i true because P 0 ii = 1 Symmetry: If i j then j i If i j then P n ij > 0 and P m ji > 0 from where j i Transitivity: If i j and j k, then i k Just notice that P n+m ik Pij n Pjk m > 0 Partitions set of states into disjoint classes (as all equivalences do) What are these classes? (start with a brief detour) Stoch. Systems Analysis Markov chains 45

46 Expected number of visits to states Define N i as the number of visits to state i given that X 0 = i N i := I {X n = i} n=1 If X n = i, this is the last visit to i with probability 1 f i Prob. revisiting state i exactly n times is (n visits no more visits) P [N i = n] = f n i (1 f i ) Number of visits N i has a geometric distribution with parameter f i Expected number of visits is E [N i ] = nfi n (1 f i ) = 1 1 f i n=1 For recurrent states N i = almost surely and E [N i ] = (f i = 1) Stoch. Systems Analysis Markov chains 46

47 Alternative transience/recurrence characterization Another way of writing E [N i ] E [N i ] = n=1 [ ] E I {X n = i} = n=1 Recall that: for transient states E [N i ] = 1/(1 f 1 ) for recurrent states E [N i ] = Therefore proving Theorem State i is transient if and only if n=1 Pn ii < State i is recurrent if and only if n=1 Pn ii = Number of future visits to transient states is finite If number of states is finite some states have to be recurrent P n ii Stoch. Systems Analysis Markov chains 47

48 Recurrence and communication Theorem If state i is recurrent and i j, then j is recurrent Proof. If i j then there are l, m such that P l ji > 0 and P m ij > 0 Then, for any n we have P l+n+m jj P l jip n ii P m ij Sum for all n. Note that since i is recurrent n=1 Pn ii = ( ) PjiP l ii n Pij m = Pji l Pii n Pij m = n=1 P l+n+m jj n=1 Which implies j is recurrent n=1 Stoch. Systems Analysis Markov chains 48

49 Recurrence and transience are class properties Corollary If state i is transient and i j then j is transient Proof. If j were recurrent, then i would be recurrent from previous theorem Since communication defines classes and recurrence is shared by elements of this class, we say that recurrence is a class property Likewise, transience is also a class property States of a MC are separated in classes of transient and recurrent states Stoch. Systems Analysis Markov chains 49

50 Irreducible Markov chains A MC is called irreducible if it has only one class All states communicate with each other If MC also has finite number of states the single class is recurrent If MC infinite, class might be transient When it has multiple classes (not irreducible) Classes of transient states T1, T 2,... Classes of recurrent states R1, R 2,... If MC initialized in a recurrent class Rk, stays within the class If starts in transient class Tk, might stay on T k or end up in a recurrent class R l For large time index n, MC restricted to one class Can be separated into irreducible components Stoch. Systems Analysis Markov chains 50

51 Example T R R T R Three classes T := {T 1, T 2 }, class with transient states R 1 := {R 1, R 2 }, class with recurrent states R 2 := {R 3 }, class with recurrent states Asymptotically in n suffices to study behavior for the irreducible components R 1 and R 2 Stoch. Systems Analysis Markov chains 51

52 Example: Right-biased random walk Step right with probability p > 1/2, left with probability (1-p) p p p p i 1 i i +1 1 p 1 p 1 p 1 p Write current position of random walker as X n = n m=1 S m S m are the steps taken. Note that E [S m ] = 1 2p and E [ S 2 m] = 1 From Central Limit Theorem [ n m=1 P S m n(1 2p) n ] a Φ(a) Stoch. Systems Analysis Markov chains 52

53 Example: Right-biased random walk (continued) Choose a = (1 2p) n [ n ] P [X n 0] = P S m 0 Summing up n=1 State 0 is transient m=1 n=1 ( Φ (1 2p) ) n < e (1 2p)2 n P00 n < P [X n 0] < e (1 2p)2n < n=1 Since all states communicate, all states are transient Stoch. Systems Analysis Markov chains 53

54 Messages States of a MC can be transient of recurrent A MC can be partitioned in classes of states reachable from each other Elements of the class are either all recurrent or all transient A MC with only one class is irreducible If not irreducible can be separated in irreducible components Stoch. Systems Analysis Markov chains 54

55 Limiting distributions Markov chains. Definition and examples Chapman Kolmogorov equations Gambler s ruin problem Queues in communication networks: Transition probabilities Classes of States Limiting distributions Ergodicity Queues in communication networks: Limit probabilities Stoch. Systems Analysis Markov chains 55

56 Limiting distributions MCs have one-step memory. Eventually they forget initial state What can we say about probabilities for large n? π j := lim n P [ X n = j X0 = i ] = lim n Pn ij Implicitly assumed that limit is independent of initial state X 0 = i We ve seen that this problem is related to the matrix power P n P := P 2 := ( ) ( ) P 7 := P 30 := ( ( ) ) Matrix product converges probs. independent of time (large n) All columns are equal probs. independent of initial condition Stoch. Systems Analysis Markov chains 56

57 Periodicity The period of a state i is defined as (ḋ is set of multiples of d) { } d = max d : Pii n = 0 for all n / ḋ State i is periodic with period d if and only if P n ii 0 only if n is a multiple of d (n ḋ) d is the largest number with this property Positive probability of returning to i only every d time steps If period d = 1 state is aperiodic (most often the case) Periodicity is a class property p 1 p State 1 has period 2. So do 0 and 2 (class property) One dimensional random walk also has period 2 Stoch. Systems Analysis Markov chains 57

58 Positive recurrence and ergodicity Recall: state i is recurrent if chain returns to i with probability 1 Proved it was equivalent to n=1 Pn ii = Positive recurrent when expected value of return time is finite E [return time] = npii n n=1 m=0 n 1 (1 Pii m ) < Null recurrent if recurrent but E [return time] = Positive and null recurrence are class properties Recurrent states in a finite-state MC are positive recurrent Ergodic states are those that are positive recurrent and aperiodic An irreducible MC with ergodic states is said to be an ergodic MC Stoch. Systems Analysis Markov chains 58

59 Example of a null recurrent MC 1/2 1/3 1/ /2 2/3 3/4... P [return time = 2] = 1 2 P [return time = 4] = = P [return time = 3] = P [return time = n] = (n 1) n It is recurrent because probability of returning is 1 (use induction) n P [return time = m] = m=2 n m=2 1 (m 1) m = n 1 1 n Null recurrent because expected return time is infinite np [return time = n] = n=2 n=2 n (n 1) n = n=2 1 (n 1) = Stoch. Systems Analysis Markov chains 59

60 Limit distribution of ergodic Markov chains Theorem For an irreducible ergodic MC, lim n P ij exists and is independent of the initial state i. That is π j = lim n Pn ij exists Furthermore, steady state probabilities π j 0 are the unique nonnegative solution of the system of linear equations π j = π i P ij, i=0 π j = 1 j=0 As observed, limit probs. independent of initial condition exist Simple algebraic equations can be solved to find π j No periodic states, transient states, multiple classes or null recurrent Stoch. Systems Analysis Markov chains 60

61 Algebraic relation to determine limit probabilities Difficult part of theorem is to prove that π j = lim n P n ij exists To see that algebraic relation is true use theorem of total probability (omit conditioning on X 0 to simplify notation) P [X n+1 = j] = = P [ X n+1 = j Xn = i ] P [X n = i] i=1 P ij P [X n = i] i=1 If limits exists, P [X n+1 = j] P [X n = j] π j (sufficiently large n) π j = π i P ij The other equation is true because the π j are probabilities i=0 Stoch. Systems Analysis Markov chains 61

62 Vector/matrix notation: Matrix limit More compact and illuminating on vector/matrix notation Finite MC with J states First part of theorem says that lim n P n exists and π 1 π 2... π J π 1 π 2... π J lim n Pn =.... π 1 π 2... π J Same probs. for all rows independent of initial state Probability distribution for large n. lim p(n) = lim n n PT n p(0) = [π 0, π 1,..., π J ] T Independent of initial condition Stoch. Systems Analysis Markov chains 62

63 Vector/matrix notation: Eigenvector Define vector stationary distribution π := [π 0, π 1,..., π J ] T Limit distribution is unique solution of (1 = [1, 1,...] T ) π = P T π, π T 1 = 1 π eigenvector associated with eigenvalue 1 of P T Eigenvectors are defined up to a constant Normalize to sum 1 All other eigenvectors of P T have modulus smaller than 1 If not, P n diverges, but we know P n contains n-step transition probs. π eigenvector associated with largest eigenvalue of P T Computing π as eigenvector is computationally efficient and robust in some problems Stoch. Systems Analysis Markov chains 63

64 Vector/matrix notation: Rank Can also write as (I is identity matrix, 0 = [0, 0,...] T ) ( I P T ) π = 0 π T 1 = 1 π has J elements, but there are J + 1 equations overdetermined If 1 is eigenvalue of P T, then 0 is eigenvalue of I P T I P T is rank deficient, in fact rank(i P T ) = J 1 Then, there are in fact only J equations π is eigenvector associated with eigenvalue 0 of I P T π spans null space of P T (not much significance) Stoch. Systems Analysis Markov chains 64

65 Example: Aperiodic, irreducible Markov chain MC with transition probability matrix P := Does P correspond to an ergodic MC? All states communicate with state 2 (full row and column P2j 0 and P j2 0 for all j) No transient states (irreducible with one recurrent state and finite) Aperiodic (period of state 2 is 1) Then, there exist π 1, π 2 and π 3 such that π j = lim n P n ij Limit is independent of i Stoch. Systems Analysis Markov chains 65

66 Example: Aperiodic, irreducible MC (continued) How do we determine limit probabilities π j? Solve system of linear equations π j = i=0 π ip ij and j=0 π j = 1 π 1 π 2 π 3 1 = The upper part of matrix above is P T There are three variables and four equations Some equations might be linearly dependent Indeed, summing first three equations: π1 + π 2 + π 3 = π 1 + π 2 + π 3 Always true, because probabilities in rows of P sum up to 1 This is because of rank deficiency of I P T Solution yields π 1 = , π 2 = and π 3 = π 1 π 2 π 3 Stoch. Systems Analysis Markov chains 66

67 Stationary distribution Limit distributions are sometimes called stationary distributions Select initial distribution such that P [X 0 = i] = π i for all i Probabilities at time n = 1 follow from theorem of total probability P [X 1 = i] = P [ X 1 = j X0 = i ] P [X 0 = i] i=1 Definitions of P ij, and P [X 0 = i] = π i. Algebraic property of π j P [X 1 = i] = P ij π i = π j i=1 Probability distribution stayed unchanged Proceeding recursively, system initialized with P [X 0 = i] = π i, Probability distribution constant, P [X n = i] = π i for all n MC stationary in a probabilistic sense (states change, probs. do not) Stoch. Systems Analysis Markov chains 67

68 Ergodicity Markov chains. Definition and examples Chapman Kolmogorov equations Gambler s ruin problem Queues in communication networks: Transition probabilities Classes of States Limiting distributions Ergodicity Queues in communication networks: Limit probabilities Stoch. Systems Analysis Markov chains 68

69 Ergodicity Define T (n) i as fraction of time spent in i-th state up to time n T (n) i := 1 n n I {X m = i} m=1 Compute expected value of T (n) i [ ] E T (n) i = 1 n E [I {X m = i}] = 1 n n m=1 n P [X m = i] π i m=1 As time n, probabilities P [X m = i] approach π i. Then [ ] lim E T (n) 1 n t i = lim P [X m = i] = π i t n m=1 For ergodic MCs same is true without expected value ergodicity lim T (n) 1 n i = lim n n n I {X m = i} = π i, m=1 a.s. Stoch. Systems Analysis Markov chains 69

70 Example: Ergodic Markov chain Recall transition probability matrix P := Visits to states, nt (n) i Ergodic averages, T (n) i 25 State 1 State 2 State State 1 State 2 State number of visits ergodic averages time time Ergodic averages slowly converge to limit probabilities Stoch. Systems Analysis Markov chains 70

71 Function s Ergodic average Use of ergodic averages is more general than T (n) i Theorem Consider an irreducible Markov chain with states X n = 0, 1, 2,... and stationary probabilities π j. Let f (X n ) be a bounded function of the state X (n). Then, with probability 1 1 lim n n n f (X m ) = f (i)π i m=1 i=1 T (n) i is a particular case with f (X m ) = I {X m = i} Think of f (X m ) as a reward associated with state X (m) (1/n) n m=1 f (X m) is the time average of rewards Stoch. Systems Analysis Markov chains 71

72 Function s Ergodic average (proof) Proof. Because I {X m = i} = 1 if and only if X m = i we can write ( 1 n f (X m ) = 1 n ) f (i)i {X m = i} n n m=1 m=1 i=1 Change order of summations. Use definition of T (n) i ) 1 n n f (X m ) = f (i) m=1 i=1 Let n in both sides ( 1 n n I {X m = i} m=1 = i=1 f (i)t (n) i Use ergodic average result for lim n T (n) i = π i [cf. page 69] Stoch. Systems Analysis Markov chains 72

73 Ensemble and ergodic averages There s more depth to ergodic results than meets the eye Ensemble average: across different realizations of the MC E [f (X n )] = f (i)p (X n = i) i=1 f (i)π i i=1 Ergodic average: across time for a single realization of the MC f (n) = 1 n n f (X n ) m=1 These quantities are fundamentally different but their values coincide asymptotically in n Observing one realization of the MC provides as much information as observing all realizations Practical consequence: Observe/simulate only one path of the MC Stoch. Systems Analysis Markov chains 73

74 Ergodicity in periodic MCs In some sense, still true if MC is periodic For irreducible positive recurrent MC (periodic or aperiodic) define π j = π i P ij, i=0 π j = 1 j=0 A unique solution exists (we say π j are well defined) The fraction of time spent in state i converges to π i lim T (n) 1 n i = lim n n n I {X m = i} π i m=1 If MC is periodic, probabilities oscillate, but fraction of time spent in state i converges to π i Stoch. Systems Analysis Markov chains 74

75 Example: Periodic irreducible Markov chain Matrix P and state diagram of a periodic MC P := MC has period 2. If initialized with X 0 = 1, then P 2n+1 11 = P [ X 2n+1 = 1 X 0 = 1 ] = 0, P 2n 11 = P [ X 2n = 1 X0 = 1 ] = 1 0 Define π := [π 1, π 2, π 3 ] T as solution of π π 2 π 3 = Normalized eigenvector for eigenvalue 1 (π = P T π, π T 1 = 1) π 1 π 2 π 3 Stoch. Systems Analysis Markov chains 75

76 Example: Periodic irreducible MC (continued) Solution yields π 1 = 0.15, π 2 = 0.50 and π 3 = Visits to states, nt (n) i State 1 State 0 State Ergodic averages, T (n) i State 1 State 0 State number of visits ergodic averages time Ergodic averages T (n) i time converge to the ergodic limits π i Stoch. Systems Analysis Markov chains 76

77 Example: Periodic irreducible MC (continued) Powers of the transition probability matrix do not converge P 2 = P 3 = = P In general we have P 2n = P 2 and P 2n+1 = P At least one other eigenvalue of the transition probability matrix has modulus 1 eig 2 ( P T ) = 1 In this example, eigenvalues of P T are 1, 1 and 0 Stoch. Systems Analysis Markov chains 77

78 Reducible Markov chains If MC is not irreducible it can be decomposed in transient (T k ), ergodic (E k ), periodic (P k ) and null recurrent (N k ) components All of these are class properties Limit probabilities for transient states are null P [X n = i] 0, for all X n T k For arbitrary ergodic component E k, define conditional limits π i = lim n P [ X n = i X 0 E k ], i Ek Result in page 60 is true with this (re)defined π i Stoch. Systems Analysis Markov chains 78

79 Reducible Markov chains (continued) Likewise, for arbitrary periodic component P k (re)define π j as π j = i P k π i P ij, j P k π j = 1, j P k A conditional version of the result in page 74 is true lim T (n) 1 n i := lim n n n I { X m = i } X 0 P k πi m=1 For null recurrent components limit probabilities are null P [X n = i] 0, for all X n N k Stoch. Systems Analysis Markov chains 79

80 Example: Reducible Markov chain Transition matrix and state diagram of a reducible MC 0.3 P := States 1 and 2 are transient T = {1, 2} States 3 and 4 form an ergodic class E 1 = {3, 4} State 5 is a separate ergodic class E 2 = {5} 0.4 Stoch. Systems Analysis Markov chains 80

81 Example: Reducible MC - matrix powers 10-step and 20 step transition probabilities P 5 = P10 = Transition into transient states is vanishing (columns 1 and 2) Transition from 3 and 4 into 3 and 4 only If initialized in ergodic class E1 = {3, 4} stays in E 1 Transition from 5 only into 5 From transient states T = {1, 2} can go into either ergodic component E 1 = {3, 4} or E 2 = {5} Stoch. Systems Analysis Markov chains 81

82 Example: Reducible MC - matrix decomposition Matrix P can be separated in blocks P = = P T P T E1 P T E2 0 P E P E2 Block P T describes transition between transient states Blocks P E1 and P E2 describe transitions in ergodic components Blocks P T E1 and P T E2 describe transitions from T to E 1 and E 2 Powers of n can be written as P n P n T Q T E1 Q T E2 = 0 P n E P n E 2 The transient transition block converges to 0, lim n P n T = 0 Stoch. Systems Analysis Markov chains 82

83 Example: Reducible MC - limiting behavior As n grows the MC hits an ergodic state with probability 1 Henceforth, MC stays within ergodic component P [ X n+m E i Xn E i ] = 1, for all m For large n suffices to study ergodic components MC behaves like a MC with transition probabilities P E1 Or like one with transition probabilities P E2 We can think that all MCs as ergodic Ergodic behavior cannot be inferred a priori (before observing) Becomes known a posteriori (after observing sufficiently large time) Culture micro: Something is known a priori if its knowledge is independent of experience (MCs exhibit ergodic behavior). A posteriori knowledge depends on experience (values of the ergodic limits). They are inherently different forms of knowledge (search for Immanuel Kant) Stoch. Systems Analysis Markov chains 83

84 Queues in communication systems Markov chains. Definition and examples Chapman Kolmogorov equations Gambler s ruin problem Queues in communication networks: Transition probabilities Classes of States Limiting distributions Ergodicity Queues in communication networks: Limit probabilities Stoch. Systems Analysis Markov chains 84

85 Non concurrent communication queue Communication system: Move packets from source to destination Between arrival and transmission hold packets in a memory buffer Example problem, buffer design: Packets arrive at a rate of 0.45 packets per unit of time and depart at a rate of How many packets the buffer needs to hold to have a drop rate smaller than 10 6 (one packet dropped for every million packets handled) Time slotted in intervals of duration t During each time slot n A packet arrives with prob. λ, arrival rate is λ/ t A packet is transmitted with prob. µ, departure rate is µ/ t No concurrence: No simultaneous arrival and departure (small t) Stoch. Systems Analysis Markov chains 85

86 Queue evolution probabilities (reminder) Future queue lengths depend on current length only Probability of queue length increasing P [ q n+1 = i + 1 q n = i ] = λ, for all i Queue length might decrease only if q n > 0. Probability is P [ q n+1 = i 1 qn = i ] = µ, for all i > 0 Queue length stays the same if it neither increases nor decreases P [ q n+1 = i qn = i ] = 1 λ µ, for all i > 0 P [ q n+1 = 0 qn = 0 ] = 1 λ No departures when q n = 0 explain second equation Stoch. Systems Analysis Markov chains 86

87 Queue as a Markov chain (reminder) MC with states 0, 1, 2,.... Identify states with queue lengths Transition probabilities for i 0 are P i,i 1 = λ, P i,i = 1 λ µ, P i,i+1 = µ For i = 0 P 0,0 = 1 λ and P 01 = λ 1 λ 0 1 λ µ 1 λ ν 1 λ µ λ λ λ λ... i 1 i i µ µ µ Stoch. Systems Analysis Markov chains 87

88 Numerical example: Limit probabilities Build matrix P truncating at maximum queue length L = 100 Arrival rate λ = 0.3. Departure rate µ = 0.33 Find eigenvector of P T associated with largest eigenvalue (i.e., 1) Yields limit probabilities π = lim n p(n) linear scale logarithmic scale Limiting probabilities Limiting probabilities state state Limit probabilities appear linear in logarithmic scale Seemingly implying an exponential expression π i α i Stoch. Systems Analysis Markov chains 88

89 Limit distribution equations 1 λ 1 λ µ 1 λ ν 1 λ µ λ λ λ λ 0... i 1 i i µ µ µ Limit distribution equations for state 0 (empty queue) For the remaining states π 0 = (1 λ)π 0 + µπ 1 π i = λπ i 1 + (1 λ µ)π i + µπ i+1 Propose candidate solution π i = cα i Stoch. Systems Analysis Markov chains 89

90 Verification of candidate solution Substitute candidate solution π i = cα i in equation for π 0 cα 0 = (1 λ)cα 0 + µcα 1 1 = (1 λ) + µα The above equation is true if we make α = λ/µ Does α = λ/µ verify the remaining equations? From the equation for generic π i (divide by cα i 1 ) cα i = λcα i 1 + (1 λ µ)cα i + µcα i+1 µα 2 (λ + µ)α + λ = 0 The above quadratic equation is satisfied by α = λ/µ And α = 1, which is irrelevant Stoch. Systems Analysis Markov chains 90

91 Compute normalization constant Determine c so that probabilities sum 1 ( i=0 π i = 1) π i = i=0 Used geometric sum J c(λ/µ) i c = 1 λ/µ = 1 i=0 Solving for c and substituting in π i = cα i yields π i = (1 λ/µ) ( ) i λ µ The ratio µ/λ is the queues stability margin Larger µ/λ larger probability of having few queued packets Stoch. Systems Analysis Markov chains 91

92 Queue balance equations Rearrange terms in equation for limit probabilities [cf. page 89] λπ 0 = µπ 1 (λ + µ)π i = λπ i 1 + µπ i+1 λπ 0 is average rate at which the queue leaves state 0 Likewise (λ + µ)π i is the rate at which queue leaves state i µπ 0 is average rate at which the queue enters state 0 λπ i 1 + µπ i+1 is rate at which queue enters state i Limit equations prove validity of queue balance equations Rate at which leaves = Rate at which enters 1 λ 0 1 λ µ 1 λ ν 1 λ µ λ λ λ λ... i 1 i i µ µ µ Stoch. Systems Analysis Markov chains 92

93 Concurrent arrival and departures Packets may arrive and depart in same time slot (concurrence) Queue evolution equations remain the same, [cf. 35] But queue probabilities change [cf. 86] Probability of queue length increasing (for all i) P [ q n+1 = i + 1 qn = i ] = P [A n = 1] P [D n = 0] = λ(1 µ) Queue length might decrease only if q n > 0 (for all i > 0) P [ q n+1 = i 1 qn = i ] = P [D n = 1] P [D n = 0] = µ(1 λ) Queue length stays the same if it neither increases nor decreases P [ q n+1 = i qn = i ] = λµ + (1 λ)(1 µ) = ν, for all i > 0 P [ q n+1 = 0 qn = 0 ] = (1 λ) + λµ Stoch. Systems Analysis Markov chains 93

94 Limit distribution / queue balance equations Write limit distribution equations queue balance equations Rate at which leaves = rate at which enters λ(1 µ)π 0 = µ(1 λ)π 1 ( λ(1 µ) + µ(1 λ) ) πi = λ(1 µ)π i 1 + µ(1 λ)π i+1 Propose exponential solution π = cα i (1 λ) + λν ν ν ν λ(1 µ) λ(1 µ) λ(1 µ) λ(1 µ) 0... i 1 i i µ(1 λ) µ(1 λ) µ(1 λ) Stoch. Systems Analysis Markov chains 94

95 Solving for limit distribution Substitute candidate solution in equation for π 0 λ(1 µ)c = µ(1 λ)cα α = λ(1 µ) µ(1 λ) Same substitution in equation for generic π i µ(1 λ)cα 2 + ( λ(1 µ) + µ(1 λ) ) cα + λ(1 µ)c = 0 which as before is solved for α = λ(1 µ)/µ(1 λ) Find constant c to ensure i=0 cαi = 1 (geometric series). Yields π i = (1 α)α i = ( 1 λ(1 µ) µ(1 λ) ) ( ) i λ(1 µ) µ(1 λ) Stoch. Systems Analysis Markov chains 95

96 Limited queue size Packets dropped if there are too many packets in queue Too many packets in queue, then delays too large, packets useless when they arrive. Also preserve memory Equation for state J requires modification (rate leaves = rate enters) µ(1 λ)π J = λ(1 µ)π J 1 π i = cα i with α = λ(1 µ)/µ(1 λ) also solve this equation (Yes!) (1 λ) + λν ν ν ν (1 µ) + λν λ(1 µ) λ(1 µ) λ(1 µ) λ(1 µ) i 1 i i +1 J µ(1 λ) µ(1 λ) µ(1 λ) µ(1 λ) Stoch. Systems Analysis Markov chains 96

97 Compute limit distribution Limit probabilities are not the same because constant c is different To compute c, sum a finite geometric series 1 = J i=0 cα i = c 1 αj+1 1 α c = 1 α 1 α J+1 Limit distributions for the finite queue are then π i = 1 α 1 α J+1 αi (1 α)α i with α = λ(1 µ)/µ(1 λ), and approximation valid for large J Approximation for large J yields same result as infinite length queue As it should Stoch. Systems Analysis Markov chains 97

98 Simulations Arrival rate λ = 0.3. Departure rate µ = Resulting α 0.87 Maximum queue length J = 100. Initial state q 0 = 0 (queue empty) Not the same as initial probability distribution Queue lenght as function of time queue length time(s) Stoch. Systems Analysis Markov chains 98

99 Simulations: Average occupancy and limit distribution Average time spent at each queue state is predicted by limit distribution For i = 60 occupancy probability is π Explains inaccurate prediction for large i 60 states 20 states 10 1 limit probs ergodic average limit probs ergodic average Limit probabs/ergodic average Limit probabs/ergodic average state state Stoch. Systems Analysis Markov chains 99

100 Buffer overflow If λ = 0.45 and µ = 0.55 how many packets the buffer needs to hold to have a drop rate smaller than 10 6 (one packet dropped for every million packets handled) What is the probability of buffer overflow? Packet discarded if queue is in state J and a new packet arrives P [overflow] = λπ J = 1 α 1 α J+1 pαj (1 α)pα J Withλ = 0.45 and µ = 0.55, α 0.82 J 57 A final caveat Still assuming only 1 packet arrives per time slot Lifting this assumption requires introduction of continuous time Markov processes Stoch. Systems Analysis Markov chains 100