ISE/OR 760 Applied Stochastic Modeling

ISE/OR 760 Applied Stochastic Modeling Topic 2: Discrete Time Markov Chain Yunan Liu Department of Industrial and Systems Engineering NC State University Yunan Liu (NC State University) ISE/OR 760 1 / 60

Outline Examples Definitions Markov property Transition probabilities Transient behavior Chapman-Kolmogorov equation Gambler s ruin problem State classification Communication Periodicity Recurrence and transience Steady state of good DTMC Balance equation πp = π DTMC having absorbing states Canonical form Time reversibility Random walks on weighted graphs Yunan Liu (NC State University) ISE/OR 760 2 / 60

Examples 1 Economics (Asset pricing, market crashes) 2 Chemistry (Eenzyme activity, growth of certain compounds) 3 Internet Applications (Google PageRank) 4 Physics 5 Information sciences 6 Sports, etc. Figure: A DTMC system modeling a a remote medical assistance system Yunan Liu (NC State University) ISE/OR 760 3 / 60

Ab Example: PageRank Algorithm Application: Best-known search engine algorithm Idea: Counting the number of links to a webpage to estimate its importance Developer: Larry Page Yunan Liu (NC State University) ISE/OR 760 4 / 60

Definition and Transient Performance Yunan Liu (NC State University) ISE/OR 760 5 / 60

Markov Processes Figure: Andrei Markov (1856-1922) Markov Property: Given present state, forecast of future is independent of past P(Future Present, Past) = P(Future Present) Yunan Liu (NC State University) ISE/OR 760 6 / 60

Markov Processes A process that follows the Markov Property is called a Markov Process. { Discrete State Space Continuous [ Time {}}{ Discrete Continuous DTMC e.g., random walks CTMC e.g., Brownian motions ] This class will focus on the discrete space Markov processes. (DTMC and CTMC) Examples: Discrete State Space Number of customers in queue Inventory Level Location (with discrete elements) Continuous State Space Temperature Level Geographic Location Amount of time left in process Yunan Liu (NC State University) ISE/OR 760 7 / 60

Definition Definition (DTMC) A discrete time discrete space stochastic process {X n, n = 0, 1,...} is called a DTMC if P(X n+1 = j X n = i, X n 1 = i n 1... X 0 = i 0 ) = Markov Property - Given present (today), future (tomorrow) is independent of past (yesterday) Example: Tomorrow will be sunny with an 80% chance if today is also sunny or with a 40% chance if today is rainy. Set up a DTMC for weather forecasting. Yunan Liu (NC State University) ISE/OR 760 8 / 60

Transition Probability Matrix (TPM) Let P be a transition probability matrix (TPM) with elements defined as P i,j = P(X n+1 = j X n = i) Properties of TPM P: Nonnegativity: Row sum: Remark: DTMC may in general be time nonhomogeneous: P i,j (n) = P(X n+1 = j X n = i). In this class, we focus on time homogeneous DTMC. Yunan Liu (NC State University) ISE/OR 760 9 / 60

Dynamics of DTMC {X n, n 0} evolves in a Markovian manner: The chain stays in a state for 1 time unit; When in state i, X n moves to a state j w.p. P i,j, i, j S. Sample path of a DTMC with state space S = {1, 2, 3, 4, 5, 6}. Yunan Liu (NC State University) ISE/OR 760 10 / 60

Questions of Interest 1 Transient performance (short-term behavior) Transition probabilities shortly after the process begins? P(X1 = j X 0 = i) =? P(X1 = i, X 4 = j, X 7 = k) =? 2 Steady-state performance (long-run behavior) Asymptotic performance after many transitions: P(X100 = j X 0 = i)? Long-run proportion of time (steps): for N large, 1 N N n=1 1 {X n=j}? Termination Probability of which state the process will terminate in Expected number of transitions until termination. Yunan Liu (NC State University) ISE/OR 760 11 / 60

Transient Behavior Yunan Liu (NC State University) ISE/OR 760 12 / 60

Chapman-Kolmogorov Equation Define the n-step transition probability from i j P (n) i,j n-step TPM P (n) has element P (n) i,j P (1) i,j = Review: P(A) = P(A B) = Yunan Liu (NC State University) ISE/OR 760 13 / 60

Derivation of C-K Equation (P (m+n) ) i,j = P (m+n) i,j = P(X n+m = j X 0 = i) = k P(X n+m = j X m = k, X 0 = i) P(X m = k X 0 = i) = k P(X n+m = j X m = k) P(X m = k X 0 = i) = k = P (n) i,k P(m) k,j ( P (m) P (n)) i,j Chapman-Kolmogorov (C-K) equation P (m+n) = P (m) P (n) = P (n) P (m) (in matrix notation) The C-K Equation allows us to relate the (m + n) step transition matrix to the product of two smaller step-size (m) and (n) transition matrices. Yunan Liu (NC State University) ISE/OR 760 14 / 60

Derivation of C-K Equation A quick implication of the C-K Equation: ( ) P (n) = P (n 1) P (1) = P (n 1) P = P (n 2) P P ( ) ( ) = P (n 3) P P P = = P (2) P P P }{{} n 3 terms = P P... P }{{} n terms Summary: n-step TPM is the n th power of the 1-step TPM P (n) = P n Yunan Liu (NC State University) ISE/OR 760 15 / 60

Finite-Dimensional Distribution (F.D.D.) For 0 n 1 n 2 n m, i 1, i 2,..., i m S P(X n1 = i 1, X n2 = i 2,..., X nm = i m ) P(X 10 = 1, X 7 = 5, X 3 = 10) = P(A B C) = P(A B C) P(B C) = P(A B C) P(B C) P(C) = P(X 10 = 1 X 7 = 5, X 3 = 10) P(X 7 = 5 X 3 = 10) P(X 3 = 10) = P(X 10 = 1 X 7 = 5) P(X 7 = 5 X 3 = 10) P(X 3 = 10) = P (3) 5,1 P(4) 10,5 k S P(X 3 = 10 X 0 = k) P(X 0 = k) = P (3) 5,1 P(4) 10,5 Only need k S P (3) k,3 P(X 0 = k) = (P 3 ) 5,1 (P 4 ) 10,5 (P 3 ) k,3 P(X 0 = k) Distribution of initial state P(X 0 = k) 1-step TPM P. k S Yunan Liu (NC State University) ISE/OR 760 16 / 60

An Example: Pooh Bear A bear of little brain named Pooh is fond of honey. Bees producing honey are located in three trees: tree A, tree B, and tree C. Tending to be somewhat forgetful, Pooh goes back and form among these three honey trees randomly: from A, Pooh goes next to B or C with probability 1/2; from B, Pooh goes next to A w.p 3/4 and to C w.p. 1/4; from C, Pooh always goes next to A. Construct a DTMC and specify its TPM. Yunan Liu (NC State University) ISE/OR 760 17 / 60

State Classification Yunan Liu (NC State University) ISE/OR 760 18 / 60

Communication Two states i, j S Accessibility: denoted by i j State j is accessible from state i if there exists n 0 s.t. P (n) i,j > 0 Communication: denoted by i j States i and j communicate if i j, j i Properties: (i) i i (note when n = 0, P (0) = I) (ii) if i j, then j i (iii) if i j, j k, then i k Proof of (iii): i j there exists m s.t. P (m) i,j > 0 P (m+n) i,k j k there exists n s.t. P (n) j,k > 0 = j S P (m) i,j P (n) j,k > P(m) i,j P (n) j,k > 0 Yunan Liu (NC State University) ISE/OR 760 19 / 60

Irreducibility and Periodicity 1 Class: We say i, j belong to the same class, if i j 2 Irreducible: We say a DTMC is irreducible if there is only one class. Periodicity: 1 The period of a state i, denoted d(i), is the greatest common divisor of all n 0 s.t. P (n) i,i > 0 2 If d(i) = 1 we say i is aperiodic. Property of Periodicity: Periodicity (aperiodicity) is a class property, that is if i j, then d(i) = d(j) Yunan Liu (NC State University) ISE/OR 760 20 / 60

Periodicity Proof: Assume P (s) i,i > 0 for some s > 0 Yunan Liu (NC State University) ISE/OR 760 21 / 60

Ever Hitting Probabilities Define the probability of visiting state j from state i in exactly n steps: f (n) i,j = P(X n = j, X k j, 1 k n 1 X 0 = i) = P(1 st transition to j occurs at step n starting in i) Next define the probability that state j will ever be visited after leaving state i: Note: f i,j = f i,j > 0 if i j; f i,j = 0 if i j n=1 f (n) i,j = P(j is ever visited after leaving i) Yunan Liu (NC State University) ISE/OR 760 22 / 60

State Classification A state j of DTMC is recurrent if f j,j = 1 (Starting in j, will always return to j w.p.1.) transient if f j,j < 1 Starting in j, will ever (never) return to j w.p. f j,j (w.p. 1 f j,j > 0). N j,j = total number of visits in j after leaving j. {, if j is recurrent N j,j = Geo(1 f j,j ) 1, if j is transient A necessary and sufficient condition: j is recurrent n=1 P(n) j,j = j is transient n=1 P(n) j,j < Proof: [ ] E[N j,j ] = E 1(X n = j X 0 = j) = P(X n = j X 0 = j) = n=1 n=1 n=1 P (n) j,j Yunan Liu (NC State University) ISE/OR 760 23 / 60

State Classification Both transience & recurrence are class properties. If i j (i, j belong to the same class) Proof: i is recurrent j is recurrent i is transient j is transient Because P (m+n+l) j,j i j P (n) i,j > 0, P (m) j,i > 0 for some m 0, n 0 P (m) j,i P (l) i,i P(n) i,j, we have k=1 P (k) j,j k>m+n P (k) j,j = l=1 P (m+n+l) j,j l=1 P (m) j,i P (l) i,i P(n) i,j = P (m) j,i P (n) i,j l=1 P (l) i,i. Yunan Liu (NC State University) ISE/OR 760 24 / 60

Gambler s Ruin Problem A gambler who at each round has probability p of winning one unit and probability q = 1 p of losing one unit. The game ends either the gambler s total fortune reaches level N > 0, or his fortune reaches 0 (the gambler is broke). Let X 0 be the initial fortune. Define the X i, i 1, as the outcome of i th bet: { +1, w.p. p X i = 1, w.p. q = 1 p The gambler s fortune after the n th bet: S n = n X i. Question: Is {S n, n 0} a DTMC? If yes, how many classes? Which are transient and which are recurrent? What are the periods? i=0 Yunan Liu (NC State University) ISE/OR 760 25 / 60

Probability of Winning Let P i be the probability the gambler reaches target fortune N before going bankrupt, given a starting fortune of i: P i = P(hits N before 0 X 0 = i) = P(hits N before 0 X 1 = 1, X 0 = i) p + P(hits N before 0 X 1 = 1, X 0 = i) q Therefore, P i p + P i q = P i = P i+1 p + P i 1 q P i+1 P i = γ(p i P i 1 ), γ = q p. A recursion with boundary conditions P 0 = 0 and P N = 1 P 2 P 1 = γ(p 1 P 0 ) = γp 1 P 3 P 2 = γ(p 2 P 1 ) = γ 2 P 1. P i P i 1 = γ(p i 1 P i 2 ) = γ i 1 P 1. P N P N 1 = γ(p N 1 P N 2 ) = γ N 1 P 1 Yunan Liu (NC State University) ISE/OR 760 26 / 60

Probability of Winning Summing up the i 1 equations yields: P i = [ 1 + q p + + ( q p ) i 1 ] Summing up all N equations yields: 1 = P N = [ 1 + q p + + ( q p ) N 1 ] P 1 = P 1 = { 1 ( q p ) i 1 q p P 1, if p q i P 1, if p = q. { 1 ( q p ) N 1 q p P 1, if p q N P 1, if p = q. Finally, P i = 1 ( q p ) i, if p q (unfair game) 1 ( q p ) N, if p = q = 1/2 (fair game). i N Yunan Liu (NC State University) ISE/OR 760 27 / 60

State Classification (Continued) Denote T j,j = total # of steps until returning to j the 1 st, after leaving j Definition: A recurrent state j is positive recurrent if E[T j,j ]< ; null recurrent if E[T j,j ]=. It is clear that if a class is transient, then w.p. 1 f j,j > 0 DTMC never returns. Summary, for a state j, =, if j is transient E[T j,j ] <, if j is positive recurrent =, if j is null recurrent Remark: A null recurrence state will always (w.p.1.) return to itself, but it will take nearly forever to do so... Yunan Liu (NC State University) ISE/OR 760 28 / 60

An Example: One-Dimensional Simple Random Walks Consider a simple random walk X 1, X 2,... I.I.D. P(X 1 = 1) = p, P(X 1 = 1) = 1 p, 0 < p < 1. S 0 = 0, S n = X 1 + + X n, n 1. The process {S n, n 0} is a simple symmetric (asymmetric) random walk if p = 1/2 (p 1/2). Questions: Is {S n, n 0} a DTMC? If so, what is the state space? Is it irreducible? Classify the states (transient, null-recurrent, positive-recurrent). Yunan Liu (NC State University) ISE/OR 760 29 / 60

Summary of State Classification A state i is transient: if f j,j < 1 k=1 P(k) j,j < N j,j Geo(1 f j,j ) recurrent: if f j,j = 1 k=1 P(k) j,j = N j,j = positive recurrent: if E[Tj,j ] < null recurrent: if E[Tj,j ] = All following features are class properties: 1 Communication 2 Transience 3 Recurrence (null-recurrence, positive-recurrence) 4 Periodicity and aperiodicity Yunan Liu (NC State University) ISE/OR 760 30 / 60

Long-Term Performance Yunan Liu (NC State University) ISE/OR 760 31 / 60

Stationary Distribution Definition (Stationary distribution) A vector π = (π 1, π 2, π 3,...) s.t. π i 0, i S π i = 1 is called the stationary distribution for a DTMC with TPM P, if πp = π or π i P i,j = π j for all j S. i S Implication: If P(X 0 = j) = π j, j S P(X 1 = j) = i S P(X 1 = j X 0 = i)p(x 0 = i) = i S P i,j π i = π j P(X 2 = j) = P(X 2 = j X 1 = i)p(x 1 = i) = P i,j π i = π j i S i S P(X n = j) = π j for all n 1. If a DTMC is initially in stationarity, it is in stationarity for all time n. Yunan Liu (NC State University) ISE/OR 760 32 / 60

Theorem 1: Convergence to Steady State Theorem (steady-state distribution) consider an (i) irreducible and (ii) aperiodic DTMC {X n : n 0}. (i) if all states are transient or null-recurrent, = P(X n = j X 0 = i) n 0. P (n) i,j (ii) if all states are positive recurrent, = P(X n = j X 0 = i) n π j > 0, P (n) i,j π = (π 1, π 2,... ) is the unique solution to Equivalently, (X n X 0 = i) D X where P(X = j) = π j. πp = π π i = 1 i S (balance equation) Proof: omitted here, see p.175 177 of Blue Ross. Yunan Liu (NC State University) ISE/OR 760 33 / 60

Theorem 1: Convergence to Steady State Remarks of Theorem 1: Steady state is defined through a limit as n. In case (i), there is no steady state; In case (ii), convergence occurs (to a limit). Define: ergodicity = aperiodic + irreducibility + positive-recurrent. Given convergence in (ii), the limit must be π. π j = lim n P(n+1) i,j = lim n k P (n) i,k P k,j = k lim n P(n) i,k P k,j = k π k P k,j. Claim: All states of a finite state space irreducible DTMC are positive-recurrent. If a DTMC is good (irreducible and positive-recurrent), then the balance equation has a unique solution (aperiodicity is not necessary) Yunan Liu (NC State University) ISE/OR 760 34 / 60

Theorem 1: Implication Corollary (Existence of good state in a finite DTMC) Consider a DTMC with a finite state space. There must exist at least one positive recurrent state. Proof: Assume no state is positive recurrent - all states are either transient or null recurrent to reach a contradiction. By Theorem 1: lim n P (n) i,j = 0 for all j S = {1, 2,..., N}. Row sum N j=1 P(n) i,j = 1 and take n : 1 = lim n N j=1 P (n) i,j = N N lim n P(n) i,j = 0 = 0. j=1 Contradiction! There must exists at least one positive recurrent state! Remark: For an irreducible and finite-state DTMC, all states are positive recurrent. Yunan Liu (NC State University) ISE/OR 760 35 / 60 j=1

Theorem 2: Long-Run Proportion of Time Theorem (long-run proportion of time) (i) For an irreducible, positive-recurrent DTMC, BE has a unique solution, and the long-run proportion of time DTMC is in a state k: 1 π k = lim N N N 1(X n = k) (ii) Let r( ) be a bounded (cost) function on S, then the long-run average cost: 1 lim n N n=1 N r(x n ) = r(j)π j = r(j)p(x = j) = E[r(X )] j S j S n=1 Remarks: (i) is a special case of (ii) when r(x ) = 1(X = k) If r(x ) = X, then (ii) becomes long-run average of {X n }: 1 lim N N N n=1 X n = j S jπ j = E[X ] Yunan Liu (NC State University) ISE/OR 760 36 / 60

Proof of Theorem 2 Define: N n (i) = N n (i, j) = N n (i, j) = n k=1 n 1(X k = i) = time (days) in state i by n. k=1 N n(i) l=1 Partial proof (i): let πj 1 n N n(j) = 1 n 1(X k = j) = 1 n n k=1 1(X k 1 = i, X k = j) = # of direct trans from i j by n 1(next visit is j after l th visit in i) = N n(i) }{{} A (l) i,j = lim n N n (j)/n. n 1(X k 1 = i, X k = j) k=1 i S l=1 1 A (l) i,j = 1 n 1(X k 1 = i, X k = j) = 1 n n N n(i, j) = N 1 n(i) 1 (l) n A i,j i S k=1 i S i S l=1 = ( ) Nn (i) 1 N n(i) 1 (l) n N n (i) A πi E[1 Ai,j ] = πi P i,j. i,j i S }{{} l=1 i S i S }{{}?? Yunan Liu (NC State University) ISE/OR 760 37 / 60

Proof of Theorem 2 Continued The total cost by time n: n r(x k ) = k=1 k=1 n 1(X k = j)r(x k ) = k=1 j S n = 1(X k = j)r(j) = j S k=1 j S n 1(X k = j)r(j) k=1 j S r(j) n 1(X k = j) Dividing n on both sides and taking n : 1 n lim r(x k ) = ( ) 1 n r(j) lim 1(X k = j) = r(j)π j. n n n n j S j S k=1 k=1 Yunan Liu (NC State University) ISE/OR 760 38 / 60

Theorem 3: Mean First Passage Time Theorem (Mean First Passing Time) Consider an irreducible and positive-recurrent DTMC: (i) If aperiodic: P (n) i,j π j = 1 E[T j,j ] (ii) If periodic with period d > 1: (n) The sequence P j,j doesn t converge; The subsequence P n d j,j dπ j = d E[T j,j ] Intuition: A cycle: time from j to j In each cycle: spend exactly 1 time unit (day) in j long-run proportion of time in j: 1/E[cycle length] = 1/E[T j,j ]. Proof: will be given in Review Theory (last topic) Yunan Liu (NC State University) ISE/OR 760 39 / 60

Example: DTMC in Hospitals When a patient is hospitalized at the NC State hospital, his/her health condition can be described by a DTMC with 3 states: low risk (L), medium risk (M) and high risk (H). Patients are admitted into the hospital in low-, medium- and high-risk conditions with probability p 1 = 0.6, p 2 = 0.3 and p 3 = 0.1, respectively. Their health status are updated once every day according to the TPM We are interested in: P = H M L 0.50 0.40 0.1 0.35 0.30 0.35 0.05 0.15 0.8 1 Long-run proportion of days a patient is in low (medium, high) risk. 2 Daily medication costs are independent exponential r.v. s with rates µ 1 = 1/2, µ 2 = 1/4 and µ 3 = 1/10 for a low-, medium- and high-risk patient. What is long-run average daily cost?. Yunan Liu (NC State University) ISE/OR 760 40 / 60

Example: Google PageRank Develop a DTMC: 1 Each state represents a web page; 2 Create direct transition (an arc) from state i to state j if page i has a link to page j; 3 If page i has k > 0 outgoing links, then set the probability on each outgoing arc to be 1 k ; 4 Solve the DTMC to determine the stationary distribution. Pages are then ranked based on their π i. Yunan Liu (NC State University) ISE/OR 760 41 / 60

Reducible DTMCs Yunan Liu (NC State University) ISE/OR 760 42 / 60

Canonical Form of Reducible DTMC Consider a DTMC with both transient states and absorbing states. Define A = set of absorbing states = {i S, s.t. P i,i = 1} T = set of transient states Suppose A = n, T = m Relabel the states and put the TPM P in its canonical form: P = A T [ A T In n 0 n m R m n Q m m ] Q m m : transient-to-transient TPM R m n : transient-to-absorbing TPM 0 n m : 0 matrix I n n : identity matrix Yunan Liu (NC State University) ISE/OR 760 43 / 60

An Example: Markov Mouse in an Open Maze Question: Starting in room 1, what is the mean number of steps until escaping the maze? what is the probability of exiting from door 3? In general, for a reducible DTMC E[number of steps until absorption X 0 = i], for i T P(being absorbed by state j X 0 = i), for j A, i T Yunan Liu (NC State University) ISE/OR 760 44 / 60

Mean Number of Steps until Absorption For transient states i, j T N i,j = total # of steps in j starting in i = n=0 1(X n = j X 0 = i). Define the fundamental matrix S R m m : [ ] S i,j E[N i,j ] = E 1(X n = j X 0 = i) = E [1(X n = j X 0 = i)] In matrix notation n=0 n=0 n=0 = P(X n = j X 0 = i) = δ i,j + = δ i,j + n=1 n=1 P (n) i,j Q (n) i,j, where δ i,j = 1 {i=j} S = I + Q + Q 2 + Q 3 + (geometric sequence) Yunan Liu (NC State University) ISE/OR 760 45 / 60

Mean Number of Steps until Absorption Solving the fundamental matrix What s missing? Fundamental matrix: S = I + Q + Q 2 + Q 3 + QS = SQ = Q + Q 2 + Q 3 + Q 4 + S(I Q)= (I Q)S = I. S = (I Q) 1 Define M i = mean number of steps until absorption, starting in i T. M i = S i,j, or M = S e, where e = [1,..., 1] T. }{{} j T m Yunan Liu (NC State University) ISE/OR 760 46 / 60

Probability of Absorption For i T, l A. Define B i,l = P(eventually being absorbed by state l X 0 = i) = P(X 1 = l X 0 = i) + P(X 2 = l, X 1 T X 0 = i) +P(X 3 = l, X 2 T X 0 = i) + = R i,l + P(X 2 = l X 1 = j, X 0 = i)p(x 1 = j X 0 = i) j T + j T = R i,l + j T In matrix notation: P(X 3 = l X 2 = j, X 0 = i)p(x 2 = j X 0 = i) + R j,l Q i,j + j T R j,l Q (2) i,j + = (R) i,l + (QR) i,l + (Q 2 R) i,l + B = R + QR + Q 2 R + = (I + Q + Q 2 + ) R = SR = (I Q) 1 R. Yunan Liu (NC State University) ISE/OR 760 47 / 60

Ever Hitting Probabilities Revisited Using the fundamental matrix S, we can compute ever-hitting probability f i,j = P(j ever visited after leaving i) < 1. For i, j T, i j S i,j = E[# of visits in j X 0 = i] = E[# of visits in j j is ever visited, X 0 = i] f i,j = E[# of visits in j X 0 = j] f i,j = S j,j f i,j For i = j T, S j,j = E[# of visits in j X 0 = j] = E[# in j j ever again,x 0 = j]f j,j + E[# in j j never again,x 0 = j](1 f j,j ) = [1 + S j,j ] f j,j + 1 (1 f j,j ) = 1 + S j,j f j,j f i,j = { Si,j S j,j, if i j; S j,j 1 S j,j, if i = j. Yunan Liu (NC State University) ISE/OR 760 48 / 60

Yunan Liu (NC State University) ISE/OR 760 49 / 60

Time Reversibility Yunan Liu (NC State University) ISE/OR 760 50 / 60

Reverse Time DTMC Definition (Reverse-Time DTMC) Consider a DTMC {X n } in stationary with TPM P and steady state π. Define its reverse-time process Properties: { X n} = {X n, X n 1, X n 2,... } (i) Inverse-time process { X n} is a DTMC, that is, P(X m = j X m+1 = i, X m+2 = i m+2, X m+3 = i m+3,...) = P(X m = j X m+1 = i) (ii) The transition probability of { X n} is P i,j P(X m = j X m+1 = i) = π jp j,i π i. Yunan Liu (NC State University) ISE/OR 760 51 / 60

Reverse Time DTMC Proof of property (i): Given present, future past (and past future). More rigorously: LHS = P(X m = j, X m+1 = i, X m+2 = i m+2, X m+3 = i m+3,... ) P(X m+1 = i, X m+2 = i m+2, X m+3 = i m+3,... ) = P(X m+2 = i m+2, X m+3 = i m+3,... X m+1 = i, X m = j)p(x m+1 = i, X m = j) P(X m+2 = i m+2, X m+3 = i m+3,... X m+1 = i)p(x m+1 = i) = P(X m+1 = i, X m = j) = P(X m = j X m+1 = i) P(X m+1 = i) Proof of property (ii): P i,j= P(X m = j X m+1 = i) = P(X m = j, X m+1 = i) P(X m+1 = i) P(X m+1 = i X m = j)p(x m = j) P(X m+1 = i) = π jp j,i π i. Yunan Liu (NC State University) ISE/OR 760 52 / 60

Time Reversibility Definition (Time Reversibility) We say DTMC {X n } is time reversible (T-R), if { X n } and {X n } have the same probability law ( P = P, P i,j = P i,j= P ) j,iπ j, π i or equivalently, π i P i,j = π j P j,i for all i, j S. Interpretation: A DTMC is T-R if, for all i, j S, long-run rate (prop. of transitions) from i j = long-run rate from j i Yunan Liu (NC State University) ISE/OR 760 53 / 60

Example 1 In general, a DTMC may not be T-R, that is, {X n } and { X n } may have different dynamics ( P i,j P i,j ). Example: DTMC with S = {1, 2, 3} P = 0 1 0 0 0 1 1 0 0, π = ( 1 3, 1 3, 1 3 ) Checking the T-R criterion: 1 3 1 = π 1P 1,2 π 2 P 2,1 = 1 3 0 Hence, the DTMC is NOT time reversible. Yunan Liu (NC State University) ISE/OR 760 54 / 60

Example 2: Random Walk on a Bounded Set Consider a random walk in a bounded set S = {0, 1,..., N} with transition probabilities P i,i+1 = p i = 1 P i,i 1 for 1 i N 1, P N,N 1 = P 0,1 = 1. Let S n denote the position at time n, then {S n, n 0} is a DTMC. Question: Is the DTMC time-reversible? Yunan Liu (NC State University) ISE/OR 760 55 / 60

Time Reversibility Proposition (A Sufficient Condition) If there exists a vector π = (π 1, π 2,... ), s.t., π k 0, π k = 1, satisfying πi P i,j = πj P j,i, for all i, j S (1) then 1 π = π (that is, π is the stationary distribution) 2 The DTMC is T-R. Proof: summing both sides of (1) over i yields πi P i,j = πj P j,i = πj i i which is the balance equation! Hence, π = π. In addition, we have P j,i = πj, i so that DTMC is T-R. π i P i,j = π j P j,i, Yunan Liu (NC State University) ISE/OR 760 56 / 60

Random Walks on Weighted Graphs Definition (RWoWG) A graph has n nodes and m arcs. Each arc (undirected) (i, j) has a weight w i,j = w j,i. A particle moves from node i j with a probability that is proportional to the weight w i,j, that is, P i,j = w i,j k w, for all i, j. i,k Let X n be the location of the particle at time n. If the RWoWG is connected, then {X n, n 0} is a DTMC which is irreducible and positive-recurrence. Yunan Liu (NC State University) ISE/OR 760 57 / 60

Random Walks on Weighted Graphs Theorem (Steady state of RWoWG) The DTMC of a connected RWoWG 1 is time-reversible, 2 has steady state Intuition: π i = k S π i = k S k S w i,k l S w k,l, i S. (2) (sum of weights on all incoming arcs to i) (sum of weights on all outgoing arcs from k) Proof: We verify that π i in (2) indeed solves Equation (1): π i P i,j = l S k S w i,k k S w l,k w i,j k S w i,k = l S where the 2nd equality holds because w i,j = w j,i. k S w j,k k S w l,k w j,i k S w j,k = π j P j,i, Yunan Liu (NC State University) ISE/OR 760 58 / 60

Examples of RWoWG: Markov Mouse in a Closed Maze Revisited Questions: Is the DTMC time reversible? Are there easy ways to compute π? Yunan Liu (NC State University) ISE/OR 760 59 / 60

Examples of RWoWG: Random Walk on a Bounded Set Revisited Consider a random walk in a bounded set S = {0, 1,..., N} with transition probabilities P i,i+1 = p i = 1 P i,i 1 for 1 i N 1, P N,N 1 = P 0,1 = 1. Let S n denote the position at time n, then {S n, n 0} is a DTMC. Questions: Can we transform this problem to a RWoWG? If yes, give an expression of π i, i = 0, 1,..., N. Yunan Liu (NC State University) ISE/OR 760 60 / 60