MATH 56A: STOCHASTIC PROCESSES CHAPTER 3

MATH 56A: STOCHASTIC PROCESSES CHAPTER 3 Plan for rest of semester (1) st week (8/31, 9/6, 9/7) Chap 0: Diff eq s an linear recursion (2) n week (9/11...) Chap 1: Finite Markov chains (3) r week (9/18...) Chap 1: Finite Markov chains (4) th week (9/25...) Chap 2: Countable Markov chains (5) th week (oct 3,4,5) Chap 3: Continuous time Markov chains (6) th week (oct 9,11,12) Ch 4: Stopping time (7) th week (oct 16,18,19) Ch 5: Martingales (8) th week (oct 23,25,26) Ch 6: Renewal processes (9) th week (oct 30,1,2) Ch 7: Reversible Markov chains (10) th week (nov 6,8,9 ) Ch 8: Weiner process (11) th week (nov 13,15,16) Ch 8: more (12) th week (nov 20,22) (short week) Ch 9: Stochastic integrals (13) th week (nov 27,29,30,4) (extra ay) Ch 9: more 3. Continuous Markov Chains The iea of continuous Markov chains is to make time continuous instea of iscrete. This iea only works when the system is not jumping back an forth at each step but rather moves graually in a certain irection. 3.1. making time continuous. On the first ay I iscusse the problem of converting to continuous time. In the iscrete Markov chain we have the transition matrix P with entries p(i, j) giving the probability of going from i to j in one unit of time. The n-th power, say P 5 has entries p 5 (i, j) = P(X 5 = j X 0 = i) We want to interpolate an figure out what happene for all positive time t. (Negative time is iscusse in Chapter 7.) We alreay know how to o that. You write: P = QDQ 1 Date: October 9, 2006. 1

2 MATH 56A: STOCHASTIC PROCESSES CHAPTER 3 where D is a iagonal matrix whose iagonal entries are the eigenvalues of P an Q is the matrix of right eigenvectors of P. The first eigenvector of P is 1 an the first right eigenvector is the column vector having all 1 s. If the eigenvalues are all positive then we can raise them to arbitrary values: P t = QD t Q 1 Usually you take logarithms. For example, if there are 3 states: D = 1 0 0 0 2 0 = e0 0 0 0 e ln 2 0 0 0 3 0 0 e ln 3 Then P t = e ta where A = Q(ln D)Q 1 = Q 0 0 0 0 ln 2 0 Q 1 0 0 ln 3 This uses: Theorem 3.1. P = QDQ 1 = Qe ln D Q 1 Q ln DQ 1 = e Proof. Let L = ln D then Conjugate by Q: D = e L := I + L + L2 2 + L3 3! + Qe L Q 1 = QQ 1 + QLQ 1 + QL2 Q 1 + QL3 Q 1 + 2 3! This is equal to e QLQ 1 since QL n Q 1 = (QLQ 1 ) n. The other theorem I pointe out was: Theorem 3.2. t P t = P t A = AP t Proof. This is just term by term ifferentiation. t P t = Qt n L n Q 1 t n! = Qnt n 1 L n Q 1 n(n 1)! = QLQ 1 Qt n L n Q 1 = AP t n!

MATH 56A: STOCHASTIC PROCESSES CHAPTER 3 3 3.2. Poisson processes. On the secon ay I explaine continuous Markov chains as generalizations of Poisson processes. A Poisson process is an event which occurs from time to time is time homogeneous (i.e., the probability that it will occur tomorrow is the same as the probability that it will occur toay) an the occurrences are inepenent The inepenence of occurrences of a Poisson event means that the probability of future occurrence is inepenent of both past an present. Markov processes are inepenent of the past. They epen only on the present. We will transform a Poisson processes so that it looks more like a Markov process. Here is an example where a Poisson event occurs three times in a time interval t = t 1 t 0. (We put t 0 = 0 in class so that t = t 1.) x + 3 x + 2 T x + 1 x a Poisson event (birth) occurs three times in a time interval each Poisson event (birth) gives a jump in the Markov state (total population) Figure 1. Poisson to Markov The Poisson process has one parameter λ calle the rate. This is measure in inverse time units (number of occurrences per unit time). Thus λ t is the expecte number of occurrences in any time interval of length t. 3.2.1. variables associate to a Poisson process. There are two ranom variables associate with a Poisson process:

4 MATH 56A: STOCHASTIC PROCESSES CHAPTER 3 Poisson variable (nonnegative integer) X = number of occurrences in t P(X = k) = e λt 1 λk t k 1 k! Exponential variable (positive real) T = time until 1st occurrence P(event oes not occur) = P(event oes not occur) = P(X = 0) = e λt 1 = 1 λt 1 + λ2 t 2 1 2 P(T > t 1) = e λt 1 P(event occurs) = P(event occurs in time t) = P(X 1) = 1 e λ t = λ t + o( t) P(T t) = 1 e λ t λ t Here the book uses the little oh notation o( t) to enote anything which vanishes faster than t: o( t) 0 as t 0 t 3.2.2. Poisson to Markov. There are two changes we have to make to transform a Poisson process into a continuous time Markov process. a) Every time an event occurs, you nee to move to a new state in the Markov process. Figure 1 shows an example where the state is the total population: X t := population at time t = X 0 + #births #eaths b) The rate α(x) epens on the state x. For example, the rate at which population grows is proportional to the size of the population: α(x) = λx, λ : constant Notice that, when the rate increases, the events will occur more frequently an the waiting time will ecrease. So, there is the possibility of explosion, i.e., an infinite number of jumps can occur in a finite amount of time. 3.3. Definition of continuous Markov chain. This is from Lawler, p. 69. We nee to start with an infinitesmal generator A which is a matrix with entries α(x, y) for all states x, y S so that α(x, y) 0 for x y an α(x, x) 0 an so that the sum of the rows is zero: α(x, y) = 0 y S

MATH 56A: STOCHASTIC PROCESSES CHAPTER 3 5 We use the notation α(x) = α(x, x) = y x α(x, y) Definition 3.3. A continuous time Markov chain with infinitesmal generator A = (α(x, y)) is a function X : [0, ) S so that (1) X is right continuous. I.e., X t is equal to the limit of X t+ t as t goes to zero from the right (the positive sie). (2) P(X t+ t = x X t = x) = 1 α(x) t + o( t) (3) P(X t+ t = y X t = x) = α(x, y) t + o( t) for y x. (4) X t is time homogeneous (5) X t is Markovian (X t epens on X t but is inepenent of the state before time t.) I pointe out that the numbers α(x), α(x, y) were necessarily 0 an that α(x) = y x α(x, y) since X t+ t must be in some state. The (x, x) entry of the matrix A is α(x, x) = α(x). So, the rows of the matrix A a up to zero an all negative numbers lie on the iagonal. 3.4. probability istribution vector. At any time t we have a probability istribution vector telling what is the probability of being in each state. p x (t) := P(X t = x) This shoul not be confuse with the time epenent probability transition matrix: p t (x, y) := P(X t = y X 0 = x) Theorem 3.4. The time erivative of the probability istribution function p x (t) is given by t p x(t) = p y (t)α(y, x) y S In matrix notation this is p(t) = p(t)a t The unique solution of this ifferential equation is: p(t) = p(0)e ta This implies that P t := e ta is the time t probability transition matrix.

6 MATH 56A: STOCHASTIC PROCESSES CHAPTER 3 Proof. The ifference p x (t + t) p x (t) is equal to the probability that the state moves into x minus the probability that it will move out of x in the time perio from t to t + t. So, p x (t+ t) p x (t) = P(X t+ t = x, X t = y x) P(X t+ t = y x, X t = x) = y x P(X t = y)p(x t+ t = x X t = y) y x P(X t = x)p(x t+ t = y X t = x) y x p y (t)α(y, x) t y x p x (t)α(x, y) t = y x p y (t)α(y, x) t p x (t)α(x) t = y p y (t)α(y, x) t So, p x (t + t) p x (t) p y (t)α(y, x) t y Take the limit as t 0 to get the theorem. 3.5. example. What is the probability that X 4 = 1 given that X 0 = 0 if the infinitesmal generator is ( ) 1 1 A =? 2 2 The answer is given by the (0, 1) entry of the matrix e 4A. The given information is that p(0) = (1, 0) an the question is: What is p 1 (4)? The solution in matrix terms is the secon coorinate of p(4) = (p 0 (4), p 1 (4)) = (1, 0)e 4A We worke out the example: Since the trace of A is 1+ 2 = 3 = 1 + 2 an 1 = 0 we must have 2 = 3. So, A = QDQ 1 where ( ) 0 0 D = 0 3 an Q is the matrix of right eigenvectors of A: ( ) ( ) 1 1 2/3 1/3 Q =, Q 1 2 1 = 1/3 1/3 The time 4 transition matrix is ( ) ( ) ( ) 1 1 1 0 2/3 1/3 e 4A = Qe 4D Q 1 = 1 2 0 e 12 1/3 1/3

MATH 56A: STOCHASTIC PROCESSES CHAPTER 3 7 So, the answer is 2 + e 12 = P 4 = 3 2 2e 12 3 p 1 (4) = 1 e 12 3 1 e 12 3 1 + 2e 12 3.6. equilibrium istribution, positive recurrence. An equilibrium istribution oes not change with time. In other wors, the time erivative is zero: π(t) = π(t)a = 0 t So, π(t) = π(0) is the left eigenvector of A normalize by: π(x) = 1 x S Since π oes not change with time, we forget the t an write π(x) for π x (t). Recall that irreucible Markov chains are positive recurrent if an only if there is an equilibrium istribution. The example above is irreucible an finite, therefore positive recurrent. The equilibrium istribution is π = (2/3, 1/3). 3.7. birth-eath chain. Definition 3.5. A birth-eath chain is a continuous Markov chain with state space S = {0, 1, 2, 3, } (representing population size) an transition rates: α(n, n + 1) = λ n, α(n, n 1) = µ n, α(n) = λ n + µ n representing births an eaths which occur one at a time. Notice that the total flow between the set of states {0, 1, 2,, n} to the states {n + 1, n + 2, } is given by the birth rate λ n an the eath rate µ n+1. So, π(n) is an equilibrium if an only if Solving for π(n + 1) gives: π(n + 1) = π(n)λ n = π(n + 1)µ n+1 λ n µ n+1 π(n) = λ nλ n 1 µ n+1 µ n π(n 1) = = λ nλ n 1 λ 0 µ n+1 µ n µ 1 π(0) If we can normalize these numbers we get an equilibrium istribution. So, 3

8 MATH 56A: STOCHASTIC PROCESSES CHAPTER 3 Theorem 3.6. The birth-eath chain is positive recurrent if an only if λ n λ n 1 λ 0 < µ n+1 µ n µ 1 3.8. birth an explosion. If there is no eath, the birth-eath chain is obviously transient. The population is going to infinity but how fast? Suppose that T n is the time that the population stays in state n. Then (when µ n = 0) T n is exponential with rate λ n. So, Theorem 3.7. a) If < then explosion occurs with probability one. b) If 1 λ n 1 λ n E(T n ) = 1 λ n = then the probability of explosion is zero. For example, in the Yule process with λ n = nλ, explosion will not occur since 1 = 1 λ n λn = 1 1 λ n = 3.9. transient birth-eath chains. Recall that an irreucible Markov chain is transient if an only if there is a right eigenvector of P with entries converging to zero corresponing to eigenvector 1. In the continuous time case, this is the same as a right eigenvector of A corresponing to eigenvalue 0. So, we want numbers a(n) such that a(n 1)µ n + a(n)( λ n µ n ) + a(n + 1)λ n = 0 This equation can be rewritten as [a(n + 1) a(n)]λ n = [a(n) a(n 1)]µ n [a(n + 1) a(n)] = µ n [a(n) a(n 1)] = µ nµ n 1 µ 1 [a(1) a(0)] λ n λ n λ n 1 λ 1 a(k + 1) is the sum of these numbers: k k µ n µ n 1 µ 1 a(k+1) = a(0)+ [a(n+1) a(n)] = a(0)+ [a(1) a(0)] λ n λ n 1 λ 1 Theorem 3.8. A birth-eath chain is transient if an only if k µ n µ n 1 µ 1 < λ n λ n 1 λ 1 Proof. Let L be this limit. Let a(0) = 1 an a(1) = 1 1/L. Then a(k + 1) given by the above formula will converge to zero. Conversely, if the a(k + 1) goes to zero, the infinite sum must converge.