Discrete time Markov chains Discrete Time Markov Chains, Limiting Distribution and Classification DTU Informatics 02407 Stochastic Processes 3, September 9 207 Today: Discrete time Markov chains - invariant probability distribution Classification of states Classification of chains Next week Poisson process Two weeks from now Birth- and Death Processes Regular Transition Probability Matrices Interpretation of π j s P = P, Regular: If P k > 0 for some k In that case lim n P (n) = π j 0 i, j N Theorem 4. (Page 68) let P be a regular transition probability matrix on the states 0,,..., N. Then the limiting distribution π = (π 0, π, π N ) is the unique nonnegative solution of the equations Limiting probabilities lim n P (n) = π j Long term averages lim n Stationary distribution π = πp m n= P(n) = π j N π j = π k P, k=0 π = πp N π k =, π = k=0
A Social Mobility Example Son s Class Lower Middle Upper Lower 0.40 0.50 0.0 Father s Middle 0.05 0.70 0.25 Class Upper 0.05 0.50 0.45 P 8 = 0.0772 0.6250 0.2978 0.0769 0.6250 0.298 0.0769 0.6250 0.298 π 0 = 0.40π 0 + 0.05π + 0.05π 2 π = 0.50π 0 + 0.70π + 0.50π 2 Classification of Markov chain states States which cannot be left, once entered - absorbing states States where the return some time in the future is certain - recurrent or persistent states The mean time to return can be finite - postive recurrence/non-null recurrent infinite - null recurrent States where the return some time in the future is uncertain - transient states States which can only be visited at certain time epochs - periodic states π 2 = 0.0π 0 + 0.25π + 0.45π 2 = π 0 + π + π 2 Classification of States j is accessible from i if P (n) > 0 for some n If j is accessible from i and i is accessible from j we say that the two states communicate Communicating states constitute equivalence classes (an equivalence relation) i communicates with j and j communicates with k then i and k communicates First passage and first return times We can formalise the discussion of state classification by use of a certain class of probability distributions - first passage time distributions. Define the first passage probability f (n) = P{X j, X 2 j,..., X n j, X n = j X 0 = i} This is the probability of reaching j for the first time at time n having started in i. What is the probability of ever reaching j? f = n= f (n) The probabilities f (n) constitiute a probability distribution. On the contrary we cannot say anything in general on n= p(n) (the n-step transition probabilities)
State classification by f (n) Classification of Markov chains A state is recurrent (persistent) if f (= ) n= f (n) = A state is positive or non-null recurrent if E(Ti ) <. E(T i ) = (n) n= nf = µ i A state is null recurrent if E(Ti ) = µ i = A state is transient if f <. In this case we define µ i = for later convenience. A peridoic state has nonzero p (nk) for some k. A state is ergodic if it is positive recurrent and aperiodic. We can identify subclasses of states with the same properties All states which can mutually reach each other will be of the same type Once again the formal analysis is a little bit heavy, but try to stick to the fundamentals, definitions (concepts) and results Properties of sets of intercommunicating states (a) i and j has the same period (b) i is transient if and only if j is transient (c) i is null persistent (null recurrent) if and only if j is null persistent A set C of states is called (a) Closed if p = 0 for all i C, j / C (b) Irreducible if i j for all i, j C. Theorem Decomposition Theorem The state space S can be partitioned uniquely as S = T C C 2... where T is the set of transient states, and the C i are irreducible closed sets of persistent states Lemma If S is finite, then at least one state is persistent(recurrent) and all persistent states are non-null (positive recurrent)
8 7 6 5 4 3 2 0 0 20 30 40 50 60 70 8 7 6 5 4 3 2 0 0 20 30 40 50 60 70 8 7 6 5 4 3 2 0 0 20 30 40 50 60 70 8 7 6 5 4 3 2 0 0 20 30 40 50 60 70 Basic Limit Theorem Theorem 4.3 The basic limit theorem of Markov chains (a) Consider a recurrent irreducible aperiodic Markov chain. Let P (n) be the probability of entering state i at the nth transition, n =, 2,..., given that X 0 = i. By our earlier convention P (0) =. Let f (n) be the probability of first returning to state i at the nth transition n =, 2,..., where f (0) = 0. Then lim n P(n) = (n) n=0 nf (b) under the same conditions as in (a), lim n P (n) ji = lim n P (n) for all j. = m i An example chain (random walk with reflecting barriers) P = 0.6 0.4 0.0 0.0 0.0 0.0 0.0 0.0 0.4 0.0 0.0 0.0 0.0 0.0 0.0 0.4 0.0 0.0 0.0 0.0 0.0 0.0 0.4 0.0 0.0 0.0 0.0 0.0 0.0 0.4 0.0 0.0 0.0 0.0 0.0 0.0 0.4 0.0 0.0 0.0 0.0 0.0 0.0 0.4 0.0 0.0 0.0 0.0 0.0 0.0 0.7 With initial probability distribution p (0) = (, 0, 0, 0, 0, 0, 0, 0) or X 0 =. Properties of that chain A number of different sample paths X n s We have a finite number of states From state we can reach state j with a probability f j 0.4 j, j >. From state j we can reach state with a probability f j j, j >. Thus all states communicate and the chain is irreducible. Generally we won t bother with bounds for the f s. Since the chain is finite all states are positive recurrent A look on the behaviour of the chain
The state probabilities Limiting distribution For an irreducible aperiodic chain, we have that p (n) µ j as n, for all i and j 0.9 0.8 0.7 0.6 0.5 0.4 0.2 0. 0 0 0 20 30 40 50 60 70 Three important remarks If the chain is transient or null-persistent (null-recurrent) p (n) 0 If the chain is positive recurrent p (n) µ j The limiting probability of X n = j does not depend on the starting state X 0 = i p (n) j The stationary distribution Stationary distribution A distribution that does not change with n The elements of p (n) are all constant The implication of this is p (n) = p (n ) P = p (n ) by our assumption of p (n) being constant Expressed differently π = πp Definition The vector π is called a stationary distribution of the chain if π has entries (π j : j S) such that (a) π j 0 for all j, and j π j = (b) π = πp, which is to say that π j = i π ip for all j. VERY IMPORTANT An irreducible chain has a stationary distribution π if and only if all the states are non-null persistent (positive recurrent);in this case, π is the unique stationary distribution and is given by π i = µ i for each i S, where µ i is the mean recurrence time of i.
The example chain (random walk with reflecting barriers) P = 0.6 0.4 0.0 0.0 0.0 0.0 0.0 0.0 0.4 0.0 0.0 0.0 0.0 0.0 0.0 0.4 0.0 0.0 0.0 0.0 0.0 0.0 0.4 0.0 0.0 0.0 0.0 0.0 0.0 0.4 0.0 0.0 0.0 0.0 0.0 0.0 0.4 0.0 0.0 0.0 0.0 0.0 0.0 0.4 0.0 0.0 0.0 0.0 0.0 0.0 0.7 Elementwise the matrix equation is π i = j π jp ji π = π 0.6 + π 2 π 2 = π 0.4 + π 2 + π 3 π 3 = π 2 0.4 + π 3 + π 4 π = πp Or π = π 0.6 + π 2 π j = π j 0.4 + π j + π j+ π 8 = π 7 0.4 + π 8 0.7 π 2 = 0.6 π π j+ = (( )π j 0.4π j ) Can be solved recursively to find: π j = ( ) 0.4 j π The normalising condition We note that we don t have to use the last equation We need a solution which is a probability distribution 8 π j =, j= Such that 8 j= N a i = i=0 ( 0.4 ( ) 0.4 8 = π 0.4 ) j π = π 7 k=0 a N+ a N <, a N + N <, a = a N =, a < π = 0.4 ( 0.4 ( ) 0.4 k ) 8 The solution of π = πp More or less straightforward, but one problem if x is a solution such that x = xp then obviously (kx) = (kx)p is also a solution. Recall the definition of eigenvalues/eigen vectors If Ay = λy we say that λ is an eigenvalue of A with an associated eigenvector y. Here y is a right eigenvector, there is also a left eigenvector
The solution of π = πp continued The vector π is a left eigenvector of P. The main theorem says that there is a unique eigenvector associated with the eigenvalue of P In practice this means that the we can only solve but a normalising condition But we have the normalising condition by j π j = this can expressed as π =. Where =. Roles of the solution to π = πp For an irreducible Markov chain, (the condition we need to verify) The stationary solution. If p (0) = π then p (n) = π for all n. The limiting distribution, i.e. p (n) π for n (the Markov chain has to be aperiodic too). Also p (n) π j. The mean recurrence time for state i is µ i = π i. The mean number of visits in state j between two successive visits to state i is π j π i. The long run average probability of finding the Markov chain in state i is π i. π i = lim n n n k= p(k) i also true for periodic chains. Example (null-recurrent) chain P = p p 2 p 3 p 4 p 5... 0 0 0 0... 0 0 0 0... 0 0 0 0... 0 0 0 0... 0 0 0 0..................... For p j > 0 the chain is obviously irreducible. The main theorem tells us that we can investigate directly for π = πp. π = π p + π 2 π 2 = π p 2 + π 3 π j = π p j + π j+ π = π p + π 2 π 2 = π p 2 + π 3 π j = π p j + π j+ we get π 2 = ( p )π π 3 = ( p p 2 )π π j = ( p p j )π j π j = ( p p j )π π j = π p i π j = π p i i= i=j Normalisation i π j = π p i = π p i = π ip i j= j= i=j i= j= i=
Reversible Markov chains p p22 p2 2 p2 [ p p P = 2 p 2 p 22 ] Solve sequence of linear equations instead of the whole system Local balance in probability flow as opposed to global balance Nice theoretical construction Local balance equations π i = j π j p ji Term for term we get π i = j π j p ji π i p = j j π i p = π j p ji π j p ji π i p = j j π j p ji If they are fulfilled for each i and j, the global balance equations can be obtained by summation.
Why reversible? Another look at a similar question P{X n = i X n = j} = P{X n = i}p{x n = j X n = i} and for a stationary chain = P{X n = i}p π i p For a reversible chain (local balance) this is π i p = π j p ji = P{X n = j}p{x n = i X n = j} = P{X n = j X n = i} the reversed sequence. P{X n = j X n = i} = P{X n = j X n = i} P{X n = i} = P{X n = j}p{x n = i X n = j} P{X n = i} For a stationary chain we get π j p ji π i = P{X n = j}p ji P{X n = i} The chain is reversible if P{X n = j X n = i} = p leading to the local balance equations p = π jp ji π i Exercise 0 (6/2/9 ex.) In connection with an examination of the reliability of some software intended for use in control of modern ferries one is interested in examining a stochastic model of the use of a control program. The control program works as " state machine " i.e. it can be in a number of different levels, 4 are considered here. The levels depend on the physical state of the ferry. With every shift of time unit while the program is run, the program will change from level j to level k with probability p jk. Two possibilities are considered: The program has no errors and will run continously shifting between the four levels. The program has a critical error. In this case it is possible that the error is found, this happens with probality q i, i =, 2, 3, 4 depending on the level. The error will be corrected immediately and the program will from then on be without faults. Alternatively the program can stop with a critical error (the ferry will continue to sail, but without control). This happens with probability r i, i =, 2, 3, 4. In general q i + r i <, a program with errors can thus work and the error is not nescesarily discovered. It is assumed that detection of an error, as well as the apperance of a fault happens coincidently with shift between levels. The program starts running in level, and it is known that the program contains one critical error.
Solution: Question Question - continued Formulate a stochastic process ( Markov chain) in discrete time describing this system. The model is a discrete time Markov chain. A possible definition of states could be 0: The programme has stopped. -4: The programme is operating safely in level i. 5-8: The programme is operating in level i-4, the critical error is not detected. The transition matrix A is A = 0 0 0 P 0 r Diag(q i )P Diag(S i )P r = The model is a discrete time Markov chain. Where P = {p } r r 2 r 3 r 4 Diag(S i ) = Diag(q i ) = S 0 0 0 0 S 2 0 0 0 0 S 3 0 0 0 0 S 4 q 0 0 0 0 q 2 0 0 0 0 q 3 0 0 0 0 q 4 S i = r i q i Question - continued Solution question 2 A = Or without matrix notation: 0 0 0 0 0 0 0 0 0 p p 2 p 3 p 4 0 0 0 0 0 p 2 p 22 p 23 p 24 0 0 0 0 0 p 3 p 32 p 33 p 34 0 0 0 0 0 p 4 p 42 p 43 p 44 0 0 0 0 r q p q p 2 q p 3 q p 4 S p S p 2 S p 3 S p 4 r 2 q 2 p 2 q 2 p 22 q 2 p 23 q 2 p 24 S 2 p 2 S 2 p 22 S 2 p 23 S 2 p 24 r 3 q 3 p 3 q 3 p 32 q 3 p 33 q 3 p 34 S 3 p 3 S 3 p 32 S 3 p 33 S 3 p 34 r 4 q 4 p 4 q 4 p 42 q 4 p 43 q 4 p 44 S 4 p 4 S 4 p 42 S 4 p 43 S 4 p 44 Characterise the states in the Markov chain. With reasonable assumptions on P (i.e. irreducible) we get State 0 Absorbing Positive recurrent 2 Positive recurrent 3 Positive recurrent 4 Positive recurrent 5 Transient 6 Transient 7 Transient 8 Transient
Solution question 3 We now consider the case where the stability of the system has been assured, i.e. the error has been found and corrected, and the program has been running for long time without errors. The parameters are as follows. P i,i+ = 0.6 i =, 2, 3 P i,i = 0.2 i = 2, 3, 4 P i,j = 0 i j > q i = 0 3i r i = 0 3i 5 Characterise the stochastic proces, that describes the stable system. The system becomes stable by reaching one of the states -4. The process is ergodic from then on. The procces is a reversible ergodic Markov chain in discrete time. Solution question 4 For what fraction of time will the system be in level. We obtain the following steady state equations π i = 3 i π 4 3 i π = 40π = i= π = 40 The sum 4 i= 3i can be obtained by using 4 i= 3i = 34 3 = 40. 4 i= 3 i π = 34 3 π =