Stochastic Models: Markov Chains and their Generalizations

Transcription

1 Scuola di Dottorato in Scienza ed Alta Tecnologia Dottorato in Informatica Universita di Torino Stochastic Models: Markov Chains and their Generalizations Gianfranco Balbo e Andras Horvath Outline Introduction to Stochastic Processes Markov Chains Hidden Markov Models References E. Cinlar, Introduction to Stochastic Processes, Prentice Hall, 1975; V.G. Kulkarni, Modeling and Analysis of Stochastic Systems, Chapman & Hall, 1995 Aprile Maggio, 2009 Introduction to Stochastic Processes Discrete Time Markov Chains Definitions Transient Solutions State Classification Steady State Solutions (Limiting Behaviour) Basic Defintions and Notation (1) Characterization A multivariate discrete random variable (X 0, X 1,, X n ) is completely described by its joint pmf. Using conditional probabilities, we can write First Passage Times Lumpability Basic Defintions and Notation (2) Markov Property Discrete time, Discrete state Stochastic Process (Discrete Time Stochastic Chain) {X n, n2 N } X n is called the present state of the process; X 0, X 1,, X n-1 is called the past of the process (the history) X n+1, X n+2, is called the future of the process Basic Defintions and Notation (3) Markov Property Using the previous result, the characterization of a multivariate discrete variable, when the Markov property holds, becomes The Markov property states that If the present state of the process is known, the future of the process is independent of its past A discrete time stochastic chain that exhibits the Markov property is called a Discrete Time Markov Chain () which formally means

2 Basic Defintions and Notation (4) When the state space is countable, the Markov property can be stated in a simpler (more convenient) form Basic Defintions and Notation (5) Transition Probability Matrix Transition probability A {X n, n2 N } is called time-homogeneous if the transition probabilities are independent of n Transition probability matrix Transition probability matrix P is called stochastic if Transition diagram P P P 21 P 22 Basic Defintions and Notation (6) examples and applications (1) For the complete characterization of the time evolution of the process, an initial state space distribution is needed Machines with failures and repairs One machine p u 1-p u Given P and {π [0] i, i 2 S }, it is possible to compute the state space distribution at step n (also called marginal distribution) in the following way Two machines working in parallel Up Down p d 1-p d Independently of the initial distribution, the time spent by the in any of its states, called Sojourn Time, has a geometric distribution with parameter q i p u 2 P u p d +(1-p u )(1-p d ) (1-p u ) 2 2p u (1-p u ) p d (1-p u ) U,U U,D D,D p u (1-p d ) 2p d (1-p d ) p d 2 (1-p d ) 2 examples and applications (2) examples and applications (3) Random walks Brand Switching P 12 P 22 p -4 r r 0 r 1 r 2 r -3 r -2 r -1 3 p -3 p -2 p -1 p 0 p 1 p 2 p P P 21 P 32 P 31 P 13 q -3 q -2 q -1 q 0 q 1 q 2 q 3 q 4 3 P 23 P 33 Gambler s Ruin Success Run p p p p p p 0 p 1 p 2 p N-2 N-1 N q 1 q 2 q 3 q4 q q q q q

3 Transient Solution (1) Computation of Marginal Distributions {π [n] i, i 2 S }, that represents the probability of finding the in state i after n steps, is expressed in a concise manner by the matrix expression introduced earlier Transient Solution (2) Power method (direct multiplication) the n-th power of P can thus be interpreted as the n steps transition probability matrix where represents the probability of moving from state i to state j in n steps Chapman-Kolmogorov Equations Finite matrix Power-2 method Use binary representation on n Compute P, P 2, P 4, Multiply factors corresponding to 1 s in binary representation of n Sparse matrix considerations Transient Solution (3) Z-Transform Method for computing {P n } Definition Observation Compute z-transform Extract n-th component Definition But, is this always true? Observe again Irreducible Discrete Time Markov Chains Communicating Classes State j is accessible from state i ( i j )if State i communicates with state j ( i $ j ) if (i j ) and (j i ) A set C ½ S is a communicating class if In this case a limit distribution does not exist and the fluctuates A communicating class C is said to be closed if A is irreducible if all its states belong to a single closed communicating class; otherwise it is reducible

4 Classification of states (1) Periodicity A state i is said to be periodic with period d if d is the largest integer such that A state i is said to be aperiodic if it has period d=1 Let T i = min {n>0: X n = i}. A state i is said to be periodic with period d if d is the largest integer such that If state i is periodic with period d and ( i $ j ), then j is periodic with the same period d.. Classification of states (2) Let T i = min {n>0: X n = i}. T i represents the first time the visits state i and is an integer-valued random variable Let and When f i < 1, m i is necessarily 1. However, when f i =1, m i is not necessarily finite A state i is said to be Recurrent if f i =1 Transient if f i < 1 A recurrent state i is said to be Positive recurrent if m i < 1 Null recurrent if m i = 1 Classification of states (3) Alternatively, it is not difficult to show that (i) State i is recurrent iff Classification of states (4) Transience and recurrence are easy to recognize in the case of finite communicating classes (ii) State I is transient iff Let v j [i] (n) represent the expected number of visits to state j starting from state i per unit of time up to time n Let C be a finite closed communicating class. Then all states in C are positive recurrent Let C be a finite communicating class that is not closed. Then all states in C are transient. A recurrent state I is positive recurrent iff A recurrent state i is null recurrent iff There are no null recurrent states in a finite state-space Not all the states in a finite state-space can be transient Classification of states (5) Transience and recurrence are more difficult to recognize in the case of infinite communicating classes For transient and null recurrent s Infinite communicating classes may be transient, recurrent or recurrent null even when they are closed Several methods exist for determining the characteristics of the states of a, all essentially based on the computation of f i for an arbitrary state i that is chosen in order to make the computation convenient. An irreducible is positive recurrent if and only if there exist a solution to with Specific results can be obtained for classes of that are characterized by particular structures of their State Transition Diagram Moreover,

5 Limiting Behaviour (2) Computational aspects Irriducible with infinite state space Recursive methods Transform method Limiting Behaviour (3) Particularly interesting is the case in which the state space S of the can be decomposed in several communicating classes such that S = C 1 U C 2 U U C M U C T where communicating classes C i, i=1,2,,m are closed, while C T is non-closed. Consider the case of a finite state space whose transition probability matrix of the can be organized so that Irriducible with finite state space Standard method for the soultion of a system of linear equations (Gauss Elimination, Gauss-Seidel Iteration, ) Using the same class partitioning the initial and limit probability distributions can be expressed in the following manner Limiting Behaviour (4) In order to compute the limiting state probability distribution we must compute the infinite power of the transition probability matrix P that assumes the following form Limiting Behaviour (5) Using the previous results, the explicit expression of P 1 becomes Considering that each P r (r=1,2,,m) can be considered as the transition probability matrix of an irreducible, we can compute For each state i of the transient class C T, we can compute the probability of being absorbed from class C r (r=1,2,,m) in the following manner Limiting Behaviour (6) Applying the standard expression We obtain and