LECTURE 19 Readings: Finish Section 5.2 Lecture outline Markov Processes I Checkout counter example. Markov process: definition. -step transition probabilities. Classification of states.
Example: Checkout Counter Discrete time Customer arrivals: Bernoulli( ) Geometric interarrival times. Customer service times: Geometric( ) State : number of customers at time. 0 1 2 3 9 10
Finite State Markov Models : state after transitions Belongs to a finite set, e.g. is either given or random. Markov Property / Assumption: Given the current state, the past does not matter. Modeling steps: Identify the possible states. Mark the possible transitions. Record the transition probabilities.
-step Transition Probabilities State occupancy probabilities, given initial state : Time 0 Time n-1 Time n i l k j m Key recursion: Random initial state:
Example 1 2
Generic Question Does converge to something? 1 2 3 Does the limit depend on the initial state? 1 2 3 4
Recurrent and Transient States State is recurrent if: Starting from, and from wherever you can go, there is a way of returning to. 5 3 4 6 7 1 2 8 If not recurrent, a state is called transient. If is transient then as. State is visited only a finite number of times. Recurrent Class: Collection of recurrent states that communicate to each other, and to no other state.
Periodic States The states in a recurrent class are periodic if: They can be grouped into groups so that all transitions from one group lead to the next group. 4 3 2 8 1 5 6 7 In this case, cannot converge.
LECTURE 20 Readings: Section 6.3 Lecture outline Markov Processes II Markov process review. Steady-state behavior. Birth-death processes.
Review Discrete state, discrete time, time-homogeneous Transition probabilities Markov property State occupancy probabilities, given initial state : Key recursion:
Recurrent and Transient States State is recurrent if: Starting from, and from wherever you can go, there is a way of returning to. If not recurrent, a state is called transient. If is transient then as. State is visited a finite number of times. Recurrent Class: Collection of recurrent states that communicate to each other, and to no other state. 9 3 4 6 7 5 1 2 8
Periodic States The states in a recurrent class are periodic if: They can be grouped into groups so that all transitions from one group lead to the next group. 4 5 6 9 3 1 2 7 8
Steady-State Probabilities Do the converge to some? (independent of the initial state ) Yes, if: Recurrent states are all in a single class, and No periodicity. Start from key recursion: Take the limit as : Additional equation:
Example 1 2
Example 1 2 Assume process starts at state 1.
Visit Frequency Interpretation (Long run) frequency of being in : Frequency of transitions : Frequency of transitions into : l k j m
Random Walk (1) A person walks between two ( -spaced) walls: To the right with probability To the left with probability Pushes against the walls with the same probabilities. 0 1 2 m Locally, we have: i i + 1 Balance equations:
Justification: Random Walk (2)
Random Walk (3) Define: Then: To get, use:
Birth-Death Process (1) General (state-varying) case: 0 1 2 3 m Locally, we have: i i + 1 Balance equations: Why? (More powerful, e.g. queues, etc.)
Birth-Death Process (2) Special case: and for all and, again, define (called load factor ). Less general (but more so than the random walk). Assume and (in steady-state)
LECTURE 21 Readings: Section 6.4 Lecture outline Markov Processes III Review of steady-state behavior Queuing applications Calculating absorption probabilities Calculating expected time to absorption
Review Assume a single class of recurrent states, aperiodic. Then, where does not depend on the initial conditions. can be found as the unique solution of the balance equations: together with
General case: Birth-Death Process 0 1 2 3 N Locally, we have: i i + 1 Balance equations: Why? (More powerful, e.g. queues, etc.)
M/M/1 Queue (1) Poisson arrivals with rate Exponential service time with rate server Maximum capacity of the system = Discrete time intervals of (small) length : 0 1 i-1 i N-1 N Balance equations: Identical solution to the random walk problem.
M/M/1 Queue (2) Define: Then: To get, use: Consider 2 cases!
The Phone Company Problem (1) Poisson arrivals (calls) with rate Exponential service time (call duration), rate servers (number of lines) Maximum capacity of the system = Discrete time intervals of (small) length : 0 1 i-1 i N-1 N Balance equations: Solve to get:
The Phone Company Problem (2) 0 1 i-1 i N-1 N Balance equations: Solution: Consider the limiting behavior as. Therefore: (Poisson)
M/M/m Queue Poisson arrivals with rate Exponential service time with rate servers Maximum capacity of the system = Discrete time intervals of (small) length : 0 1 i-1 i m j-1 j N-1 N Balance equations:
Gambler s Ruin (1) Each round, Charles Barkley wins 1 thousand dollars with probability and looses 1 thousand dollars with probability Casino capital is equal to He claims he does not have a gambling problem! 0 1 2 m Both and are absorbing!
Calculating Absorption Probabilities Each state is either transient or absorbing Let be one absorbing state Definition: Let be the probability that the state will eventually end up in given that the chain starts in state For For For all other :
Gambler s Ruin (2) 0 1 2 m
Expected Time to Absorption 3 1 2 4 What is the expected number of transitions until the process reaches the absorbing state, given that the initial state is? For all other :