Readings: Finish Section 5.2

Transcription

1 LECTURE 19 Readings: Finish Section 5.2 Lecture outline Markov Processes I Checkout counter example. Markov process: definition. -step transition probabilities. Classification of states.

2 Example: Checkout Counter Discrete time Customer arrivals: Bernoulli( ) Geometric interarrival times. Customer service times: Geometric( ) State : number of customers at time

3 Finite State Markov Models : state after transitions Belongs to a finite set, e.g. is either given or random. Markov Property / Assumption: Given the current state, the past does not matter. Modeling steps: Identify the possible states. Mark the possible transitions. Record the transition probabilities.

4 -step Transition Probabilities State occupancy probabilities, given initial state : Time 0 Time n-1 Time n i l k j m Key recursion: Random initial state:

5 Example 1 2

6 Generic Question Does converge to something? Does the limit depend on the initial state?

7 Recurrent and Transient States State is recurrent if: Starting from, and from wherever you can go, there is a way of returning to If not recurrent, a state is called transient. If is transient then as. State is visited only a finite number of times. Recurrent Class: Collection of recurrent states that communicate to each other, and to no other state.

8 Periodic States The states in a recurrent class are periodic if: They can be grouped into groups so that all transitions from one group lead to the next group In this case, cannot converge.

9 LECTURE 20 Readings: Section 6.3 Lecture outline Markov Processes II Markov process review. Steady-state behavior. Birth-death processes.

10 Review Discrete state, discrete time, time-homogeneous Transition probabilities Markov property State occupancy probabilities, given initial state : Key recursion:

11 Recurrent and Transient States State is recurrent if: Starting from, and from wherever you can go, there is a way of returning to. If not recurrent, a state is called transient. If is transient then as. State is visited a finite number of times. Recurrent Class: Collection of recurrent states that communicate to each other, and to no other state

12 Periodic States The states in a recurrent class are periodic if: They can be grouped into groups so that all transitions from one group lead to the next group

13 Steady-State Probabilities Do the converge to some? (independent of the initial state ) Yes, if: Recurrent states are all in a single class, and No periodicity. Start from key recursion: Take the limit as : Additional equation:

14 Example 1 2

15 Example 1 2 Assume process starts at state 1.

16 Visit Frequency Interpretation (Long run) frequency of being in : Frequency of transitions : Frequency of transitions into : l k j m

17 Random Walk (1) A person walks between two ( -spaced) walls: To the right with probability To the left with probability Pushes against the walls with the same probabilities m Locally, we have: i i + 1 Balance equations:

18 Justification: Random Walk (2)

19 Random Walk (3) Define: Then: To get, use:

20 Birth-Death Process (1) General (state-varying) case: m Locally, we have: i i + 1 Balance equations: Why? (More powerful, e.g. queues, etc.)

21 Birth-Death Process (2) Special case: and for all and, again, define (called load factor ). Less general (but more so than the random walk). Assume and (in steady-state)

22 LECTURE 21 Readings: Section 6.4 Lecture outline Markov Processes III Review of steady-state behavior Queuing applications Calculating absorption probabilities Calculating expected time to absorption

23 Review Assume a single class of recurrent states, aperiodic. Then, where does not depend on the initial conditions. can be found as the unique solution of the balance equations: together with

24 General case: Birth-Death Process N Locally, we have: i i + 1 Balance equations: Why? (More powerful, e.g. queues, etc.)

25 M/M/1 Queue (1) Poisson arrivals with rate Exponential service time with rate server Maximum capacity of the system = Discrete time intervals of (small) length : 0 1 i-1 i N-1 N Balance equations: Identical solution to the random walk problem.

26 M/M/1 Queue (2) Define: Then: To get, use: Consider 2 cases!

27 The Phone Company Problem (1) Poisson arrivals (calls) with rate Exponential service time (call duration), rate servers (number of lines) Maximum capacity of the system = Discrete time intervals of (small) length : 0 1 i-1 i N-1 N Balance equations: Solve to get:

28 The Phone Company Problem (2) 0 1 i-1 i N-1 N Balance equations: Solution: Consider the limiting behavior as. Therefore: (Poisson)

29 M/M/m Queue Poisson arrivals with rate Exponential service time with rate servers Maximum capacity of the system = Discrete time intervals of (small) length : 0 1 i-1 i m j-1 j N-1 N Balance equations:

30 Gambler s Ruin (1) Each round, Charles Barkley wins 1 thousand dollars with probability and looses 1 thousand dollars with probability Casino capital is equal to He claims he does not have a gambling problem! m Both and are absorbing!

31 Calculating Absorption Probabilities Each state is either transient or absorbing Let be one absorbing state Definition: Let be the probability that the state will eventually end up in given that the chain starts in state For For For all other :

32 Gambler s Ruin (2) m

33 Expected Time to Absorption What is the expected number of transitions until the process reaches the absorbing state, given that the initial state is? For all other :