Markov Chains. Chapter 16. Markov Chains - 1

Transcription

1 Markov Chains Chapter 16 Markov Chains - 1

2 Why Study Markov Chains? Decision Analysis focuses on decision making in the face of uncertainty about one future event. However, many decisions need to consider uncertainty about a sequence of future events. Uncertain demand for GM SUVs each month over the next year Uncertain weather in Napa valley every week over the grape season Uncertain daily evolution of stock prices We need probability models for systems that evolve over time in a probabilistic manner stochastic processes Markov chains are special stochastic processes: Probabilities indicating how the process will evolve in the future depending only on the present state of the process They provide the conceptual foundation for Markov Decision Processes, perhaps the most widely used probabilistic decision models Decision Analysis-2

3 Overview Stochastic process Markov chains Chapman-Kolmogorov equations State classification First passage time Long-run properties Absorption states Markov Chains - 3

4 Event vs. Random Variable What is a random variable? Recall: a sample space is the set of all possible outcomes of an experiment A random variable takes numerical values depending on the outcome of the experiment Examples of random variables: X = number on a die (integer values) X = number of customers purchasing an item (integer values) X = inches of rain (could be integer or real-valued) X = time until a customer gets served (real-valued) Markov Chains - 4

5 Stochastic Processes Suppose now we take a series of observations of that random variable, X 0, X 1, X 2, A stochastic process is an indexed collection of random variables {X t }, where t is the index from a given set T. (The index t often denotes time.) Examples: Roll a die 10 times, X i = number on die on i th roll, i=1,2,,10. Note that X i takes integer values from 1 to 6. The stochastic process { X t } = {X 1,X 2,..} denotes the sequence of rolls. Sales of an item, X t = number of items sold on day t, t=1,2, Then the stochastic process { X t } = {X 0, X 1,X 2,..} provides a mathematical representation of how the sales evolve starting today Markov Chains - 5

6 Gambler s Ruin Example Consider a gambling game where you win $1 with probability p, and lose $1 with probability 1-p on each turn. The game ends when you either accumulate $3 or go broke. You start with $1. Let X t denote your fortune after t turns of the game. Then the stochastic process {X t }= {X 1, X 2, } describes how your gambling fortune evolves. Questions you might want to answer: Should you play? Will the game eventually end? What is the probability you win $3 or go broke? How does everything change with p? Markov Chains - 6

7 Space of a Stochastic Process The value of X t is the characteristic of interest X t may be continuous or discrete, but we ll focus on discrete Example: X t = number of defective items on day t This graph is a realization of a stochastic process X t t Here, X 4 = 2 we say the state of our stochastic process at time t=4 is 2. Markov Chains - 7

8 States of a Stochastic Process Note that a stochastic process denotes how the state of a system evolves over discrete time points Here, we have discrete states AND discrete time points In fact, we ll consider a finite number of possible states. We label the states (or values) 0, 1, 2,, M These states will be mutually exclusive and exhaustive What do those mean? Mutually exclusive: States have no intersection cannot be in two different states at the same time Exhaustive: All possible outcomes are included in the states Markov Chains - 8

9 Types of Stochastic Processes There are several types of stochastic processes depending on how future values probabilistically depend on present and past values. In general, future values may depend on the present value as well as all the past values (for example, stock prices may depend on past values) On the other hand, future values may be completely independent of present and past values (as in fair coin tossing or fair die rolling). In some cases, future values may be independent of past values and depend only on the present value (as in the gambling example). In INDE 411, we will focus on this last category of stochastic processes (called Markov chains). Hence, our stochastic processes {X t } are called discrete time finite state Markov chains Markov Chains - 9

10 Weather Example Let X t be a random variable that takes value 0 if the weather is dry on day t and value 1 if the weather is rainy on day t. Then the stochastic process { X t }={X 0, X 1,X 2,..} provides a mathematical representation of how the weather evolves starting today (t=0), and the state of the system is dry or rainy. Suppose the probability that tomorrow is dry is 0.8 if today is dry, but is 0.6 if it rains today. We write: P(dry tomorrow dry today) = 0.8 = P(X 1 =0 X 0 =0) P(dry tomorrow rainy today) = 0.6 = P(X 1 =0 X 0 =1) Or, for any day t, we write: P(X t+1 =0 X t =0) = 0.8 and P(X t+1 =0 X t =1) = 0.6 Markov Chains - 10

11 Weather Example, continued Suppose we are given the states of weather on days 0,1,2,3. That is, suppose we know that X 0 =0 X 1 =0 X 2 =1, X 3 =0 (dry, dry, rainy, dry). What is the probability that X 4 =0? Mathematically, what is P(X 4 =0 X 3 =0, X 2 =1, X 1 =0, X 0 =0)? We have P(X 4 =0 X 3 =0) = 0.8, and, in writing this number we did not care about the values of X 2 X 1 X 0 This observation is true for any values of X 3 X 2 X 1 X 0 and in fact for any t. Intuitively, given today s weather and the weather in the past, the conditional probability of tomorrow s weather is independent of weather in the past and depends only on today s weather (this is called the Markovian property). Markov Chains - 11

12 Markovian Property A stochastic process {X t } satisfies the Markovian property if P(X t+1 =j X 0 =k 0, X 1 =k 1,, X t-1 =k t-1, X t =i) = P(X t+1 =j X t =i) for all t = 0, 1, 2, and for every possible state, i,j What does this mean? Future depends only on the present, not on the past Or, given the current state and the past states, the conditional probability of the next state is independent of past states and depends only on the current state. Markov Chains - 12

13 Markov Chain Definition A stochastic process {X t } for t = 0, 1, 2, is a Markov chain if it satisfies the Markovian property. Markov Chains - 13

14 One-Step Transition Probabilities The conditional probabilities P(X t+1 =j X t =i) are called the one-step transition probabilities One-step transition probabilities are stationary if for all t P(X t+1 =j X t =i) = P(X 1 =j X 0 =i) = p ij Interpretation: the conditional probabilities don t change over time, they are the same for all t X t j i t t+1 Markov Chains - 14

15 One-step Transition Probabilities for the Weather Markov Chain The weather chain p 00 =P(X t+1 = 0 X t = 0) = p 10 =P(X t+1 = 0 X t =1) = p 01 =P(X t+1 = 1 X t = 0) = 1-P(X t+1 = 0 X t =0) = p 11 =P(X t+1 = 1 X t =1) = 1-P(X t+1 = 0 X t =1) = One-step transition matrix: arrange the four one-step transition probabilities in a one-step transition matrix P whose rows and columns correspond to states and entries are p ij =P(X t+1 = j X t = i) State Markov Chains - 15

16 One-step Transition Probabilities for the Weather Markov Chain The weather chain p 00 =P(X t+1 = 0 X t = 0) = 0.8 p 10 =P(X t+1 = 0 X t =1) = 0.6 p 01 =P(X t+1 = 1 X t = 0) = 1-P(X t+1 = 0 X t =0) = 0.2 p 11 =P(X t+1 = 1 X t =1) = 1-P(X t+1 = 0 X t =1) = 0.4 One-step transition matrix: arrange the four one-step transition probabilities in a one-step transition matrix P whose rows and columns correspond to states and entries are p ij =P(X t+1 = j X t = i) State p 00 = 0.8 p 01 = p 10 = 0.6 p 11 = 0.4 Markov Chains - 16

17 Transition Matrix Stationary one-step transition probabilities can be represented using a one-step transition matrix P, p ij = P(X t+1 =j X t =i) for i, j {0,1,,M} " $ $ P = $ $ $ # p 00 p p 0M p 10 p p 1M!! p (M!1)M p M 0 p M1... p MM % ' ' ' ' ' & Markov Chains - 17

18 Markov Chain State Transition Diagram A Markov chain with its stationary transition probabilities can also be illustrated using a state transition diagram Weather example: 0 1 Weather Dry 0 Rain 1! # " $ & % 0.8 Dry Rain Markov Chains - 18

19 Weather Example with Variable Probabilities State Transition Diagram p Dry 0 1-p Rain 1 1-q Probability Transition Matrix 0 1 Dry 0 " p 1! p $ Rain 1 # $ q 1! q q % ' &' Markov Chains - 19

20 Gambler s Ruin Stochastic Process Consider again the gambling game with probability p=0.4 of winning on any turn, and you start with $1, stop when you go broke or have $3 What are the random variables of interest, X t? X t =$fortune on turn t What are the possible values (states) of the random variables? {0,1,2,3} What is the index t? turn of the game Markov Chains - 20

21 Gambler s Ruin as a Markov Chain Does the Gambler s Ruin stochastic process satisfy the Markovian property? Yes, intuitively, given your current gambling fortune and all past gambling fortunes, the conditional probability of your gambling fortune after one more gamble is independent of your past gambling fortunes and depends only on your current gambling fortune. More formally, P(X 5 =0 X 4 =1, X 3 =2, X 2 =1, X 1 =2, X 0 =1) = 0.6. In writing this number, you did not care about values of X 3 X 2 X 1 X 0 Is the Gambler s Ruin stochastic process stationary? Yes, intuitively, the probability of winning is the same for all turns of the game. More formally, P(X t+1 =0 X t =1) = 0.6 for all t. Markov Chains - 21

22 Gambler s Ruin Markov Chain Suppose the probability of winning on any turn is p=0.4 State transition diagram: One-step transition matrix P: " % $ ' $ ' $ ' $ # ' & Markov Chains - 22

23 Gambler s Ruin Example with Variable Probability Probability p of winning on any turn State Transition Diagram p p p 1-p Probability Transition Matrix # & % ( % 1" p 0 p 0 ( % 0 1" p 0 p( % $ ( ' Markov Chains - 23

24 Inventory Example A camera store stocks a particular model camera Orders may be placed on Saturday night and the cameras will be delivered first thing Monday morning The store uses an (s, S) policy: If the number in inventory is less than s, order enough to bring the supply up to S If the number of cameras in inventory is greater than or equal to s, do not order any cameras The store set the policy with s = 1 and S = 3 If zero cameras on hand on Saturday night, order 3 cameras If one or more cameras on hand on Saturday night, do not order any cameras Markov Chains - 24

25 Inventory Example What are the random variables of interest, X t? X t = number of cameras in inventory on Saturday night of week t What are the possible values (states) of these random variables? {0,1,2,3} What is the index, t? weeks Markov Chains - 25

26 Inventory Example Graph one possible realization of the stochastic process X t Sat night X 0 = 3 Sat night X 1 = 2 Sat night X 2 = 0 Sat night X 3 = 1 t Markov Chains - 26

27 Inventory Example Describe X t+1 as a function of X t, the number of cameras on hand at the end of the t th week, under the (s=1, S=3) inventory policy X 0 represents the initial number of cameras on hand Let D i represent the demand for cameras during week i Assume D i s are independent and identically distributed (iid) random variables X t+1 = Max {3 - D t+1, 0} Max {X t - D t+1, 0} if X t = 0 (Order) if X t 1 (Don t order) Markov Chains - 27

28 State Transition Diagram Inventory Example P(D = 0) P(D =1) P(D = 2) P(D = 2) P(D " 3) P(D "1) P(D =1) P(D =1) P(D = 0) P(D = 0) P(D " 2) P(D " 3) Probability Transition Matrix P(D = 0) # P(D " 3) P(D = 2) P(D =1) P(D = 0) & % ( % P(D "1) P(D = 0) 0 0 ( % P(D " 2) P(D =1) P(D = 0) 0 ( % $ P(D " 3) P(D = 2) P(D =1) P(D = 0) ( ' Markov Chains - 28

29 Demand Probabilities with Poisson Distribution Assume D t ~ Poisson(λ=1) for all t Recall, the Poisson pmf is P( X n) P( D = 0) =1 0 e "1 0! = e"1 = P( D =1) =1 1 e "1 1! P( D = 2) =1 2 e "1 P D #1 = e "1 = ! = e"1 2 = ( ) =1" P( D = 0) = ( ) =1" P( D = 0) " P( D =1) = ( ) =1" P( D = 0) " P( D =1) " P( D = 2) = P D # 2 P D # 3 " e n!! n = =! n = 1, 2, Markov Chains - 29

30 Inventory Example Transition Probabilities Write P, the one-step transition matrix P = # P(D " 3) P(D = 2) P(D = 1) P(D = 0) & % ( % P(D " 1) P(D = 0) 0 0 ( % P(D " 2) P(D = 1) P(D = 0) 0 ( % $ P(D " 3) P(D = 2) P(D = 1) P(D = 0) ( ' = # & % ( % ( % ( % $ ( ' Markov Chains - 30