Uncertainty Runs Rampant in the Universe C. Ebeling circa Markov Chains. A Stochastic Process. Into each life a little uncertainty must fall.

Size: px

Start display at page:

Download "Uncertainty Runs Rampant in the Universe C. Ebeling circa Markov Chains. A Stochastic Process. Into each life a little uncertainty must fall."

Aileen Noreen McCarthy
5 years ago
Views:

1 Uncertainty Runs Rampant in the Universe C. Ebeling circa 2000 Markov Chains A Stochastic Process Into each life a little uncertainty must fall.

2 Our Hero - Andrei Andreyevich Markov Born: 14 June 1856 in Ryazan, Russia Died: 20 July 1922 in Petrograd (now St Petersburg), Russia

3 Brand Switching The wine of the month club offers two brands of wine Wine A and Wine B Currently Wine A has 20% of the market Customers in the club order monthly either A or B Anyone buying Wine A one month has a 75% chance of buying it the next month therefore a 25% chance of switching to Wine B Anyone buying Wine B one month has a 55% chance of buying it the next month therefore a 45% chance of switching to Wine A Wine A What is the expected market share of Wine A at the end of the next several months?

4 Brand Switching and Markov a look ahead Next Month: Wine A Wine B This Month: Wine A Wine B q (0) = (.2.8); P= initial state vector transition matrix

looking further ahead.75.25 = =.2.8 =.51.49.45.55 (1) (0) q q P (2) (1) q q P ( ) ( ).75.25 = =.51.49 =.603.397.45.55 ( ) ( ) Month: 2 (2) (0) 2.

5 looking further ahead = =.2.8 = (1) (0) q q P (2) (1) q q P ( ) ( ) = = = ( ) ( ) Month: 2 (2) (0) q = q P =.2.8 = Wow, using matrix-vectors Month: 1 Month: 2 ( ) ( ) q = q P= q P P= q P (2) (1) (0) (0) 2 is so easy. ( )

Still looking further ahead.75.25 = =.603.397 =.6309.3691.45.

6 Still looking further ahead = = = (3) (2) q q P Month: 3 ( ) ( ) Month: 3 (3) (0) q = q P =.2.8 = ( ) ( ) Month: 10 (10) (0) q = q P =.2.8 = ( ) ( ) Look, both rows are now the same!!!

7 Stochastic Process Indexed collection of random variables index is often time {X t }, t = 0, 1, 2, 3,. Examples X t = number of demands in month t X t = number of customers served on day t X t = number of errors on page t of a manuscript X t = A or B, selected brand for month t X t = 0,1,2,3,... ; the state of system at time t

8 What do we do with these stochastic processes? I think we answer questions with them. What we really need to model a stochastic process is the joint probability distribution.

9 The Markovian Property P{X t = a t X 1 = a 1, X 2 = a 2,, X t-1 = a t-1 } = P{X t = a t X t-1 = a t-1 } I call this the Markovian or memoryless Property

10 A Stationary Process Assume for all t: P{X t = j X t-1 = i} = P{X 1 = j X 0 = i} = P ij I call these the onestep transition probabilities

11 Markov Chains Look, these Markov processes are really great for predicting stock market movements! Two types steady-state processes absorption state processes Characteristics finite number of states future state depends only on current state (Markovian or memoryless property) stationary process (probabilities constant over time) discrete transitions (time periods)

12 Transition Probabilities The conditional probability that a Markov chain will be in state j at time t+1 given that it is in state i at time t is denoted p ij and is called the one-step transition probability, i.e., p ij = P[X t+1 = j X t = i]. The matrix of one-step transition probabilities is denoted as P and is given by { P} ij P p11 p12 p13 p1 m p p p p p p p p m = = m1 m2 m3 mm where m j= 1 and p 0 ij p ij = 1 for all i

Stochastic matrix { P} ij p11 p12 p13 p1 m p p p p p p p p 21 22 23 2m = P= m1 m2 m3 mm where m j= 1 and p 0 ij p ij = 1 for all i A

13 Stochastic matrix { P} ij p11 p12 p13 p1 m p p p p p p p p m = P= m1 m2 m3 mm where m j= 1 and p 0 ij p ij = 1 for all i A matrix whose rows sum to one is called a stochastic matrix. If the columns also sum to one, then it is called a doubly stochastic matrix.

14 n-step transition probabilities P{X t+n = j X t = i} = P{X n = j X 0 = i} = P ij (n) where j= 1 ( n) ij = 1 for all i ( n) and p 0 for n= 1, 2,3... ij m p

15 The n-step transition matrix { n } P ( ) ( n) ij ( n) ( n) ( n) ( n) p p p p p p p p m (n) ( n) ( n) ( n) = P = m p p p p ( n) ( n) ( n) ( n) m1 m2 m3 mm

16 Chapman-Kolmogorov Equations m ( n) v ( n v) ij ik kj k = 1 P = PP i, jn,,0< v< n in particular m m ( n) ( n 1) ( n 1) ij = ik kj = ik kj k= 1 k= 1 P P P P P Provides a method for computing the n-step transition probabilities

17 More on the Chapman-Kolmogorov Equations P m ( n) ( n 1) ij = Pik Pkj k = 1 in particular m (2) (1) (1) ij = ik kj k = 1 P P P This is definition of the (i,j) th element if the matrix P is squared. That is, {P ij (2) } = P (2) = P x P

18 Generalizing P (2) = P P = P 2 P (3) = P P (2) = P 3 I see now how we can compute the n- step transition matrix. and in general P (n) = P P (n-1) = P (n-1) P = P n

19 Whoa Chuck! We need another example of these so called Markov chains before you go any further.

20 Our very first complete example Joe college has decided when on spring break to travel among his favorite three cities: Dayton, Columbus, and Cleveland. Each day, he will randomly determine where he will spend the following day. If he is currently in Dayton, he will either stay in Dayton or travel to Columbus with a probability. If he is in Columbus, he will either stay in Columbus or travel to Cleveland with a probability. If he is in Cleveland, he will travel to Dayton with a.75 probability or travel to Columbus with a.25 probability.

21 Our very first complete example continued: Define state 1: Dayton state 2: Columbus state 3: Cleveland P = Joe College going on his spring break.

22 Our very first complete example continued some more P (2) = = Day 2 of the spring break

23 Our very first complete example continued even more P = (3) = Day 3 of the spring break

24 Our example, yes continued P (4) = = Day 4 of the spring break

25 Our example, you guessed it (5) (2) (3) P = P P = = Day 5 of the spring break

26 Our example, keeps on truck-in Joe College decides to begin his spring in Dayton with probability.2, Columbus with probability.3, and Cleveland with probability.5. q (0) = (.2,.3,.5) is referred to as the initial state vector and q (n) is the state vector after n transitions (5) (0) (5) q = q P = (.2,.3,.5) = (.3356,.443,.2214)

state vector q = q, q, q,, q 1 2 3 m the probability that a Markov chain is in state j at time n is

27 Initial State Probabilities In some cases, we may not know in which state a Markov chain starts, so we describe the initial state (0) probability: q = P X = i i [ ] 0 ( ) (0) (0) (0) (0) (0) Given the initial state vector q = q, q, q,, q m the probability that a Markov chain is in state j at time n is called the unconditional state probability and can be found via: This is a remarkable result. q (n) = q (0) P (n)

28 Steady-State Probabilities The steady-state probabilities of a Markov chain, when they exist, is given by: π j lim ( n) = P j = 1,..., ij n m

29 Joe College s spring break revisited Joe decides to continue his spring break indefinitely and not return to classes. In the long run, what is the average percent of time that he will spend in each city? P = After 10 days P =

30 Steady-State Equations π m j= 1 m = π p for j = 1,..., m j i ij i= 1 π j = 1 There are m+2 equations and only m+1 unknowns. One of the equations must be redundant.

31 Back to the example.5.5 0,,,, [ π π π ] = [ π π π ] π 1 =.5 π π 3 π 2 =.5 π 1 +.5π π 3 eliminate one π 3 =.5 π 2 π 1 + π 2 + π 3 =1 need in order to avoid all zero solution

32 Back to the example some more 1. π 1 =.5 π π 3 2. π 2 =.5 π 1 +.5π π 3 3. π 3 =.5 π 2 4. π 1 + π 2 + π 3 =1 From 1: π 3 = 50 π 1 /75 = (2/3) π 1 From 3: π 2 = 2 π 3 = (4/3) π 1 Using 4: π 1 + (4/3) π 1 + (2/3) π 1 = 1 or (1 + 4/3 + 2/3) π 1 = 3 π 1 =1 π 1 = 1/3, π 2 = 4/9 =.4444, π 3 = 2/9 =.2222

33 Expected Recurrence Times let μ jj = 1 π j μ jj are expected recurrence times. The average number of transitions before returning to state j. from example: μ 11 = 1/π 1 = 3/1 = 3 days μ 22 = 1/π 2 = 9/4 = 2.25 days μ 33 = 1/π 3 = 9/2 = 4.5 days

34 This is great stuff Chuck. Can you do another example? Please!!!!

35 It s a rat race A rat moves randomly in the maze below. What are the long term probabilities of the rat being in each region?

36 Transition Matrix p =

37 Steady-State equations p = π1 = π π = π + π + π π3 = π π π + π + π + π = Solution: π1, π2, π3, π4 =,,, ( )

38 Another example In a particular society, the movement of a family from one social class to another is governed by the following transition probabilities where a transition takes place from one generation to the next. class: upper middle lower P = upper middle lower

39 State Transition Diagram Upper class Lower class Middle class.2.7

40 After 2 and 3 generations P 2 class: upper middle lower = upper middle lower P 3 class: upper middle lower = upper middle lower

41 The Steady-State solution ( π ) ( ) 1, π2, π3 = 1, 2, π π π π =.4 π +.1π Equations: π =.1 π +.2 π +.6π π + π + π = 1 Solution: π1, π2, π3 =,, =.0975,.585, ( ) ( )

Expected recurrence times 1 1 1 41 41 41,, =,, = 10.25,1.71,3.

42 Expected recurrence times ,, =,, = 10.25,1.71,3.15 π1 π2 π ( ) I see that if my family were to leave the lower class we would return in 3.15 generations. How many generations are expected before my family reaches the upper class?

43 Quick, go to the first passage times This is really going to be good. Comeon

Chapter 16 focused on decision making in the face of uncertainty about one future

Chapter 16 focused on decision making in the face of uncertainty about one future 9 C H A P T E R Markov Chains Chapter 6 focused on decision making in the face of uncertainty about one future event (learning the true state of nature). However, some decisions need to take into account