9 - Markov processes and Burt & Allison 1963 AGEC

Size: px

Start display at page:

Download "9 - Markov processes and Burt & Allison 1963 AGEC"

Fay Julianna Nichols
5 years ago
Views:

1 This documet was geerated at 4:51 PM o Wedesday, October 09, 2013 Copyright 2013 Richard T. Woodward 9 - Markov processes ad Burt & Alliso 1963 AGEC I. What is a Markov Chai? A Markov chai is a discrete-time stochastic process i which the probability of movig from state x i j t to x t + 1 is writte p ij with p = 1. If the trasitio probabilities do ot j ij chage over time, the the dyamic system is said to be statioary. A Markov trasitio matrix, P, for a system with 3 states, therefore, might take the form: x \ x t +1 t I this case, the probability of movig from state x 1 to x 2 i oe period is 0.1. What is the probability of movig to states 2 or 3 from state 3 i oe period? Note that addig up the elemets of the i th row gives the total probability of goig to all states give that you have started i the i th state. Hece, all rows must sum to 1. Now let s cosider the probability of movig from state 1 to state 2 i two periods. This would be equal to p11 p12 + p12 p22 + p13 p32 = = 0.15, probability of goig from 1 to 1 ad the 1 to 2 probability of goig from 1 to 2 ad the 2 to 2 probability of goig from 1 to 3 ad the 3 to 2 i.e. the sum of the probability across all possible paths from 1 to 2 i two steps. Note that this equatio is the 1 st row times the 2 d colum. Similarly the probability of movig from state 2 to state 3 i two periods is equal to the 2 d row times the 3 rd colum. This is matrix multiplicatio. Hece, the complete probability trasitio matrix of movig from state i to state j i two periods ca be foud i the matrix P 2 =P P, or x x \ t+ 2 t I geeral, the elemet pij of the -step probability trasitio matrix, P, defies the probability of beig i state j after periods, give that you iitially started i state i. P is foud by multiplyig P by itself, times. The limitig probability distributio, if oe exists, is the Markov matrix P = lim P. If this limit exist, the it tells us the probability of beig i each state i the distat future.

2 9-2 x After 4 periods x \ t + 4 t x After 10 periods x \ t + 10 t We see that after 4 iteratios, P 4, the rows are begiig to coverge ad by the time we reach P 10, the rows are essetially equal. Beyod 10 periods, the probability of beig i each of the three states (out to 3 decimal poits) is the same, regardless of the state i which we start. Lookig out to period 10 ad beyod, there is a 19.2% chace that the variable will be i state 2, regardless of the state i which we started. Idetifyig the limitig probability distributio ca be helpful for presetig the results of stochastic systems. There are two ways to fid the limitig probability distributio. The cetral characteristic of P is that if you multiply it by P, it does ot chage, i.e., P = P P. Hece, usig matrix algebra we fid that ( I P) P = 0, which implies a system of liear equatios that ca easily be solved to obtai the elemets of the vector P. The alterative 1 approach is simply to recursively multiply P times itself may, i.e. P + = P P, util 1 the product does ot chage ay more, i.e. P + P 0. Why might the limitig probability distributio ot exist? Suppose that the sigle-stage MTM is \ x x t+ t I this case from state 1 we always move to state 2, from 2 to 3 ad from 3 back to 1. If you start i state 1, the after 2 periods, you will be i state 3. At ay time t, the probability is always either zero or oe, ad if it is oe, the ext period the probability is zero. There is o limitig probability distributio for systems that exhibit such cyclical behavior. Although there are other iterestig properties of Markov trasitio matrices, this will be sufficiet for the aalysis required here. II. Why look at Burt & Alliso 1963? It is worth askig why we should eve bother to look at a 1963 paper o computatioal methods. Certaily give the icredible advaces that have bee made i both hardware ad software, papers that discuss computatioal work 50 years old are of questioable value. Noetheless, I ca thik of three basic reasos why lookig at this paper is useful: The paper is a clear ad simplistic formulatio of a stochastic dyamic programmig problem, makig it ideal for pedagogical purposes;

3 9-3 The paper presets a simple agricultural applicatio that ca easily be built o for more elaborate aalysis; It was a watershed paper i Agricultural Ecoomics ad much of the dyamic programmig i the field ca be traced back to Oscar Burt ad/or this paper. III. Burt ad Alliso s farm maagemet problem The applied problem that these authors cosider is the optimal fallow-platig decisio by a farmer who faces stochastically chagig soil moisture levels. Early i the paper the authors mistakely call the previous period s decisio the state variable, but the correct state variable is referred to i Table 1: the soil moisture cotet. The choice variable is whether to fallow the field, or to plat it with wheat. Payoffs (i.e. the beefit fuctio) are depedet o the state ad the decisio, State Soil moisture level Net retur uder Fallow π(z=f) Net retur with platig π(z=w) or more Depedig o whether the decisio is made to fallow or harvest, two distict Markov trasitio matrices ca be foud i Table 1: Choice-cotiget Markov Trasitio Matrices MTM - Fallow P(F) MTM - Wheat P(W) /20 5/20 7/20 7/20 1 9/23 7/23 7/ /20 5/20 14/20 2 9/23 7/23 7/ /20 19/20 3 9/23 7/23 7/ /23 7/23 7/ /23 7/23 7/ Give these values, the authors the solve for the optimal policy of a ifiite-horizo optimizatio problem for the farmer seekig to maximize the et preset value of his or her et returs usig a 6% discout rate. Maximum likelihood estimates of trasitio probabilities Where might these MTMs have come from? They must be estimated. Burt ad Alliso estimate these probabilities usig precipitatio data, expert judgmets, ad assumptios. More geerally, oe ca thik about the problem of estimatig each of the choice p z usig observatios of what has actually cotiget trasitios probabilities ij ( ) occurred. Suppose that, startig at poit i, a choice z has bee observed ki ( ) the umber of times each of the possible states was reached is k ( ) ij z such that z times ad

4 ij =. For a give set of trasitio probabilities, p ( ) ( ) ( ) k z k z j ij observig this data would be j k ( p ) ij ij L =. Maximizig L (or l(l)) subject to the costrait that p ( z ) = 1 ij z, the likelihood of ij will yield the j 9-4 maximum likelihood estimate of the trasitio probabilities. It turs out, that this reduces kij ( z) to the simple relatioship pˆ ij ( z) = ki ( z), i.e., the maximum likelihood estimates of the trasitio probabilities are simply the relative frequecies. If oe eeds to actually estimate trasitio probabilities, perhaps based o limited prior iformatio, the payig attetio to the statistical foudatio for such estimatio ca be quite importat. IV. The solutio algorithm Burt ad Alliso employ a two-step solutio algorithm that made sese whe computers were slow, but I will explai here the policy iteratio approach that is at the heart of their solutio approach. Step 1: The first step, preseted i Table 2 of the paper, is to idetify a array of cadidate optimal choices. They do this by startig out by assumig that the value for 0 V x = 0 for all x. I this case, ay soil moisture cotet i the ext period is zero, i.e., ( ) sice the payoffs from wheat are greater tha the payoffs from fallowig i all states, it is 1 z x = [ W W W W W ]. We will call this a cadidate ever optimal to fallow, ( ) optimal policy vector. Let R 1 be the first cadidate optimal vector of the returs at each 1 R x = Fially, we poit i the state space, i.e. ( ) [ ] also obtai the cadidate optimal Markov trasitio matrix, P 1, which is costructed by takig the rows of the Markov trasitio matrix associated with the vector of optimal policies. Sice z 1 (x)=w for all x, it is simply equal to MTM associated with platig, P(W): /23 7/23 7/ /23 7/23 7/ /23 7/23 7/ /23 7/23 7/ /23 7/23 7/ Step 2: The ext step is to use the cadidate optimal policy idetified i step 1 to obtai a estimate of the value fuctio. The idea here is to calculate what the value fuctio would be if we followed this policy forever. First, imagie that we followed it for just 1 R x, the trasitio two periods. I this case, after obtaiig the beefits as defied by ( )

5 9-5 matrix P 1 would determie the probability of where we eded up ext period. Suppose 1 z x you would ear i that you start i state 2 at time t=1. Followig the policy ( ) the first period, but you would t kow with certaity where you will ed up i period 2. That is, there s a 9/23 chace you ll be i state 1, a 7/23 chace i state 2, etc. The 1 z x for 2 periods expected value of beig i state 2 ad followig the cadidate policy ( ) is β [(9/ )+(7/ )+(7/ )+( )+( )]. Usig matrix otatio, we ca write the value for the etire vector of states, V = R + β P R where V 1 is our first estimate of the cadidate value fuctio assumig the cadidate policy vector z 1 (x). If we followed this policy for 3 periods our estimate of the value fuctio would be closer to the ifiite-horizo outcome: ( ) V 1 = R 1 + β P 1 R 1 + β P 1 R 1 = R 1 + β P 1 R 1 + β 2 P 1 P 1 R 1. Followig this process for periods we would have 2 ( ) ( ) V = R + β P R + β P R + + β P R. Takig the limit as, this sum ca actually be writte quite cocisely. You probably remember that 1 1 β = = t 1 1 r r = + 1 β or Markov Trasitio matrices is t T t = 0 t= 0 t 1 β =. The aalog to this for 1 β t t β ( β ) 1, Vˆ = limv = lim P R = I P R where I is the idetity matrix ad V ˆ is the vector of values that would result if a cadidate policy that gives rise to P ad R is followed ad ifiitum. Hece, give the cadidate payoff vector R 1 ad cadidate MTM P 1, V 1 ca be foud by solvig a liear system of equatios, I β P V = R, or, (1) ( ) ( β ) 1 V 1 = I P 1 R 1. (2) This step is typically referred to as the policy iteratio step, because it takes a cadidate policy vector ad the fids the preset value of the ifiite stream of all future beefits assumig that that policy is iterated forever. (A VB program that implemets matrix iversio ad matrix multiplicatio is available from the class web site) Solvig this equatio for the cadidate policy preseted above yields V 1 =

6 9-6 Step 3: Our ext step is to use the cadidate value fuctio to fid a ew cadidate policy. To do this, we put V 1 o the right had side of the Bellma s equatio, ad for each state determie whether we re better off with a policy of F or W. It turs out, that whe V 1 is o the RHS of the value fuctio, i state 1 the future payoff from fallowig more tha makes up for the short term loss. Hece, our ew cadidate policies, payoffs ad Markov trasitio matrix, are as follows: z 2 R 2 P 2 1 F /20 5/20 7/20 7/20 2 W /23 7/23 7/ W /23 7/23 7/ W /23 7/23 7/ W /23 7/23 7/ Agai usig (2), we ca solve for a ew cadidate value fuctio assumig that z 2 is 2 V = followed ad ifiitum. I this case we get [ ] 2 We repeat the process agai, with the V o the RHS of the Bellma s equatio. I this case, it turs out that z 3 is the same as z 2. Hece, if we decided to follow the policy rule [ F W W W W ] from t=2 oward, it would also be optimal to follow that same policy rule at t=1. This meas that the cadidate policy z 2 (x) is the optimal policy for a ifiite horizo, z * (x). It is worth poitig out, as do Burt ad Alliso, that i fact soil moisture is ot a discrete variable. Hece, this problem is actually ot a true DD problem because although the true choice variable is discrete plat or fallow the true state variable is cotiuous. Discretizig the state space as they did is oly oe of a umber of ways to approximate the solutio of a cotiuous state problem. We will discuss other methods to deal with cotiuous state variables i the ext lecture. V. What does it mea for the value & policy fuctios to be costat over time? There is frequetly some cofusio about what is ad what is ot costat over time i a ifiite horizo dyamic optimizatio problem. It is the fuctios that will be costat over time ot the policies themselves. For example, at the optimum of B&A s problem, over the first 4 periods, the value ad policy fuctios would be costat as show below. x t=1 t=2 t=3 t=4 t=1 t=2 t=3 t=4 V(x) V(x) V(x) V(x) z * (x) z * (x) z * (x) z * (x) F F F F W W W W W W W W W W W W W W W W

7 9-7 However, i a actual realizatio of the problem, the state would chage over time depedig o the policy chose ad the radom evet distributed accordig to the MTM s P(F) ad P(W). For example, we might see a path like the oe preseted below. Startig i period 1 i state 1 the optimal policy is to fallow. There are the four states i which we might ed up ext period, state 2 with probability 1/20, state 3 with probability 5/20, state 4 with probability 7/20 ad state 5 with probability 7/20, but oly oe will actually occur. I the example below we assume that we have good luck ad the soil moisture icreases to state 5 i period 2. I period 2, sice x=5, platig is optimal. I this case the soil moisture drops, say to state 2. The optimal policy i this state is W ad there s a 9/23 chace that we will ed up i state 1 i the followig period, which we assume is what happes. Hece, i period 4 we are back i state 1 ad fallowig would oce agai be optimal. x t=1 t=2 t=3 t=4 t=1 t=2 t=3 t=4 V(x) V(x) V(x) V(x) z * (x) z * (x) z * (x) z * (x) F F F F W W W W W W W W W W W W W W W W R=-2.33 R=47.63 R=32.07 R=-2.33 Simulatig a optimal policy path I presetig your results, it is ofte ecessary to use Mote Carlo simulatio of a umber of paths ad preset those results either graphically or umerically. To begi thikig about how oe might actually simulate a optimal policy path, cosider first the followig thought experimet. Suppose that there are 6 states of the world, ad give your optimal choice i each of these states, the chace of edig up i each of these states is equal to 1/6. So you could simulate this radom path by rollig a die. Suppose you start i state 3. The you roll your die ad, if it comes up with a 1, you advace to state 1. If you roll a 5, you ed up i state 5, ad so o. I this way, rollig your die, time after time, you could obtai a radom path that coicides with the optimal policy. Further, if you wated to simulate a secod radom path, you could go back to period 1 ad state 3 ad start over agai. I this way you could obtai the followig simulated path. t=1 t=2 t=3 t=4 t=5 t=6 t=7 t=8 t=9 t=10 Simulatio #1 x t =3 x t =5 x t =4 x t =2 x t =5 x t =1 x t =3 x t =2 x t =2 x t =4 Simulatio #2 x t =3 x t =3 x t =6 x t =2 x t =1 x t =4 x t =5 x t =2 x t =3 x t =5 Now, how could we do this with a computer istead of die? We could use a radom umber geerator, which would geerate a umber betwee 0 ad 1. If the value draw is less tha 1/6, this is treated as a 1 o the die, betwee 1/6 ad 2/6 is a 2, ad so o. I other words, we draw a radom umber, say ε, ad the carry out the followig logical test:

8 9-8 if 1/6 > ε the go to state 1, else if 2/6 > ε the go to state 2, else if 3/6 > ε the go to state 3, I this very simple case, the Markov Trasitio matrix would look like this. x t+1 =1 x t+1 =2 x t+1 =3 x t+1 =4 x t+1 =5 x t+1 =6 x t =1 1/6 1/6 1/6 1/6 1/6 1/6 x t =2 1/6 1/6 1/6 1/6 1/6 1/6 x t =3 1/6 1/6 1/6 1/6 1/6 1/6 x t =4 1/6 1/6 1/6 1/6 1/6 1/6 x t =5 1/6 1/6 1/6 1/6 1/6 1/6 x t =6 1/6 1/6 1/6 1/6 1/6 1/6 So for ay x t, the cumulative probabilities would be as follows: x t+1 =1 x t+1 =2 x t+1 =3 x t+1 =4 x t+1 =5 x t+1 =6 x t 1/6 2/6 3/6 4/6 5/6 6/6 We ca ow geeralize the problem as follows. Suppose that a optimal Markov Trasitio matrix has bee idetified, say P, i which, P(x t,x t+1 ) is the probability of goig from x t to x t+1 give that the optimal choice is beig made. I ay period, the state variable x ca take o values, x,..., 1 x. Usig P, a radomly chose policy path for T periods ca be geerated usig the followig algorithm:

9 9-9 Pseudo-code for stochastic policy simulatio Set x t =x 0, a chose value For every period to be simulated, t==1, T Geerate a uiform radom umber, ε [0.1] Set the variable CDF = 0 for every possible value of x t+1,x i = x 1,x 2,, x CDF = CDF+P(x t,x i ) Time loop State loop Probability check If CDF ε, the x t+1 =x i Exit State Loop Ed if Ed of State loop Record x t, z t * (x t ), ad x t+1 Set x t =x t+1 Ed of Time loop VB-code for stochastic policy simulatio ix = 3 ' iitial value for the idex For it = 1 To t ' Store values i spreadsheet Cells(it, 1) = it Cells(it, 2) = xgrid(ix) ' Geerate a radom # betwee 0 & 1. ' Rd must ot be dimesioed It is a special VB fuctio that geerates a uiform distributed # eps = Rd ' Figure out which cell was chose usig the Optimal Markov Trasitio Matrix, say mtmstar. As soo as cdf>eps, we ve foud x t+1 cdf = 0# For ix1 = 1 To x cdf = cdf + mtmstar (ix, ix1) If cdf > eps The ix = ix1 ' We ll go to ix1 ext Ed If Next i Next it Exit For stop loopig over ix1 I practice (ad i your problem sets) you would eed to ru may simulatios to evaluate the distributio of your optimal paths over time. It ca also be useful to preset the limitig probability distributio. I the Burt ad Alliso case, followig the optimal policy leads to the followig limitig probability distributio so that the most commo state is state 3. Aother way to iterpret this is that i the log ru fallowig will be optimal 28% of the time. State Limitig Probability [VB subrouties to carry out matrix multiplicatio are cotaied i the MATRIX 2.3 library, accessible from the AGEC 637 web site. Additioal subrouties that carry out policy iteratio ad other simple operatios are available o request.] VI. Refereces Burt, O.R. ad J.R. Alliso "Farm Maagemet Decisios With Dyamic Programmig." Joural of Farm Ecoomics 45:

10 9-10 Rust, Joh Numerical Dyamic Programmig i Ecoomics. I H. Amma, D. Kedrick ad J. Rust (eds.), Hadbook of Computatioal Ecoomics. New York: North Hollad. VII. Readigs for ext class Judd (1998) Read the sectio, Limits of discretizatio methods o p. 430, the look over the begiig of the sectio startig o p Read pages

9 - Markov processes and Burt & Allison 1963 AGEC

This document was generated at 8:37 PM on Saturday, March 17, 2018 Copyright 2018 Richard T. Woodward 9 - Markov processes and Burt & Allison 1963 AGEC 642-2018 I. What is a Markov Chain? A Markov chain