14.451 Lecture Notes 8 Guido Lorenzoni Fall 29 1 Stochastic dynamic programming: an example We no turn to analyze problems ith uncertainty, in discrete time. We begin ith an example that illustrates the poer of recursive methods. Take an unemployed orker ith linear utility function. The orker is draing age-o ers from a knon distribution ith continuous c.d.f. F () on [; ]. At any point in time, he can stop and accept the o er. If he accepts he gets to ork at age and then orks forever getting utility = (1 ). Sequence setup: history is sequence of observed o ers t = ( ; 1 ; :::; t ) : A plan is to stop or not after any possible history, i.e., choose ( t ) 2 f; 1g. The stopping time T is a random variable that depends on the plan (:): T is the rst time here ( t ) = 1. Objective is to choose (:) to maximize " # T E 1 T : Recursive setup. State variable: did you stop in the past? if yes hat age did you accept? So the state space is no X = funemployedg [ R +. The value after stopping at age is just V () = = (1 ). So e need to characterize V (unemployed), hich e ill denote V U. Each period decision after never stopped is max 1 ; V U or, equivalently, max 2f;1g 1 + (1 ) V U : So optimal policy is to stop if > ^, not stop if < ^, and indi erence if = ^, here ^ = (1 ) V U. 1
Bellman equation V U = max 1 ; V U df () : We can rerite it in terms of the cuto ^ and e have ^ = (1 ) max 1 ; V U df () = max f; ^g df () nd xed point, here simply nd ^ that solves here Properties of this map: T (v) ^ = T ( ^) it is continuous increasing on [; ]; has derivative max f; vg df () : T (v) = F (v) vf (v) + vf (v) = F (v) 2 [; ] for v 2 (; ) (here e use continuous distribution); has T () = E [] and T ( ) =. Therefore, a unique xed point ^ exists and is in (; ) (you can use contraction mapping to prove it). Comparative statics 1. An increase in increases the cuto ^. Just look at T (v) = max f; vg df () and see that it is increasing in both and v at the xed point ^. Comparative statics 2. A rst-order stochastic shift in the distribution F leads to a (eak) increase in ^. Comparative statics 3. A second-order stochastic shift in the distribution F leads to a (eak) increase in ^. What is rst-order and second-order stochastic dominance? Take to distributions F and G on R. 2
De nition 1 The distribution F dominates the distribution G in the sense of 1st order stochastic dominance i h (x) df (x) h (x) dg (x) for all monotone functions h : R! R. De nition 2 The distribution F dominates the distribution G in the sense of 2nd order stochastic dominance i h (x) df (x) h (x) dg (x) for all convex functions h : R! R. Sometimes you see stochastic dominance (1st and 2nd order) de ned in terms of comparisons of the c.d.f. of F and G and then the de nitions above are theorems! Exercise: using the de nitions above prove comparative statics 2 and 3. Characterizing the dynamics. Let us make the problem more interesting (and stationary) by assuming that hen employed agents lose their job ith exogenous probability. The state space is still X = funemployedg [ R +. No the Bellman equation(s) are V () = + V U + (1 ) V () V U = max V () ; V U df () From the rst e get V () = + V U 1 (1 ) and e have to nd V U from + V V U U = max 1 (1 ) ; V U df () Exercise: prove that this de nes a contraction ith modulus. So e still have a cuto given by ^ = (1 ) (1 ) V U : No the ne thing is that the optimal policy de nes a Markov process for the state x t 2 X. No let us simplify assuming the distribution of ages is a discrete distribution ith J possible realizations f! 1 ;! 2 ; :::;! J g and probabilities f 1 ; 2 ; :::; J g (the c.d.f. is no a step function). Suppose!^ 1 < ^ <!^. 3
No e have a Markov chain ith transition probabilities given as follos: Pr (x t+1 = unemployed j x t = unemployed) = X ^ 1 j j=1 Pr (x t+1 =! j j x t = unemployed) = for j = 1; :::; ^ 1 Pr (x t+1 =! j j x t = unemployed) = j for j = ^ ; :::; J Pr (x t+1 = unemployed j x t =! j ) = for all j Pr (x t+1 =! j j x t =! j ) = 1 for all j Pr (x t+1 =! j j x t =! j ) = for all j 6= j and all j We can then address questions like: suppose you have a large population of agents (ith independent age dras and separation shocks) and you start from some distribution on the state X, if the economy goes on for a hile do you converge to some invariant distribution on X? This is the analogous of the deterministic dynamics, but the notion of convergence is di erent. No steady state but invariant distribution. Example: f! 1 ;! 2 ;! 3 g ith ^ = 2, then X = funemployed;! 1 ;! 2 ;! 3 g and transition matrix: M = 2 6 4 1 2 3 1 1 1 Suppose you start from distribution. 1; ; 2; ; 3; ; 4; What happens to the distribution after t periods? 2 3 2 3 1;t 1; 6 2;t 4 3;t 5 = M t 6 2; 4 3; 5 4;t 4; Does it converge? 3 5 : 4
MIT OpenCourseWare http://oc.mit.edu 14.451 Dynamic Optimization Methods ith Applications Fall 29 For information about citing these materials or our Terms of Use, visit: http://oc.mit.edu/terms.