Lecture 4: The Bellman Operator Dynamic Programming

Transcription

1 Lecture 4: The Bellman Operator Dynamic Programming Jeppe Druedahl Department of Economics 15th of February 2016 Slide 1/19

2 Infinite horizon, t We know V 0 (M t ) = whatever { } V 1 (M t ) = max u(m t, C t ) + βv 0 (M t+1 ) V 2 (M t ) = max V 3 (M t ) = max { } u(m t, C t ) + βv 1 (M t+1 ) { } u(m t, C t ) + βv 2 (M t+1 )... { } lim n Vn (M t ) = max u(m t, C t ) + βv n 1 (M t+1 )? where M t+1 = Γ(M t, C t ) Does the limit exist? 15th of February 2016 Slide 3/19

3 Operator notation Write the on the following general form V n (M t ) = max u(m t, C t ) + V n 1 (Γ(M t, C t )) for all M t M Alternatively in operator form V n (M t ) = J(V n 1 )(M t ) for all M t M A fixed point is a function V such that V(M t ) = J(V)(M t ) for all M t M Is there always a fixed point, and is it unique? 15th of February 2016 Slide 4/19

4 Contraction mapping requirement Let F(M) be the space of bounded continuous functions Theorem Assume u(m t, C t ) is real-valued, continuous and bounded, 0 < β < 1 and the constraint set, C(M t ) is non-empty, compact-valued and continuous, then J has a unique fixed point V F(M), and for all V 0 F(M) J n (V 0 ) V β n V 0 V, n = 0, 1, 2, 3,... Full proof: Lucas and Stokey (1989), theorem 4.6 Main idea: Apply Blackwell s contraction mapping theorem requiring that J is 1 Monotone 2 Discounted 15th of February 2016 Slide 5/19

5 Montone (requirement 1) V(M t ) Q(M t ), M t M J(V)(M t ) J(Q)(M t ), M t M CV (M t) arg max u(m t, C t ) + βv(γ(m t, C t )) CQ (M t) arg max u(m t, C t ) + βq(γ(m t, C t ) Insert into J(V)(M t ) J(V)(M t ) = max u(m t, C t ) + βv(γ(m t, C t )) = u(m t, CV (M t)) + βv(γ(m t, CV (M t)) u(m t, CQ (M t)) + βv(γ(m t, CQ (M t)) u(m t, C Q (M t)) + βq(γ(m t, C Q (M t)) = J(Q)(M t ) 15th of February 2016 Slide 6/19

6 Discounted (requirement 2) We have γ (0, 1) : J(V + k)(m t ) J(V)(M t ) + γk. J(V + k)(m t ) = max u(m t, C t ) + β(v(γ(m t, C t )) + k) What could break down here? = max u(m t, C t ) + βv(γ(m t, C t )) + βk = J(V)(M t ) + βk J(V)(M t ) + γk for γ = β (0, 1) 15th of February 2016 Slide 7/19

7 Summarize 1 The uniqueness of the value function can be proven 2 Iteration on the value function can be proven to converge at a rate of β 3 Further properties: 1 Monotonicity in states expanding the choice set 2 Concavity if choice set is convex and u is concave 3 Differentiability (e.g. Benvenste and Scheinkman (1979), Clausen and Strub (2016)) 4 Unique policy function typically requires that the choice set is convex and u is strictly concave 5 Boyd s Weighted Contraction Mapping Theorem can be used if returns are unbounded (see Carroll (2012)) 15th of February 2016 Slide 8/19

8 (VFI) Algorithm 11: Find the fixed point V input : tol. = output: V[ ] C [ ] 1 V[ ] = 0 all m M 2 while? do 3 V [ ] = V[ ] 4 V[ ], C [ ] = find V(V[ ]) 5 δ = max( V [:] V[:] ) 15th of February 2016 Slide 10/19

9 Policy Think of step n in VFI where we for all M t M set ] V n (M t ) = u(m t, C n (M t )) + βe t [V n 1 (Γ(M t, C n (M t )) [ ] C n (M t ) = arg max u(m t, C t ) + βe t V n 1 (Γ(M t, C t )) C t Alternative: Simulate forward for k periods using C n ( ) as decision rule, and update by V n (M t ) = k β j u(m t+j, C n (M t+j )) j=0 +β k+1 V n 1 (Γ(M t+k+1, C t+k+1 )) Better convergence? Yes, in terms of speed. No, in terms of pool of atraction Everything is discrete: The simulation can be replaced by inversion of a matrix! [Bertel will show you] 15th of February 2016 Slide 12/19

10 Guess and verify Consider the neoclassical growth model V(K t ) = max C t log C t + βv(k t+1 ) s.t. K t+1 = AK α t C t Assume that V(K t ) = a + b log K t such that a + b log K t = max K t+1 log(ak α t K t+1 ) + β(a + b log K t+1 ) The FOC then is 1 AK α t K t+1 = βb K t+1 K t+1 = βb 1 + βb AKα t Inserrt FOC and solve for a and b (independent of K t ) a + b log K t = log(akt α βb βb 1 + βb AKα t ) + β(a + b log( 1 + βb AKα t )) 15th of February 2016 Slide 14/19

11 Guess and verify is only possible for very special models Value and policy functions might, however, be well approximated by parametric functions (typically polynomials, Weierstrass theorem) Solve for the parameters numerically instead of solving the maximization problems (relying on the first-order conditions instead) 15th of February 2016 Slide 15/19

12 The The model: 1 A household gets utility from consumption and disutility from labor 2 The household s income dependent on whether it works or not 3 The household accumulates human capital by working 4 It can save in an acount with an interest rate of r Task: Write up the on the white board for your choice of utility function, wage process and human capital accumulation equation 15th of February 2016 Slide 17/19

13 Ensure that you understand: Algorithm 11 How to set up a Go to PadLet and ask or answer a question ( druedahl/dynamic programming) Think about: What is the problem with having respectively: 1 Multiple states 2 Multiple choices 3 Multiple shocks 15th of February 2016 Slide 19/19