SZG Macro 2011 Lecure 3: Dynamic Programming SZG macro 2011 lecure 3 1
Background Our previous discussion of opimal consumpion over ime and of opimal capial accumulaion sugges sudying he general decision problems on he nex pages, where c = conrol (consumpion) k = sae x = exogenous variable In each case, we migh solve he dynamic opimizaion problem using a discree ime opimal conrol approach as in prior lecures. SZG macro 2011 lecure 3 2
Finie horizon problem T å { c},{ k} b uk x c = 0 max (,, ) u: momenary objecive s.. k - k = g( k, x, c ) g : sae law of moion + 1 k k 0 ³ b( x ) T+ 1 T+ 1 SZG macro 2011 lecure 3 3
Infinie horizon problem å { c},{ k} b uk x c = 0 max (,, ) s.. k + - k = g( k, x, c ) = 0,1,... 1 k 0 SZG macro 2011 lecure 3 4
We migh aack he finie horizon problem by forming L = T å = 0 T å = 0 b u( k, x, c ) + bl[ k + g( k, x, c )-k ] + 1 +Q [ k - b( x )] T+ 1 + 1 + 1 SZG macro 2011 lecure 3 5
We migh aack he infinie horizon problem by forming L = å = 0 å = 0 b u( k, x, c ) + bl[ k + g( k, x, c )-k ] + 1 SZG macro 2011 lecure 3 6
Eiher case Oucome is sequence of opimal conrol {c }, opimal sae {k }, and opimal shadow prices { } ha saisfy FOCs and TC give he pah {x }. Desirable because we have boh real oucomes and a means of deriving supporing prices. SZG macro 2011 lecure 3 7
Why? We will sudy an alernaive, dynamic programming, nex Alernaive ool for oolki DP is beer for cerain problems wih uncerainy DP logic applied in oher, equilibrium conexs LS: he imperialism of recursive mehods SZG macro 2011 lecure 3 8
Ouline 1. Cerainy opimizaion problem used o illusrae: a. Resricions on exogenous variables {x } b. Value funcion c. Policy funcion d. The Bellman equaion and an associaed Lagrangian e. The envelope heorem f. The Euler equaion SZG macro 2011 lecure 3 9
Ouline Con d 2. Adding uncerainy 3. Applicaions opimal consumpion over ime opimal consumpion under uncerainy. SZG macro 2011 lecure 3 10
1. A cerainy dynamic problem and he DP approach Maximize å = 0 b uk (, x, c) Subjec o k+ 1 - k = g( k, x, c) conrolled sae and x = x( V ) exogenous variable V = m( V - 1) exogenous sae SZG macro 2011 lecure 3 11
Wha s differen from background seup? Immediae jump o infinie horizon problem, no essenial bu maches presenaion in LS chaper (noe differences in noaion, hough). The exogenous (x) variable(s) are now funcions of a vecor of exogenous sae variables, which evolve according o a difference equaion (perhaps nonlinear, perhaps in a vecor). The laer is a key par of he vision of Richard Bellman, he invenor of DP: his experience in oher areas (such as difference equaions) led him o hink in erms of describing dynamics in erms of sae variables. SZG macro 2011 lecure 3 12
Recursive policies Suppose conrols are funcions of saes, c ( k, ) policy funcion k 1 k g( k, x, c) k g( k, x( ), ( k, )) Then, he sae vecor evolves according o a recursion k 1 k g( k, x( ), ( k, )) s 1 M( s) 1 m( ) ha can be used o generae fuure saes from given iniial condiions SZG macro 2011 lecure 3 13
Evaluaing he objecive Under any recursive policy, we can see ha all of he erms which ener in he objecive are a funcion of he iniial sae (s 0 ) so ha he objecive is also a funcion of he iniial sae å b = 0 = 0 uk (, x, c) = b uk (, x( V ), p( k, x( V ))) = Wk (, V ) = Ws ( ) å 0 0 0 SZG macro 2011 lecure 3 14
Recursion for objecive under arbirary recursive policy Ws ( ; p) = ws ( ; p) + bws ( ; p) 0 0 1 ws ( ; p) = ukx (, ( V), p( kx, ( V))) s + 1 = M( s ) SZG macro 2011 lecure 3 15
Noice he swich Given ha here is a policy funcion ( ), he objecive is now a funcion of he sae vecor. We have made he change we are now hinking in erms of funcions raher han sequences. Bu we haven opimized ye! We could be calculaing he objecive wih a very bad policy. SZG macro 2011 lecure 3 16
Bellman s core idea Subdivide complicaed ineremporal problems ino many wo period problems, in which he rade-off is beween he presen now and laer. Specifically, he idea was o find he opimal conrol and sae now, aking as given ha laer behavior would iself be opimal. SZG macro 2011 lecure 3 17
The Principle of Opimaliy An opimal policy has he propery ha, whaever he sae and opimal firs decision may be, he remaining decisions consiue an opimal policy wih respec o he sae originaing from he firs decisions Bellman (1957, pg. 83) SZG macro 2011 lecure 3 18
Following he principle, The naural maximizaion problem is max{ uc (, k, x( V )) + bv( k, V )} c, k + 1 + 1 + 1 s.. k = k + g( k, x, c ) V + 1 + 1 = m( V ) Where he righ hand side is he curren momenary objecive (u) plus he consequences (V) for he discouned objecive of behaving opimally in he fuure. SZG macro 2011 lecure 3 19
Noing ha ime does no ener in an essenial way We someimes wrie his as (wih meaning nex period) max{ uckx (,, ( V)) + bv( k', V')} ck, ' s.. k' = k + g( k, x( V), c) V' = m( V) So hen he Bellman equaion is wrien as ck, ' Vk (, V) = max{ uckx (,, ( V)) + bvk ( ', V')} s.. k' = k + g( k, x( V), c) V' = m( V) SZG macro 2011 lecure 3 20
Afer he maximizaion We know he opimal policy (which we will call as above, bu wih he proviso ha i is opimal) and can calculae he associaed value, so ha here is now a Bellman equaion of he form Vk (, V) = {((, up kv), kx, ()) V + bvk ( + gkx (, ( V), p( k, V)), V')} A funcional equaion is defined, colloquially, as an equaion whose unknowns are funcions. In our conex, he unknowns are he policy and value funcions. SZG macro 2011 lecure 3 21
How o do he opimizaion? You are free o choose, depending on he applicaion Someimes we ake he Euler roue, subsiuing in he consrain and maximizing direcly over k Oher imes we wan o use a Lagrange approach, puing a muliplier on he consrain governing k SZG macro 2011 lecure 3 22
The associaed Lagrangian Takes he form L { u( c, k, x( )) V( k', ')} [ k g( k, x( ), c) k'] The opimal policy, sae evoluion and relaed muliplier are obained by maximizing wih respec o c,k and minimizing wih respec o. Hence hese are all funcions of he sae variables. SZG macro 2011 lecure 3 23
For an opimum (off corners) We mus have L u(, c k, x()) g(, k x(),) c 0 c c c L V( k', ') 0 k' k' L [ k g( k, x( ), c) k'] 0 And, a he values which solve hese equaions, V=L SZG macro 2011 lecure 3 24
The envelope heorem (Benvenise-Scheinkman) Quesion: wha is he effec of an infiniessimal change in k on V? Answer: I is given by V u(, c k, x()) g(, k x(),) c [ 1] k k k when we evaluae a he opimal policy and he associaed muliplier. As in LS, his may also be wrien a form which does no involve V u( c, k, x( )) V( k', ') g( k, x( ), c) he muliplier, [ 1] k k k' k SZG macro 2011 lecure 3 25
Ouline of proof Nonrivial o show differeniabiliy of V Bu if we have his (as we will frequenly assume) hen V L u( c, k, x( )) c u( c, k, x( )) { } k k c k k V( k', ') k' k' k [ k g( k, x( ), c) k'] k g( kx, ( ), c) g( kx, ( ), c) c k' [1 ] [ ] k k k k While his looks ugly, all erms involving behavior are muliplied by coefficiens ha are se o zero by he FOCs. SZG macro 2011 lecure 3 26
Deails on ET V u( c, k, x( )) g( k, x( ), c) [1 ] k k k uckx (,, ( )) gckx (,, ( )) c { } c c k V( k', ') k' { } k' k { [ k g( k, x( ), c) k']} k SZG macro 2011 lecure 3 27
Ieraing on he Bellman Equaion Under specific condiions on he funcions u and g, he Bellman equaion has a unique, sricly concave (in k) soluion. Under hese condiions, i can be calculaed by considering he limi Vj 1 ( k, ) max c, k' { u( k, x( ), c) Vj( k', ')} s.. k' k g( k, x( ), c) These ieraions are inerpreable as calculaing he value funcions for a class of finie horizon problems, wih successively longer horizons. SZG macro 2011 lecure 3 28
3. A Sochasic dynamic problem and he DP approach Maximize å = 0 E{ b u( k, x, c )} ( k, V ) 0 0 Subjec o k+ 1 - k = g( k, x, c) and Markovian exogenous sae variables x = x( V ) (, V B) = prob( V Î B V = V) + 1 SZG macro 2011 lecure 3 29
Markov examples Markov chains (LS, Chaper 1) Linear sae space sysems Nonlinear difference equaions wih iid shocks, m(, e ) 1 1 We won be more explici unil necessary. Key poin: saes are enough o compue expecaions. SZG macro 2011 lecure 3 30
Bellman Equaion Uncerainy case is minor modificaion of cerainy case Vk (, V) = max{ uckx (,, ( V)) + bevk ( ', V') ( k, V)} ck, ' s.. k' = k + g( k, x( V), c) SZG macro 2011 lecure 3 31
Proceeding as above Lagrangian L {(, u c k, x()) EV( k', ') (, k )} [ k g( k, x( ), c) k'] FOCs ET is unchanged L u(, c k, x()) g(, k x(),) c c c c L EV( k', ') 0 k' k' L [ k g( k, x( ), c) k'] 0 0 SZG macro 2011 lecure 3 32
Implicaions for opimal policies Funcions of saes and sae evoluion c ( k, ) k k g( k, x( ), ( k, )) 1 ( k, ) Sae evoluion is now a larger Markov process. For example, s k k g( k, x( ), ( k, )) M( s, e ) 1 1 1 1 m(, e 1) SZG macro 2011 lecure 3 33
Value Funcion Since c,k,x depend on saes, he value funcion also is V(s). I is he maximized RHS of he Bellman equaion. SZG macro 2011 lecure 3 34
Wha we ve covered in his lecure Inroducion o DP under cerainy Bellman Equaion Associaed Lagrangian FOCs and he ET DP wih exogenous variables ha are funcions of a Markov process (exogenous sae vecor) Wha follows: Opimal consumpion over ime via dynamic programming: calculaion of policy and value funcions in a simple case Seing up opimal consumpion problem wih uncerain income SZG macro 2011 lecure 3 35
3A. Opimal consumpion over ime Simple case (no k,x in u) å = 0 uc ( ) Accumulaion of asses b a = Ra [ + y -c ] + 1 y = y + r( y -y) + 1 And R=1 (level consumpion) SZG macro 2011 lecure 3 36
Bellman Equaion V( a, y) max { u( c) V( a', y')} ca, ' s.. a' R[ a y c] y' y ( y y) SZG macro 2011 lecure 3 37
Taking an Euler Roue 1 V( a, y) max ca, '{ u( a y a') V( a', y')} R s.. y' y ( y y) 1 1 V( a', y') EE :0 uc( a y a ') R R a' V( a, y) 1 ET : uc ( a y a ') a R SZG macro 2011 lecure 3 38
Learning abou consumpion Updae ET and inser in EE o ge 1 1 uc( a y a') uc( a' y' a'') c c' R R Suppose here is a linear policy funcion c ( y y) a y a c' ( y' y) a' y ( y y) R[ a y c] y a a ( y y) R[ a y ( y y) a] y a y a SZG macro 2011 lecure 3 39
Requiring c=c, we have equaions ha resric undeermined coefficiens ( y y) a y a ( y y) R[ a ( y y) y ( y y) a] y a y a Ry ( ) a R(1 ) R/[1 R] y y a y y a a R 1 a ar[1 a] a ( ) R y y a 1 (1 ) R SZG macro 2011 lecure 3 40
Economic Rules Consume he normal level of income (y) Consume he ineres from asse sock, leaving he asse sock unchanged period o period (excep as noed nex) Consume based on he presen value of deviaions from normal income, reaing his as if i were anoher source of wealh; allow variaions in asse posiion on his basis. SZG macro 2011 lecure 3 41
Could have goen hese rules more direcly 1 j 1 j j ( ) c a ( ) [ y ( y y)] R R j 0 j 0 1 1 1 c a y ( y y ) 1 1 1 1 1 R R R R 1 1 c y [ a ( y y)] R 1 R SZG macro 2011 lecure 3 42
Quesions & Answers If we could have goen hem more easily, hen why do we need DP? Because here are many problems ha we canno solve so easily and DP is a procedure for solving hem. Wha is he value funcion? V 1 1 1 ( a, y ) ( ( )) 1 u a 1 y y y 1 1 R Easy o deermine in his case because c is consan over ime; V inheris properies of u Check: ake his v, inser in Bellman equaion as v, show opimal form c has specified form, show v has his form. R SZG macro 2011 lecure 3 43
3B. Opimal consumpion wih flucuaing income: seing up a DP Simple case (no k,x in u) å = 0 E{ b u( c )} s Accumulaion of asses (don necessarily resric R) a+ 1 = Ra [ + y -c] 0 Income process y( V ) V : Markov SZG macro 2011 lecure 3 44
One version of he Bellman equaion V( a, ) max {( u( c) EV( a', ')} ca, ' 1 s..[ a y( ) c a'] 0 R SZG macro 2011 lecure 3 45
FOCs and ET Make sure you can work hese ou following he recipe above, c: u ( c) 0 c 1 EV ( a ', ') a': E{ } 0 R a' 1 : [ a y( ) c a'] 0 R ET : EV ( a, ) a SZG macro 2011 lecure 3 46
Implicaions for policies Opimal consumpion depends on (a) wealh; and (b) he variables ha are useful for forecasing fuure income. ca (, ) Bu solving for his funcion is no longer easy. Raionalizes SL s discussion of numerical mehods, a opic ha we will consider furher laer. SZG macro 2011 lecure 3 47
Implicaion for value funcion Value funcion is objecive evaluaed a opimal consumpion policy, which is a funcion of a Markov process, so ha 0 V0 = å b p V 0 V0 = 0 Va (, ) E{ u( ( a, ))} ( a, ) Value funcion saisfies he Bellman funcional equaion. Va (, V) = max {( uc ( ) + EVa ( ', V')} ca, ' 1 s..[ a + y( V) -c- a' = 0] R = ( u( p( a, V)) + EV( Ra [ + y( V) -p( a, V)], V') ( a, V) SZG macro 2011 lecure 3 48