Chapter 4 Introduction to Dynamic Programming

Transcription

1 Chaper 4 Inroducion o Dynamic Programming An approach o solving dynamic opimizaion problems alernaive o opimal conrol was pioneered by Richard Bellman beginning in he lae 1950s Bellman emphasized he economic applicaions of dynamic programming righ from he sar Unlike opimal conrol, dynamic programming has been fruifully applied o problems in boh coninuous and discree ime I is generally more powerful han opimal conrol for dealing wih sochasic problems, and i does no always require some of he differeniabiliy and coninuiy assumpions inheren o opimal conrol Dynamic programming can also deal wih problems ha arise concerning ime inconsisency, in ways ha are difficul o deal wih in opimal conrol In his chaper we lay ou he ground work for dynamic programming in boh deerminisic and sochasic environmens We will see how o characerize a dynamic programming problem and how o solve i We will also presen a series of heorems ha are exremely useful for characerizing he properies of soluion for he many cases in which an explici analyical soluion canno be obained Subsequen chapers presen numerous applicaions of he mehods developed here 1 Deerminisic Finie-Horizon Problems Consider he following finie-horizon consumpion problem: subjec o max T { c } T = 0 = 0 β uc ( ), (11)

2 INTRODUCTION TO DYNAMIC PROGRAMMING 121 k+ 1 = f ( k) c (12) Capial depreciaes a he rae of 100 percen per period Equaion (1) is maximized subjec o he furher consrain ha k f ( k ), 0 T, k 0 given, (13) which saes ha capial can neiher be negaive nor exceed oupu Subsiuing (12) ino (11) yields max T { k } T + 1 = 0 = 0 β u( f ( k) k+ 1), (14) so we have changed he problem from maximizing by choice of consumpion in each period o one of choosing nex period s capial sock 1 One approach o solving his problem is by brue-force opimizaion This is possible because here are a finie number, T, of choices o make To see his, maximize (14) wih respec o k +1 o obain he firs-order condiion 2 β u ( f ( k ) k ) + β + 1 u f ( k ) k f k =, or ( ) ( ) ( ) ( ) u ( f ( k ) k ) = βu f ( k ) k f k (15) This firs-order condiion mus be saisfied for each =0,1,, T 1 I is clear ha he f k k + Equaion (15) hus represens T equaions in T unknowns The variables k 0 and k T+1 appear in opimal soluion for k T+1 is zero, since i only appears in he erm ( T) T 1 wo of hese equaions, bu we already know wha hey are To inerpre (15), replace f ( k) k + 1 wih c o ge ( ) ( ) ( ) β u c = u c f k (16) 1 This is no necessary o do, bu i ofen makes he problem easier o deal wih algebraically 2 We are assuming ha f(k) and u(c) have he forms necessary o ensure an inerior soluion, so we do no need o worry abou he Kuhn-Tucker inequaliy consrains (Wha are hese assumpions?)

3 INTRODUCTION TO DYNAMIC PROGRAMMING 122 The lef-hand side is he marginal uiliy of consumpion in period The righ-hand side consiss of he produc of presen value of he marginal uiliy consumpion in period +1 and he marginal produciviy of capial One uni of consumpion foregone in period increases he capial sock in period +1 by one uni, and his raises oupu in period +1 by an amoun equal o he marginal produc of capial Convering his o uiliy measures and discouning back o period, (16) saes ha he marginal uni in consumpion mus have equal value across wo adjacen periods We will soon be ineresed in exending his model o allow for an infinie planning horizon The difficuly is ha he erminal condiion k T+1 =0 goes away, leaving us wih T equaions and T+1 unknowns However, i urns ou ha here is an alernaive approach o solving his finie-horizon problem ha is useful no only for he problem a hand, bu also for exending he model o he infinie-horizon case This is he dynamic programming approach Suppose we obained he soluion o he period-1 problem, max T { k } + 1 = 1 T 1 β u( f ( k) k+ 1), = 1 k 1 >0 given Whaever he soluion o his problem is, le V1( k 1) denoe he value obained from period 1 onwards Noe ha he value depends on he iniial capial sock I hen follows ha he period-0 problem can be wrien as V ( k ) = max u( f ( k ) k ) + βv ( k ) { } 0 0 k In fac, for any we can define an analogous equaion V ( k ) = max u( f ( k ) k ) + βv ( k ), (17) { } k k+ 1 f k subjec o ( ), k given, for =T, T 1,, 0 Equaion (17) is a paricular applicaion of Bellman s Principle of Opimaliy: Theorem 11 (Bellman s Principle of Opimaliy) An opimal policy has he propery ha, whaever he iniial sae and iniial decision are, he remaining decisions mus consiue an opimal policy wih regard o he sae resuling from he iniial decision

4 INTRODUCTION TO DYNAMIC PROGRAMMING 123 Bellman and Dreyfus (1962), among ohers, provide a proof of he principle, bu i is so inuiive ha we won boher o show i here The recursive sequence in (17) begins in he final ime period wih V ( k ) T+ 1 T = 0 Noe ha solving (17) sequenially will yield exacly he same se of equaions as (15) In period T, we have { + 1 } V ( k ) = max u( f ( k ) k ), (18) T T T T kt + 1 which implies ha k T+1 =0 In period T 1 we have { } ( ) max ( ( ) ) β ( ) V k = u f k k + V k, T 1 T 1 T 1 T T T kt which gives he firs-order condiion ( ( ) ) β ( ) 0 = u f kt 1 kt + VT kt ( ( ) ) β ( ( )) ( ) T 1 T T T = u f k k + u f k f k, where he second line comes from differeniaing (18) Repeaedly solving (17) for each ime period yields he sysem of T equaions in (15) EXERCISE 11 (Cake eaing) Suppose you have a cake of size x, wih x 0 given In each period, =1, 2, 3,, T, you can consume some of he cake and save he remainder Le c be your consumpion in period and le u ( c ) represen he flow of uiliy from his consumpion Assume ha u() is differeniable, sricly increasing and concave, wih lim c 0 u ( c) = Le lifeime uiliy be represened by 0 β uc () Characerize he opimal pah of consumpion = T { c } = 0, (a) by he direc mehod, (b) by he mehod of dynamic programming

5 INTRODUCTION TO DYNAMIC PROGRAMMING Deerminisic Infinie-Horizon Problems So how does he dynamic programming approach help us in he infinie-horizon case? Consider again he period-1 version of he consumpion problem, bu now wrien for an infinie planning horizon: 1 V1( k1) = max β u( f ( k) k + 1) (21) { k } + 1 = 1 = 1 Assume for he momen ha a soluion o his problem exiss Le he maximized value of he objecive funcion be V 1 (k 1 ) Then, according o Bellman s principle of opimaliy, he period-0 problem can be wrien as V ( k ) = max u( f ( k ) k ) + βv ( k ) (22) 1 { } 0 0 k subjec o 0 k1 f( k0), wih k 0 given Noe ha we could re-index ime in (21) by replacing wih s+1 o obain β s β u( f( ks) ks+ 1) = V0( k0) (23) s= 0 I hen becomes clear ha V 0 (k 0 ) and V 1 (k 1 ) mus be exacly he same funcion because (21) and (23) differ only by noaion Tha is, if a soluion exiss, i mus saisfy V ( k ) = max u( f ( k ) k ) + βv ( k ) 1 { } 0 k Because ime does no maer direcly in his problem, we can drop he subscrip noaion and le k ' denoe nex period s value of k: V ( k) = max u( f ( k) k ') + βv ( k '), (24) k ' { } subjec o 0 k ' f( k), k 0 given Equaion (24) is usually referred o as he Bellman equaion of dynamic programming The firs-order condiion for his maximizaion problem is ( () ') β ( ') u f k k = V k, (25)

6 INTRODUCTION TO DYNAMIC PROGRAMMING 125 which is no oo helpful as i sands because we do no know he funcion V ( k ') However, we can use he envelope heorem o make some more progress Differeniae he value funcion in (24) wih respec o k, yielding 3 ' V () k = u ( f() k k ') f () k + u ( f() k k ') βv ( k ') dk + dk = u ( f() k k ') f () k (26) The erm in square brackes is equal o zero from he firs-order condiion (25) (his is he applicaion of he envelope heorem) Updae (26) by one period, V ( k ') = u ( f ( k ') k '') f ( k '), and subsiue ino (25) o obain u ( f ( k) k ') = βu ( f ( k ') k '') f ( k ') (27) In erms of dae subscrips, we have u ( f ( k ) k ) = βu f ( k ) k f k, ( ) ( ) which is he soluion (15) we arrived a before for he finie horizon case EXERCISE 21 (Cake eaing forever) Exend he cake-eaing example (11) o an infinie planning horizon (a) Derive he Bellman equaion and use i o characerize he opimal policy (b) Assume ha uiliy is given by u(c )=ln(c ) Use he mehod of undeermined coefficiens o show ha he value funcion akes he linear form Vx ( ) = A+ Bln( x) (c) Show ha he opimal policy is o ea a consan fracion 1 β of he cake in each period (d) Wha is he opimal policy when u(c)=c? 3 This requires, of course, ha V(k) be differeniable I urns ou ha if u is differeniable hen V(k) is also differeniable under quie general condiions The resul was esablished by Benvenise and Scheinkman (1979), so (26) is someimes referred o in he lieraure as he Benvenise- Scheinkman condiion

7 INTRODUCTION TO DYNAMIC PROGRAMMING 126 In oher infinie-horizon dynamic programs, i may no be reasonable o assume ha ime does no maer, and so he ime subscrips on he problem are fundamenally imporan For example, in he consumpion problem we have jus seen, he resource consrain could ake he form k+ 1 = Af ( k) c, where A is a echnology parameer ha flucuaes wih ime In his case, he value funcion depends on ime, which we denoe in he following way: V() k = max { u( Af ( k) k ') + β V + 1 ( k ')} k ' Despie he apparen added complexiy, he general approach o finding he opimal policy remains he same The firs-order condiion is u ( Af() k k ') = V ( k ') β + 1 The envelope heorem ells us ha ( ) V () k = u Af() k k ' Af () k Updaing by one period, V ( ( ) ) + 1 ( k ') = u A+ 1f k ' k '' A+ 1f ( k '), and subsiuing ino he firs-order condiion yields ( ) ( ) u ( Af ( k) k ') = u A f ( k ') k '' A f k ' (28) β Wih he excepion of A and A +1, his is exacly he same as he resul obained in (27) The nex example is a somewha more complicaed applicaion of hese ideas EXAMPLE 22 (Eekhou and Jovanovic [2002]) Firms produce oupu, y, using human capial, k, according o he producion funcion y = A ( k) k The erm A (k) is a produciviy parameer ha changes over ime, and capures knowledge spillovers o he firm from is compeiors The greaer a firm s own level of human capial, he less i has o learn from ohers Hence, i is assumed ha A () k < 0 Firms face a cos of adjusmen

8 INTRODUCTION TO DYNAMIC PROGRAMMING 127 o k ha is proporional o oupu Given k unis his period, he firm can have nex period a a cos of yc ( k ' k ) I is assumed ha c > 0 and c > 0 (a) Derive he Bellman equaion for his problem k ' V() k = max 1 c A() k k β V + 1 ( k ') + k ' k Because A (k) may vary from period o period, he ime subscrips on he value funcion are imporan (b) Derive a difference equaion in k ha characerizes he opimal policy The firs-order condiion is k ' c A () k = β V + 1 ( k ') k k ' unis The envelope heorem says k ' k ' k ' A ( k) k V () k = A() k c 1 c k k k A () k Updaing by one period and subsiuing back ino he firs-order condiion gives k ' k '' k '' k ' A 1( k ') k ' c A() k βa 1 ( k ') c c 1 = + + k k ' k ' k ' A ( k '), (29) which implicily defines a second-order difference equaion in k (c) Le ka () k ε = A denoe he absolue value of he elasiciy of A wih respec o k Le xk = k+ 1 k denoe he growh facor for human capial, and le xa = A+ 1 A denoe he growh facor for produciviy Assume ha ε, x k and x A are consan for all Derive a saionary soluion relaing he elasiciy ε o x k

9 INTRODUCTION TO DYNAMIC PROGRAMMING Subsiue ε = A 1 ( k ') k ' A + 1 ( k ') ino (29), divide hroughou by A (k); replace A+ 1 ( k ' ) A() k wih x A, and le k '' k ' = k ' k = xk : 1 x k c x βx A ε = 1 1 c x ( ) k ( ) k (210) (d) Why is i reasonable o assume βx A x k <1? Oupu is y=a(k)k If A and k grow oo fas, he presen value of oupu will become infinie Consraining x A and x k ensures ha he presen value of oupu diminishes o zero for periods far enough in he fuure (e) Show ha, if ε is large enough, dx k dε<0 Inerpre his finding Direc differeniaion of (210) gives (1 ) ( ) 2 dε c 1 cc + c = x k 2 dxk 1 c β x A (1 c) Rearranging and making use of (210) allows us o wrie dxx dε 1 xk βxa = c 1 (1 ε) (1 ε) x k x k βx + A c βx A As βx A x k <1 he erm in square brackes is unambiguously posiive, as is he numeraor Thus, if ε>1, dx k dε<0 There are wo effecs of human capial growh Firs, for given A, oupu is increased Second, A is reduced as i becomes more difficul o absorb knowledge from oher firms If he laer effec is large enough (ie if ε is large enough), he firm reduces is invesmen in human capial, preferring o free ride on he knowledge developed by oher firms Eekhou and Jovanovic use his insigh o develop an equilibrium model of inequaliy

10 INTRODUCTION TO DYNAMIC PROGRAMMING 129 Alhough we can usually make good progress in characerizing opimal policies defined implicily by equaions such as (27) and (28), in mos cases i will no be possible o obain an explici soluion for he opimal policy This is unforunae because we would usually like an explici soluion in order o solve for he value funcion When explici soluions are no available we mus ake a more indirec roue o ask some of our basic quesions, including Can we prove formally exisence and uniqueness of he value funcion? Can we prove here is a unique opimal policy for he choice or sae variable? Wha oher properies of he value funcion can be derived? We will address hese quesions in he remainder of he secion I should be noed ha, in he ineress of racabiliy, we will be saing heorems ha may be more resricive han necessary The sandard reamen of he following maerial a is mos general level is o be found in Sokey and Lucas (1989), a raher difficul and ime-consuming book A Conracion Mapping Theorem for Bounded Funcional Equaions Recall from Chaper 3 he following conracion mapping heorem for fixed poin expressions: THEOREM [ch 3] 37 (Conracion mapping heorem) Le f(x) denoe a coninuous funcion which maps a value x from a closed, bounded inerval ino a closed, bounded inerval If f(x) is a conracion mapping, hen here exiss exacly one fixed poin x* = f( x*) To explore uniqueness and exisence of a soluion o he Bellman equaion, we will replace x and x -1 in our difference equaions, wih funcions, f(x) and g(x) Tha is, we wrie gx ( ) = Tf( x' ), (211)

11 INTRODUCTION TO DYNAMIC PROGRAMMING 130 where T denoes some operaion on he firs funcion f ha yields he second funcion g Equaion (211) is called a funcional equaion, because from i we wan o solve for he funcion g(x) for all values of x raher han for any paricular value of x For example, in he consumpion problem we have Vk () = max u( fk () k' ) + βv ( k' ) (212) k ' In (212), he operaor T is he ac of muliplying V by β, adding u( f() k k ') o i, and hen maximizing he resuling funcion by choice of k ' We would like find a unique funcion V(k) ha saisfies he recursive represenaion in (212) for all k This may be very difficul so, before we proceed, we would like o know if one exiss Forunaely, he conracion mapping heorem also applies o such funcional equaions Alhough we will leave ou he echnical deails associaed wih he heorem, we do need o inroduce a new disance funcion, known as he supremum norm and denoed by x y Le f(x) and g(x) denoe wo funcions of x [ a, b], hen he supremum norm, f g denoes he maximum absolue difference beween he wo funcions observed as x akes on differen values in he closed inerval [a,b] An operaor on a funcion is a conracion mapping whenever applying he operaor o wo such funcions brings hem closer ogeher for any admissible values of x Using he supremum norm as our measure of disance, if T is a conracion mapping hen Tfx ( ) Tgx ( ) < fx ( ) gx ( ) 4 This will require ha he funcions are coninuous Addiionally, for he supremum norm o exis, he funcions f(x) and g(x) mus have well-defined maxima and minima, and he conracion mapping heorem applies o ses of funcions ha have hem Tha is, he conracion mapping heorem applies o ses of coninuous funcions mapping closed bounded inervals ino closed bounded inervals For such ses, he supremum norm always exiss 5 4 The idea here is ha as he supremum norm goes o zero he wo funcions ge closer and closer ogeher and evenually become he same funcion 5 A se of funcions wih hese properies is called a "complee meric space" We don' need o ge ino where his name comes from, bu if you come across he erm now you know wha i means

12 INTRODUCTION TO DYNAMIC PROGRAMMING 131 The simples way o ensure ha a maximum exiss is o bound he one-period payoffs in some way For example, in (212) above, here could be some maximum value, u <, of he uiliy funcion regardless of how much capial he consumer has Given discouning, i hen follows ha V < u (1 β) < so V is bounded from above Even so, we sill need o ensure ha V(k) can acually aain is maximum, and his requires ha k mus be able o aain he value ha maximizes V(k) Imagine ha his is a he boundary of he inerval from which k is drawn Then we need o ensure ha k can acually aain he value a his boundary Tha is, we need o ensure ha he inerval for k includes is own boundary Pu anoher way, k mus be drawn from a closed, bounded inerval For example, here migh be some maximum feasible amoun of k, say k, such ha u( f ( k) k ') u < for any k ' 0, k If k is bounded, hen assuming ha u is coninuous everywhere ensures ha u is bounded One also needs o worry abou he lower bound Imagine, for example, ha u(c)=ln(c) Then u(0) and V will no be bounded below However, his is ofen no a pracical concern because we have a maximizaion problem In he consumpion problem, for any k>0, we will always wan o ensure ha c>0 for all Hence, as u=ln(c) is coninuous and c can be bounded above zero, u(c) is bounded below and so is V Assuming such bounds leads o he following exisence and uniqueness heorem: THEOREM 21 (Conracion mapping heorem for bounded reurns) Le C[a,b] be he se of all coninuous funcions mapping values from a bounded closed inerval ino a bounded closed inerval Le f(x) and g(x) be wo arbirary funcions from his se Now consider an operaor, T, on ha funcion, such ha g(x)=tf(x) If T is a conracion mapping hen here exiss exacly one funcion saisfying f ( x) = Tf( x) PROOF We will no prove exisence, which is ricky, bu uniqueness is easy Suppose here were wo funcions f*(x) and g*(x), saisfying f*(x)=t f*(x) and g*(x)=t g*(x) for all x Then, as T is a conracion we have Tf *( x) Tg *( x) < f *( x) g *( x) Bu as f*(x)=t f*(x) and g*(x)=t g*(x), his implies f *( x) g *( x) = Tf *( x) Tg *( x) < f *( x) g *( x), a conradicion Thus here canno be wo fixed poins

13 INTRODUCTION TO DYNAMIC PROGRAMMING 132 The conracion mapping heorem is, of course, a special ype of fixed poin heorem In fac, anoher name for i is he Banach fixed poin heorem The main difficuly is ha deciding wheher he operaor T is a conracion mapping direcly is likely o be a very hard problem And ha is why i is so nice o be helped ou by a handy lile heorem from Mr Blackwell: THEOREM 22 (Blackwell's conracion mapping heorem) (a) The operaor T is a conracion if i has he following wo properies: (Monooniciy) If f ( x) g( x) for all x, hen Tf ( x) Tg( x) for all x (Discouning) Le a be a posiive consan There exiss some δ (0,1) such ha T( f + a)( x) Tf( x) + δa (b) If T is a conracion mapping, and he oher assumpions of Theorem 21 are saisfied, he unique fixed poin of he funcional equaion gx ( ) = Tfx ( ), which we will denoe by f *( x) = Tf *( x), can be found by repeaedly applying he operaor T o he funcion f Tha is, f * ( x) = lim T n f( x) n PROOF If f ( x) g( x) for all admissible values of x, hen f ( x) g( x) + f( x) g( x), because f ( x) g( x) is a disance funcion and herefore is non-negaive If monooniciy +, and if discouning holds his inequaliy holds, we have Tf( x) T ( g( x) f( x) g( x) ) furher implies Tfx ( ) Tgx ( ) + β fx ( ) gx ( ) for some β<1 Subracing Tg(x) from boh sides of he inequaliy, we have Tf( x) Tg( x) β f ( x) g( x) This inequaliy holds for any admissible x, including he value ha makes for he larges difference beween Tf(x) and Tg(x) Thus, applying he supremum norm, Tfx ( ) Tgx ( ) β fx ( ) gx ( ), which is he definiion of a conracion mapping I is easies o show wha his heorem means by example Example 22 is paricularly simple Example 23 coninues he consumpion problem Boh examples deal wih he exisence quesion in par (a) of he heorem

14 INTRODUCTION TO DYNAMIC PROGRAMMING 133 EXAMPLE 22 Le C be he se of all coninuous and bounded funcions on he closed inerval [0,1] and equipped wih he supremum norm Le he funcional equaion be given by g(x)=05f(x), where f and g are any wo funcion drawn from he se C We will firs use Blackwell's heorem o show ha he operaor T in Tf(x)=05f(x) is a conracion mapping (i is obvious ha his is he case, because Tf ( x) Tg( x) = 05 f ( x) g( x) < f ( x) g( x), bu we will go ahead and use he heorem anyway Sep one is o verify ha he candidae funcions f and g saisfy he requiremens of Theorem 21 They do by assumpion in his example, bu we will normally have o verify ha his is he case (for example, if f and g are boh uiliy funcions, we will have o check ha hey are bounded) Sep 2 is o check Blackwell's monooniciy condiion Consider a pair of funcions such ha f ( x) g( x) for all x [0, 1] Then, i mus be he case ha 05 f ( x) 05 g( x) in he same domain, and hus ha Tf ( x) Tg( x) for all x [0, 1] Thus, monooniciy holds Sep 3 is o show ha discouning holds T( f + a)( x) = 05( f( x) + a) = 05 f ( x) + 05a < Tf( x) + δa for any δ (05,1) Thus, discouning holds We herefore conclude ha T is a conracion mapping and ha here exiss a fixed poin funcion saisfying f(x)=tf(x) for all x Par (b) ells us how o find his n n funcion, as f * ( x) = lim n T fx ( ) = limn 05 fx ( ) = 0 Thus, he only bounded funcion f*(x) ha saisfies f(x)=tf(x) for all x [0,1] is he zero funcion, f*=0 for all x EXAMPLE 23 Coninuing our consumpion example, k ( ) β ( ) Vk () = max u fk () k + V k

15 INTRODUCTION TO DYNAMIC PROGRAMMING 134 Firs, we assume ha f and u are such ha u is bounded below u Then, as V canno exceed he discouned presen value of receiving u forever, i follows ha V u (1 β) <, so V is bounded and herefore i has a maximum Nex, we show monooniciy, which saes ha if here exis wo funcions V(k) and Q(k) such ha Vk () Qk () for all k, hen TV TQ for all k This is sraighforward o esablish because of he maximizaion involved in he problem Le k k ' Q denoe he opimal choice of when i is he funcion Q(k) ha is being maximized Then, because we are in fac maximizing V(k), i mus be he case ha TV ( )( k') = max u( f( k) k' ) + βv ( k' ) k ' u( f() k k ' Q) + βv ( k ' Q), because ' Q u( f() k k ' Q) + βq( k ' Q), because V ( k ' Q) Q( k ' Q) TQ ( )( k') k is no he maximum of V Thus, monooniciy holds Finally, we need o show ha discouning holds This is again easy, in his case because we have discouning in our problem Le a be some posiive consan Then TV ( + a) ( k' ) = max u( f( k) k' ) + β ( V( k' ) + a) k ' = TV ( k ') + βa Hence, we have shown ha here exiss a unique soluion o he funcional equaion I should be apparen from his example ha he monooniciy and discouning condiions of Blackwell s heorem can virually be confirmed by casual inspecion of he model In essence, if you have a dynamic maximizaion problem wih discouning of fuure reurns, hen Blackwell s heorem will apply o any problem in which he undiscouned reurns are bounded and he sae variable can ake on any value in a closed bounded inerval Finally, he heorem also gives us a way o solve he dynamic programming problem, which may be useful in cerain seings Define an arbirary funcion Q(k) and apply he

16 INTRODUCTION TO DYNAMIC PROGRAMMING 135 conracion repeaedly o obain Vk ( ) = lim TQk ( ) This funcion so obained will n saisfy he fixed poin funcional equaion Vk ( ) = TVk ( ) and be he unique soluion o he dynamic programming problem Bu how useful is his soluion echnique? In pracice i ofen is no very useful, because no one really has enough ime o do an infinie amoun of algebra! However, for problems wih specific funcional forms, i can work if you can make a guess of he general form he soluion will ake Then, afer a few ieraions you may see a paern arising, allowing you o jump he remaining (infinie number of) seps n A Theorem for Unbounded Reurns In Example 22 we jus assumed ha u(c) was bounded Bu wha if i is no? In principle, capial can grow wihou bound and so can uiliy, and hen i is no obvious ha he value funcion will be bounded (which is, afer all, wha we really care abou) The problem is ha boundedness is an essenial componen of Theorem 22 Sokey and Lucas (1989) discuss his case in some deail (see heir Theorem [414]) We provide here a more resricive heorem ha will, for many applicaions, suffice THEOREM 23 (A heorem for unbounded reurns) Consider he general dynamic program x ' { β } TV( x ') = sup h( x, x ') + V( x ') 6 Assume ha he erm β h( x, x ) { } exiss and is finie for any feasible pah = 0 x = given x 0 Then, if T is a conracion mapping, here is a unique soluion o he dynamic opimizaion problem 6 The erm sup refers o supremum Unil we know ha he value funcion is bounded, we canno be sure ha he maximum value exiss If V is unbounded, we can ge arbirarily close o infiniy, bu we canno acually aain i The erm supremum applies o such cases Once we know ha he value funcion is bounded, we can replace sup wih max

17 INTRODUCTION TO DYNAMIC PROGRAMMING 136 Theorem 23 resrics he admissible one-period payoffs o sequences ha canno grow oo rapidly relaive o he discoun facor As = 0 β hx (, x+ 1) < by assumpion, hen V ( x ) = max β h( x, x ) is also bounded Thus, V ( x ) has a maximum 0 = and he remainder of he heorem can be applied The only difficuly wih Theorem 23 is his: you need o solve he dynamic programming problem o find he pah of he sae variable, ye you don know if he soluion echnique works unil you have shown ha he pah of he one-period payoff funcion is finie in presen value erms The way ou of his impasse can be shown by example: 0 EXAMPLE 24 We reurn o he consumpion example again, his ime wihou assuming ha reurns are bounded Assume ha f ( k) = k α for some α<1 and u(c)=ln(c) Then, as α ( ) ln( ) ln( ) α uc = c = k k+ 1, and k+ 1 = k c we can make wo observaions Firs, he mos rapidly ha he capial sock can grow is o choose zero consumpion a each poin in ime This implies an upper bound o he capial sock given by 1 1 α α + 0 lnk+ lnk lnk (213) Second, he larges one-period pay-off is found by consuming all he oupu, so ha α α ( ) ( ) ( ) ( ) uc = ln c = ln k k ln k = α lnk (214) + 1 So, if we combine he mos rapid growh in capial (213) wih he larges payoff in each period (214), we have 1 uc ( ) αlnk α + lnk I hen follows ha α lnk0 β uc ( ) α( αβ) lnk0 = < 1 αβ = 0 = 0 0 (215) for any finie k 0 Noe ha he acual sequence of payoffs mus be less han his in presen value We have combined in (215) a sequence of uiliies from consuming everyhing wih a sequence of capial socks from consuming nohing, and one canno have boh simulaneously So any feasible sequence of payoffs mus be bounded in presen value, and

18 INTRODUCTION TO DYNAMIC PROGRAMMING 137 his implies ha he value funcion mus also be bounded The funcion Vk ( ) herefore has a maximum, and he reminder of he heorem can be applied as before EXERCISE 22 An agen can produce wo goods, x and y according o he producion funcions x x = l and y x y = l The agen is endowed wih one uni of y labor ime in each period, so l + l = 1 Good x canno be sored, bu good y is indefiniely sorable Producion of good x is immediae, so ha quaniies of x produced in period are consumed in period Producion of good y akes ime, so ha producion of y in period canno be consumed unil period +1 a he earlies Uiliy in each period is given by u x y cc =, and he agen maximizes he discouned lifeime uiliy over he infinie horizon, wih a discoun facor β<1 (a) Show ha he value funcion is bounded [Hin: consruc an infeasible sequence of reurns ha mus exceed any feasible sequence] (b) Now assume ha sored y depreciaes a he rae δ per ime period Show, his ime using a more direc approach, ha he value funcion is bounded for his case A corollary o Theorem 23 in Sokey and Lucas (1989) also gives us a way o solve for he value funcion regardless of wheher we can show ha T is a conracion mapping: THEOREM 24 (Solving models wih unbounded reurns) Consider he general dynamic program TV( x ') = sup { h( x, x ') + βv ( x ')}, x ' and, for any given x 0, le ˆ( ) max Vx β hx (, x ) for all admissible x, and = < Then, if TVˆ() x Vˆ() x = 0 n ˆ n TVx Vx ( ) = lim ( ) yields a well-defined finie-valued funcion, hen V(x) is he unique soluion o he dynamic programming problem

19 INTRODUCTION TO DYNAMIC PROGRAMMING 138 Theorem 24 says ha we can find he soluion o he dynamic opimizaion problem by firs defining a funcion ha we know gives a value for any x ha is greaer han he soluion Then, repeaedly applying he operaor T o his funcion, we see if we converge ono a well-defined finie-valued funcion This will work as long as TVˆ() x Vˆ() x This heorem offers a soluion echnique under differen assumpions han we saw for from Theorem 22 The earlier heorem saed ha if T were a conracion mapping, you could sar wih any value funcion and ierae o find he unique soluion Bu doing so would only work if T is a conracion mapping Theorem 24 saes ha all you need is ha T reurns a funcion ha is smaller a each value of x Bu his will only work if you know you are saring wih a funcion ha is larger han V(x) for each value of x EXAMPLE 25 We will apply Theorem 24 o he consumpion example wih f ( k) = u(c)=ln(c) From Example 24, define Vk ˆ( ) = αln k(1 αβ), and recall ha he operaor is defined by ˆ α TV( k ') = max ln k k ' + βvˆ k ' 0 k' k ( ) ( ) α αβ ln k ' = max ln ( k k ') + (216) 0 k' k 1 αβ The maximum of his expression if found upon seing k ' = αβk α, so ha on subsiuing back ino (216) we ge αβ α TVˆ( k ') = ln(1 αβ ) + ln( αβ ) + ln k 1 αβ 1 αβ As αβ (0,1), ln(1 αβ) < 0 and ln( αβ ) < 0, clearly TVˆ() k < Vˆ() k as Theorem 24 requires Now, apply he operaor a second ime: updae he equaion, replacing k wih k ', muliply by β, add ln ( k α k '), and again ake he maximum wih respec o k ' : 2 ˆ( ') max ln α αβ α TVk = ( k k ' ) + β ln(1 αβ ) + ln( αβ ) + ln k ', k ' 1 αβ 1 αβ which again gives he opimaliy condiion k ' = αβk α Subsiuing back yields 2 αβ α TVk ˆ( ') = (1 + β ) ln(1 αβ ) + ln( αβ ) + ln k 1 αβ 1 αβ k α

20 INTRODUCTION TO DYNAMIC PROGRAMMING 139 Repeaing his process n imes 7, we ge n 1 i αβ n α TVk ˆ( ') = β ln(1 αβ ) + ln( αβ ) + ln k 1 αβ 1 αβ i= 0 n 1 1 β + αβ α = ln(1 αβ) + ln( αβ) + lnk, 1 β 1 αβ 1 αβ which converges as n o ˆ 1 n αβ α lim TVk ( ') Vk ( ) = ln(1 αβ) + ln( αβ) + ln k (217) n 1 β 1 αβ 1 αβ By Theorem 24, his represens he soluion o our fixed poin problem Of course, now ha we know wha V(k) is, we can easily solve for he opimal policy upon noing ha α 1 αβ α Vk ( ) = max ln ( k k' ) + β ln(1 αβ) + ln( αβ) + ln k', k ' 1 β 1 αβ 1 αβ and he firs-order condiion yields k ' = αβk α No one said ha explicily solving dynamic programming problems would be easy! There is, however, an alernaive way o solve he problem in Example 24, using a echnique wih which you are already familiar: he mehod of undeermined coefficiens EXAMPLE 26 We will solve he consumpion problem from Example 25 using he mehod of undeermined coefficiens We guess a soluion of he form Vk () = A+ Blnk for consan A and B o be deermined (from equaion [217] we know his guess is correc) Given his guess, he nex ask is o derive he opimal policy The Bellman equaion mus ake he form k ' α { ( ) β β } A+ B ln k = max ln k k ' + A+ Bln k ', (218) 7 This is very edious and is easy o make algebraic misakes However, afer wo or hree rounds you will spo a paern ha allows you o wrie T n

21 INTRODUCTION TO DYNAMIC PROGRAMMING 140 so he firs-order condiion yields βb α k ' = k 1 βb (219) + Nex, subsiue (219) ino (218) o obain βb βb A+ Blnk = ln 1 + αlnk + βa+ βbln + αβblnk 1 βb 1 βb + + This expression mus hold for any k Hence, maching coefficiens on lnk, we ge B = α 1 αβ Maching coefficiens on he consans, we ge 1 βb 1 A= βbln + ln 1 β 1+ βb 1+ βb 1 αβ = ln(1 αβ) + ln( αβ) 1 β (1 αβ) Hence, 1 Vk ( ) A Blnk ln(1 ) αβ α = + = αβ + ln( αβ) + lnk, 1 β 1 αβ 1 αβ which is he same as in (217) So now we have wo ways ha we may be able o use o find an explici soluion o a dynamic programming wih specific funcional forms The firs is o ierae an infinie number of imes using he operaor TV ˆ( k ) The second is o guess he funcional form and use he mehod of undeermined coefficiens In mos cases, neiher of hese approaches is easy The infinie ieraion approach is algebraically edious and requires a bi of luck: you need o spo a paern developing before hings ges oo messy The benefi of his approach is ha you don have o guess he form of he soluion in advance, alhough you eiher have o find a funcion Vk ˆ( ) saisfying Vk ˆ( ) > Vk () for all k, or show ha T is a conracion mapping The second mehod is algebraically easier, bu i requires luck (and experience) in guessing he funcional form In Example 25, we correcly

22 INTRODUCTION TO DYNAMIC PROGRAMMING 141 guessed ha V = A+ Blnk only because we had already seen he soluion Wihou ha raher large help, how many guesses would you have ried before hiing on he righ funcional form? EXERCISE 23 Consider he wo-good producion problem of Exercise 22, wihou depreciaion of he sored good Guess ha he soluion akes he form Vy () = A B+ y for unknown consans A and B Find he parameers A and B as a funcion of he discoun facor Show ha he producion of x is inversely relaed o he discoun facor Condiions for Uniqueness of he Policy Funcion The conracion mapping heorem gives condiions for exisence and uniqueness of he value funcion However, i need no generae a unique policy funcion In his secion, we provide a heorem ha gives he condiions under which he policy funcion is also unique The general dynamic programming problem, Vx ( ) = max { f( xx, ') + βv ( x' )}, x ' has he firs-order condiion fx ' ( xx, ') + βv ( x' ) = 0 (220) For his o consiue a uniquely-defined maximum, we would naurally urn o he second order condiion, fx' x' ( xx, ') + βv ( x' ) < 0 (221) So, one hing we need for uniqueness is clearly ha he funcion f be sricly concave So he only difficuly is checking he condiions under which V ( x ') < 0 Condiions under which V is concave are easy o come by I is also very generally rue ha we can differeniae V once (and hence ha our whole soluion echnique is valid) However, V may no be wice differeniable, so a saemen such as (221) may no have much meaning

23 INTRODUCTION TO DYNAMIC PROGRAMMING 142 However, even in his case we can provide condiions for concaviy of V, and (220) will coninue o define a unique maximum I urns ou ha if f is sricly concave hen V will also be a sricly concave funcion wih one addiional assumpion, ha he se X from which x and x are drawn is sricly convex Before we sae he heorem formally and prove i, i migh be useful o offer he following reminder of he meaning of concaviy of a funcion and convexiy of a se A funcion f is sricly concave if, for any valid inpus ino he funcion, { x, x ' } and { x, x ' }, and any hird se of inpus { x, x ' } 1 1 x ' θ = θx ' + (1 θ) x ' for any θ (0,1), hen 0 1 f ( x, x ' ) f ( x, x ' ) (1 ) f ( x, x ' ) θ θ > θ + θ θ θ saisfying xθ = θx0 + (1 θ) x1 and (Plo his for a concave funcion wih a single argumen) A se X is convex if, for any x 0 and x 1 belonging o he se, hen x θ also belongs o he se Inuiively, he boundary of a convex se is a concave funcion, and a convex se has no holes By far he mos imporan and common convex se we will deal wih in economic modeling is a bounded inerval of real numbers: if wo numbers belong in an inerval hen so does a weighed average of hem THEOREM 25 (Concaviy of he value funcion and uniqueness of he policy funcion) Given he general dynamic programming problem Vx ( ) = max { f( xx, ') + βv ( x' )}, if f is a sricly concave funcion, he se X of admissible values for x is convex, and he opimal sequence for { x } = 0 involves an inerior soluion in every period, hen (i) V(x) is a sricly concave funcion, and (ii) he opimal policy is unique PROOF Le xθ = θx0 + (1 θ) x1, and assume ha x 0 and x 1 are admissible values for he sae variable Then, as he se of admissible values is convex, x θ is also admissible and we can wrie { } TV ( x ) = f ( x, x ' ) + βv ( x ' ) θ θ θ θ ( f ( x, x ' ) V ( x ' )) (1 )( f ( x, x ' ) V ( x ' )) > θ + β + θ + β (sric concaviy of f) x '

24 INTRODUCTION TO DYNAMIC PROGRAMMING 143 TV ( x, x ' ) (1 θ) TV ( x, x ' ) = θ +, so he operaor is also sricly concave This proves par (i) To prove par (ii), noe ha he sum of wo sricly concave funcions is also sricly concave Hence, f ( xx, ') + βv ( x' ) is sricly concave Moreover (and his will be familiar from sandard opimizaion problems), if a sricly concave funcion has a maximum, he maximum is unique Hence, he maximum idenified by (218) is unique if f is sricly concave Furher Properies of he Value Funcion Two more useful properies can be esablished when we have a unique soluion o he dynamic programming problem We sae hese wihou proof THEOREM 26 (Furher properies of he value funcion) (i) If he one-period payoff funcion f ( xx, ') is monoonically increasing [decreasing] in he curren value of he sae variable, x, hen V(x) is also monoonically increasing [decreasing] in x (ii) If here exiss a parameer, α, such ha f ( xx, '; α ) is monoonically increasing [decreasing] in α, hen V(x;α) is also monoonically increasing [decreasing] in α PROOF We will provide a proof of par (i), which is easy Le x ' i denoe he opimal value of nex period's sae when oday's value is x i, and consider wo values for oday's sae, x 1 < x 2 Then, V ( x1) = f ( x1, x ' 1) + βv ( x ' 1) < f ( x, x ' ) +βv ( x ' ) f ( x2, x ' 2) + βv ( x ' 2) =V ( x 2 ) The firs inequaliy is because x 1 < x 2 and f is sricly increasing in x The second is because he value funcion obained on responding opimally o a curren value x 2 mus exceed any value funcion obained by responding subopimally

25 INTRODUCTION TO DYNAMIC PROGRAMMING 144 This secion has developed a lo of conceps Becoming comforable wih heir use requires pracice and will ake ime I will herefore be useful o see he conceps in acion To ha end, we close his secion wih an example ha makes use of much of he maerial developed here EXAMPLE 27 (Convex invesmen coss) In his example I describe a general invesmen problem, and hen see wha I can say abou is properies The example highlighs he use of he heorems in making precise saemens abou quie general problems In his case, also, checking ha he value funcion is bounded is a lile difficul The cos of invesmen, c(i), is sricly increasing, sricly convex and differeniable wih c(0)=0 The firm produces oupu according o he producion funcion f(k), wih k 0 and f(0)=0, and where f is differeniable, sricly increasing and sricly concave The producion funcion furher saisfies lim k 0 f ( k) =+, so we can resric aenion o inerior soluions, and lim k f ( k) = 0 Capial mus be purchased one period ahead of is use, and i depreciaes a he consan rae δ (0,1) The price of oupu is p, he discoun facor is β (0,1), he ineres rae is r, and used capial can always be sold a he price q The firm's problem is { ( ) ( )} max β pf k c k 1 (1 δ) k +, { k } + 1 = 0 = 0 and he associaed Bellman equaion is k { ( ) ( δ ) β ( )} Vk () = max pf k ck' (1 ) k + V k' (222) To show exisence and uniqueness of a soluion, I firs need o show ha (222) maps bounded coninuous funcions ino bounded coninuous funcions This is a lile ricky in his case, and I have o hink abou he naure of he opimal soluion before I acually solve he model Noe ha capial bough in period can be resold in period +1 for a price (1 δ)q Thus capial will only be accumulaed as long as V () k > (1 δ) q I need o show ha his inequaliy canno hold for any quaniies of capial, bu I am going o have o do i in a roundabou away I claim ha he following inequaliy holds:

26 INTRODUCTION TO DYNAMIC PROGRAMMING 145 pf () k V ( k ') < (223) 1 β If his claim is rue, I can show ha V(k) is bounded I will hen characerize he soluion o he model assuming i is rue, and use his characerizaion o verify he claim laer Given (223), coninued invesmen in capial requires ha (1 β)(1 δ) q f () k > (224) p However, lim k f ( k) = 0, so here mus exis some k < such ha (224) is no longer saisfied Thus, k is bounded beween zero (by assumpion) and k < As he funcions f and c are coninuous, boundedness of capial implies ha he one-period reurn is bounded, while discouning hen implies ha he value funcion is bounded Thus, (222) maps bounded coninuous funcions ino bounded coninuous funcions Then, by Theorem 21 (conracion mapping heorem for bounded reurns), if he operaor defined by (222) is a conracion mapping, he funcion V (k) is uniquely deermined I can herefore use Theorem 22 (Blackwell's Theorem) o verify exisence and uniqueness of a fixed poin Monooniciy and discouning are boh saisfied for his model, he former because he operaor involves maximizaion, and he laer because we are discouning fuure reurns by he facor β<1 (hese claims can be verified by exacly following he seps in Example 22) Consequenly, here is a unique value funcion saisfying (222) The one-period reurn, pf ( k) c ( k ' (1 δ) k) is a sricly concave funcion of k (because c is convex, c is concave) Hence, by Theorem 25, V ( k ') is sricly concave, and he policy funcion obained from he firs order condiion defines a unique invesmen sraegy The firs-order condiion is given by c ( k ' (1 δ) k) = βv ( k ') (225) Before applying he envelope heorem, I will use he firs-order condiion o show uniqueness of he policy funcion direcly As c is differeniable and sricly convex, he lef hand side of (225) is coninuous and sricly increasing in k ' As V ( k ') is a leas once differeniable and sricly concave, he righ hand side of (225) is coninuous and sricly

27 INTRODUCTION TO DYNAMIC PROGRAMMING 146 decreasing in k ' Thus, here exiss a unique k ' saisfying (225) Moreover, he lef hand side of (223) is decreasing in k for any k ' I have now shown ha opimal policy, k ', is increasing in k (You migh like o draw he graph o verify hese argumens) The envelope heorem gives V ( k) = pf ( k) + (1 δ) c ( k ' (1 δ) k) (226) Updaing one period, V ( k ') = pf ( k ') + (1 δ) c ( k '' (1 δ) k '), and subsiuing ino he firs-order condiion yields c ( k ' (1 δ) k) = βpf ( k ') + (1 δ) βc ( k '' (1 δ) k '), a second-order difference equaion ha fully characerizes he unique ime pah of he opimal invesmen sraegy Finally, I need o use hese resuls o verify claim (223) Subsiue (225) ino (226) o eliminae c : V ( k) = pf ( k) + (1 δβ ) V ( k ') I do no need o worry abou he case where k > k ' because if i were ever opimal o reduce he capial sock he desired quaniy could be sold immediaely a a price q Hence, resricing aenion o he case where, k k ', concaviy of he value funcion implies ha V ( k ') V ( k) Tha is, V ( k ') pf ( k) + (1 δβ ) V ( k '), so ha pf ( k) pf ( k) V ( k ') < 1 (1 δβ ) 1 β, as claimed in (223) Alhough here is relaively lile srucure o he model, we have been able o esablish some imporan properies To do so, we made use of Theorems 21, 22, and 25 Having esablished ha he one-period reurn funcion was bounded, Theorems 23 and 24 for unbounded reurns were no relevan We found ha he one-period reurn is in-

28 INTRODUCTION TO DYNAMIC PROGRAMMING 147 creasing in he capial sock By Theorem 26, hen, he value of a firm is also increasing in he size of is capial sock Moreover, we have shown ha he value of nex period's capial sock is increasing in he size of he sock his period Thus, here is persisence in firm size if a firm were o receive a posiive shock o is capial sock oday, ha shock would persis for some ime We have also shown ha here is an upper limi o he amoun of capial ha a firm will accumulae, and hence ha here is an upper bound o firm size and value This finding ells us ha, as long as demand is sufficienly large, no one firm would ge o dominae any indusry exhibiing diminishing reurns and convex adjusmen coss 3 Dynamic Programming and Opimal Conrol 8 Alhough dynamic programming mos ofen is carried ou in discree-ime seings, i can also be used in coninuous ime In his secion we show he equivalence of dynamic programming and opimal conrol soluions o coninuous-ime, deerminisic, dynamic opimizaion problems coincide Consider he following familiar coninuous-ime invesmen problem for a firm: T max ( ), ( ), x () 0 ( ) u k x d, subjec o k () = f( k (), x (), ), k(0) = k 0 (31) Define V ( 0, k( 0) ) as he bes value for he firm ha can be aained a ime 0 given ha he capial sock a ime 0 is k( 0 ) This funcion is defined for all 0 [ 0, T] and any feasible k( 0 ) Tha is, T V (, k( )) = max u( k( ), x( ), ) d, (32) 0 0 x 0 8 This secion can be omied wihou loss of coninuiy

29 INTRODUCTION TO DYNAMIC PROGRAMMING 148 subjec o (31) Noe ha V ( T, k( 0) ) = 0 by definiion Break up he inegral in (32) ino wo pars, one covering he shor inerval [, + ], and he oher covering he inerval ( T] : 0, T V ( 0, k( 0) ) = max u( k( ), x( ), ) d + V ( 0, k( 0) ) + u( k( ), x( ), ) d, x 0 0+ By Bellman's Principle of Opimaliy, he invesmen pah x(), ( + T], mus be opimal for he problem beginning a ime 0 + Tha is, 0, 0 + T V ( 0, k( 0) ) = max u( k( ), x( ), ) d max u( k( ), x( ), ) d + x (), 0 x (), 0+ < T, 0 0+ subjec o (31) Pu anoher way, 0 + V ( 0, k( 0) ) = max u( k( ), x( ), ) d + V ( 0 +, k( 0) + k) x (), 0, (33) 0 which saes ha he value of he opimal policy is equal o he reurn o choosing an opimal policy over he inerval [ 0, 0 + ] plus he reurn from coninuing opimally hereafer As is assumed o be small, hen he following approximaions are reasonable (as hey will be exac in a momen when we le ) 0 + u( k( ), x( ), ) ) d u( k( 0), x( 0), ), xd () x0 0 Tha is, as is a small inerval, hen we can approximae he wo inegrals by assuming ha u and x are consan over he inerval Now, in discree ime modeling we would le = 1, and assume ha wihin each period of lengh 1, he chosen policy mus be consan Doing so yields { } (, ( )) max ( ( ), ( ), ) ( 1, ( 1) ) V k = u k x + V + k +, (34) x ()

30 INTRODUCTION TO DYNAMIC PROGRAMMING 149 where u now measures he payoff during a single period from choosing invesmen x() and beginning wih capial sock k() This is he key funcional equaion for discree ime ha we have already seen Bu wha I wan o do righ now is o hink abou he coninuous-ime problem and relae i o opimal conrol In fac, we can go from (33) o opimal conrol wih he simple assumpion ha V (, k( )) is wice differeniable The assumpion allows us o ake a Taylor expansion of (33) around = 0 : { V ( 0, k( 0) ) max u( k( 0), x( 0), 0) + V ( 0, k( 0) ) + V ( 0, k( 0) ) + Vk ( 0, k( 0) ) k} x ( 0 ) Subrac V ( 0, k( 0) ) from boh sides and divide hrough by : k 0 = max { u( k( 0), x( 0), 0) + V ( 0, k( 0) ) + Vk ( 0, k( 0) ) } x ( 0 ) Finally, we le 0, yielding 0= max { u( k (), x (), ) + V ( k, ()) + Vk ( k, ()) k ()}, (35) x () where we can now, wihou inaccuracy, drop he zero subscrip on curren ime Le λ() denoe he cosae variable from opimal conrol We know ha λ() has he meansing of he marginal value of he sae variable, and hence ha λ()=v k (k(),) Using his fac in (35), we have V (, k() ) = max u( k(), x(), ) + λ()() k x () { } = max { u( k( ), x( ), ) + λ( ) f ( k( ), x( ), ) } (36) x () Equaion (36) is known as he Hamilon-Jacobi-Bellman equaion, and represens he fundamenal parial differenial equaion obeyed by he opimal value funcion Noe, ha he righ hand side of (36) mus be maximized by choice of in he language of opimal conrol he conrol variable, x() Bu he righ hand side is simply he Hamilonian of opimal conrol, and is firs-order condiion is u x + λf = 0 (37) x The opimaliy condiion for λ() is also readily derived Equaion (36) mus be rue even if k() is modified Thus, we can differeniae (36) wih respec o k() o ge

31 INTRODUCTION TO DYNAMIC PROGRAMMING 150 V = u + λ f + λf (38) k k k k where he erm V = V = λ ( ) Now, noing ha he oal derivaive of V (, k( )) k wih respec o ime is (, k( ) ) k dvk λ () = = Vk + Vkk k () = Vk + Vkk f = Vk + λk f d, (39) we can combine (38) and (39) o ge k or λ + λ f = u + λ f + λf, k k k k u k + λf = λ k Hence, if V(k,) is wice differeniable, hen opimal conrol and dynamic programming give equivalen opimaliy condiions 4 Sochasic Dynamic Programming One of he mos aracive feaures of dynamic programming is he relaive ease wih which sochasic elemens can be incorporaed We herefore now exend he mehods of secion 2 o incorporae sochasic feaures ino our models The exension is in principle sraighforward: one adds a judiciously-placed random variable such ha a ime pas realizaions are known bu fuure realizaions are no Thus, he curren value funcion depends upon he disribuion of fuure values of he random shock, and he way in which he shock affecs fuure reurns This uncerainly is handled wih he expecaions operaor Consider, for example, he cake-eaing problem of Exercise 11 In his problem, he naural source of uncerainy concerns random variaions in he agen s preferences For example, i may be ha uiliy in each period is given by v( c ) = z u( c ) where z is a random variable The correc formulaion for he Bellman equaion in his problem depends upon wha we assume is known abou he random process One assump-

32 INTRODUCTION TO DYNAMIC PROGRAMMING 151 ion is ha he realized value of z is known when period- consumpion is chosen, in which case we wrie { β + 1 } ( ) max ( ) ( ) V x = z u c + E V x c (recall ha x is he size of cake remaining) An alernaive assumpion is ha he ase shock for he curren period is no known a he ime he consumpion decision is being made, in which case we wrie { β + 1 } ( ) = max ( ) + ( ) V x E z u c V x c and his ime we canno ake he expecaions operaor inside o he second erm In boh cases, E denoes he expecaion of he value funcion condiional upon informaion ha is known when period- decisions are made I is up o he modeler o be clear abou wha belongs in he period- informaion se, because differen assumpions may lead o drasically differen behavior The cake-eaing example adds shocks o he reurns in each period bu, condiional on he consumpion choice, he evoluion of he sae variable is deerminisic A second common way o inroduce sochasic elemens is o suppose ha he payoff funcion is deerminisic once he value of he sae variable is given, bu he evoluion of he sae variable is subjec o random shocks Consider a sochasic version of he simple infiniehorizon consumpion problem wih capial accumulaion (equaion [21]): β ( ( ) + ) max E u z f k k 0 1 = 1 Here, oupu is subjec o random produciviy shocks, z, so ha k+ 1 = zf ( k) c However, once c and k are given, he one-period reurn is fixed We assume here ha z is known a he ime c is chosen, so ha nex period s capial sock is also known However, nex period s value funcion remains sochasic because i will depend upon he realizaion of z +1 Thus, he Bellman equaion is: { } V ( k) = max u( c) + βe V ( k ') c { uc βev ( zf k c) } = max ( ) + ( ) c, (41)