THE BELLMAN PRINCIPLE OF OPTIMALITY

THE BELLMAN PRINCIPLE OF OPTIMALITY IOANID ROSU As I undersand, here are wo approaches o dynamic opimizaion: he Ponrjagin Hamilonian) approach, and he Bellman approach. I saw several clear discussions of he Hamilonian approach Barro & Sala-i-Marin, Blanchard & Fischer, D. Romer), bu o my surprise, I didn see any clear reamen of he Bellman principle. Wha I saw so far Duffie, Chapers 3 and 9, and he Appendix of Mas-Colell, Whinson & Green) is confusing o me. I guess I should ake a look a some dynamic opimizaion exbook, bu I m oo lazy for ha. Insead, I m going o ry o figure i ou on my own, hoping ha my freshness on he subjec can be pu o use. The firs four secions are only abou local condiions for having a finie) opimum. In he las secion I will discuss global condiions for opima in Bellman s framework, and give an example where I solve he problem compleely. As a bonus, in Secion 5, I use he Bellman mehod o derive he Euler Lagrange equaion of variaional calculus. 1. Discree ime, cerainy We sar in discree ime, and we assume perfec foresigh so no expecaion will be involved). The general problem we wan o solve is max f, k, c ) 1) c ) =0 s.. k +1 = g, k, c ). In addiion, we impose a budge consrain, which for many examples is he resricion ha k be evenually posiive i.e. lim inf k 0). This budge consrain excludes explosive soluions for c, so ha we can apply he Bellman mehod. I won menion he budge consrain unil he las secion, bu we should keep in mind ha wihou i or some consrain like i), we migh have no soluion. The usual names for he variables involved is: c is he conrol variable because i is under he conrol of he choice maker), and k is he sae variable because i describes he sae of he sysem a he beginning of, when he agen makes he decision). In his paper, I call he equaion k +1 = g, k, c ) he sae equaion, I don know how i is called in he lieraure. To ge some inuiion abou he problem, hink of k as capial available for producion a ime, and of c as consumpion a. A ime 0, for a saring level of capial k 0, he consumer chooses he level of consumpion c 0. This deermines he level of capial available for he nex period, k 1 = g0, k 0, c 0 ). So a ime 1, he consumer decides on he level of c 1, which ogeher wih k 1 deermines k 2, and he cycle is repeaed on and on. The infinie sum =0 f, k, c ) is o be hough of as he oal uiliy of he consumer, which he laer is supposed o maximize a ime 0. Bellman s idea for solving 1) is o define a value funcion V a each = 0, 1, 2,... V, k ) = max fs, k s, c s ) s.. k s+1 = gs, k s, c s ), c s) s= Dae: April 8, 2002. 1

2 IOANID ROSU which represens he consumer s maximum uiliy given he iniial level of k. Then we have he following obvious resul Theorem 1.1. Bellman s principle of opimaliy) For each = 0, 1, 2,... [ 2) V, k ) = max f, k, c ) + V + 1, g, k, c ) )]. c This in principle reduces an infinie-period opimizaion problem o a wo-period opimizaion problem. Bu is his he whole sory? How do we acually solve he opimizaion problem 1)? Here is where he exbooks I menioned above are less clear 1. Well, le s ry o squeeze all we can ou of Bellman s equaion 2). We denoe parial derivaives by using subscrips. A sar superscrip denoes he opimum. Then he firs order condiion from 2) reads 3) f c, k, c ) + V k + 1, g, k, c ) ) g c, k, c ) = 0. Looking a his formula, i is clear ha we would like o be able o compue he derivaive V k + 1, k +1 ). We can ry o do ha using again he formula 2). Since we are differeniaing a maximum operaor, we apply he envelope heorem 2 and obain 4) V k, k ) = f k, k, c ) + V k + 1, g, k, c ) ) g k, k, c ). From 3), we can calculae V k + 1, g, k, c ) ), and subsiuing i in 4), we ge 5) V k, k ) = f k f ) c g k, k, c ). g c Finally, subsiue his formula ino 3) and obain a condiion which does no depend on he value funcion anymore: 6) f c, k, c ) + g c, k, c ) f k f ) c g k + 1, gk, c ), c g +1) = 0. c Noice ha his formula is rue for any k, no necessarily only for he opimal one up o ha poin. Bu in ha case, c and c +1 are he opimal choices given k. In any case, from now on we are only going o work a he opimum, k, c ). The previous formula can be wrien as follows: 7) f k + 1) f c + 1) g c + 1) g k + 1) = f c) g c ). This is he key equaion ha allows us o compue he opimum c, using only he iniial daa f and g ). I guess equaion 7) should be called he Bellman equaion, alhough in paricular cases i goes by he Euler equaion see he nex Example). I am going o compromise and call i he Bellman Euler equaion. For he purposes of comparison wih he coninuous-ime version, wrie g, k, c ) = k + h, k, c ). Denoe φ = φ+1) φ). Then we can rewrie he Bellman Euler equaion 7) 8) fc ) h c ) ) = f k + 1) f c + 1) h c + 1) h k + 1). 1 MWG only discusses he exisence and uniqueness of a value funcion, while Duffie reas only he Example menioned above, and leaves a crucial Lemma as an exercise a he end of he chaper. 2 To sae he envelope heorem, sar wih a funcion of wo variables fx, θ), such ha for every x, he maximum max θ fx, θ) is achieved a a poin θ = θ x) in he inerior of he θ-inerval. Then d dx max fx, θ) = f θ x θ x)).

THE BELLMAN PRINCIPLE OF OPTIMALITY 3 Example 1.2. In a ypical dynamic opimizaion problem, he consumer has o maximize ineremporal uiliy, for which he insananeous feliciy is uc), wih u a von Neumann Morgensern uiliy funcion. Therefore, f, k, c ) = β uc ), and 0 < β < 1 is he discoun consan. The sae equaion k +1 = g, k, c ) ypically is given by 9) e + φk ) = c + k +1 k ), where e is endowmen e.g. labor income), and φk ) is he producion funcion echnology). As an example of he echnology funcion, we have φk ) = rk. The derivaive φ k ) = r is hen, as expeced, he ineres rae on capial. Noice ha wih he above descripion we have k +1 = g, k, c ) = k + φk ) + e c. So we ge he following formulas: c = u c ), Bellman Euler equaion 7) becomes which is he usual Euler equaion. f g c = 1, u c ) = β 1 + φ k ) ) u c +1 ), 2. Discree ime, uncerainy f k = 0, Now we assume everyhing o be sochasic, and he agen solves he problem max 10) E 0 f, k, c ) c ) =0 s.. k +1 = g, k, c ). g k = 1 + r. The As usual, we denoe by E he expecaion given informaion available a ime. Then we can define he value funcion V, k ) = max E fs, k s, c s ) s.. k s+1 = gs, k s, c s ). c s) s= The Bellman principle of opimaliy 2) becomes [ 11) V, k ) = max f, k, c ) + E V + 1, g, k, c ) )]. c Now in order o derive he Euler equaion wih uncerainy, all we have o do is replace V + 1) in he formulas of he previous secion by E V + 1) using, of course, he fac ha differeniaion commues wih expecaion). We arrive a he following Bellman Euler equaion 12) E f k + 1) f c + 1) g c + 1) g k + 1) For our paricular Example 1.2, we ge ) u c ) = β1 + r)e u c +1 ), 3. Coninuous ime, cerainy = f c) g c ). This is a bi rickier, bu he same derivaion as in discree ime can be used. The difference is ha insead of he inerval, + 1 we now look a, + d. The problem ha he decision maker has o solve is 13) max f, k, c ) c 0 s.. dk d = h, k, c ).

4 IOANID ROSU The consrain can be rewrien in differenial noaion 14) k +d = k + h, k, c )d, so we have a problem similar in form o 1), and we can solve i by an analogous mehod. Define he value funcion V, k ) = max c s) fs, k s, c s ) ds The Bellman principle of opimaliy saes ha 15) V, k ) = max c [ +d s.. dk s ds = hs, k s, c s ). fs, k s, c s ) ds + V + d, k + h, k, c )d )]. We know ha +d fs, k s, c s ) ds = f, k, c )d. The firs order condiion for a maximum is 16) f c )d + V k + d, k +d ) h c ) d = 0. This is equivalen o 17) V k + d) = f c) h c ). As we did in he firs secion, apply he envelope heorem o derive 18) V k ) = f k )d + V k + d) 1 + h k ) d ). Subsiue 17) ino 18) o obain V k ) = f c) h c ) + f k ) f ) c) h c ) h k) d. If φ is any differeniable funcion, hen φ + d)d = φ)d, so we ge he formula 19) V k + d) = f c + d) h c + d) + f k ) f ) c) h c ) h k) d. Puing equaions 17) and 19), we ge f c + d) h c + d) f c) h c ) = f k ) f ) c) h c ) h k) d. Using he formula φ + d) φ) = dφ d d, we can rewrie he above formula as ) d fc ) 20) = f k ) f c) d h c ) h c ) h k) This is he Bellman Euler equaion in coninuous-ime. One can see ha i is prey similar o our equaion 8) in discree ime. We now have o be more careful wih our noaion. By f c ) in he above formula, wha we really mean is f c, k, c ) as usual, calculaed a he opimum, bu I m going o omi he sar superscrip). Then we calculae d d f c) = f c + f kc h + f cc dc d. So we can rewrie he Bellman Euler equaion 20) as follows: 21) f c + f kc h + f cc dc ) = f c d h c h c + h kc h + h cc dc d h c h k ) h c f k.

THE BELLMAN PRINCIPLE OF OPTIMALITY 5 In general, in order o solve his, noice ha we can rewrie 21) as dc /d = λ, k, c ), so he opimum is given by he following sysem of ODE s { dc /d = λ, k 22), c ) dk /d = h, k, c ). Example 3.1. Applying he above analysis o our favorie example, we have f, k, c ) = e ρ uc ) and dk /d = h, k, c ) = e + φk ) c. The Euler equaion 20) becomes d ) e ρ u c ) = e ρ u c )φ k ), d or equivalenly [ ] 23) u c )c u dc /d = φ k ) ρ. c ) c Noice ha we ge he same equaion as 7 ) from Blanchard and Fischer Chaper 2, p. 40), so we are on he righ rack. 4. Coninuous ime, uncerainy Firs, we assume ha he uncerainy comes from he funcion h for example, if h depends on an uncerain endowmen e ). In he second par of his secion, we are going o assume ha he consrain is sochasic. Bu for now, he agen solves he problem 24) The value funcion akes he form V, k ) = max c s) max E 0 f, k, c ) c 0 s.. dk d = h, k, c ). fs, k s, c s ) ds and he Bellman principle of opimaliy 15) becomes 25) V, k ) = max c [ +d s.. dk s ds = hs, k s, c s ), fs, k s, c s ) ds + E V + d, k + h, k, c )d )]. We arrive a he following Bellman Euler equaion ) d fc 26) E = f k f c h k. d h c For our paricular Example 1.2, we ge [ ] 27) u c )c dc u E /d = φ k ) ρ. c ) c Now we assume everyhing is sochasic, and he agen solves he problem max E 28) 0 f, k, c ) c 0 s.. dk = α, k, c )d + β, k, c )dw, h c where W is a one-dimensional Wiener process Brownian moion), and α, β are deerminisic funions. If we define he value funcion V, k ) as above, he Bellman principle of opimaliy implies ha 29) [ ] V, k ) = max f, k, c ) d + E V + d, k +d ) c.

6 IOANID ROSU Now V +d, k +d ) = V +d, k +αd+βdw ) = V, k )+V d+v k αd+βdw )+ 1 2 V kkβ 2 d, where he las equaliy comes from Iô s lemma. Taking expecaion a, i follows ha 30) E V + d, k +d ) = V, k ) + V d + V k α d + 1 2 V kkβ 2 d. The Bellman principle 29) can be wrien equivalenly as c ) sup V, k ) + f, k, c )d + E V + d, k +d ) 0, wih equaliy a he opimum c. Puing his ogeher wih 30), we ge he following 31) sup D a V, y) + f, y, a) = 0, a where D a is a parial differenial operaor defined by 32) D a V, y) = V, y) + V y, y)α, y, a) + 1 2 V yyβ, y, a) 2. Equaion 31) is also known as he Hamilon Jacobi Bellman equaion. We go he same equaion as Duffie, chaper 9A, so we re fine.) This is no quie a PDE ye, because we have a supremum operaor before i. However, he firs order condiion for 31) does give a PDE. The boundary condiion comes from some ransversaliy condiion ha we have o impose on V a infiniy see Duffie). Noice ha for he sochasic case we ook a differen roue han before. This is because now we canno eliminae he value funcion anymore he reason is ha we ge an exra erm in he firs order condiion coming from he dw -erm, and ha erm depends on V as well). So his approach firs looks a a value funcion which saisfies he Hamilon Jacobi Bellman equaion, and hen derives he opimal consumpion c and capial k. 5. The Euler Lagrange equaion The reason I m reaing his variaional problem here is ha he Bellman mehod seems very well suied o solve i. The classical Euler Lagrange equaion ) d F 33) = F d ċ c solves he following problem 34) max c) b a F, c, ċ)d s.. ca) = P and cb) = Q. Suppose we know he opimum curve c only up o some poin x = c). Then we define he value funcion V, x) = max cs) b F s, c, ċ)ds s.. c) = x and cb) = Q. In he discussion ha follows, we denoe by λ a direcion in R n he curve c also has values in R n ). The Bellman principle of opimaliy says ha 35) V, x) = max λ +d F, x, λ)d + V +d x + λd). The las erm comes from he ideniy c + d) = c) + ċ)d = x + λd. The firs order condiion for his maximum is afer dividing hrough by d) dv +d dc = Fċ.

THE BELLMAN PRINCIPLE OF OPTIMALITY 7 The envelope heorem applied o equaion 35) yields dv dc = F cd + dv +d dc Replace by + d in he above equaion o ge = F c d Fċ. dv +d = F c d Fċ + d). dc Puing ogeher he wo formulas we have for dv +d /d, we obain Fċ + d) Fċ) = F c d. This implies ) d d F ċ d = F c d, which afer canceling d is he desired Euler Lagrange equaion 34). 6. Condiions for a global opimum There are wo main issues in he exisence of he soluion. One is wheher he value funcion is finie or no. This will be aken care of by he budge consrain, as we will see below. The oher issue is wheher or no he maximum in he Bellman principle of opimaliy is aained a an inerior poin in order o ge he firs order condiion o hold). This las issue should be resolved once we analyze he soluion, and see ha we have indeed an inerior maximum. Of course, afer we know ha he Bellman Euler equaion holds, we need o do a lile bi of exra work o see which of he possible soluions fis he iniial daa of he problem. Since I don wan o develop he general heory for his one can look i up in a exbook), I will jus analyze our iniial Example 1.2. Recall ha we are solving he problem max c ) β uc ) =0 s.. k +1 = 1 + r)k + e c. We assume ha he endowmen is always posiive and has a finie presen value β e <. We impose he requiremen ha he capial k be evenually posiive, i.e. ha lim inf k 0. For simpliciy, assume also ha β1 + r) = 1. Sar wih an opimum sequence of consumpion c ). Such a sequence clearly exiss, since we are looking for a c ) ha achieves he maximum uiliy a zero finie or no). Noe ha we don assume anyhing paricular abou i; for example we don assume ha he value funcion corresponding o i is finie, or ha i aains an inerior opimum. Now we sar o use our assumpions and show ha c ) does saisfy all hose properies. Using he sae equaion, we deduce ha for sufficienly large so ha k +1 becomes posiive, hence bigger han 1), c 1 + r)k + e + 1. Wrie his inequaliy for all s > and muliply hrough by β s. Denoe by P V c) = s βs c s, he presen value of consumpion a ime, and similarly for k and e. Adding up he above inequaliies for s, we ge 36) P V c) 1 + r)p V k) + P V e) + 1 1 β. We assumed ha P V e) <. Le s show ha P V c) <. If P V k) <, we are done. If P V k) =, hen we can consume ou of his capial unil P V k) becomes finie. Cerainly, by doing his, P V c) will increase, ye i will sill saisfy 36). Now all he erms on he righ hand side of 36) are finie, hence so is P V c), a he new level of consumpion. Tha

8 IOANID ROSU means ha our original P V c) mus be finie. Moreover, i also follows ha P V k) is finie. Suppose i isn. Then, we can proceed as above and increase P V c), while sill keeping i finie. Bu his is in conradicion wih he fac ha c ) was chosen o be an opimum. I have o admi ha his par is ricky, ) bu we can avoid i if we wan o be rigorous!) Since u is monoone concave, P V uc) is also finie. Tha means ha he Bellman value funcion V, k ) is finie, so our firs concern is resolved. We now look a he Bellman principle of opimaliy [ V, k ) = max β uc ) + V + 1, 1 + r)k + e c ) )]. c Here s a suble idea: if we save one small quaniy ɛ oday, omorrow i becomes 1 + r)ɛ, and we can eiher consume i, or leave for laer. This laer decision has o be made in an opimum fashion since his is he definiion of V +1 ), so we are indifferen beween consuming 1 + r)ɛ omorrow and saving i for laer on. Thus, we migh as well assume ha we are consuming i omorrow, so when we calculae V +1, only omorrow s uiliy is affeced. Therefore, for marginal purposes we can regard V +1 as equal o β +1 uc +1 ). Then, o analyze he Bellman opimum locally is he same as analyzing [β uc ɛ) + β +1 u c +1 + 1 + r)ɛ )] max ɛ locally around he opimum ɛ = 0. Since β1 + r) = 1, he Bellman Euler equaion is u c ) = u c +1 ) or equivalenly c = c +1 = c. Because u is concave, i is easy o see ha ɛ = 0 is a local maximum. Tha akes care of our second concern. We noice ha consumpion should be smoohened compleely, so ha c = c for all. Recall he sae equaion: e c = k +1 1 + r)k. Muliplying his by β = 1/1 + r) and summing over all, we ge β e c ) = 1 ) k+1 1 + r) k = lim β k +1 1 + r)k 0. During he discussion of he finieness of he value funcion, we noiced ha P V k) <. In paricular, his implies ha lim β k = 0. So we ge he formula β e 1 1 β c = 1 + r)k 0, from which we deduce ha 37) c = rβ β e + rk 0. =0 Since here is only one opimum, i mus be a global opimum, so his ends he soluion of our example.