O Q L N. Discrete-Time Stochastic Dynamic Programming. I. Notation and basic assumptions. ε t : a px1 random vector of disturbances at time t.

Transcription

1 Econ. 5b Spring 999 C. Sims Discree-Time Sochasic Dynamic Programming 995, 996 by Chrisopher Sims. This maerial may be freely reproduced for educaional and research purposes, so long as i is no alered, his copyrigh noice is reproduced wih i, and he copies are no sold. I. Noaion and basic assumpions We consider a problem defined in erms of : a ime index, wih ineger values C : a kx vecor, called he conrol vecor S : an nx vecor, called he sae vecor Γ() : a mapping from sae vecor values o subses of R k, defining consrains on he choice of C : he informaion se a, consising of Cs (), Ss (),(), ε s all s ε : a px random vecor of disurbances a ime. The objecive is o maximize l E N M = l β U( C, S ) by choice of C, S+, =,..., q. We assume ha he infinie sum inside he brackes in () is well-defined for each choice of C s ha saisfies he consrains below and ha he expecaion of he sum is well-defined for each such choice of C s. The choice of C s is consrained in four ways: A) S is given, no subjec o choice; B) for each =,...,, S is deermined from pas hisory and curren ε according o = S f( C, S, ε ). (2) C) for each, C is consrained o lie in he se Γ( S ); D) for each, C is allowed o depend only on informaion in, and only in such a way ha C and U( C, S) are well-defined random variables. Noe ha (2) means ha, hough we describe he problem as ha of choosing boh C and S o maximize he objecive subjec o (A)-(D), effecively we choose only C, since a each, once C is chosen, S + is deermined by (2). Noe also ha (D) means ha he formal mahemaical problem here is no choosing a sequence of numbers lq, C bu choosing a sequence of funcions P q ()

2 nc () s such ha a each dae, C ( ω) = C Cs, Ss, εs, all s informaion se ino our bes choice for C. cl q h maps our posiion ω in he To complee he specificaion we need assumpions on he random disurbances ε. The sandard dynamic programming framework requires ha E) for each, ε + is independen of C and of all he random variables in 2 and ha F) he random variables lε, =,..., q are muually independen and idenically disribued (i.i.d.). This means ha no choice made before ime can influence he realizaion of he random variable ε. Noe ha his does no mean ha ε and C s are independen for s. Since ε`s daed earlier han s are in s, hey may influence our choice of C s ; and his will creae dependence in he join probabiliy disribuion of C s and ε for s. Now observe ha he range of probabiliy disribuions we can generae for values of C and S + for hrough our choice of C funcions depends on only via S. Since we are allowed o make he choice of C depend on anyhing in ha we like, we can creae dependencies beween he acual fuure values of S and C and, say, ε 3 if we like. Bu since his dependence can ake any form we like, and since he daa in are all fixed and known o us a he ime when we choose C, he range of disribuions for fuure C and S ha we can achieve does no depend on, excep hrough he fac ha S eners he version of (2) for =. We denoe by he se of all possible values of S, and we mean by calling all possible values of S ha no only is every value of S wih which we migh be confroned in, bu also for every S in and every way of choosing C's ha saisfies (A)-(D), S lies in for all wih probabiliy one. Thus for every S in here will be a unique, possibly infinie, leas upper bound for he aainable values of he objecive funcion. [Noe ha, hough he fuure C's and S's are unknown and random a ime, he objecive funcion includes an expecaion operaor, so is value is a number, no a random variable.] We denoe by V () he funcion mapping S's in ino he leas upper bound of achievable values of he objecive funcion. If he problem is well defined, V(S) exiss for each S in, hough i is imporan in pracice o check ha he infinie sum in () indeed converges for all feasible choices of acions. V is called he value funcion. The echnically sophisicaed reader may noe ha (D) direcly rules ou he kind of nonmeasurabiliy ha concerns Sokey and ucas in heir chaper Since C is required by (D) above o depend only on random variables in, (E) could omi he of C and phrase. 2

3 II. The principle of opimaliy: necessiy and sufficiency Theorem : (The Principle of pimaliy) Suppose ha V is he value funcion for he problem of maximizing () subjec o (A)-(F). Then for each S in, EVbf( CS,,εg exiss for all C in Γ( S ) (wih ±infiniy allowable values), and V( S) = l.u.b. U( C, S) + βe V f( C, S, ε Cin Γ( S) n b gs (3) Remark: In (3) we have omied daes on C, S, and ε, bu we mean here o ake he expecaion wih respec o he disribuion of ε, which is he same for all, and o rea S as nonrandom. If (3) is rue in his form for every S in, hen of course i will also be rue a every wih C, S and he E operaor given subscrips and ε given a + subscrip. For (3) o make sense, EVbC, S,εg mus be defined, hough possibly infinie, as he heorem assers. Proof: Noe ha he objecive () can be wrien as s + + = UC (, S) βe E β UC (, S ). (4) + N M The erm in (4) in brackes, ogeher wih he preceding E operaor, is exacly he same in form as (), excep wih all he ime subscrips advanced by. Since he consrains are all of he same form a all daes, he leas upper bound of his erm for a given value of S is V( S ). In a well-defined problem, (4), being he value of he objecive funcion, mus iself be well-defined for every S in and every feasible choice of acions. Bu one paricular feasible choice of acions is o choose C in Γ( S) arbirarily, hen o choose C for daes = and laer so ha he second addiive erm in (4), for every possible value of S = f( C, S, ε ), is a leas V( S) δ when V( S ) is finie and a leas /δ when V ( S ) is infinie, where δ is an arbirarily small posiive number. Wih his paricular way of choosing C s, we will have, herefore, when V( S ) is finie wih probabiliy one, R s l U UC (, S) βvs ( ) q UC (, S) βe β UC ( +, S+ ) [, δ]. (5) + S + T N M = P PV W Since he second erm in brackes is a random variable whose expecaion we know exiss, as i is he value of he objecive funcion for a feasible choice of acions, (5) bounds is firs erm in brackes above and below by random variables whose expecaions exis and are arbirarily close o each oher. Thus he expecaion of he firs erm exiss as well. When V( S ) is infinie wih non-zero probabiliy, our choice of C s gives an arbirarily large value of he objecive funcion, implying ha he firs erm, being bounded below by random variables wih arbirarily large expecaion, iself has a well-defined infinie expecaion. Now suppose (3) were no rue. Then eiher here would be, for some S in, a choice CS ~ ( ) of C making he righ-hand-side of (3) exceed V( S), or here would be some S in such ha he righ-hand side of (3) is bounded away from V( S) from below. Suppose ha (3) fails hrough 3

4 he righ-hand side being larger han V( S ~ ), where S ~ is he paricular value of S a which (3) fails. Now consider his way of choosing C's when S = S ~ : a ime, choose C = C ~ ( S ~ ) ; a all laer daes, choose C's according o a scheme ha makes he erm in brackes in (4) very close o V( S ). By doing so, we can make (4) as close as we like o he righ-hand side of (3). Bu hen we will have succeeded in choosing C s in such a way ha he objecive funcion value exceeds V( S ), a conradicion. If (3) fails he oher way, so ha for some S ~ he righ-hand side of (3) is bounded away from V( S ~ ) from below, a parallel argumen, again using (4) shows ha here is no way o choose C s o bring he objecive funcion value arbirarily close o V( S ~ ) when S = S ~, again a conradicion wih he definiion of V, which complees he proof. There is an addiional necessary condiion on he value funcion ha characerizes is long run rae of growh. Theorem 2: Suppose ha V is he value funcion for he problem of maximizing () subjec o (A)-(F). Then for every δ >, i is possible o choose a policy funcion C δ ( ) such ha, for each S in wih V(S) finie, he value of he objecive funcion aained using C δ ( ) is a leas V( S) δ, and he sequence of S, =,..., generaed by seing S = S and saisfies S = f( Cδ ( S ), S, ε ), =,..., (6) β EV ( S ). (7) Remark: In he (usual) special case where here is a policy funcion C ( ) ha acually generaes an objecive funcion value equal o V( S ) (raher han jus arbirarily close o i), (7) mus hold for he S sequence generaed by C ( ) from every iniial S for which V(S) is finie. Proof: Firs observe ha we can cerainly find C δ ( ). Since V saisfies (3) by Theorem, we can for each S ha delivers finie V(S) choose C ( S ) o saisfy δ δ d δ i. (8) U( C ( S), S) + βe V f( C ( S), S, ε V( S) ( β) δ If (8) holds for each S wih finie V(S), hen we can apply (8) o he erm in brackes on is own lef-hand side o obain UC ( δ( S), S) + βe UC ( δ( S), S) + βvs bg 2 VS ( ) ( β) δ ( β) δβ, (9) where we are assuming ha he S sequence is being generaed from (6). Repeaedly applying (8) his way will give us he desired conclusion, ha he realized value of he objecive funcion using C δ ( ) is a leas V( S) δ. 4

5 This resul suggess a mehod for solving hese problems: keep guessing forms for he V funcion unil we find one ha saisfies (3) for all S in. Since in checking wheher (3) is saisfied for all S, we will ordinarily be finding, for every S, he C ( S) ha maximizes he righ-hand side, we will have he policy rule C ( ) immediaely a hand when we have found he righ V. The problems wih his sraegy are, firs, ha i is hopelessly inefficien unil we find some sysemaic way o locae V's ha migh saisfy (3), and, second, ha so far we know only ha he V ha represens he maximum aainable objecive funcion value -- he value funcion of he problem -- saisfies (3). We have no ye shown ha here canno be oher funcions V ha also saisfy (3). I urns ou ha generally here are oher V's, besides he acual value funcion, ha saisfy (3), bu ha we can pick ou he rue value funcion by applying some addiional side condiions. Theorem 3: Suppose here is a funcion V addiion ha saisfies (3) for every S in and ha in i) for every S in here is a value C ( S) for C ha aains he maximum on he righ-hand side of (3) wih V = V ; ii) for every S in, if S, =,...,, is generaed from hen + + S = f( C ( ), S, ε ) () β E V S ( ) ; and () iii) for any V V ha solves (3) for every S in, here is some δ> such ha for any associaed C δ saisfying d i (2) δ δ U( C ( S), S) + βe V f( C ( S), S, ε V ( S) δ for every S in, if S, =,...,, are generaed from S = f( Cδ ( S ), S, ε ), =,...,, hen (3) lim β EV ( S ) =. (4) Then V is he value funcion for he problem. 3 Remark: The heorem assers ha if he necessary condiions of Theorems and 2 are me for V, hen V is he value funcion unless here is some oher soluion o (3) ha violaes (4). I 3 Condiion (iii) can easily be relaxed o require only ha lim inf β EV ( S ). The proof hen is longer, bu has essenially he same form. 5

6 is sraighforward, bu somewha edious, o exend his heorem o he case where no C is available and we insead have o sele for a sequence of C δ s ha approach he upper bound, as we used in Theorem 2. Proof: Suppose here is some soluion V V o (3). If for all S in, V ( S) V ( S), wih sric inequaliy for some S, hen V can be he value funcion. This follows because, since i saisfies (ii), V does represen an aainable value of he objecive funcion for each possible value of is argumen, so a policy ha delivers a lower V insead canno be opimal. Bu hen suppose ha, for some S, V ( S) V ( S) = γ >. Consider he following policy: for =,..., T, se C = C δ ( S) chosen as in (iii), hen for T se C = C ( S),. The value of he objecive funcion under his policy saring from S = S is N T T EM U( Cδ ( S), S) βp + β EV ( ST ) V ( S) T δ, (5) = where by making δ small enough we can make V δ ( S) as close as we like o V ( S). convergence in (5) follows from (iii) and he definiion of C δ. Bu noice ha δ (6) V ( S) = U( C ( S), S) + βe V ( S ) U( C ( S), S) + βe V ( S ) (Noe ha in (6) we are implicily reaing he wo S values as generaed by he C choice a ime zero in each expression -- so he S s in he wo expressions are no he same.) Applying he same argumen again o V ( S ) in (6), and so on recursively T imes allows us o conclude N M The T T E U( Cδ ( S), S) βp + β EV ( ST ) < V ( S). (7) = Bu wih δ arbirarily small, (5) and (7) ogeher imply V ( S) > V ( S), which conradics our iniial assumpion. So V ( S) V ( S) for all S, which we have already noed means ha V is no he value funcion. Since he argumen applies o arbirary V V, V is he unique opimal soluion and we have compleed he proof. Corollary: If U is bounded below, he necessary condiions of Theorems and 2 are also sufficien. Proof: Because he objecive funcion is discouned, i is bounded below when U is bounded below. This makes (4) hold auomaically. III.Value Ieraion The argumens of he previous secion poin o a concepually simple mehod for approximaing he value funcion and hence C, he opimal policy rule. The mehod is called, for reasons ha will be obvious, value funcion ieraion, and proceeds as follows. Begin by 6

7 guessing a form V for he value funcion. Then for each ieraion n, n=,2,3,..., se, for each S in, Vn( S) = l.u.b. U( C, S) + βe Vn ( f( C, S, ε)) C in Γ m r. (8) Coninue unil Vn( S) = Vn ( S), all S, o wihin some crierion for numerical accuracy. A ha poin a V saisfying he principal of opimaliy will have been arrived a, and he oher necessary and sufficien condiions can be checked. I is no necessarily rue ha value funcion ieraions converge, however. When U is unbounded eiher above or below, i can easily happen ha value funcion ieraion convergence fails. Since many of he sandard uiliy funcions of macroeconomic models -- logarihmic C γ UC ( ) = log( C) and CRR U( C)= for example -- fail o saisfy such a boundedness γ condiion, we mus generally be wary ha value ieraion migh no converge. I is worh knowing, hough, ha when U is bounded (which we already know is a sufficien condiion o guaranee ha a soluion o he opimaliy equaion is he value funcion) value ieraion necessarily converges. The argumen goes as follows. Firs we noe ha V ( S) V ( S) l.u.b. U( C, S) U( C, S) + βe V f( C, S, ε) V f( C, S, ε) n n b g b gs. (9) n n n 2 C in Γ( S) This follows from (8) and he fac ha l.u.b. a () l.u.b. b () l.u.b. a () b () If we inroduce as a norm on he space of V s l q l q m r. (2) V = sup V( S), (2) S (9) can be used o produce Vn Vn β Vn Vn 2. (22) This implies ha he sequence of value funcion ieraes lq V n is a Cauchy sequence on he space of bounded funcions, and hence ha i converges o some bounded funcion. The reason he argumen fails when U is unbounded is ha hen V is generally unbounded (no infinie -- i jus ges arbirarily large or small as we change is argumen S) and herefore does no allow us o use (2). Even if we sar wih a bounded V, he unboundedness of U generally can make V unbounded. IV. Consrains a Infiniy We could have se up our problem wih wo sors of addiional erms. The objecive funcion could have been expanded o he form EM β U( C, S) P + lim β E W( S ), (23) N = 7

8 and he consrains could have been expanded o include he requiremen ha lim β GS ( ). (24) Noe ha in (24) here is no expecaion operaor, so we are requiring ha he inequaliy hold wih cerainy. Wih hese addiions, he problem reains is recursive srucure and he Bellman equaion sill funcions in he same way as a necessary and (wih some side condiions) sufficien condiion for an opimum. You migh go hrough he argumens yourself sep by sep o be sure ha his is rue. V.Exogenous Saes In many economic models he assumpion off i.i.d. shocks ε, (F) in he firs par of he noes, is a source of some difficuly. The objecive funcion is ofen inerpreable as a uiliy funcion or profi funcion and he consrains (2) on he evoluion of he sae are ofen inerpreable as budge consrains or producion relaions. Bu in such objecs he serial dependence properies of "exogenous disurbances" is no naurally aken o be resriced. "Technology" in a producion funcion, for example, is ofen aken o drif upward in a serially correlaed way, and "income" in a consumer's opimizaion problem is naurally aken o be serially correlaed. Such siuaions can be accomodaed by including in he equaions for evoluion of he sae (2) a descripion of he serial dependence in he exogenous disurbances. In his case he exogenous componens of he problem migh be labeled Z, and heir sochasic evoluion described by an equaion of he form Z = g( Z, η ) (25) + + in which η saisfies boh (E) and (F). The sae vecor in he problem is hen aken o include boh endogenous saes, which we migh label K, and exogenous saes Z. The full sae vecor S = [ K Z ]. The equaions of evoluion given in (2) can hen be expanded o he specialized form K+ = f( C, K, Z, ε + ). (26) Z = g( Z, η ) + + Here η is playing he role of a componen of ε in he more general specificaion (2). We poin ou his special case here no only because i shows how serial dependence in sochasic componens of he model can be accommodaed wihin a dynamic programming seup, bu also because i ofen helps simplify finding and inerpreing soluions o recognize a srucure like ha in (25)-(26) when i is presen. VI. Firs rder Condiions The usual echniques of calculus, in paricular he Kuhn-Tucker condiions for an opimum, can be applied o he Bellman equaion (3) when U and V are differeniable and have appropriae concaviy properies and when he consrains defined in (C) also have differeniable forms. 8

9 V Slighly abusing noaion, we will use V o sand for DV S = for each value S of he sae vecor and each C in Γ( S ), in wha follows. Assume ha EV ( S) D f( CS,, ε ), DU( CS, ) (27) C are well-defined. Assume also ha he se Γ( S ) can be characerized by he inequaliy (or vecor of inequaliies) HCS (, ). (28) The usual Kuhn-Tucker heorem hen assers ha a necessary condiion for (3) o hold (assuming ha he lub in (3) is aained) is ha NM UCS (, ) f HCS (, ) + βev ( f( CS,, ε) λ( S) C CP = C, (29) wih λ( S ) > for hose elemens of λ corresponding o which he inequaliy in (28) is an equaliy, and λ( S ) = for hose where he inequaliy is sric. Recognizing ha opimal C is a funcion C ( S) of S, we can differeniae lef and righ-hand sides of (3) wih respec o S o obain NM U f = + P + + V S EV S S S C U E V f C S S C ( ) β ( ) β ( ( ( ),, ε) In wriing (3) we assume ha C is differeniable. Noice ha he erm in brackes following C is exacly he lef-hand side of (29). The fac ha his erm is zero in he case wih no consrains on C is a special case of a heorem called he envelope heorem, and (3) wih ha erm subsiued ou is someimes called he "envelope condiion" in dynamic programming jargon. C e j. (3) Because (3) involves he unknown funcion C, i is no direcly very useful. Using (29) we can conver (3) o he form NM P + U f HCS (, ) C ( S) V ( S) = + βev ( f( CS,, ε)) λ( S ) C. (3) Then we observe ha, if λ( S ) >, so he consrain ha H= is binding, HC ( ( S), S) C ( S) = DH ( C ( S), S) + DHC 2 ( ( S), S) =. (32) Using (32) in (3) hen gives us he usual form of he envelope condiion V U f = + EV f CS S S S HCS (, ) β b (,, ε) g λ( ). (33) NM Equaions (29) and (33) ogeher are someimes called he Euler equaions for he problem. Noe ha one way o remember hem is o form he Hamilonian-like expression P 9

10 V( S) U( C, S) βe V( f( C, S, ε)) + λ H( C, S) (34) Equaions (29) and (33) are hen he parial derivaives of (34) wih respec o C and S, respecively. When (28) holds wih equaliy, (28), (29) and (33) are a sysem wih as many equaions as he sum of he dimensions of he vecors C, S and λ. If i were no for he expecaion operaors in he sysem, we could solve i for V ( S), C and λ for any given S. Because of he expecaion operaor, we have o rea he sysem as a funcional equaion. ne way o use i compuaionally, for example, would be o posulae a funcional form for V and use he posulaed form in compuing he expecaions in (29) and (33). Then he sysem can ordinarily be solved joinly for C, λ and V ( S) for any given S, producing (if we solve i for many values of S) a new candidae guess for V. Ieraing his process we migh hope o arrive a a V funcion ha saisfies he equaions, and in he process a a C funcion ha defines opimal decisions. Mehods like his are, or can be made o be, numerically more efficien han value funcion ieraion when he problem is smooh enough o allow heir applicaion.