Eon. 511b Sprng 1993 C. Sms I. Th Opmaon Problm CONTINUOUS TIME DYNAMIC PROGRAMMING W onsdr h problm of maxmng subj o and EU(C, ) d (1) j ^ d = (C, ) d + σ (C, ) dw () h(c, ), (3) whr () and (3) hold for all n (, ). In solvng h problm w ak as gvn and known, whl h pahs of C and hrby for > ar subj o ho. W assum ha C and pahs mus b hosn so ha dsons mad a dpnd only on nformaon avalabl a. Tha s, our ho a mus b xprssd as a mappng from wha w wll know a, whh s {C s, s,w s, s }, o a ral vor C of hos mad a. Of ours () hn drmns s bhavor a. W assum and C ar boh vors, so σ s a marx. II. Rursv Sruur Th problm dfnd by (1)-(3) sars a m. W ould as asly dfn h problm as sarng a som arbrary da. Furhrmor, for vry nal valu of for whh h problm has a soluon, hr wll b som orrspondng valu of h objv funon, V(). Tha s, hr s a funon V() = EU(C *, ) -βs ds, (4) j +s +s whr h * ndas ha C * s hosn opmally. Th funon V s alld h valu funon. No ha h opmal ho of C n opmng (4) mus dpnd only on, no on ohr nformaon avalabl a, sn (4) slf and h ons- 1
rans do no pnd on h pas xp hrough. W wr C * =γ( ), and γ s h poly funon solvng h dynam programmng problm. Now obsrv ha (1) an b wrn as T EU(C, ) d =EU(C, ) d + (5) j j -βt EU(C, ) -βs ds. j T+s T+s T Th las rm on h rgh of (5) has h sam sruur as h orgnal ngral on h lf of (5), xp ha s shfd n m. Sn our hos of C T+s for s> hav no ff on h valu of h frs ngral on h rgh of (5), w an maxm h ovrall ngral n wo sps. Frs, maxm h sond rm on h rgh of (5), akng T as gvn. Thn, akng aoun of how T affs wha uly s obanabl from h sond rm on h rgh of (5), maxm ovr h (,T) nrval. Ths s Bllman s prnpl and an b sad mor onsly as h assron ha h problm (1)-(3) s uvaln o h problm of maxmng T EU(C, ) d + -βt V. (6) j T wh rsp o C and subj o () and (3). III. Condons Drmnng an Opmum Now assumng ha, undr h opmal ho of C, h objv funon (1) has a wll-dfnd fn valu, w an dfn a nw sohas pross Z = EU(γ, ) -βs ds. (7) j s s Th pross Z, bng of h form E X, s by onsruon a marngal and hrfor has
^ Z =. (8) Bu now usng (6) and Io s lmma, w an form anohr xprsson for Z,.. 1 Z = U(γ(),) - βv() + D V +----- rσ D V σ =. (9) 9 9 Of ours o form (9) w mus assum ha V s w dffrnabl. Ths provds us wh a nssary ondon for V o b h valu funon and γ o b h opmal poly funon, namly U(γ(),) - βv() + D V 1 +----- rσ D V σ =, (1) 9 for all possbl valus of. Euaon (1) by slf s no an uaon w an solv for V or γ. solubl uaon, w apply Bllman s uaon, whh s for hs sup To arrv a a 1 max { U-βV+D V +----- rσ D V σ } =. (11) h(c,) 9 9 If, for som V and γ, (11) and (1) hold for vry possbl valu of, hn (subj o som rgulary ondons) V s h valu funon and γ s h opmal poly. To s hs, suppos φ s a sohas pross for C ha maks dpnd only on pas nformaon and, ha, whn s usd n () o gnra, sasfs (3). L and ~ = (1) ~ ~ ~ d = (φ,) d + σ (φ,)dw. (13) Furhr l Q = ~ -βs ~ U(φ, ) ds + V. (14) j s s No ha Q =V and ha 3
EQ ~ -βs L EU(φ, ) ds. (15) j s s In assrng (15) w ar assumng an mporan rgulary ondon ha aually fals n som vrsons of h frs smpl xampl w wll onsdr blow, h lnar-uadra prmann nom modl. Th ondon s ha for any fasbl Now E V ----------L. (16) L ~ 1 Q= U(φ,) - βv +D V+----- rσ D σ V, (17) 9 9 whr h mpl C and argumns n (17) ar φ and. ~ Obvously f (11) holds for all possbl valus of, Q, all. Ths, wh (15), mpls ha φ as h ho for C ylds a valu of h objv funon no grar han V( ). Sn hs holds for any Φ, V( ) s n fa h maxmum aanabl valu of h objv funon and s aand wh h poly rul γ. No ha whl (11) s a sandard, rlavly asy o rmmbr form for h Bllman uaon, supprsss, for h sak of smpl noaon, h mporan pons ha U, σ and ar boh funons of C and, whl V dpnds on alon. If U,, and σ ar dffrnabl n C, (11) mpls h frs-ordr ondon D U(C,) + D V D (C,) +.5 rd V D σ (C,)σ (C,) C C = µ D h, (18) C C 9 whr µ s a uhn-tukr mulplr ha vanshs for h(c,)<. For h as of a on-dmnsonal (18) aks h smplr form D U(C,) + V D (C,) + V σ (C,) D σ (C,) = µ D h. (19) C C C C Euaon (18) or (19) an b solvd, n prnpl, for C as a funon of, V, and V. Subsung hs bak no (1) gvs us a dffrnal uaon n V, V, 4
V, and alon. Ths s a sond ordr dffrnal uaon (paral dffrnal uaon n h as of a non-salar ) n V, and an n prnpl b solvd, gvn appropra boundary ondons. Ofn n fndng a soluon s hlpful, nsad of ombnng (1) wh (18) o oban a dffrnal uaon n V, o ombn (18) wh h drvav of (1) wh rsp o. Ths s usful baus ofn h rsulng sysm an b rdud o a dffrnal uaon n γ, whh s ofn mor drly usful han V. Also, h drvav of (1) wh rsp o has an nrpraon as a dffrnal uaon n m (rahr han ) ha an somms b ombnd wh () o oban a sysm of dffrnal uaons wh rsp o m n C and. Ths may allow us o prod drly o soluons for or hararaons of C and as funons of m, whh may agan b of mor dr nrs han hr V or γ. Dffrnang (1) wh rsp o gvs D U - βd V + D V D +.5rD V D σ σ () 9 9 + D V +.5 rd V σ σ 9 +D Uγ +D VD γ +.5rD V D σ σ C C γ =. C 9 9 No ha whn µ=, (18) abov mpls ha h las ln of () (all h rms nvolvng γ ) s dnally ro. Whn µ, h onsran h(γ(),)= holds, so D C h γ = -D h. (1) Thus h rms on h las ln of () always ar ual o -µd h. Ths fa and h rsulng smplfaon of () s wha s known as h nvlop horm. I lavs us wh D U - βd V + D V D +.5rD V D σ σ () 9 9 +D V+.5rD V σ σ =-µd h. 9 No ha Io s lmma mpls ha h las wo rms on h lf of () ar xaly D V, so ha () an b wrn 5
-D V = D U - βd V + D V D +.5rD V D σ σ + µ D h (3) 9 9 In fa, f w dfn λ=d V, w an rwr (18) and (3) as D U + λ D +.5 rd λ D σ σ C C - µ D h = (4) C C 9 -λ =D U-βλ + λ D +.5rD λ D σ σ + µ D h. (5) 9 9 In h drmns as, whr σ, or n any ohr as whr σ s onsan, (), (4) and (5) form h usual Hamlonan frs-ordr ondons. Whl hs sysm s sll oasonally of som us n nrprng sohas problms, s no so drly usful baus of h apparan of D λ n h sohas vrson. Ths prvns h sysm from bng nrprd as a s of dffrnal uaons n h m pahs of C, and λ. IV. Th Lnar-Quadra Prmann Inom Modl L s apply wha w v dvlopd n h prdng sons o h onnuous m vrson of h sandard prmann nom modl. W onsdr h problm of maxmng subj o Spalng (4) and (5) o hs as ylds Combnng (8) and (9) gvs us E(C-.5C ) d (6) j ------- da = (ra + Y - C) d + σ dw. (7) 1-C=-λ (8) -λ = (r-β)λ. (9) C = (r-β)(1-c) (3) 6
whh n h ladng as of r=β produs Hall s onluson ha onsumpon s a marngal. Rwrng (8) n rms of γ() (o g bak o (3) as appld o hs problm) gvs us γ A+.5γ σ = (r-β)(1-γ), (31) an ordnary (nonlnar) sond-ordr dffrnal uaon n γ. Euaon (31) has a las wo vry smpl soluons. On s γ() 1. Ths s h poly of sng C a s saaon lvl forvr. Th ohr maks γ lnar, so ha γ = and γ=a+ba for som a and b. I s asy o hk ha (31) hn mpls ------- b (ra + Y - a - ba) = (r - β) (1 - a - ba). (3) From (3) w onlud ha f b, b=r-β. In ha as w an onlud furhr ha ------- ------- a=(β/r-1)y+β/r-1. In h spal as β=r, hs rdus o h rul C=rA+Y. Ths wo soluons boh dsplay wha s alld rany uvaln. Tha s, baus h rm n σ, h only on affd by h prsn of unrany, dsappars from (31) for hs soluons, hy ar soluons also o h vrson of h problm ha has σ =. W ould hav found hs soluons by gnorng h prsn of unrany, and hy would nonhlss hav bn orr whn unrany was nrodud. Ths rsul s obvously a spal as. I always arss whn U s uadra n s argumns, s lnar n s argumns, and hr s no sd onsran h, h lnar-uadra as. [A ompl vrson of hs nos would go on o dsuss how h prsn of wo soluons ha work for all an b ronld wh h opmaly prnpl and wha o mak of all h ohr, nonlnar soluons o (31).] 7