The generation of successive approximation methods for Markov decision processes by using stopping times

Transcription

1 The geerati f successive apprximati methds fr Markv decisi prcesses by usig stppig times Citati fr published versi (APA): va Nue, J. A. E. E., & Wessels, J. (1976). The geerati f successive apprximati methds fr Markv decisi prcesses by usig stppig times. (Memradum COSOR; Vl. 7622). Eidhve: Techische Hgeschl Eidhve. Dcumet status ad date: Published: 01/01/1976 Dcumet Versi: Publisher s PDF, als kw as Versi f Recrd (icludes fial page, issue ad vlume umbers) Please check the dcumet versi f this publicati: A submitted mauscript is the versi f the article up submissi ad befre peer-review. There ca be imprtat differeces betwee the submitted versi ad the fficial published versi f recrd. Peple iterested i the research are advised t ctact the authr fr the fial versi f the publicati, r visit the DOI t the publisher's website. The fial authr versi ad the galley prf are versis f the publicati after peer review. The fial published versi features the fial layut f the paper icludig the vlume, issue ad page umbers. Lik t publicati Geeral rights Cpyright ad mral rights fr the publicatis made accessible i the public prtal are retaied by the authrs ad/r ther cpyright wers ad it is a cditi f accessig publicatis that users recgise ad abide by the legal requiremets assciated with these rights. Users may dwlad ad prit e cpy f ay publicati frm the public prtal fr the purpse f private study r research. Yu may t further distribute the material r use it fr ay prfit-makig activity r cmmercial gai Yu may freely distribute the URL idetifyig the publicati i the public prtal. If the publicati is distributed uder the terms f Article 25fa f the Dutch Cpyright Act, idicated by the Tavere licese abve, please fllw belw lik fr the Ed User Agreemet: Take dw plicy If yu believe that this dcumet breaches cpyright please ctact us at: peaccess@tue.l prvidig details ad we will ivestigate yur claim. Dwlad date: 19. Mar. 2019

2 EINDHOVEN UNIVERSITY OF TECHNOLOGY Departmet f Mathematics PROBABILITY THEORY, STASCS AND OPERAONS RESEARCH GROUP Memradum COSOR The geerati f successive apprximati methds fr Markv decisi prcesses by usig stppig times by J.A.E.E. va Nue ad J. Wessels Eidhve, Nvember 1976 The Netherlads

3 The geerati f successive apprximati methds fr Markv decisi prcesses by usig stppig times J.A.E.E. va Nue ad J. Wessels Summary I this paper we will csider several variats f the stadard successive apprximati techique fr Markv decisi prcesses. It will be shw hw these variats ca be geerated by stppig times. Furthermre it will be demstrated hw this class f techiques ca be exteded t a class f value rieted techiques. This latter class ctais as extreme elemets several variats f Hward's plicy iterati methd. Fr all methds preseted extraplatis are give i the frm f MacQuee's upper ad lwer buds. 1. Itrducti I [lj we itrduced the stadard successive apprximati methd fr Markv decisi prcesses with respect t the ttal expected reward criteri. I fact there exist sme variats f this methd. These variats differ i the plicy imprvemet prcedure: the stadard prcedure may be replaced by a Gauss-Seidel prcedure (see e.g. Hastigs [10J, Kusher ad Kleima [13J), a verrelaxati prcedure (see Reetz[7J ad Schellhaas [9J) r sme ther variats (see Va Nue [4J). I [3J it has bee shw that such variats ca be geerated by stppig times. This apprach has bee geeralized i [2J. I secti 2 we will itrduce the mai idea f this apprach. Plicy iterati -with its several variats- as itrduced by Hward [12J is usually t viewed up as a successive apprximati techique. Hwever, i [5J it has bee shw t be a extreme elemet f a class f exteded successive apprximati techiques, the s-called value-rieted methds. This apprach has bee cmbied i [6J with the stppig time apprach. I [2J a further geeralizati has bee give (maily with respect t the cditis). Value-rieted methds will be treated i secti 3. Secti 4 will be devted t upper ad lwerbuds fr the techiques

4 - 2 - preseted i the earlier secti. Furthermre sme remarks umerical aspects will be made. I this paper we will use the same tatis as i [lj, hwever, i rder t keep the prfs simple, we will wrk uder smewhat strger assumptis. I fact, ur assumptis are the same as thse i [2J. Fr details we will refer repeatedly t [2J. Assumptis. Our assumptis are the same as the assumptis i [lj, with assumpti 2.3 (i) replaced by 00 (b) fr all i EO S. These strger assumptis make the spaces V ad W superfluus. As remarked i [lj (remark 5.1) e may replace r i the assumptis (ad defiiti f - V) by a vectr b with b - r EO W. We will d s i this paper i rder t facilitate referrig t [2J. 2. Stppig times ad successive apprximatis I this secti we will shw that each stppig time characterized by a gahead fucti 0 fr the sequece {X} a iduces a peratr Uf: = v V, such that U is mte ad (usually) ctractig. Furthermre all these ctractig peratrs V have the same uique fixed pit v * S we have fr ay va EO V ad ay 0: 00 fr = 1,2, ad v -+ v Defiiti 2.1. A (radmized) g ahead fuati 0 is a fucti which maps G 00 := 00 u Sk k=l it [O,lJ.

5 - 3 - By 6 we dete the set f all g ahead fuctis. 1 - (so,sl,,s) will be iterpreted as the prbability t stp the = s ad the prcess has prcess at time, give that X = so""'x t bee stpped earlier. Defiiti 2.2. (a) 0 E 6 is said t be radmized if (a) E {0,1} fr all (b) a E 6 is said t be zer if (i) > > 0 fr sme ad all i E S (c) 0 E 6 is said t be trasiti memryzess if a(a) ly depeds the last tw etries f a, fr thse a with at least tw etries ad satisfyig a(so,,sk) F 0 fr all k <, if a = so,..,s' S fr a trasiti memryless g ahead fucti the stppig prbability ly depeds the mst recet trasiti. The relevace f this ti will becme clear i the curse f this secti. Examples 2.1. Belw sme examples f zer g ahead fuctis will be give. These examples will be used repeatedly i this paper. (a) Defie the g ahead fucti 0 ( = 1,2, ) by a (a) := 1 if a ctais less tha + 1 etries, therwise a (a) := O. The g ahead fuctis a are radmized, a is ly trasiti memryless if = 1. {b) defie a R by a R (s,s,,s) fiite legth, ar(a) := 1 fr all s ad all sequeces f := 0 therwise. R is radmized ad trasiti memryless. (e) 0H is defied by ah(so"",s) := 1 if S < 51 <, < s (ay ), therwise ah(a) := O. 0H is radmized ad trasiti memryless. (d) a (i) = ~ fr all i E S, (a) := 0 elsewhere. r r is trasiti memryless. r

6 - 4 - Sice we itrduced a prbabilistic g ahead ccept, we have t icrprate it i the prbability space ad measure. Therefre we exted the space (S x A)~ (see [lj secti 2) t (S x E x A)~, with E := {a,l}. Furthermre ~... the stchastic prcess {xt,zt}t=a t {xt,yt,zt}t=o' where Y t as the prcess may g ahead. = a as lg Nw ay startig state i, ay g ahead fucti 0, ad ay decisi rule... ~ determie a prbability measure (S x E x A) with the required prperties i a bvius way (see [2J fr details). This prbability measure will be deted by JP ~, O. Expectatis will be deted by :IE ~, a. Nte ~ ~ that :II? ~, 0 ad 1I?~ ~ ~ are equal fr evets which d t deped the variables Y t I fact the g ahead ccept iduces a stppig time Defiiti 2.3. The radm variable t defied by takig values i {0,1,.~.,... } is t = ~ Y = = Y = a ad Y = 1-1 Y t = 0 fr all t = 0,1,. t is a radmized stppig time with respect t X O,X 1 ' Nw we will itrduce ur peratrs. Defiiti 2.4. Fr each 0 E ~ ad each strategy (= radmized decisi rule) ~ ~ the peratr La V is defied by if t =... fr v E V, with v(x) := a t Lemma 2.1. L6 is mte ad (fr zer 0) strictly ctractig V. Therefre L~ pssesses a uique fixed pit v~ i V.

7 - 5 - Prf. The ctracti factr f L6 is II P 6 () II =: p 6 ' where P 6 () is the matrix with (i,j) etry JP~,6('r <a,x. = j). P~<l if ad ly if 6 is zer. Examples. Take fr a arbitrary statiary strategy (f,f, ). (a) (L 6 v) (i) 1 = r(i,f(i» + Lpf(i) (i,j)v(j) ; j (b) (L 6 v) (i) R = [1 - pf(i) (i,i)]-l[r(i,f(i» + 2 pf(i) (i,j)v(j)] j7'i (c) v) (i) = r(i,f(i» H (L + L pf(i) (i,j) (L; v) (j) + 2 pf(i) (i,j)v(j) jj<i H j~i (d) (L v) (i) r = ~v(i) + ~[r(i,f(i» + 2 pf(i) (i,j)v(j)j j (e) let 6 be zer, the v 6 = v(), idepedet f 6. Remark 2.1. If is a statiary strategy the there exist values fr {pa(i,j), r(i,a)} ad g ahead fuctis 6' ad 0" such that v~, 7' v~" (see lemma i [2J). We w cme t the peratrs U 6 Defiiti 2.5. The peratr U V is defied by := sup L v, where the supremum is take cmpetwise. Nte that L has ly bee defied fr strategies, s the supremum is ly take ver the strategies (= radmized decisi rules). Extesi t the radmized decisi rules wuld t affect the value f Uv Therem 2.1. Let 0 E A, the U is mte ad (ly fr zer 0) strictly ctractig with ctracti radius 'V 0 := sup P~ Therefre U pssesses

8 - 6 - (fr zer 0) a uique fixed pit. v * is fixed pit fr all U with zer. Prf. Fr details we refer t the prf f therem i [2J. With respect t the last statemet we remark: if 7T = (f,f, ) Sice f may be chse such that V(7T) ~ v * we btai Uv * ~ v * If we had Uv * > v * I cstruct a strategy 7T' with V(7T * 1 ) > v - j.l ([lj therem 3.1 (ii)), the it wuld be pssible t This therem serves as the basis fr a -based succes.sive apprximati algrithm, sice v := U v _ cverges i rm t v 1 * if va C V. I the defiiti f U we take the supremum ver all strategies. Oe wuld aturally prefer t restrict eself t Markv strategies ad eve use the algrithm fr cstructig -ptimal statiary strategies. The fllwig therem (fr the prf we refer t [2J therem ad 5.2.3X shws that the ccept f trasiti memryless g ahead fuctis plays a crucial rle i this prblem. Therem 2.2. (a) Let 0 be trasiti memryless, > 0, V C V. The there exists a plicy f, such that (b) Let 0 be t trasiti memryless, the there exist values fr the parameters {pa(i,j), r(i,a)}, such that fr sme v c V ad sme there is f c F with f L0v ~ U0v - j.l Hece, if 0 is trasiti memryless we have

9 - 7 - where the sup is t ecessarily cmpetwise. Whereas if 0 is t trasiti memryless f sup LV may t be defied. Fr zer ad trasiti memryless g ahead fuctis we w btai the fllwig iterati prcedures (a) (if sup L~V is attaied fr sme f) f Chse V V, defie v := U v _ l ad chse f such that v the (i) Ilv - v* II s; \)0 * 11 v0 - v II (ii) Ilv - v (f ) II s; -1 (1 - \)0) vllv - v -l II (iii) v(f ) * if V satisfies Uv O ~ v' the v 1 s; v s; s; v - (b) Chse e > 0 ad V V with V s; Uv O - ev. Chse f defie ( = 1, ) such that f L V _ 1 ~ max{v _ l, U v _ l - e(l - \))V} the (i) IIv * - v II < e fr sufficietly large (ii) v 1 s; v s; v(f ) - s; v * I fact, as i the case f 0 1, mre efficiet lwer ad upperbuds ca be btaied (see secti 4).

10 - 8 - Examples 2.3. The examples 2.2 (a)-(b) iduce umerically well-executable plicy imprvemet prcedures. I fact 01 iduces the stadard successive apprximatis techique based Gauss-Jrda-iterati; R iduces Jacbi iterati (cmpare Prteus [14J); 0H yields Gauss-Seidel iterati; ther chices f yield verrelaxati ad cmbiatis f verrelaxati ad Gauss-Seidel iterati (i this respect lemma i [2J has iterestig csequ,eces). 3. Value rieted methds I the fregig secti we develped a whle class f plicy imprvemet prcedures r successive apprximatis techiques. As we saw i secti 2, at the -th stage f ay plicy imprvemet prcedure the best estimate fr the ptimal strategy is the statiary strategy f This makes the ext plicy imprvemet mre efficiet if L~e value v is earer t v(f ). I fact plicy iterati techiques we their high efficiecy i the plicy imprvemet part t the fact that they have v = v (f ). A dis-. advatage f plicy iterati is i fact the cmputati f these v Hwever, there is a arbitrary way cmbiig the advatages f plicy iterati ad successive apprximatis. Namely suppse that f such that is chse the defie v O. {1,2,,c}) Nte that f lim(l.1'/'v _ 1 = v(f ), A~ u s by the chice f A we i fact determie hw gd v apprximates v(f ). The chice A = 1 gives the successive apprximati f se9ti 2, whereas the chice A = c gives fr ay trasiti memryless ad ~zer g ahead fucti a variat f the plicy iterati techique. Belw we give a mre frmal treatmet.

11 - 9 - Defiiti 3.1. Let 0 be zer ad trasiti memryless ad suppse that the sup L~v is attaied fr sme plicy if v E V. Furthermre we f assume that we have a uique way f desigatig such a plicy. We defie the peratrs U~A) V fr A = 1,Z,,~ by U (A) v if the sup i Uv is attaied fr f. Nte that (~) f U v = lim (L ) v = v(f) -+-> ().) It des t seem revlutiary t cjecture that v := U v _ 1 cverges * t v if V E V. awever, e becmes smewhat mre prudet as s as e real;zes that U;A) u. ;s ej.'ther ecessarj. '1y mt e, r ecessarj. '1y ctractig as e ca see i the fllwig simple example fpr 0 = 0 1 ' S = {l,z}, II :: 1, A = {l,z}: pl(i,z) =p2(i,1) =0.99, r(i,l) =1, ther prbabilities ad rewards beig zer. Nw e btais fr v := (O,O)T, w whereas lim U~A)W = (O,a)T. ).-+-0> We will w prve that the prpsed iterati step leads t a cvergig algrithm. Therem 3.1. Let the situati be such that U(A) va E V with Uv a ~ va (A) The v := * U~ v 1 cverges i rm t v u - ad is defied ad chse v 1 ~ v ~ v(f ) ~ v* - where f f L v _ 1 is the plicy (uique, pssibly after tie breakig) which maximizes

12 Prf. By assumpti we have Hece Sice U v l = L (f 2 )v l ~ L (f 1 )v l, e btais v l S V z S v(f 2 ). By iducti this gives v 1 S v :s; v (f ) :s; v * O the ther had - * v ~ u~v' which teds t v* fr + ~. Therefre v + v ad IIv * :s; V IIv O - v II I the same way as i [lj fr the stadard algrithm e may btai Fre sphisticated buds (see secti 4). Furthermre the assumpti that the sup i Uv is attaied ca be weakeed as i [lj by itrducig apprximatis (i rm) f the sup. This ca be exteded i several ways. Fr a detailed descripti f these pssibilities see [2J. As already stated, the case A = ~ represets a variety f plicy iterati prcedures. I fact the prcedures (fr ay zer trasiti memryless 0) geerate sequeces f plicies with icreasig value. Hece a ptimal plicy is btaied after a fiite umber f iteratis if the state ad acti spaces are fiite. If 0 = 0, the we have the stadard plicy iterati algrithm as itrduced by Hward i [12J fr the fiite state, fiite acti discuted 1 case. If 0 = 0H' the we have the Gauss-Seidel variat as itrduced by Hastigs E1 0J 4. Sme remarks umerical ad ther aspects Fr the algrithms based the peratrs we prved gemetric cvergece. Hwever, (A) U (secti 2) ad U (secti 3) the extraplati based the cvergece rate ly are usually t very gd. As i the case f U 1 (see [lj) e ca btai better buds rather easily. Fr the case the sup i Uv is attaied ad exactly cmputed i the algrithm based U~ A) (A = 1,2,, ~) we btai, if U0v 0 ~ v0 :

13 * v +(1- Pf ) 1IL.r(f 1)V -v li$v(f 1)$v::; + 1 \) where Pf : = if II-1 (i) L p f (i) (i, j hd j), II v "-: = if ].I -1 (i) v (i). f j f Fr a mre detailed descripti we refer t [2J. The prf i this case is cmpletely similar t the prf i the case <5 = 0 1, Fr umerical experiece it appears that value rieted methds ca give a csiderable gai i cmputatial efficiecy. This is especially true if the plicy imprvemet prcedure requires may peratis. Geerally speakig e may say that U-based successive apprximatis methds ly eed a small umber f iteratis t reach a ear-ptimal plicy, hwever, the prf f this ear-ptimality requires relatively may additial iteratis. S i quite a lt f iteratis f des t chage substatially. Therefre it is efficiet t chse A greater tha e. I fact it is still mre prfitable t icrease the value f A i subsequet situatis. T give a idea f the gai i cmputatial efficiecy we meti that we fud i a umber f examples with 0 = 0 1 a savig i cmputig time f 20-40% whe we tk A = 5 istead f A = 1 (i bth situatis we used a subptimality test; the umbers f states raged betwee 40 ad 1000), see [4J ~ I all prcedures (all 0 ad all A) the stadard subptimality test is allwed ad als the mre sphisticated ad mre efficiet subptimality test which is described i the paper by Hastigs ad Va Nue [10J i this vlume. Istead f defiig -based peratrs U e may trasfrm the data i the prblem ad slvig the trasfrmed prblem by the stadard successive apprximati methds. This apprach has bee preseted by Prteus i [14J. I ur tati the trasfrmati is r (f) T-l := lef, Lr(X,Z ) =O

14 (see prf f lemma 2.1) By itrducig the matrices Q(f) with Q(f) (i,j) := pf(i,j)o(i,j) we btai c P (f) = L Qk (f)[p (f) - Q(f) J, k=o '" ref) = c L k=o k Q (f)r(f), beig exactly Prteus' preiverse trasfrmati. I fact we shwed i secti 2, that the trasfrmed prblem pssesses the same ptimal value vectr as the rigial prblem. I fact sme extesi is pssible with respect t the cditisude. (A) which the U - ad U -based prcedures cverge. We metied already O the kid f cditis f [1J. Ather apprach is i csiderig a fixed 0 ad require strict r N-stage ctracti fr U V r W. I [8J Reetz chses such a apprach fr 0 = 0a' Oe might cjecture that -as i the case f 0 (see [1J) - N-stage ctracti implies 1-stage 1 etracti with respect t a differet rm. Refereces [1J J.A.E.E. va Nue, J. Wessels, Markv decisi prcesses with ubuded rewards. I this vlume. [2J J.A.E.E. va Nue, Ctractig Markv decisi prcesses. Mathematical Cetre Tract 71, Amsterdam [3J J. Wessels, Stppig times ad Markv prgrammig. Trasactis f the seveth Praque Cferece Ifrmati Thery, Statistical Decisi Fuctis, Radm Prcesses (icludig 1974 Eurpea Meetig f Statisticias) Academia, Prague (t appear). [4J J;A.E.E. va Nue, A set f successive apprximati methds fr discuted Markvia decisi prblems. Zeitschrift fur Operatis Res. 20 (1976)

15 [5J J.A.E.E. va Nue, Imprved successive apprximati methds fr discuted Markv decisi prcesses. p i A. Prekpa (ed.), Prgress i Operati Research, Amsterdam, Nrth-Hllad Publ. Cmp [6J J.A.E.E. va Nue, J. Wessels, A priciple fr geeratig ptimizati prcedures fr discuted Markv decisi prcesses. p i the same vlume as [5J. [7J D. Reetz, Sluti f a Markvia decisi prblem by verrelaxati. Z.f.O.R. 4 (1973) [8J D. Reetz, A decisi exclusi algrithm fr a class f Markvia decisi prcesses. Zeitschrift fur Operatis Research, Vl. 20, 1976, [9J H. Schellhaas, Zur Extraplati i Markffsche EtscheidUgsmdelle mit Disktierug. Z.f.O.R. ~ (1974) [10J N.A.J. Hastigs, Sme tes dyamic prgrammig ad replacemet. Opere Res. Q. ~ (1968) [11J N.A.J. Hastigs, J.A.E.E. va Nue, The acti elimiati algrithm fr Markv decisi prcesses. I this vlume. [12J R.A. Hward, Dyamic prgrammig ad Markv decisi prcesses. Cambridge (Mass.), M.I.T.-Press, [13J H.J. Kusher, A.J. Kleima, Accelerated prcedures fr the sluti f discrete Markv ctrl prblems. IEEE-Tras. Aut. Ctr. A.C. ~ (1971) [14J E.L. Prteus, Buds ad trasfrmatis fr discuted fiite Markv decisi chais. Opere Res. ~ (1975)