Online Bellman Residual and Temporal Difference Algorithms with Predictive Error Guarantees

Transcription

1 Onlin Bllman Rsidual and mporal Diffrnc Algorihms wih Prdiciv Error Guarans Wn Sun and J. Andrw Bagnll Roboics Insiu, Carngi Mllon Univrsiy, Pisburgh, USA {wnsun, Absrac W sablish conncions from opimizing Bllman Rsidual and mporal Diffrnc Loss o worscas long-rm prdiciv rror. In h onlin larning framwork, larning aks plac ovr a squnc of rials wih h goal of prdicing a fuur discound sum of rwards. Our firs analysis shows ha, oghr wih a sabiliy assumpion, any no-rgr onlin larning algorihm ha minimizs Bllman rror nsurs small prdicion rror. Our scond analysis shows ha applying h family of onlin mirror dscn algorihms on mporal diffrnc loss also nsurs small prdicion rror. No saisical assumpions ar mad on h squnc of obsrvaions, which could b non- Markovian or vn advrsarial. Our approach hus sablishs a broad nw family of provably sound algorihms and provids a gnralizaion of prvious wors-cas rsuls for minimizing prdiciv rror. W invsiga h ponial advanags of som of his family boh horically and mpirically on bnchmark problms. 1 Inroducion Rinforcmn larning (RL) is an onlin paradigm for opimal squnial dcision making whr a agn inracs wih nvironmns, aks acions, rcivs rward and ris o maximiz is long-rm rward, a discound sum of all h rwards ha will b rcivd from now on. An imporan par of RL is policy valuaion, h problm of valuaing h xpcd long-rm rwards of a fixd policy. mporal Diffrnc (D) larning [Suon, 1988] is prhaps h bs known family of algorihms for policy valuaion. I has bn obsrvd ha whn combind wih funcion approximaion, D may divrg and lad o poor prdicion. h Rsidual Gradin (RG) was proposd [Baird, 1995] o addrss hs concrns. RG amps o minimiz h Bllman Error (BE) (s dfiniion in Sc. ), ypically wih linar funcion approximaion, using sochasic gradin dscn. Comparison bwn h family of D algorihms and RG has rcivd rmndous anion, alhough mos of h analyss havily rly on crain sochasic assumpions of h nvironmn such as ha h squnc of obsrvaions ar Markovian or from a saic Markov Dcision Procss (MDP). For insanc [Schoknch and Mrk, 003] showd ha D convrgs provably fasr han RG if h valu funcions ar prsnd by abular form. [Schrrr, 010] shows ha Bllman Rsidual minimizaion njoys a guarand prformanc whil D dos no in gnral whn sas ar sampld from arbirary disribuions (off-policy) ha may no corrspond o rajcoris akn by h sysm. [Schapir and Warmuh, 1996] and [Li, 008] providd wors-cas analysis of long-rm prdiciv rror for varians of h linar D and RG undr a non-probabilisic onlin larning sing. hir rsuls rly on h spcral analysis of a marix ha is rlad o spcific upda ruls of h D and RG algorihms undr linar funcion approximaion. Unforunaly, his approach maks i mor difficul o xnd hir wors-cas (assumpion fr) analysis o broadr familis of algorihms and rprsnaions ha arg Bllman and mporal Diffrnc rrors. Following [Schapir and Warmuh, 1996] and [Li, 008] s onlin larning framwork, w prsn wo simpl, gnral conncions bwn long-rm prdiciv rror and no-rgr onlin larning ha amps o minimiz BE and D. h cnral ida is ha mhods such as D and RG should b fundamnally undrsood as onlin algorihms as opposd o sandard gradin mhods, and ha on canno simulanously mak consisn prdicions in h sns of D and BE whil doing a poor job in rms of long-run prdicions. Similar o [Schapir and Warmuh, 1996] and [Li, 008], our analysis dos no rly on any saisical assumpions abou h undrlying sysm. his allows us o analyz difficul scnarios such as MDP wih ransiion probabiliis changing ovr im or vn wih ach ransiion chosn nirly advrsarial. h main conribuion of h papr is h analysis of h conncions bwn onlin long-rm rward prdicion and no-rgr onlin larning. Paricularly, h firs analysis on BE shows ha any no-rgr and sabl [Ross and Bagnll, 011] onlin larning algorihms, whn arging opimizing BE, nsur small prdicion rror. h scond analysis focuss on D and shows ha whn applying h family of Onlin Mirror Dscn (OMD) on D, w can also achiv small prdicion rror. W addiionally show ha Implici Onlin Larning is anohr propr algorihm ha can b usd for opimizing D o achiv small prdicion rror. hs wo analysis consqunly suggss a broad nw family of algorihms.

2 Paricularly, our analysis on BE gnralizs h RG algorihm from [Baird, 1995] in a sns ha RG is a spcific xampl of our family of algorihms ha runs Onlin Gradin Dscn (OGD) [Zinkvich, 003] on a squnc of BE loss funcions. For D, our analysis gnralizs D (0) from [Schapir and Warmuh, 1996] by showing ha running OGD a spcial form of OMD, rvals h upda rul of D (0). Prliminaris W considr h squnial onlin larning modl prsnd in [Schapir and Warmuh, 1996; Li, 008] whr no saisical assumpions abou h squnc of obsrvaions ar mad. h squnc of h obsrvaions can ihr b Markovian as ypically assumd in RL problm sings or vn advrsarial. W dfin h obsrvaion a im sp as x R n, which usually rprsns h faurs of h nvironmn a. hroughou h papr, w assum ha faur vcor x is boundd as x X. h corrsponding rward a sp is dfind as r R, whr w assum ha rward is always boundd r R R +. Givn a squnc of obsrvaions {x } and a squnc of rwards {r }, h long-rm rward a is dfind as y = k= γk r k, whr γ [0, 1) is a discound facor. Givn a funcion spac F h larnr chooss a prdicor f a ach im sp from F for prdicing long-rm rwards. hroughou his papr, w assum ha any prdicion mad by a prdicor f a a sa x is uppr boundd as f(x) P R +, for any f F and x. A im sp = 0, h larnr rcivs x 0, iniializs a prdicor f 0 F and maks prdicion ŷ 0 of y 0 as f 0 (x 0 ). Rounds of larning hn procds as follows: h larnr maks a prdicion ŷ of y a sp as f (x ); h larnr hn obsrvs a rward r and h nx sa x +1 ; h larnr updas is prdicor o f +1. his inracion rpas and is rminad afr sps. hroughou his papr, w call his problm sing as onlin prdicion of long-rm rward. W firs dfin h signd Bllman Error a sp for prdicor f as b = f (x ) r γf (x +1 ), which masurs ffcivly how slf consisn f is in is prdicions bwn im sp and + 1. W dfin h corrsponding Bllman Loss a im sp wih rspc o prdicor f as: l b (f) := (f(x ) r γf(x +1 )). (1) W also dfin signd mporal Diffrnc Error (signd D rror) a sp for prdicor f as d = f (x ) r γf +1 (x +1 ). W dfin D Loss a sp as: l d (f) := (f(x ) r γf +1 (x +1 )). () h Signd Prdicion Error of long-rm rward a for f is dfind as = f (x ) y and = f (x ) y for f accordingly. W will ypically b inrsd in bounding h Prdicion Error (PE) of a givn algorihm in rms of h bs possibl PE. o lighn noaion in h following scions, all sums ovr im indics implicily run from 0 o 1 unlss xplicily nod ohrwis. 3 Onlin Larning for Long-rm Rward Prdicion In his scion, w firs propos a nw prspciv of RG algorihm and D algorihm: w show ha RG and D boh could b undrsood as running Onlin Gradin Dscn on Bllman loss l b and D loss l d, rspcivly. A vry im sp, afr rciving h Bllman loss l b (f), l us apply OGD on l b (f): f +1 = f µ b ( f f (x ) γ f f (x +1 )), (3) whr w dno f f(x) as h funcional gradin of h valuaion funcional f(x) a funcion f. 1 Now for linar funcion approximaion whr f(x) is rprsnd as w x, h upda sp in Eq. 3 xacly rvals h RG algorihm proposd by [Baird, 1995]. Now, l us apply OGD o h D loss l d (f), w g h following upda sp: f +1 = f µ d f f (x ). (4) No ha h abov upda rul is implici in a sns ha h Righ Hand Sid (RHS) and h Lf Hand Sid (LHS) boh hav f +1 (d has f +1 ). o g h xplici upda rul for f +1, on nds o solv f +1 from Eq. 4. If w subsiu h linar funcion approximaion f(x) = w x ino Eq. 4 and solv for w +1, on can xacly rval h D (0) upda rul proposd in [Schapir and Warmuh, 1996]. Onlin Gradin Dscn is on of h popular no-rgr onlin larning algorihms. h abov prspciv suggss ha RG and D could b undrsood as applying a spcial no-rgr onlin algorihm OGD, o Bllman loss and D loss. A naural qusion ha on would lik o know is ha whhr any ohr no-rgr onlin algorihms, such as Onlin Nwon sp [Hazan al., 006], Onlin Frank Wolf [Hazan and Kal, 01] and implici onlin larning [Kulis al., 010], can b applid o Bllman loss l b and D loss l d, and achiv similar guarans on PE. 3.1 Opimizing Bllman Loss In his scion, w sablish a conncion bwn opimizing Bllman loss and wors cas long-rm prdiciv rror. Paricularly, w show ha opimizing Bllman loss wih any sabl and no-rgr onlin algorihms nsurs small prdicion rror for long-rm rward prdicion. W firs dfin h sabiliy condiion: Dfiniion 3.1 Onlin Sabiliy: For h gnrad squnc of prdicors f, w say h algorihm is onlin sabl if: 1 lim (f (x +1 ) f +1 (x +1 )) = 0. (5) Inuiivly, h onlin sabiliy mans ha on avrag h diffrnc bwn succssiv prdicors is vnually small. ha is, h diffrnc bwn f (x +1 ) and f +1 (x +1 ) is small on avrag. Onlin sabiliy is a gnral condiion and dos no svrly limi h scop of h onlin larning algorihms. For insanc, whn f is linar, h dfiniion of sabiliy of onlin larning in [Saha al., 01] (s Eq. 3 1 W assum h funcion f f(x) blongs o F. his is ru for funcion classs such as Rproducing Krnl Hilbr Spac (RKHS). his is why w call l d (f) D loss, sinc wih OGD, i rvals h D (0) algorihm, no h D algorihm. Howvr, narly idnical rsuls can b sablishd for D loss which rplacs f +1 by f in l d, and classic D can b rcovrd by OGD on D loss.

3 in [Saha al., 01]) and [Ross and Bagnll, 011] implis our form of onlin sabiliy. In fac, w can show ha many popular no-rgr onlin larning algorihms including OGD, ONS, OWF, implici onlin larning, and FRL saisfy our onlin sabiliy condiion. W rfr radr o [Sun and Bagnll, 015] for h daild sudy of h onlin sabiliy condiion for h abov mniond no-rgr onlin algorihms. Dfin ɛ = f (x +1 ) f +1 (x +1 ), wih h onlin sabiliy condiion, w now rady o sa h main horm: horm 3. Assum a squnc of prdicors {f } is gnrad by running som onlin algorihm on h squnc of Bllman loss {l b }. For any prdicor f F, h sum of prdicion rrors can b uppr boundd as: whr (1 γ) (b b ) + γ ɛ + (1 + γ) + M, (6) M = (γ + γ )( 0 ) (γ γ)( 0). By running a no-rgr and onlin sabl algorihm, as, h avrag prdicion rror is hn asympoically uppr boundd by a consan facor of h bs possibl prdicion rror in h funcion class: lim (1 + γ) (1 γ). (7) h proof of h horm only consiss of asy applicaion of lscoping ricks and Cauchy-Schwar inqualiis. W rfr radrs o [Sun and Bagnll, 015] for h daild proof of h abov horm. W mphasiz ha h abov analysis is indpndn of h paricular form of funcion approximaion. Whn = 0,, from horm. 3., i is asy o s ha no-rgr ra of (1/ ) (b b ) and h onlin sabiliy ra of (1/ ) ɛ oghr drmin h ra of h convrgnc of (1/ ). Whn and γ 1 (spcifically whn γ (1/ )), our uppr bound analysis in Eq. 7 is asympoically ighr han h uppr bound in [Li, 008] (Eq. 1) providd for RG. Sinc a larg numbr of popular norgr onlin algorihms saisfy h onlin sabiliy condiion, our horm ssnially xpnds h family of algorihms ha can b usd o larn prdicors of long-rm rwards. W mphasiz hr ha sabiliy of onlin algorihms is ssnial for our rsuls h no-rgr propry can b shown by counr-xampl o b insufficin o achiv low prdiciv rror [Sun and Bagnll, 015]. 3. Opimizing D Loss h analysis in Sc. 3.1 is gnral nough such ha almos any xising no-rgr onlin larning algorihm can b usd for opimizing Bllman loss and nsurs small prdicion rror on long-rm rwards. hough w wish such a nic gnralizaion also xiss for D, w could no sablish i. Insad w show ha a broad family of onlin larning algorihms Onlin Mirror Dscn (OMD), whn applid o D loss, nsurs small prdicion rror similar in form o [Schapir and Warmuh, 1996]. W also show ha implici onlin gradin dscn, a spcial form of implici onlin larning, can also b usd for opimizing D loss. h proofs of h horms prsnd in his scion ar in h appndix. 3 Onlin Mirror Dscn for D loss L us dfin R(f) as a rgularizaion and assum ha R(f) is a boh smooh and srongly convx funcion wih rspc o f wih norm, dfind by h innr produc associad wih F as f = f, f. A funcion R(f) is α-smooh and β-srongly convx if and only if: β f f +1 R(f ) R(f +1 ) R(f +1 )(f f +1 ) α f f +1. (8) Wihou loss of gnraliy, w assum ha R(f) is 1-srongly convx (ohrwis simply scal i) and α-smooh funcion wih rspc o f wih norm. For insanc, whn f is linar, w / is 1-srongly convx and 1-smooh. Whn applying OMD on D loss, w hav h following upda rul, which w dno as OMD-D : f = arg min f, θ + 1 R(f); f µ (9) θ +1 = θ + (ŷ r γŷ +1 ) f f (x ). (10) No ha whn w compu f using Eq. 9, h RHS of Eq. 9 acually implicily dpnds on ŷ, which is qual o f (x ) and hnc dpnds on f. Hr, w assum ha hough f appars on boh sids of Eq. 9, w can sill solv for f from Eq. 9 as D (0) dos. In pracic, whhr or no w can solv f from Eq. 9 could dpnd on h form of R(f). For insanc, whn R(f) = f / and f blongs o a Rproducing Krnl Hilbr Spac (RKHS) (.g., linar funcion f(x) = w x), w can achiv closd-form upda of f. In fac, whn f(x) = w x, R(w) = w, i is asy o show h upda rul from Eq. 9 rvals h D (0) algorihm. h following horm shows opimizing D loss wih OMD nsurs small long-rm prdicion rror: horm 3.3 Wih µ = O( 1 ) and F bing a RKHS, OMD-D (Eq. 9 and 10) has h following bound: + γ (1 γ) + O( ). (11) For h avrag prdicion rror /, w hav: lim + γ (1 γ). (1) Implici Onlin Larning for D Loss h OMD framwork gnralizs qui a fw popular onlin algorihms such as Onlin Gradin Dscn, Normalizd Exponnial Gradin (normalizd EG), OGD wih lazy projcion and p-norm algorihm [Shalv-Shwarz, 011]. Howvr, OMD is concpually diffrn from anohr family of onlin algorihms Implici Onlin Larning [Kulis al., 010]. Implici onlin larning algorihms usually ar mor sabl and robus compard o algorihms wih xplici upda ruls. 3 Availabl a hp:// wnsun

4 h ida of implici upda has bn applid o classic D [amar al., 014], whr h auhors show h algorihm wih implici upda is mor sabl han classic D in a sns ha i is no snsiiv o larning sp siz. Brifly, givn h squnc of loss l (f), implici onlin larning updas f as f +1 = arg min f l (f) + 1 µ D R (f, f ), whr D R (f, f ) is h Brgman divrgnc gnrad from rgularizaion R. For spcial cas whr f is in RKHS, D loss l d (f) is acually a quadraic loss wih rspc o f. Hnc, w propos o apply h implici Onlin Gradin Dscn on spcial form of implici onlin larning, o D loss. S R(f) = f /, w hav h following upda rul: f +1 = arg min f l d (f) + 1 µ f f, (13) No ha h abov upda rul is implici sinc ŷ +1 (burid in l d ) dpnds f +1. Dpnding on h form of f, w can achiv closd-form soluion for f +1 from Eq. 13. Blow, w dmonsra a closd-form upda rul for linar funcion f(x) = w x wih R(w) = w. Rplac f wih w in Eq. 13, ak h drivaiv wih rspc o w, s i o zro, and solv for w +1, w will g: µ w +1 = w 1 + µ x (w x r γŷ +1 )x. (14) No ha ŷ +1 implicily dpnds on w +1. o solv for w +1, w firs do produc x +1 on boh sids of h abov quaion (h LHS bcoms ŷ +1 ), solv for ŷ +1 and hn subsiu ŷ +1 back o h quaion and solv for w +1. his givs us h following Implici-D upda sp: µ w +1 = w 1 + µx (x γx +1 ) b x, (15) whr b = (w x r γw x +1 ). h corrsponding upda rul for RKHS wih krnl K(, ) is: µ f +1 = f 1 + µk(x, x γx +1 ) b K(x, ), (16) whr b = (f (x ) r γf (x +1 )). Implici-D has h following uppr bound on PE: horm 3.4 Wih µ = O( 1 ) and F bing a RKHS, Implici-D (Eq. 15 and 16) has h following bound: (1 + γ) ( + γ ) (1 γ) + O( ). (17) For h avrag prdicion rror /, w hav: lim (1 + γ) ( + γ ) (1 γ) (18) 3.3 Discussion h bound of OMD-D is h ighs compard o Implici- D and RG. hough our OMD-D bound is no as igh as h on from [Schapir and Warmuh, 1996], our analysis is mor gnral. Our bound of RG is asympoically ighr han h on from [Li, 008] whn γ 1. Exprimnally w find ha Implici-D prforms rally wll, which indicas ha our wors-cas bound for Implici-D may b no igh. Avrag Prdicion Error 80 Random Walk (f(x) = w x),. = 0:95, n = implici OGD (D) implici OGD (BR) ONS (D) ONS (BR) OFW (D) OFW (BR) D(0) RG Numbr of Sps (a) Random Walk Avrag Prdicion Error 70 Puddl World (f(x) = w x),. = 0:99, n = implici OGD (D) implici OGD (BR) ONS (D) ONS (BR) OFW (D) OFW (BR) D(0) RG Numbr of Sps (b) Puddl World Figur 1: Convrgnc of prdicion rror. W applid a s of onlin algorihms on Bllman loss {l b (w)} (do lin) and D -loss funcions {l d (w)} (solid lin) for Random walk (lf) and Puddl World (righ). 4 Exprimns W applid svral onlin larning algorihms o wo simulad policy valuaion problms: (1) Random Walk wih a ring chain, which is a varian of h Hall problm inroducd in [Baird, 1995], () PuddlWorld adopd from [Suon and Baro, 1998]. W sd svral popular no-rgr and sabl onlin larning algorihms, including implici onlin gradin dscn (implici OGD), onlin Nwon sp (ONS) [Hazan al., 006], onlin Frank Wolf (OFW) [Hazan and Kal, 01] and classic onlin gradin dscn [Zinkvich, 003], on boh D loss and Bllman loss. Fig. 1 shows h convrgnc of avrag prdicion rror wih rspc o numbr of im sps. W no ha ONS and implici OGD giv good convrgnc spd in gnral. hroughou h xprimns, w found ha implici OGD works wll for boh D loss and Bllman loss. Our xprimnal rsuls also show ha our approachs hav h possibiliy o achiv smallr prdicion rror han D(0) (.g., Fig. 1b). No ha whn opimizing D loss, ONS and OFW acually achiv good prformanc, hough our analysis on D loss currnly dos no suppor ONS or OFW. h xprimn rsuls for RKHS can b found a [Sun and Bagnll, 015], whr w also dmonsrad hs algorihms on a simulad hlicopr hovr domain [Coas al., 008]. 5 Conclusion W inroducd a nw prspciv for RG and D hy could b undrsood as running spcial no-rgr onlin algorihm on Bllman loss and D loss, rspcivly. his nw prspciv nabls us o driv wo gnralizaions, on for RG and on for D in h onlin sing, whr no saisical assumpions ar placd on h obsrvaions. Paricularly, w show ha any no-rgr and sabl onlin algorihms, whn applid o Bllman loss, nsurs small prdicion rror. For D, w connc D o wo family of onlin algorihms Onlin Mirror Dscn and Implici Onlin Larning, and w show ha opimizing D loss wih OMD and implici OGD guarans small prdicion rror. h rmaining opn problm is ha whhr hr xiss a mor gnral conncion bwn D loss and no-rgr onlin algorihms: whn opimizing D loss, ar h no-rgr propry and sabiliy sufficin o achiv low prdicion rror?

5 Rfrncs [Baird, 1995] Lmon Baird. Rsidual algorihms: Rinforcmn larning wih funcion approximaion. In Procdings of h 1h Inrnaional Confrnc on Machin Larning (ICML 1995), pags 30 37, [Coas al., 008] Adam Coas, Pir Abbl, and Andrw Y Ng. Larning for conrol from mulipl dmonsraions. In Procdings of h 5h inrnaional confrnc on Machin larning (ICML 008), pags , 008. [Hazan and Kal, 01] Elad Hazan and Sayn Kal. Projcion-fr Onlin Larning. In 9h Inrnaional Confrnc on Machin Larning (ICML 01), pags 51 58, 01. [Hazan al., 006] Elad Hazan, Ami Agarwal, and Sayn Kal. Logarihmic rgr algorihms for onlin convx opimizaion. In Procdings of h 19h annual confrnc on Compua- ional Larning hory (COL 006), pags , 006. [Kulis al., 010] Brian Kulis, Pr L Barl, Barl Ecs, and Brkly Edu. Implici Onlin Larning. In Procdings of h 7h inrnaional confrnc on Machin larning (ICML 010), pags , 010. [Li, 008] Lihong Li. A wors-cas comparison bwn mporal diffrnc and rsidual gradin wih linar funcion approximaion. In Procdings of h 5h inrnaional confrnc on Machin larning (ICML 008), pags , 008. [Ross and Bagnll, 011] Sphan Ross and J. Andrw Bagnll. Sabiliy Condiions for Onlin Larnabiliy. arxiv: , 011. [Saha al., 01] Ankan Saha, Prak Jain, and Ambuj wari. h Inrplay Bwn Sabiliy and Rgr in Onlin Larning. arxiv prprin arxiv: , pags 1 19, 01. [Schapir and Warmuh, 1996] Robr E. Schapir and Manfrd K. Warmuh. On h wors-cas analysis of mporal-diffrnc larning algorihms. Machin Larning, (1):95 11, [Schrrr, 010] Bruno Schrrr. Should on compu h mporal Diffrnc fix poin or minimiz h Bllman Rsidual? h unifid obliqu projcion viw. Inrnaional Confrnc on Machin Larning (ICML 010), 010. [Schoknch and Mrk, 003] Ralf Schoknch and Arur Mrk. D(0) Convrgs Provably Fasr han h Rsidual Gradin Algorihm. In Inrnaional Confrnc on Machin Larning (ICML 003), pags , 003. [Shalv-Shwarz, 011] Shai Shalv-Shwarz. Onlin Larning and Onlin Convx Opimizaion. Foundaions and rnds in Machin Larning, 4(): , 011. [Sun and Bagnll, 015] Wn Sun and J. Andrw (Drw) Bagnll. Onlin Bllman Rsidual Algorihms wih Prdiciv Error Guarans. In h 31s Confrnc on Uncrainy in Arificial Inllignc (UAI 015), July 015. [Suon and Baro, 1998] Richard S. Suon and Andrw G. Baro. Rinforcmn Larning: An Inroducion. MI Prss, [Suon, 1988] R S Suon. Larning o Prdic by h Mhods of mporal Diffrnc. Machin Larning, pags 9 44, [amar al., 014] Aviv amar, Panos oulis, Shi Mannor, and Edoardo M. Airoldi. Implici mporal Diffrncs. arxiv: , pags 1 6, 014. [Zinkvich, 003] Marin Zinkvich. Onlin Convx Programming and Gnralizd Infinisimal Gradin Ascn. In Inrnaional Confrnc on Machin Larning (ICML 003), pags 41 4, 003.