Temporal Difference Methods and Approximate Monte Carlo Linear Algebra

Transcription

1 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon empal Dfference Mehod and Appromae Mone Carlo Lnear Algebra Dmr P. Bereka Deparmen Elecrcal Engneerng and Compuer Scence Maachue Inue echnology RL Wkhop, Llle 8

2 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon Focu Appromae oluon lnear equaon (, where ( A b, A n n, b R n by olvng he proeced equaon y Π (y Π proecon on a ubpace ba funcon (wh repec o ome nm h he Galerkn appromaon approach, bu mulaon play a cenral and non-radonal role. We conder very large n. Sarng pon: Appromae DP/Bellman equaon/polcy evaluaon A : encode he chan rucure, b : co vec hen y Π (y he equaon olved by D mehod [D(λ, LSD(λ, LSPE(λ] We generalze o he cae where A arbrary, ubec only o I ΠA : nverble (on wk wh H. Yu - paper avalable from our web e

3 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon Benef and Challenge Generalzaon A hgher perpecve f D mehod n appromae DP Movae mprovemen n varou area: Eplaon ue Auomac generaon feaure bound Smplfed convergence analy An eenon o a va new area applcaon here are many lnear yem huge dmenon n pracce Dealng wh le rucure Lack conracon Abence a chan Ill-condonng

4 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon Oulne Proeced Equaon Appromaon he Appromae DP Cone he General Proeced Equaon Cone General LSD and LSPE-ype Alghm Fm he Alghm Choce f a Conracon Auomac Generaon Feaure Mulep Veron - λ-mehod 3 Eenon onlnear Eenon Lea Square/Bellman -ype Mehod

5 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon DP Cone/Polcy Evaluaon an Decon Problem (MDP n ae, ranon probable dependng on conrol Polcy eraon mehod; we focu on ngle polcy evaluaon Bellman equaon: A b where b: co vec A ha ranon rucure, e.g., A αp f dcouned problem, A P f average co problem

6 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon Appromae Polcy Evaluaon Appromaon whn ubpace S { r R }, Φ a mar wh ba funcon a column Proeced Bellman equaon: Π(A b bound, aumng ΠA conracon wh modulu α (, α Π Long hy, arng wh D(λ (Suon, 988 Lea quare mehod are currenly me popular

7 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon Lea Square Polcy Evaluaon (LSD Dae o 996 (Bradke and Baro, wh λ-eenon by Boyan ( Idea: Solve a mulaon-baed appromaon he proeced equaon he proeced Bellman equaon wren a Cr d LSD olve Ĉr ˆd, where are obaned ung mulaon Ĉ C, ˆd d Doe no need he conracon propery DP problem Mulep veron: LSD(λ whch LSD appled o he mappng (λ ( ( λ λ k k ( A (λ b (λ, k where A (λ ( λ λ k A k, b (λ λ k A k b k k

8 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon Proeced Value Ieraon (PVI Value Ieraon > Proecon > Value Ieraon > Proecon... (Φ S an Squar Proecon on on S o mal Subpace S ( Φ Π Π r Condonal Mea Π Π mu be a conracon - beng a conracon no enough m machng eenal: a (Eucldean proecon nm f whch a conracon here a magcal nm: he eady-ae drbuon nm (ae are weghed by he eady-ae drbuon he chan

9 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon Lea Square Polcy Evaluaon (LSPE Π Subpace S Π Subpace S ( Π Subpace S Proecon on S mal ( Π θ Condonal Subpace S Proecon on Emaon S θ Condonal Emaon mal Π Π noe wh noe wh θ Condonal Emaon mal θ Condonal Emaon mal k k k k noe wh k k k k k k k k noe wh f (z z f (z z PVI LSPE Z PVI Z o ξ (May UeLSPE Q oy Π Subpace S Proecon on S Subpace S Proecon on S ( foy ( Π opue P A Q Z (z z o ξ (May Ue Q o Probably o ξ (May fzprobably (z z Π Condonal θ Emaon ( Subpace S Emaon Proecon ( Π Subpace SonProecon on SPmal θ Condonal on S(mal Π Subpace Sranmed Proecon S oy Recever Sgnal Regon ranmed oy Recever Regon o P A osgnal P P A Probably noe wh Probably noe θ Condonal Emaon mal θ Condonal wh Emaon Π mal Π " ranmed ranmed ( S Proecon on S oy Recever ranmed Recever (Sgnal Regon Π Subpace S Proecon on S Sgnal Regon Subpace ranmed noe wh oy noe wh ( Π Subpace S Proecon on S θ Condonal Emaon mal ranmed Π " f Π " f (z z... ranmed z Z k k k k Z (z noe Π wh θmal Emaon mal fz (z θzcondonal fcondonal (z z o ξ (May Ue Q f < f Probably f < Emaon f mal θ Condonal Emaon Probably Z noe wh o P P A noe wh Probably noe wh oy Recever ranmed fz (z Probably z oy Recever ranmed Sgnal Regon f ( f S Proecon on S Π < Sgnal fregon < Π f S Proecon ( Subpace on S ( ranmed Π Subpace Subpace S Proecon on S Recever ranmed oy Recever Sgnal Regon ranmed oy Sgnal Regon ranmed Π f(z Π Probably f (z Π fz z Z (z Z z ranmed Subpace S ranmed ZEmaon µ mal Subpace θ Condonal Emaon mal aθ Condonal a b A f { A} ( f ( y Π µ b A f { A} ( f (Recever y Π µsgnal,sµy µy y (µ, µy Regon y y (µ Condonal Emaon mal zero-m oy θ ranmed θ Condonal Emaon mal θ Condonal Emaon mal Probably noe wh noe Probably wh Probably noe wh ranmed f < f y f < f a b A f ( f ( y µ µ y (µ, µy wh a hrehold noe wh ranmed Sgnal { A} Se E par Efficennoe Froner Selecon k ( Se E oy Froner hrehold k b A f { A} f ( par y µ Sgnal µrecever (µ Selecon Efficen y y Regon, µyregon oy Recever Sgnal Regon ranmed oy Recever ranmed f < f ranmed f < f fz (z z fz (z z fz (z kz (z ranmed Seg E fzpar z Efficen (zranmed zefficen Froner Selecon hrehold α β g( H g( H f (d α βselecon g( Hhrehold Seg E fzpar Froner kh f (d g( f < f Probably Probably Probably Probably A Probably A a b Proce f { A} ( ( µy y (µregon a b Proce f { A} ( f ( y µ Poble Recever Sgnal y (µregon, µy Calculaon, µy ranmed µyobervaon Space Poble Obervaon Calculaon Poer Space Poer oy ranmed αfβoy g( y HµRecever g( HSgnal g f (d f < f α β g( H g( H f (d g Sgnal ranmed oy Recever Regon Recever b AfRecever ( fregon ( y µp ranmed a b A f { A} f ( y µ oy µy yranmed (µ, µy Sgnal Regon ranm ( µ y y (µ,µ y { A} pθ ( P(Θ oy ka P(Θ k P(Θ ( ksgnal ranmed ranmed Selecon Calculaon f hrehold Proce < Poer f Θ P(Θ f kk < Se EfSpace Efficen ranmed par Froner k ranmed Poble Obervaon Se E Space par Efficen PobleFroner Obervaon Selecon Calculaonhrehold Proce Poer. Froner Se E par Efficen hrehold k k Se E par Efficen pθ (..Froner P(Θ Selecon k hrehold P(Θ k k P(Θ Selecon p kaθp(θ ( ;fθ p ( ; θ p ( ; θ p ( ; θm p (; θ p (;pθ θm ( α.. β g( ( ; p ( ; θ p (; θ p (; θ. A ( f ( y µ µ (µ, µ b p m m y y y α β g( H g(h f (d g < f H g( H f (d g { A} f f < f f ( y µα µ. y y (µ, µyf { A} ( < f H elecon H < g f α βf g( H Hypohe g( fa(db g AΘ f (d ( ; Hypohe elecon Θ Θ Space Θ Poer.. Space β fg( f θ p( ;Obervaon θpg( ( ; θ ph (; θ ppoer (; θm.. < f m p Calculaon Poble p ( ;Obervaon θ p ( ; θ pcalculaon ( ; θm Se p (; θ par p (; θeffcen m. EProce Froner Poble Selecon hrehold k Proce bp(θ fcalculaon ( a bcalculaon ( ( Froner y µppoble ( Selecon µay A P(Θ { A} yk y (µ,kµ f ( k y µ µy y (µ, µy Space ppoble Proce Space Obervaon Proce Poer { A} ( Obervaon P(Θ A fk P(Θ Poer par fkefficen SeE hrehold Θ fs (; H p ( ; θ p ( ; θm p (; θθ p (; θm elecon Θ Θ fs (; H p ( ; θ p ( ; θm p (; θhypohe p (; θm elecon Θ Θ pθ ( Hypohe P(Θ k P(Θ k pθ ( P(Θ k P(Θ k γ.α βy µg( H g( H f (d a gb Ak f { A}Froner ( f (.. y µ Selecon µy y (µhrehold, µy a bpar A f { A} ffroner f µyselecon (, µpe ghrehold k g( y f(µ y α(βnerval ph fθ( Effcen a b Aecf { A} ( fθ( Se yrule,( µ(; a b AH µp ( ;θmeffcen θymθ(µ, (; µ(; ( ;Se (d ( ;yθpar µpyp( ; (; pp. θm y θm p ( ; ( yh ; θµepp ( ;θdecon yp θ(µ pconfdence.. g( p mµ Decon Rule Confdence nerval (;.y θθm p (; fs (; p (; θm ec { A} fs(; H p ( ;aθ ( ; θ p ( ; θm p (;...θpoble f θm.. ( f ( y µ µy y (µ, µy Obervaon p ( ; θ p ( ; θ p ( ; θm p (; θ pspace (;Space θm Poble p ( ; θ pcalculaon ( ; θ p ( ; θmproce p (; θbpoer pa (; { A} Obervaon Calculaon Proce Poer Se E par Efficen Hypohe elecon Θ Θ Se E par Efficen Froner Selecon hrehold k Hypohe elecon Θ Θ Froner Selecon hrehold k Seξ E par Efficen Froner Selecon hrehold k SeP(Θ E par Efficen Froner Selecon hrehold kh p ( k P(Θ k γ α β g( g( H f (d g Decon Rule Confdence nerval ec γ α β g( H g( H f (d g Θ v log / v log ξ / pθ ec ( P(Θ elecon k P(ΘΘ Θ Hypohe elecon Decon Θ Θ Rule Confdence nerval Hypohe k Se E par Efficen Froner Selecon hrehold k (; α θβ pg( H g( H f (dproce g fs (; ( ; θ ph( ; θfm(d.pobervaon fs (;HH pg( ( ; θ ph( ; θfm pg (;αθβ pml (; θmh g( H fα (d (;θm g( Emae ML Emae Emae Space Calculaon Poer α β g( β gg( HH p g( (demae Space Poble Obervaon Calculaon Proce Poer.Poble.. pgξ(;/ / fs (; H p ( ; θ p ( ; θm p (;vθlog θpm( ; (; ( ; θ θθpm θ..m(; pθm (; vθlog θm pθ( ; ;( ; θθm pfsp( ; θhmθpp (; p pξ(; ( ; ( pθ ( ;pθ p (; pp (; ( Proce P(Θ α kcalculaon P(Θ Proce HPoer Space Poble p ( P(Θ k P(Θ k Θ Obervaon β g( H g( fk Space Poble Obervaon Calculaon Proce Poer (d g Θ Decon Rule Confdence nerval ec Poer Poble Obervaon Calculaon Proce Poer Space Poble Obervaon Calculaon Decon Rule Confdence nerval ec p ( ; H p ( ; H pspace (; θ p ( ; H p ( ; H p ( ; θ p (; θ p (; θ Emae ML Emae m m m ( ; θm p (; θ p p ( P(Θ k P(Θ k ML pec Emae Θ Hypohe k P(Θ ΘpΘ kθdecon P(Θ Rule Confdence k nerval elecon P(Θ elecon Θ pθ (Decon P(Θ Emae nerval k P(Θ (Hypohe ( Θ k P(Θec k Rule Confdence Θ Poble Obervaon Calculaon Proce.. Poer.vlogSpace H H umber head k Θ, Obervaon Pr pθ H umber head k Θ, Obervaon Prp pθ( ; ξ p / v logh ξ / ;θθ ( ;θθ m p (; p ( θm ; H p.. ( θp; H p( ( ( ;. pp p(; θ θk P(Θ θ p. ( θθp p( ; θm p (; θ p..(; m m(; (; P(Θ p θ..m.k p ( v ;log Hξ pf ( p;h( ; p;(; Θ p( ; θm m p( ; pθ(; θ p pp(; (; Θ θpθ θ ( ; θm;mθ (; θm (; (; ( ; (; / θfsh p(; θ( ; θ pp.p( ; m θm ( ; ( ;H p ( ; θ p ( Emae ; θ p ( ;ML θm Emae p (;Θ/ θ pp(; θmθ p.. ( fs; (; p( ; θvmlog pξp(; θ θpmp(;( ;θθm p.. ( ; θ p ( ; θm p (; θ p (; θm. ( θ Emae ML Emae H H umber head k Θ,ΘObervaon Pr pθ Hypohe elecon Θ Hypohe elecon Θ Θ H H umber head k Θ, Obervaon Pr p. Emae Θ Emae ML Hypohe elecon Θ Θ Emae Hypohe elecon Θ Rule Θ Hypohe Hypohe elecon elecon ΘΘ ML Θ Emae fec Decon Confdence nerval p ( ; θ p ( ; θ p ( ; θm p (; θ p (; θm.. Decon Rule Confdence nerval fθθ p Θ p ( ec ; H p ( ; H p ( ; θm p (; θ p (; θm ( ; H p ( ; H p ( ; θm p (; θ p (; θm fhypohe p(; ( ; p pp (; (;θ θp p(;θ(; Condonal p Θ f Θ Pon Emae Analy pp f Θp Pon pfθs(; mθ (;θsh θ θemae S (; pp θ( ; mθp( (; θm θm(; ( ;H ( ; p ( p;h( ; θpm ( ; (;θpmh(; θ ( (; ;H(; ( ; pθh pθh (; θ pθ ( ; θ( ; pθ( ; ec m θpfcondonal m( ; ec mf(; S(; H m m θm pθ ( ; θ( ; p θpanaly fs (;ph ph θumber pθ(; p; H θpm ps (; (; Θ m pv m m Θ ( ; elecon / ph umber head kθ, Obervaon ΘPr p H head k Θ,pf(; Obervaon p( ;Pr plog ξθh A mulaon-baed appromaon o PVI Dae o 996 (Bereka and Ife; alo n he Bereka and kl (996 book - ued n a er applcaon LSPE: mulaon noe wh Π, {z } PVI Incremenal lke D(λ - no epze unlke D(λ Same compley/ame oluon a LSD Aympocally dencal" o LSD, bu dffer n early age Allow f a favable nal gue r ; may be an advanage n opmc/few ample appromae polcy eraon Θ f ξ / v log Θ

10 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon Advanage Proeced Equaon Mehod n DP All operaon are done n low-dmenon he hgh-dmenonal vec need no be ed he proecon nm mplemened n mulaon - need no be known a pr here a proecon nm (he drbuon nm ha nduce conracon ΠA and a pr bound

11 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon General/onDP Proeced Equaon Mehod A doe no have a ranon probably rucure o chan, no conracon guaranee We may nroduce an arfcal chan f amplng/proecon Wh clever choce he chan, ΠA may be a conracon Compuable bound are avalable All operaon are done n low-dmenon he hgh-dmenonal vec need no be ed Mehod: LSD analog (doe no requre ΠA o be a conracon LSPE analog (requre ΠA o be a conracon D(λ analog (requre ΠA o be a conracon

12 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon Proeced Equaon Appromaon Mehod (LSD-lke Le Π be proecon wh repec o v u ξ n ξ, where ξ R n a probably drbuon wh pove componen Eplc fm proeced equaon Π(A b n n r arg mn r a φ( r b A r R where φ( denoe he h row he mar Φ Opmaly condon/equvalen fm: n n ξ a φ( A r {z } Epeced value n ξ φ(b {z } Epeced value he wo epeced value are appromaed by mulaon

13 Proeced Equaon Appromaon noe General LSD and LSPE-ype Alghm Eenon Smulaon Mechanm o ξ (May Ue Q o P A Π Subpace S Proecon on S ( k k k k... k k k k k.k kkk k. k k k k kk k k k. k k k k(may k... Π ξ oξ ξ Ue Q o ξ (May Ue oue ξ (May Ue o ξ (May Q Ue Q Q o o ξ (May o (May Q Ue Ch o ξ (May Ue θ Condonal Emaon o A o o P A o P mal A P A P o P noe wh ( Π S SΠ o osubpace PSubpace P S S Proecon ( A Subpace Proecon on ( Π ( Π Subpace Π Proecon on S on S Proecon S ( S on S ( Π Subpace S Proec Π Π Π Π Π S Π Subpace ( on Subpace S Proecon ( kkk k fz S Π kproecon zk k k Π k(z k k θ k k k k Condonal Squar θ Condonal Emaon mal ze θ Condonal θ k Condonal θ Condonal Emaon Emaon mal Emaon mal mal o ξ (May o Ue ξ (May Ue Q θ Condonal Emaon Erro o ξ (May Ue Q o ξ (May Ue Q noe Π Π noe wh Q wh wh noe wh Probably noe wh noe o A P A noe wh o P o A P o P A θ Condonal θrecever Emaon mal Emaon oy Sgnal Regon ξ, ranmed amplng: equence {Π Proecon,...} o.e., Condonal Π,S ( ( Π accdng Subpace Proecon S Proecon S Proecon on S on S f ( Subpace S on S (Generae Π Subpace on S fsubpace Z( f z Π fz (z zπ fz(z z Z (z z Z (z ranmed wh noe wh Π f (z z relave frequency each row ξ Π Z Probably θ Condonal θ Condonal Emaon Emaon mal mal zer Probably θcondonal Emaon mal Probably θ Condonal Emaon mal Probably equence Probably amplng: Generae (,,..oy. accdng o Sgnal,,Probably Recever ( wh noe wh noe noe Recever wh noe wh oy Recever Sgnal ra oy oy Sgnal oy Recever Regon Sgnal Recever Regon Sgnal Regon ranmed Regon ranmed ranmed f < f z oy Sgnal Regon wh ranmed ome ranon mar (z zp wh frecever Z Z (z ranmed ranmed probably ranmed f ranmed f ranmed fz (z z fz (z z Z (z z fz (z z p >Probably f a, 6 Probably f Probably Probably Probably Probably a b A f ( f ( y µ µy (µ, µy f f < f yf< f < f < f f { A} franmed < ranmed f ran oy oy Recever Sgnal Recever Regon Sgnal Regon oy.e., Recever Sgnal oy Regon Recever Sgnal Regon ranmed oy Recever Sgnal Regon oy Recever Sgnal Regon ranmed f each, he relave frequency (, p ranmed Froner ranmed ranmed ranmed Se E par Effcen Selecon hrehold k ranmed ranmed amplng may be done ung a chan wh ranon mar a b A f { A} ( f ( γ α β g( H g( H f (d g Q (unrelaed o P b A fy { A} y (µ µy a b A f a b A ( f f a(b ( A yffaµ( ( µyµf ( µfy,yµ( y (µ y, µµµyy, µ y( (µ yµ y y (µ, µ f { A} { A} b f ( yµ µy < A f< { A} f f( < par f Frone f { A} f f E < fa Se Effcen Spacealo Poble Obervaon Proce- Poer amplng may be done whouacalculaon chan u ample Se E Effcen Effcen Froner EEffcen par Froner Se par Selecon Froner hrehold Selecon khrehold Selecon k Selecon hrehold k ( f Se E par Sep ome Effcen EP(Θ <drbuon Froner par fk fpar Effcen k < Froner f β g( hrehol Θ Se E Selecon row accdng o known ξp(θ (e.g., a unfm γ α H

14 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon and o ξ (May Ue Q w o ξ (May Ue Q o P A ( P Π A Subpace S Proecon on S umn o Accdn Π on ( Π Subpace S Proecon S Π k k k... mal k k k k k θk Condonal kk.k kk k k. k k Emaon k k k k k k. Acc o ξ (May Ue M oue ξ (May Ue o ξ(may (May Ue oue ξ (May Ue o ξ (May Q Q ( QCha Marko (May Π Ch ( noe Emaon wh o ξ o Q o o ξξ(may Ue Ue θ Condonal mal o P A o o A o P Π A A P P Π ( Subpace S P o P A wh ξ (May Ue Q o o osubpace PSubpace A S o Proecon ( Subpace ( Π ( Π ( Subpace Π Proecon SΠ on S on S Proecon S Condonal on S Subpace z Π S Proecon (fz(z Π on Pθ Proecon o S A Π Π Π Π Π Subpace ( ( S Π Proecon on Subpace S S Proec noe wh θ Condonal Emao Π f (z z k k k k k k k k Z ( Π Subpace S Proecon on S k k k k kk Probably θnoe Condonal Emaon o θ Condonal k θ k Condonal θ Condonal Emaon Emaon mal Emaon mal mal zero wh o (May Ue Qm θsgnal Condonal Emaon (May Ue Q ξ ξπ (May Ue Q Π Recever Π ξ o (May o Ue ξ ranmed noe wh Probably noe wh noe oy wh noe wh o Regon o P A P A noe wh P o P o A mal θ Condonal Emaon o A ranmed y Recever Sgnal Regon ranmed θ Condonal θ Emaon mal Emaon Subpace Subpace Condonal fzproecon (z Π ( Π ( ( Proecon Π S Proecon S on Sz Err on (wh SSubpace S Proecon on SProbably noe Π Subpace on S Err amplng Sae Sequence Generaon n DP. Affec: ranmed fz (z z f z Π fz (z zπ fz(z z (z Z (z f z wh noe wh Z Π Π oy Recever Probably he proecon nm θ Condonal Condonal mal zero-m m Emaon (z Probably θcondonal Probably femaon θ f fz Emaon mal Emaon Sgnal Probably θ Condonal Probably mal <oy Z ranmed Recever Regon err Probably Wheher ΠA a conracon noe wh noe wh noe wh noe wh oy Recever Sgnal Regon oy oy Recever Sgnal oy Recever Regon Sgnal Recever Regon Sgnal Regon ranmed ranmed ranm f < f Probably ranmed oy Recever Sgnal Regon f z Z (z Z (z amplng ranon Sequence Generaon nfdp. Canzbe ranmed ranmed ranmed ranmed ranmed oy Recever Sgnal Regon ranmed fz (z Affec: z fz (z z fz (z z fz (z z oally unrelaed o rowamplng. ranmed a b A f { A} ( f ( y µ µy y (µ, µy f < f Probably Probably he amplng/mulaon Probably Probably Probably Probably b A f { A} ( f ( µynoe, µf yf y < f < k Machng Se y µ f < oy (µ Recever < f f f< f famplng Sgnal f P wh A ha an effec lke n mpance E par Effcen Froner oy Recever Sgnal Recever Regon hrehold Regon ranm oy Recever Sgnal oy Regon Sgnal Regon w oy Recever Sgnal Regon Selecon ranmed ranmed oy Recever Sgnal Regon ranmed f < f ranmed ranmed ranmed E par Effcen Selecon hrehold k a b A f ( f Froner ranmed ranmed ranmed { A}

15 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon LSD-Lke Mehod Opmaly condon/equvalen fm proeced equaon n n n ξ a φ( A r ξ φ(b {z } Epeced value {z } Epeced value he wo epeced value are appromaed by row and column amplng (bach A me, we olve he lnear equaon hen r r k φ( k φ( k a «k k φ( k r p k k φ( k b k k

16 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon Conder PVI LSPE-ype Mehod Π(A b,,,... Epreng he proecon a a lea quare mnmzaon, we have equvalenly r r arg mn r R (A b ξ,! n ξ φ(φ( {z } Epeced value n n ξ a φ( r b A {z } Epeced value Appromae he wo epeced value by row and column amplng r! φ( k φ( k k k φ( k ak k p k k φ( k r b k If ΠA a conracon wh repec o ome nm, r r «

17 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon f Conracon I Mu have Sum A o have hope conracon ΠA Propoon: Le ξ be he nvaran drbuon an rreducble Q uch ha A Q hen and Π are conracon mappng under any one he followng hree condon: ( F ome calar α (,, we have A αq. ( here e an nde ī uch ha aī < q ī f all,..., n. (3 here e an nde ī uch ha P n aī <. oe : Under condon ( and (, and Π are conracon mappng wh repec o he pecfc nm ξ oe : Apple o DP dcouned and ochac he pah problem

18 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon f Conracon II Mu have Sum A Propoon: Le ξ be he nvaran drbuon a Q wh no ranen ae. Aume A Q and ha I ΠA nverble. hen he mappng Π γ, where γ ( γi γ, a conracon wh repec o ξ f all γ (,. oe : Π γ and Π have he ame fed pon oe : Π need no be a conracon oe 3: Apple o average co problem (Yu and Bereka 6

19 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon Back o Dcouned DP/Eplaon Here A αp, where P crepond o he polcy evaluaed and α he dcoun fac If we ake Q P f row amplng, hen ΠA a conracon We may alo ue chan Q P f row amplng, o change ξ and nduce eplaon; f eample ue Polcy R (f polcy prob. β, Polcy P (on polcy prob. β he LSD-ype alghm alway apple ( doe no requre ha ΠA be a conracon If ΠA can be hown o be a conracon, he LSPE(λ- and D(λ-ype alghm apply. In parcular, we ge convergence wh no ba f: ( F all λ [, f β α ( F all β [, f λ uffcenly large

20 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon Applcaon o Dagonally Domnan Syem Conder he oluon he yem C d, where d R n and C an n n mar uch c, c c,,..., n Conver o he yem A b, where b d c ( f We have a n a o row um A c c f and c,,..., n, c Under he earler condon, ΠA a conracon.

21 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon Auomac Generaon Power A a Ba Funcon Ue Φ whoe h row where g ome vec φ( `g( (Ag( (A g( Eample n he MDP cae: Ue a feaure fne hzon co A ufcaon f A a conracon and g b: he fed pon ha an epanon he fm A k b Whle (A k g( hard o generae, can be appromaed by amplng (n effec we ue noy feaure k

22 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon Mulep Veron (Fed Sep and λ-mehod Replace by a mulep mappng wh he ame fed pon, e.g., k where k fed, (λ ( λ λ k k, A (λ ( λ λ k A k, k k where λ (, uch ha he nfne ere converge Movaon f λ-mehod, aumng ha pecral radu A (A Propoon: If I A nverble and (A, hen (A (λ <, λ (,, lm λ `A (λ A λ ncreae he conracon become ronger We mu have λ < /(A f a λ-mehod o apply. here are no rercon f a k-ep mehod

23 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon λ-mehod When he LSD/LSPE-ype mehod gven earler are appled o Π (λ hey yeld generalzaon o LSD(λ and LSPE(λ he fmula nvolve empal dfference, baed on he epanon (λ ( λ m (A m b A m A m m he enre analy D(λ, LSD(λ, and LSPE(λ f DP generalze ubec o he followng rercon: Egenvalue λa mu be whn he un crcle f LSD analog Addonal conracon aumpon f LSPE(λ and D(λ [.e., ΠA (λ a conracon]

24 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon Fm λ-mehod I and column amplng are done ung he ame chan P. Defne w k, and f m w k,m a k k p k k a k k p k k a km km p km km Eample: Dcouned DP w k,m α m, k LSPE-ype mehod r r! φ( k φ( k φ( k λ mk w k,mk d ( m, k k mk where d ( m are he empal dfference d ( m b m w m, φ( m r φ( m r,, m

25 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon Fm λ-mehod II Recurve/effcen updae f LSPE-ype mehod r r B (C r h where B B φ( φ(, h h z b, C C z `w, φ( φ(, z λw, z φ(. LSD(λ-ype mehod u r C h D(λ-ype mehod where γ he epze r r γ z d (

26 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon Convergence Reul Propoon: Aume ha P rreducble, and ha λ afe λ ma a /p <, λ [,., Le r be generaed by he LSD(λ-ype alghm. hen, r r λ wh probably he ame rue f he LSPE(λ-ype alghm [aumng alo ha (A (λ ] Here r λ he oluon he proeced equaon Π (λ Smlar reul f D(λ-ype eenon, under uable (ochac appromaon-ype condon f he epze

27 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon A onlnear Equaon wh Scalar onlneare Conder he yem ( Af ( b, where f : R n R n a mappng wh calar funcon componen he fm f ( `f (,..., f n( n. Aume ha each he mappng f : R R nonepanve: f ( f (,,..., n,, R. hen f A a conracon wh repec o a weghed Eucldean nm, alo a conracon h rucure mple favable choce a chan f mulaon

28 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon Opmal Soppng Le ( αpf ( b, where P rreducble ranon probably wh nvaran drbuon ξ, α (, a calar dcoun fac, and f ha componen f ( mn{c, },,..., n, where c are ome calar. hen ( he Q-fac equaon crepondng o a dcouned opmal oppng problem In h cae, ΠA a conracon wh repec o ξ [kl and Van Roy (999, who gave a Q-learnng alghm wh lnear funcon appromaon] he LSPE alghm ha been generalzed o h problem (Yu and Bereka 7; alo he 3rd Edon my DP e 7 here no good" LSD-ype alghm f h problem (he fed pon equaon o be appromaed nonlnear

29 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon Lnear Lea Square/Regreon/Bellman Mehod Conder olvng he problem mn r R A b ξ o appromae he weghed lea quare oluon A b. Here A : m n mar, ξ a known probably drbuon vec, b R m, and Φ an n mar ba funcon. he oluon r (Φ A ΞAΦ Φ A Ξb, where Ξ he dagonal m m mar havng ξ along he dagonal o appromae he oluon, we replace Φ A ΞAΦ and Φ A Ξb wh mulaon-baed emae

30 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon Iue n Regreon/Bellman Mehod eed o ample wo column f each row me noe Varance reducon a fm mpance amplng may be eenal Dealng wh (near ngular Φ A ΞAΦ Add a mall mulple he deny o Φ A ΞAΦ (lke a pr n a regreon eng,.e., appromae by mulaon where γ mall pove parameer Ue a promal mehod: r (Φ A ΞAΦ γi Φ A Ξb r (Φ A ΞAΦ γ I (Φ A Ξb γ r, where γ a pove parameer. h converge o he crec oluon (Φ A ΞAΦ Φ A Ξb Applcaon n nvere problem and oher area (huge dmenon - e.g., n 9, A: fully dene

31 Proeced Equaon Appromaon General LSD and LSPE-ype Alghm Eenon Concludng Remark D mehod can be naurally eended o olve lnear yem equaon In dong o, perpecve and new mehod are obaned f appromae DP he all approach very mple: Sar wh a deermnc alghm Wre n erm epeced value Appromae he epeced value by mulaon he approach apple o many lnear algebra-ype problem - beyond hoe dcued here (e.g., compung he domnan egenvalue a mar, appromang he nvaran drbuon a chan here conderable leraure and heecal wk on Mone Carlo lnear algebra mehod (arng wh von eumann he new elemen here lnear funcon appromaon and he connecon wh D mehod Ecng propec: Applcaon o lnear algebra problem huge dmenon, far beyond he DP cone