Unsupervised Cross-Domain Transfer in Policy Gradient Reinforcement Learning via Manifold Alignment

Transcription

1 Unsupevsed Coss-Doman ansfe n Polcy Gaden Renfocemen Leanng va Manfold Algnmen Haham Bou Amma Unv. of Pennsylvana hahamb@seas.upenn.edu Ec Eaon Unv. of Pennsylvana eeaon@cs.upenn.edu Paul Ruvolo Oln College of Engneeng paul.uvolo@oln.edu Mahew E. aylo Washngon Sae Unv. aylom@eecs.wsu.edu Absac he success of applyng polcy gaden enfocemen leanng RL o dffcul conol asks hnges cucally on he ably o deemne a sensble nalzaon fo he polcy. ansfe leanng mehods ackle hs poblem by eusng knowledge gleaned fom solvng ohe elaed asks. In he case of mulple ask domans, hese algohms eque an ne-ask mappng o faclae knowledge ansfe acoss domans. Howeve, hee ae cuenly no geneal mehods o lean an ne-ask mappng whou equng ehe backgound knowledge ha s no ypcally pesen n RL sengs, o an expensve analyss of an exponenal numbe of ne-ask mappngs n he sze of he sae and acon spaces. hs pape noduces an auonomous famewok ha uses unsupevsed manfold algnmen o lean neask mappngs and effecvely ansfe samples beween dffeen ask domans. Empcal esuls on dvese dynamcal sysems, ncludng an applcaon o quadoo conol, demonsae s effecveness fo coss-doman ansfe n he conex of polcy gaden RL. Inoducon Polcy gaden enfocemen leanng RL algohms have been appled wh consdeable success o solve hghdmensonal conol poblems, such as hose asng n oboc conol and coodnaon Pees & Schaal hese algohms use gaden ascen o une he paamees of a polcy o maxmze s expeced pefomance. Unfounaely, hs gaden ascen pocedue s pone o becomng apped n local maxma, and hus has been wdely ecognzed ha nalzng he polcy n a sensble manne s cucal fo achevng opmal pefomance. o nsance, one ypcal saegy s o nalze he polcy usng human demonsaons Pees & Schaal 2006, whch may be nfeasble when he ask canno be easly solved by a human. hs pape exploes a dffeen appoach: nsead of nalzng he polcy a andom.e., abula asa o va human demonsaons, we nsead use ansfe leanng L o nalze he polcy fo a new age doman based on knowledge fom one o moe souce asks. In RL ansfe, he souce and age asks may dffe n he fomulaons aylo & Sone In pacula, Copygh c 2015, Assocaon fo he Advancemen of Afcal Inellgence All ghs eseved. when he souce and age asks have dffeen sae and/o acon spaces, an ne-ask mappng aylo e al. 2007a ha descbes he elaonshp beween he wo asks s ypcally needed. hs pape noduces a famewok fo auonomously leanng an ne-ask mappng fo coss-doman ansfe n polcy gaden RL. s, we lean an ne-sae mappng.e., a mappng beween saes n wo asks usng unsupevsed manfold algnmen. Manfold algnmen povdes a poweful and geneal famewok ha can dscove a shaed laen epesenaon o capue nnsc elaons beween dffeen asks, especve of he dmensonaly. he algnmen also yelds an mplc ne-acon mappng ha s geneaed by mappng ackng saes fom he souce o he age. Gven he mappng beween ask domans, souce ask aecoes ae hen used o nalze a polcy n he age ask, sgnfcanly mpovng he speed of subsequen leanng ove an unnfomed nalzaon. hs pape povdes he followng conbuons. s, we noduce a novel unsupevsed mehod fo leanng nesae mappngs usng manfold algnmen. Second, we show ha he dscoveed subspace can be used o nalze he age polcy. hd, ou empcal valdaon conduced on fou dssmla and dynamcally chaoc ask domans e.g., conollng a hee-lnk ca-pole and a quadoo aeal vehcle shows ha ou appoach can a auomacally lean an ne-sae mappng acoss MDPs fom he same doman, b auomacally lean an ne-sae mappng acoss MDPs fom vey dffeen domans, and c ansfe nfomave nal polces o acheve hghe nal pefomance and educe he me needed fo convegence o nea-opmal behavo. Relaed Wok Leanng an ne-ask mappng has been of mao nees n he ansfe leanng communy because of s pomse of auonomous ansfe beween vey dffeen asks aylo & Sone Howeve, he maoy of exsng wok assumes ha a he souce ask and age ask ae smla enough ha no mappng s needed Baneee & Sone 2007; Kondas & Bao 2007, o b an ne-ask mappng s povded o he agen aylo e al. 2007a; oey e al he man dffeence beween hese mehods and hs pape s ha we ae neesed n leanng a mappng beween asks. hee has been some ecen wok on leanng such mappngs. o example, mappngs may be based on seman-

2 c knowledge abou sae feaues beween wo asks Lu & Sone 2006, backgound knowledge abou he ange o ype of sae vaables aylo e al. 2007b, o anson models fo each possble mappng could be geneaed and esed aylo e al Howeve, hee ae cuenly no geneal mehods o lean an ne-ask mappng whou equng ehe backgound knowledge ha s no ypcally pesen n RL sengs, o an expensve analyss of an exponenal numbe n he sze of he acon and sae vaable ses of ne-ask mappngs. We ovecome hese ssues by auomacally dscoveng hgh-level feaues and usng hem o ansfe knowledge beween agens whou suffeng fom an exponenal exploson. In pevous wok, we used spase codng, spase poecon, and spase Gaussan pocesses o lean an ne-ask mappng beween MDPs wh abay vaaons Bou Amma e al Howeve, hs pevous wok eled on a Eucldean dsance coelaon beween souce and age ask ples, whch may fal fo hghly dssmla asks. Addonally, placed escons on he ne-ask mappng ha educed he flexbly of he leaned mappng. In ohe elaed wok, Bósc e al use manfold algnmen o asss n ansfe. he pmay dffeences wh ou wok ae ha he auhos a focus on ansfeng models beween dffeen obos, ahe han polces/samples, and b ely on souce and age obos ha ae qualavely smla. Backgound Renfocemen Leanng poblems nvolve an agen choosng sequenal acons o maxmze s expeced eun. Such poblems ae ypcally fomalzed as a Makov decson pocess MDP = S, A, P 0, P,, whee S s he poenally nfne se of saes, A s he se of acons ha he agen may execue, P 0 : S [0, 1] s a pobably dsbuon ove he nal sae, P : S A S [0, 1] s a sae anson pobably funcon descbng he ask dynamcs, and : S A S R s he ewad funcon measung he pefomance of he agen. A polcy π : S A [0, 1] s defned as a condonal pobably dsbuon ove acons gven he cuen sae. he agen s goal s o fnd a polcy π whch maxmzes he aveage expeced ewad: π = ag max π = ag max π E [ 1 H H ] s, a, s +1 π =1 p π τ Rτ dτ, whee s he se of all possble aecoes wh hozon H, Rτ = 1 H s, a, s +1, and 2 H =1 p π τ = P 0 s 1 1 H Ps +1 s, a πa s. 3 =1 Polcy Gaden mehods Suon e al. 1999; Pees e al epesen he agen s polcy π as a funcon defned ove a veco θ R d of conol paamees and a veco of sae feaues gven by he ansfomaon Φ : S R m. By subsung hs paameezaon of he conol polcy no Eqn. 2, we can compue he paamees of he opmal polcy as θ = ag max θ J θ, whee J θ = p πθτ Rτ dτ. o maxmze J, many polcy gaden mehods employ sandad supevsed funcon appoxmaon o lean θ by followng an esmaed gaden of a lowe bound on he expeced eun of J θ. Polcy gaden algohms have ganed aenon n he RL communy n pa due o he successful applcaons on eal-wold obocs Pees e al Whle such algohms have a low compuaonal cos pe updae, hghdmensonal poblems eque many updaes by acqung new ollous o acheve good pefomance. ansfe leanng can educe hs daa equemen and acceleae leanng. Snce polcy gaden mehods ae pone o becomng suck n local maxma, s cucal ha he polcy be nalzed n a sensble fashon. A common echnque Pees & Schaal 2006; Agall e al fo polcy nalzaon s o fs collec demonsaons fom a human conollng he sysem, hen use supevsed leanng o f polcy paamees ha maxmze he lkelhood of he human-demonsaed acons, and fnally use he fed paamees as he nal polcy paamees fo a polcy gaden algohm. Whle hs appoach woks well n some sengs, s napplcable n seveal common cases: a when s dffcul o nsumen he sysem n queson so ha a human can successfully pefom a demonsaon, b when an agen s consanly faced wh new asks, makng gaheng human demonsaons fo each new ask mpaccal, o c when he asks n queson canno be nuvely solved by a human demonsao. he nex secon noduces a mehod fo usng ansfe leanng o nalze he paamees of a polcy n a way ha s no suscepble o hese lmaons. Ou expemenal esuls show ha hs mehod of polcy nalzaon, when compaed o andom polcy nalzaon, s able o no only acheve bee nal pefomance, bu also oban a hghe pefomng polcy when un unl convegence. Polcy Gaden ansfe Leanng ansfe leanng ams o mpove leanng mes and/o behavo of an agen on a new age ask by eusng knowledge fom a solved souce ask. In RL sengs, each ask s descbed by an MDP: ask = S, A, P 0, P, and = S, A, P 0, P,. One way n whch knowledge fom a solved souce ask can be leveaged o solve he age ask s by mappng opmal sae, acon, nex sae ples fom he souce ask no he sae and acon spaces of he age ask. ansfeng opmal ples n hs way allows us o boh povde a bee umpsa and leanng ably o he age agen, based on he souce agen s ably. Whle he pecedng dea s aacve, complexes ase when he souce and age asks have dffeen sae and/o acon spaces. In hs case, one mus defne an ne-ask mappng χ n ode o anslae opmal ples fom he souce o he age ask. ypcally aylo & Sone 2009, χ s defned by wo sub-mappngs: 1 an ne-sae mappng χ S and 2 an ne-acon mappng χ A.

3 Souce Doman Phase I: Lean coss-doman mappng GS P0 S Phase II: Coss-doman ansfe va 2. eflec age aces fom age S 1. sample nal saes P execue shaed epesenaon aces fom age Doman + 4. ansfe ackng sgnal G gue 1: ansfe s spl no wo phases: I leanng he ne-sae mappng χs va manfold algnmen, and II nalzng he age polcy va mappng he souce ask polcy. By adopng an RL famewok whee polces ae saefeedback conolles, we show ha we can use opmal sae aecoes fom he souce ask o nellgenly nalze a conol polcy n he age ask, whou needng o explcly consuc an ne-acon mappng. We accomplsh hs by leanng a pseudo-nveble ne-sae mappng beween he sae spaces of a pa of asks usng manfold algnmen, whch can hen be used o ansfe opmal sequences of saes o he age. he fac ha ou algohm does no eque leanng an explc ne-acon mappng sgnfcanly educes s compuaonal complexy. Ou appoach consss of wo phases gue 1. s, usng aces gaheed n he souce and age asks, we lean an ne-sae mappng χs usng manfold algnmen Phase I n gue 1. o pefom hs sep, we adap he Unsupevsed Manfold Algnmen UMA algohm Wang & Mahadevan 2009, as dealed n he nex secon. Second, we use χs o poec sae aecoes fom he souce o he age ask Phase II n gue 1. hese poeced sae aecoes defne a se of a ackng aecoes fo he age ask ha allow us o pefom one sep of polcy gaden mpovemen n he age ask. hs polcy mpovemen sep nellgenly nalzes he age polcy, whch esuls n supeo leanng pefomance han sang fom a andomly nalzed polcy, as shown n ou expemens. Alhough we focus on polcy gaden mehods, ou appoach could easly be adaped o ohe polcy seach mehods e.g., PoWER, REPS, ec.; see Kobe e al Leanng an Ine-Sae Mappng Unsupevsed Manfold Algnmen UMA s a echnque ha effcenly dscoves an algnmen beween wo daases Wang & Mahadevan UMA was developed o algn daases fo knowledge ansfe beween wo supevsed leanng asks. Hee, we adap UMA o an RL seng by algnng souce and age ask sae spaces wh poenally dffeen dmensons ms and m. o lean χs elang S and n S, aecoes of o saes n he souce,, ns ask, τ = s1,..., shs, ae obaned by =1 followng π, and aecoes of saes n he age ask, τ = n on,, s1, ae obaned by ulz,..., sh =1 ng π, whch s nalzed usng andomly seleced polcy paamees. o smplcy of exposon, we assume ha aecoes n he souce doman have lengh HS and hose n he age doman have lengh H ; howeve, ou algohm s capable of handlng vaable-lengh aecoes. We ae neesed n he seng whee daa s scace n he age ask han n he souce ask.e., n ns. Gven aecoes fom boh he souce and age asks, we flaen he aecoes.e., we ea he saes as unodeed and hen apply he ask-specfc sae ansfomaon o oban wo ses of sae feaue vecos, one fo he souce ask and one fo he age ask. Specfcally, we ceae he followng ses of pons: 1 1 X = Φ s1,..., Φ shs, ns ns shs s1,,φ Φ 1 1 X = Φ s1,..., Φ sh, n n Φ s1,,φ sh. Gven X RmS HS ns, X Rm H n, we can apply he UMA algohm Wang & Mahadevan 2009 wh mnmal modfcaon, as descbed nex. Unsupevsed Manfold Algnmen UMA he fs sep of applyng UMA o lean he ne-sae mappng s o epesen each ansfomed sae n boh he souce and age asks n ems of s local geomey. We use he noaon Rx Rk+1 k+1 o efe o he max of pa wse Eucldean dsances among he k-neaes neghbos of x X. Smlaly, Rx efes o he equvalen ma x of dsances fo he k-neaes neghbos of x X. he elaons beween local geomees n X and X ae epesened RnS HS n H wh n byhe max Wo w, = exp ds Rx, Rx and dsance mec ds Rx, Rx = " mn mn orx oh γ1 Rx, 4 1 h k! Rx γ2 orx oh #. We use he noaon o oh o denoe he hh vaan of he k! pemuaons of he ows and columns of he npu max, s he obenus nom, and γ1 and γ2 ae defned as: R o R R o or oh h x x x x. γ1 = γ2 = Rx Rx orx oh orx oh

4 o algn he manfolds, UMA compues he on Laplacan LX + µγ L = 1 µγ 2 µγ 3 L X + µγ 4 5 wh dagonal maces Γ 1 Γ 4 R n H n H, whee Γ 1, R n S H S n S H S and = w, and Γ 4, = w,. he maces Γ 2 R n S H S n H and Γ 3 R n H n S H S on he wo manfolds wh Γ 2, = w, and Γ 3, = w,. Addonally, he non-nomalzed Laplacans L X and L X ae defned as: L X = D X W X and L X = D X W X, whee D X R n S H S n S H S s a dagonal max wh D, = X w, and, smlaly, D, = X w,. he maces W and W epesen he smlay n he souce and age ask sae spaces especvely and can be compued smla o W. o on he manfolds, UMA fs defnes wo maces: τ Z = 0 0 τ DS 0 D = 0 D S. 6 Gven Z and D, UMA compues opmal poecons o educe he dmensonaly of he on sucue by akng he d mnmum egenvecos ζ 1,..., ζ d of he genealzed egenvalue decomposon ZLZ ζ = λzdz ζ. he opmal poecons α and α ae hen gven as he fs d 1 and d 2 ows of [ζ 1,..., ζ d ], especvely. Gven he embeddng dscoveed by UMA, we can hen defne he ne-sae mappng as: χ S [ ] = α + α [ ]. 7 he nvese of he ne-sae mappng o poec age saes o he souce ask can be deemned by akng he pseudo-nvese of Eqn. 7, yeldng χ + S [ ] = α+ α [ ]. Inuvely, hs appoach algns he mpoan egons of he souce ask s sae space sampled based on opmal souce aecoes wh he sae space exploed so fa n he age ask. Alhough acons wee gnoed n consucng he manfolds, he algned epesenaon mplcly capues local sae anson dynamcs whn each ask snce he saes came fom aecoes, povdng a mechansm o ansfe aecoes beween asks, as we descbe nex. Polcy ansfe and Impovemen Nex, we dscuss he pocedue fo nalzng he age polcy, π. We consde a model-fee seng n whch he polcy s lnea ove a se of poenally non-lnea sae feaue funcons modulaed by Gaussan nose whee he magnude of he nose balances exploaon and exploaon. Specfcally, we can we he souce and age polces as: π π s s = Φ s = Φ s θ + ɛ θ + ɛ, whee ɛ N 0, Σ and ɛ N 0, Σ. τ =1 o nalze π, we fs sample m nal age aecoes D = fom he age ask usng a { } m andomly nalzed polcy hese can be newly sampled aecoes o smply he ones used o do he nal manfold algnmen sep. Nex, we map he se of nal saes n D o he souce ask usng χ + S. We hen un he opmal souce polcy sang fom each of hese mapped nal saes o poduce a se of m opmal sae aecoes n he souce ask. nally, he esulng sae aecoes ae mapped back o he age ask usng χ S o geneae a se of efleced sae- { aecoes n he age ask, D = τ } m =1. o clay, we assume ha all aecoes ae of lengh H; howeve, hs s no a fundamenal lmaon of ou algohm. We defne he followng ansfe cos funcon: J θ = m =1 p θ τ ˆR τ, τ whee ˆR s a cos funcon ha penalzes devaons beween he nal sampled aecoes n he age ask and he efleced opmal aecoes: ˆR τ, τ = 1 H H s s = Mnmzng, Eqn. 8 s equvalen o aanng a age polcy paameezaon θ such ha π follows he efleced aecoes D. uhe, Eqn. 8 s n exacly he fom equed o apply sandad off-he-shelf polcy gaden algohms o mnmze he ansfe cos. he Manfold Algnmen Coss-Doman ansfe fo Polcy Gadens MAXD-PG famewok s dealed 1 n Algohm 1. Specal Cases Ou wok can be seen as an exenson of he smple modelbased case wh a lnea-quadac egulao LQR Bempoad e al polcy, whch s deved and explaned n he onlne appendx 2 accompanyng hs pape. Alhough he assumpons made by he model-based case seem escve, he analyss n he appendx coves a wde ange of applcaons. hese, fo example, nclude: a he case n whch a dynamcal model s povded befoehand, o b he case n whch model-based RL algohms ae adoped see Buşonu e al In he man pape, howeve, we consde he moe geneal model-fee case. Expemens and Resuls o assess MAXD-PG s pefomance, we conduced expemens on ansfe boh beween asks n he same doman as well as beween asks n dffeen domans. Also, we suded 1 Lnes 9-11 of Algohm 1 eque neacon wh he age doman o a smulao fo acqung he opmal polcy. Such an assumpon s common o polcy gaden mehods, whee a each eaon, daa s gaheed and used o eavely mpove he polcy. 2 he onlne appendx s avalable on he auhos webses.

5 Algohm 1 Manfold Algnmen Coss-Doman ansfe fo Polcy Gadens MAXD- PG Inpus: Souce and age asks and, opmal souce polcy π, # souce and age aces ns and n, # neaes neghbos k, # age ollous z, nal # of age saes m. Lean χs : 1: Sample ns opmal souce aces, τ, and n andom age aces, τ 2: Usng he modfed UMA appoach, lean α and α o poduce χs = α+ α [ ] ansfe & Inalze Polcy: 3: Collec m nal age saes s1 P0 4: Poec hese m saes o he souce by applyng χ+ S [ ] 5: Apply he opmal souce polcy π on hese poeced n om saes o collec D = τ =1 6: Poec he samples n D o he age usng χs [ ] o poduce ackng age aces D 7: Compue ackng ewads usng Eqn : Use polcy gadens o mnmze Eqn. 8, yeldng θ Impove Polcy: 0 9: Sa wh θ and sample z age ollous 10: ollow polcy gadens e.g., epsodc REINORCE bu usng age ewads R 11: Reun opmal age polcy paamees θ he obusness of he leaned mappng by vayng he numbe of souce and age samples used fo ansfe and measung he esulan age ask pefomance. In all cases we compaed he pefomance of MAXD- PG o sandad polcy gaden leanes. Ou esuls show ha MAXD- PG was able o: a lean a vald ne-sae mappng wh elavely lle daa fom he age ask, and b effecvely ansfe beween asks fom ehe he same o dffeen domans. Dynamcal Sysem Domans We esed MAXD- PG and sandad polcy gaden leanng on fou dynamcal sysems gue 2. On all sysems, he ewad funcon was based on wo facos: a penalzng saes fa fom he goal sae, and b penalzng hgh foces acons o encouage smooh, low-enegy movemens. Smple Mass Spng Dampe SM: he goal wh he SM s o conol he mass a a specfed poson wh zeo velocy. he sysem dynamcs ae descbed by wo saevaables ha epesen he mass poson and velocy, and a sngle foce ha acs on he ca n he x decon. Ca Pole CP: he goal s o swng up and hen balance he pole vecally. he sysem dynamcs ae descbed va a fou-dmensonal sae veco hx, x, θ, θ, epesenng he poson, velocy of he ca, and he angle and angula velocy of he pole, especvely. he acons conss of a foce ha acs on he ca n he x decon. h, h, h, hx, x hx, x hx, x a Smple Mass hx, x h 3, h 33, 3 h 3, 3 h 3h, 3, h 2 22, 2, 2, 2 2 h 1, h 11, 1 h 2h h 1, h 11, 1 hx, x hx, hx, x x hx, x c hee-lnk Ca Pole h, 2 hx, x hx, x hx, x b Ca Pole x e11hx,e11 e2 e ee11 e ee31 e e 21oll o1ll e l ol e2b e 2B ee12be 2Be1 B e1 e B B 1pB c pc 4 ph 4 h e3be e3b ch p4 3B ch 4 yawyaw yaw e3b oll yaw d Quadoo gue 2: Dynamcal sysems used n he expemens. hee-lnk Ca Pole 3CP: he 3CP dynamcs ae descbed va an egh-dmensonal sae veco hx, x, θ1, θ 1, θ2, θ 2, θ3, θ 3, whee x and x descbe he poson and velocy of he ca and θ and θ epesen he angle and angula velocy of he h lnk. he sysem s conolled by applyng a foce o he ca n he x decon, wh he goal of balancng he hee poles upgh. Quadoo QR: he sysem dynamcs wee adoped fom a smulao valdaed on eal quadoos Bouabdallah 2007; Voos & Bou Amma 2010, and ae descbed va hee angles and hee angula veloces n he body fame.e., e1b, e2b, and e3b. he acons conss of fou oo oques {1, 2, 3, 4 }. Each ask coesponds o a dffeen quadoo confguaon e.g., dffeen amaue lenghs, ec., and he goal s o sablze he dffeen quadoos. Same-Doman ansfe We fs evaluae MAXD- PG on same-doman ansfe. Whn each doman, we can oban dffeen asks by vayng he sysem paamees e.g., fo he SM sysem we vaed mass M, spng consan K, and dampng consan b as well as he ewad funcons. We assessed he pefomance of usng he ansfeed polcy fom MAXD- PG vesus sandad polcy gadens by measung he aveage ewad on he age ask vs. he amoun of leanng eaons n he age. We also examned he obusness of MAXD- PG s pefomance based on he numbe of souce and age samples used o lean χs. Rewads wee aveaged ove 500 aces colleced fom 150 nal saes. Due o space consans, we epo same-doman ansfe esuls hee; deals of he asks and expemenal pocedue can be found n he appendx2. gue 3 shows MAXD- PG s pefomance usng vayng numbes of souce and age samples o lean χs. hese esuls eveal ha ansfe-nalzed polces oupefom sandad polcy gaden nalzaon. uhe, as he numbe of samples used o lean χs nceases, so does boh he nal and fnal pefomance n all domans. All nalzaons esul n equal pe-eaon compuaonal cos. heefoe, MAXD- PG boh mpoves sample complexy and educes wall-clock leanng me.

6 Aveage Rewad Souce 1000 age Samples 500 Souce 500 age Samples Souce 300 age Samples Souce 100 age Samples Sandad Polcy Gadens Ieaons Ieaons Ieaons Ieaons 3000 a Smple Mass b Ca Pole c hee-lnk Ca Pole d Quadoo gue 3: Same-doman ansfe esuls. All plos shae he same legend and vecal axs label. Aveage Rewad Souce 1000 age Samples Souce 500 age Samples 500 Souce 300 age Samples 100 Souce 100 age Samples 315 Sandad Polcy Gadens Ieaons Ieaons Ieaons a Smple Mass o Ca Pole 305 b Ca Pole o hee-lnk CP c Ca Pole o Quadoo θ θ * age: SM age: CP age: 3CP Souce: SM Souce: CP Souce: 3CP Pocuses Measue d Algnmen Qualy vs ansfe gue 4: Coss-doman ansfe esuls. Plos a c depc age ask pefomance, and shae he same legend and axs labels. Plo d shows he coelaon beween manfold algnmen qualy Pocuses mec and qualy of he ansfeed knowledge. Coss-Doman ansfe Nex, we consde he moe dffcul poblem of cossdoman ansfe. he expemenal seup s dencal o he same-doman case wh he cucal dffeence ha he sae and/o acon spaces wee dffeen fo he souce and he age ask snce he asks wee fom dffeen domans. We esed hee coss-doman ansfe scenaos: smple mass o ca pole, ca pole o hee-lnk ca pole, and ca pole o quadoo. In each case, he souce and age ask have dffeen numbes of sae vaables and sysem dynamcs. Deals of hese expemens ae avalable n he appendx 2. gue 4 shows he esuls of coss-doman ansfe, demonsang ha MAXD-PG can acheve successful ansfe beween dffeen ask domans. hese esuls enfoce he conclusons of he same-doman ansfe expemens, showng ha a ansfe-nalzed polces oupefom sandad polcy gadens, even beween dffeen ask domans and b nal and fnal pefomance mpoves as moe samples ae used o lean χ S. We also examned he coelaon beween he qualy of he manfold algnmen, as assessed by he Pocuses mec Goldbeg & Rov 2009, and he qualy of he ansfeed knowledge, as measued by he dsance beween he ansfeed θ and he opmal θ paamees gue 4d. On boh measues, smalle values ndcae bee qualy. Each daa pon epesens a ansfe scenao beween wo dffeen asks, fom ehe SM, CP, o 3CP; we dd no consde quadoo asks due o he equed smulao me. Alhough we show ha he Pocuses measue s posvely coelaed wh ansfe qualy, we hesae o ecommend as a pedcve measue of ansfe pefomance. In ou appoach, he coss-doman mappng s no guaaneed o be ohogonal, and heefoe he Pocuses measue s no heoecally guaaneed o accuaely measue he qualy of he global embeddng.e., Goldbeg and Rov s 2009 Coollay 1 s no guaaneed o hold, bu he Pocuses measue sll appeas coelaed wh ansfe qualy n pacce. We can conclude ha MAXD-PG s capable of: a auomacally leanng an ne-sae mappng, and b effecvely ansfeng beween dffeen doman sysems. Even when he souce and age asks ae hghly dssmla e.g., ca pole o quadoo, MAXD-PG s capable of successfully povdng age polcy nalzaons ha oupefom saeof-he-a polcy gaden echnques. Concluson We noduced MAXD-PG, a echnque fo auonomous ansfe beween polcy gaden RL algohms. MAXD-PG employs unsupevsed manfold algnmen o lean an nesae mappng, whch s hen used o ansfe samples and nalze he age ask polcy. MAXD-PG s pefomance was evaluaed on fou dynamcal sysems, demonsang ha MAXD-PG s capable of mpovng boh an agen s nal and fnal pefomance elave o usng polcy gaden algohms whou ansfe, even acoss dffeen domans.

7 Acknowledgemens hs eseach was suppoed by ONR gan #N , AOSR gan #A , and NS gan IIS We hank he anonymous evewes fo he helpful feedback. Refeences Agall, B. D.; Chenova, S.; Veloso, M.; and Bownng, B A suvey of obo leanng fom demonsaon. Robocs and Auonomous Sysems 575: Baneee, B., and Sone, P Geneal game leanng usng knowledge ansfe. In Poceedngs of he 20h Inenaonal Jon Confeence on Afcal Inellgence, Bempoad, A.; Moa, M.; Dua, V.; and Pskopoulos, E he explc lnea quadac egulao fo consaned sysems. Auomaca 381:3 20. Bócs, B.; Csao, L.; and Pees, J Algnmen-based ansfe leanng fo obo models. In Poceedngs of he Inenaonal Jon Confeence on Neual Newoks IJCNN. Bou Amma, H.; aylo, M.; uyls, K.; Dessens, K.; and Wess, G Renfocemen leanng ansfe va spase codng. In Poceedngs of he 11h Confeence on Auonomous Agens and Mulagen Sysems AAMAS. Bouabdallah, S Desgn and Conol of Quadoos wh Applcaon o Auonomous lyng. Ph.D. Dsseaon, École polyechnque fédéale de Lausanne. Buşonu, L.; Babuška, R.; De Schue, B.; and Ens, D Renfocemen Leanng and Dynamc Pogammng Usng uncon Appoxmaos. Boca Raon, loda: CRC Pess. Goldbeg, Y.; and Rov, Y Local Pocuses fo manfold embeddng: a measue of embeddng qualy and embeddng algohms. Machne Leanng 771: Kobe, J.; Bagnell, A.; and Pees, J Renfocemen leanng n obocs: a suvey. Inenaonal Jounal of Robocs Reseach 3211: Kondas, G., and Bao, A Buldng poable opons: Skll ansfe n enfocemen leanng. In Poceedngs of he 20h Inenaonal Jon Confeence on Afcal Inellgence, Lu, Y., and Sone, P Value-funcon-based ansfe fo enfocemen leanng usng sucue mappng. In Poceedngs of he 21s Naonal Confeence on Afcal Inellgence, Pees, J., and Schaal, S Polcy gaden mehods fo obocs. In Poceedngs of he IEEE/RSJ Inenaonal Confeence on Inellgen Robos and Sysems, Pees, J., and Schaal, S Naual aco-cc. Neuocompung 717-9: Pees, J.; Vayakuma, S.; and Schaal, S Naual aco-cc. In Poceedngs of he 16h Euopean Confeence on Machne Leanng ECML, Spnge. Suon, R. S.; McAllese, D. A.; Sngh, S. P.; and Mansou, Y Polcy gaden mehods fo enfocemen leanng wh funcon appoxmaon. Neual Infomaon Pocessng Sysems aylo, M. E., and Sone, P ansfe leanng fo enfocemen leanng domans: a suvey. Jounal of Machne Leanng Reseach 10: aylo, M. E.; Kuhlmann, G.; and Sone, P Auonomous ansfe fo enfocemen leanng. In Poceedngs of he 7h Inenaonal Jon Confeence on Auonomous Agens and Mulagen Sysems AAMAS, aylo, M. E.; Sone, P.; and Lu, Y ansfe leanng va ne-ask mappngs fo empoal dffeence leanng. Jounal of Machne Leanng Reseach 81: aylo, M. E.; Wheson, S.; and Sone, P ansfe va ne-ask mappngs n polcy seach enfocemen leanng. In Poceedngs of he 6h Inenaonal Jon Confeence on Auonomous Agens and Mulagen Sysems. oey, L.; Shavlk, J.; Walke,.; and Macln, R Relaonal macos fo ansfe n enfocemen leanng. In Blockeel, H.; Ramon, J.; Shavlk, J.; and adepall, P., eds., Inducve Logc Pogammng, volume 4894 of Lecue Noes n Compue Scence. Spnge Beln Hedelbeg Voos, H., and Bou Amma, H Nonlnea ackng and landng conolle fo quadoo aeal obos. In Poceedngs of he IEEE Inenaonal Confeence on Conol Applcaons CCA, Wang, C., and Mahadevan, S Manfold algnmen whou coespondence. In Poceedngs of he 21s Inenaonal Jon Confeence on Afcal Inellgence IJCAI, Mogan Kaufmann.