Unsupervised Cross-Domain Transfer in Policy Gradient Reinforcement Learning via Manifold Alignment

Transcription

1 Proceedngs of he Tweny-Nnh AAAI Conference on Arfcal Inellgence Unsupervsed Cross-Doman Transfer n Polcy Graden Renforcemen Learnng va Manfold Algnmen Haham Bou Ammar Unv. of Pennsylvana hahamb@seas.upenn.edu Erc Eaon Unv. of Pennsylvana eeaon@cs.upenn.edu Paul Ruvolo Oln College of Engneerng paul.ruvolo@oln.edu Mahew E. Taylor Washngon Sae Unv. aylorm@eecs.wsu.edu Absrac The success of applyng polcy graden renforcemen learnng RL o dffcul conrol asks hnges crucally on he ably o deermne a sensble nalzaon for he polcy. Transfer learnng mehods ackle hs problem by reusng knowledge gleaned from solvng oher relaed asks. In he case of mulple ask domans, hese algorhms requre an ner-ask mappng o faclae knowledge ransfer across domans. However, here are currenly no general mehods o learn an ner-ask mappng whou requrng eher background knowledge ha s no ypcally presen n RL sengs, or an expensve analyss of an exponenal number of ner-ask mappngs n he sze of he sae and acon spaces. Ths paper nroduces an auonomous framework ha uses unsupervsed manfold algnmen o learn nerask mappngs and effecvely ransfer samples beween dfferen ask domans. Emprcal resuls on dverse dynamcal sysems, ncludng an applcaon o quadroor conrol, demonsrae s effecveness for cross-doman ransfer n he conex of polcy graden RL. Inroducon Polcy graden renforcemen learnng RL algorhms have been appled wh consderable success o solve hghdmensonal conrol problems, such as hose arsng n roboc conrol and coordnaon Peers & Schaal 28. These algorhms use graden ascen o une he parameers of a polcy o maxmze s expeced performance. Unforunaely, hs graden ascen procedure s prone o becomng rapped n local maxma, and hus has been wdely recognzed ha nalzng he polcy n a sensble manner s crucal for achevng opmal performance. or nsance, one ypcal sraegy s o nalze he polcy usng human demonsraons Peers & Schaal 26, whch may be nfeasble when he ask canno be easly solved by a human. Ths paper explores a dfferen approach: nsead of nalzng he polcy a random.e., abula rasa or va human demonsraons, we nsead use ransfer learnng TL o nalze he polcy for a new arge doman based on knowledge from one or more source asks. In RL ransfer, he source and arge asks may dffer n her formulaons Taylor & Sone 29. In parcular, Copyrgh c 215, Assocaon for he Advancemen of Arfcal Inellgence All rghs reserved. when he source and arge asks have dfferen sae and/or acon spaces, an ner-ask mappng Taylor e al. 27a ha descrbes he relaonshp beween he wo asks s ypcally needed. Ths paper nroduces a framework for auonomously learnng an ner-ask mappng for cross-doman ransfer n polcy graden RL. rs, we learn an ner-sae mappng.e., a mappng beween saes n wo asks usng unsupervsed manfold algnmen. Manfold algnmen provdes a powerful and general framework ha can dscover a shared laen represenaon o capure nrnsc relaons beween dfferen asks, rrespecve of her dmensonaly. The algnmen also yelds an mplc ner-acon mappng ha s generaed by mappng rackng saes from he source o he arge. Gven he mappng beween ask domans, source ask raecores are hen used o nalze a polcy n he arge ask, sgnfcanly mprovng he speed of subsequen learnng over an unnformed nalzaon. Ths paper provdes he followng conrbuons. rs, we nroduce a novel unsupervsed mehod for learnng nersae mappngs usng manfold algnmen. Second, we show ha he dscovered subspace can be used o nalze he arge polcy. Thrd, our emprcal valdaon conduced on four dssmlar and dynamcally chaoc ask domans e.g., conrollng a hree-lnk car-pole and a quadroor aeral vehcle shows ha our approach can a auomacally learn an ner-sae mappng across MDPs from he same doman, b auomacally learn an ner-sae mappng across MDPs from very dfferen domans, and c ransfer nformave nal polces o acheve hgher nal performance and reduce he me needed for convergence o near-opmal behavor. Relaed Work Learnng an ner-ask mappng has been of maor neres n he ransfer learnng communy because of s promse of auonomous ransfer beween very dfferen asks Taylor & Sone 29. However, he maory of exsng work assumes ha a he source ask and arge ask are smlar enough ha no mappng s needed Baneree & Sone 27; Kondars & Baro 27, or b an ner-ask mappng s provded o he agen Taylor e al. 27a; Torrey e al. 28. The man dfference beween hese mehods and hs paper s ha we are neresed n learnng a mappng beween asks. There has been some recen work on learnng such mappngs. or example, mappngs may be based on seman- 254

2 c knowledge abou sae feaures beween wo asks Lu & Sone 26, background knowledge abou he range or ype of sae varables Taylor e al. 27b, or ranson models for each possble mappng could be generaed and esed Taylor e al. 28. However, here are currenly no general mehods o learn an ner-ask mappng whou requrng eher background knowledge ha s no ypcally presen n RL sengs, or an expensve analyss of an exponenal number n he sze of he acon and sae varable ses of ner-ask mappngs. We overcome hese ssues by auomacally dscoverng hgh-level feaures and usng hem o ransfer knowledge beween agens whou sufferng from an exponenal exploson. In prevous work, we used sparse codng, sparse proecon, and sparse Gaussan processes o learn an ner-ask mappng beween MDPs wh arbrary varaons Bou Ammar e al However, hs prevous work reled on a Eucldean dsance correlaon beween source and arge ask rples, whch may fal for hghly dssmlar asks. Addonally, placed resrcons on he ner-ask mappng ha reduced he flexbly of he learned mappng. In oher relaed work, Bósc e al. 213 use manfold algnmen o asss n ransfer. The prmary dfferences wh our work are ha he auhors a focus on ransferrng models beween dfferen robos, raher han polces/samples, and b rely on source and arge robos ha are qualavely smlar. Background Renforcemen Learnng problems nvolve an agen choosng sequenal acons o maxmze s expeced reurn. Such problems are ypcally formalzed as a Markov decson process MDP T = S, A, P, P,r, where S s he poenally nfne se of saes, A s he se of acons ha he agen may execue, P : S [, 1] s a probably dsrbuon over he nal sae, P : S A S [, 1] s a sae ranson probably funcon descrbng he ask dynamcs, and r : S A S Rs he reward funcon measurng he performance of he agen. A polcy π : S A [, 1] s defned as a condonal probably dsrbuon over acons gven he curren sae. The agen s goal s o fnd a polcy π whch maxmzes he average expeced reward: π =argmax π =argmax π E T [ 1 H H ] rs, a, s +1 π =1 p π τ Rτ dτ, where T s he se of all possble raecores wh horzon H, Rτ = 1 H rs, a, s +1, and 2 H =1 p π τ =P s 1 1 H Ps +1 s, a πa s. 3 =1 Polcy Graden mehods Suon e al. 1999; Peers e al. 25 represen he agen s polcy π as a funcon defned over a vecor θ R d of conrol parameers and a vecor of sae feaures gven by he ransformaon Φ : S R m. By subsung hs parameerzaon of he conrol polcy no Eqn. 2, we can compue he parameers of he opmal polcy as θ = argmax θ J θ, where J θ = T p πθτ Rτ dτ. To maxmze J, many polcy graden mehods employ sandard supervsed funcon approxmaon o learn θ by followng an esmaed graden of a lower bound on he expeced reurn of J θ. Polcy graden algorhms have ganed aenon n he RL communy n par due o her successful applcaons on real-world robocs Peers e al. 25. Whle such algorhms have a low compuaonal cos per updae, hghdmensonal problems requre many updaes by acqurng new rollous o acheve good performance. Transfer learnng can reduce hs daa requremen and accelerae learnng. Snce polcy graden mehods are prone o becomng suck n local maxma, s crucal ha he polcy be nalzed n a sensble fashon. A common echnque Peers & Schaal 26; Argall e al. 29 for polcy nalzaon s o frs collec demonsraons from a human conrollng he sysem, hen use supervsed learnng o f polcy parameers ha maxmze he lkelhood of he human-demonsraed acons, and fnally use he fed parameers as he nal polcy parameers for a polcy graden algorhm. Whle hs approach works well n some sengs, s napplcable n several common cases: a when s dffcul o nsrumen he sysem n queson so ha a human can successfully perform a demonsraon, b when an agen s consanly faced wh new asks, makng gaherng human demonsraons for each new ask mpraccal, or c when he asks n queson canno be nuvely solved by a human demonsraor. The nex secon nroduces a mehod for usng ransfer learnng o nalze he parameers of a polcy n a way ha s no suscepble o hese lmaons. Our expermenal resuls show ha hs mehod of polcy nalzaon, when compared o random polcy nalzaon, s able o no only acheve beer nal performance, bu also oban a hgher performng polcy when run unl convergence. Polcy Graden Transfer Learnng Transfer learnng ams o mprove learnng mes and/or behavor of an agen on a new arge ask T by reusng knowledge from a solved source ask T S. In RL sengs, each ask s descrbed by an MDP: ask T S = S S, A S, P S, P S,r S and T = S, A, P, P,r. One way n whch knowledge from a solved source ask can be leveraged o solve he arge ask s by mappng opmal sae, acon, nex sae rples from he source ask no he sae and acon spaces of he arge ask. Transferrng opmal rples n hs way allows us o boh provde a beer umpsar and learnng ably o he arge agen, based on he source agen s ably. Whle he precedng dea s aracve, complexes arse when he source and arge asks have dfferen sae and/or acon spaces. In hs case, one mus defne an ner-ask mappng χ n order o ranslae opmal rples from he source o he arge ask. Typcally Taylor & Sone 29, χ s defned by wo sub-mappngs: 1 an ner-sae mappng χ S and 2 an ner-acon mappng χ A. 255

3 races from πs Phase I: Learn cross-doman mappng α T S shared represenaon α T χ S races from arge 3. execue πs 2. reflec arge 1. sample nal saes α T+ α T P S P S Source Doman G S Phase II: Cross-doman ransfer va χ S α T S α T+ 4. ransfer rackng sgnal Targe Doman G T gure 1: Transfer s spl no wo phases: I learnng he ner-sae mappng χ S va manfold algnmen, and II nalzng he arge polcy va mappng he source ask polcy. By adopng an RL framework where polces are saefeedback conrollers, we show ha we can use opmal sae raecores from he source ask o nellgenly nalze a conrol polcy n he arge ask, whou needng o explcly consruc an ner-acon mappng. We accomplsh hs by learnng a pseudo-nverble ner-sae mappng beween he sae spaces of a par of asks usng manfold algnmen, whch can hen be used o ransfer opmal sequences of saes o he arge. The fac ha our algorhm does no requre learnng an explc ner-acon mappng sgnfcanly reduces s compuaonal complexy. Our approach consss of wo phases gure 1. rs, usng races gahered n he source and arge asks, we learn an ner-sae mappng χ S usng manfold algnmen Phase I n gure 1. To perform hs sep, we adap he Unsupervsed Manfold Algnmen UMA algorhm Wang & Mahadevan 29, as dealed n he nex secon. Second, we use χ S o proec sae raecores from he source o he arge ask Phase II n gure 1. These proeced sae raecores defne a se of a rackng raecores for he arge ask ha allow us o perform one sep of polcy graden mprovemen n he arge ask. Ths polcy mprovemen sep nellgenly nalzes he arge polcy, whch resuls n superor learnng performance han sarng from a randomly nalzed polcy, as shown n our expermens. Alhough we focus on polcy graden mehods, our approach could easly be adaped o oher polcy search mehods e.g., PoWER, REPS, ec.; see Kober e al Learnng an Iner-Sae Mappng Unsupervsed Manfold Algnmen UMA s a echnque ha effcenly dscovers an algnmen beween wo daases Wang & Mahadevan 29. UMA was developed o algn daases for knowledge ransfer beween wo supervsed learnng asks. Here, we adap UMA o an RL seng by algnng source and arge ask sae spaces wh poenally dfferen dmensons m S and m T. To learn χ S relang S S and S, raecores of } saes n he source ask, τs = ns s,s 1,,s,S H S, are obaned by =1 followng πs, and raecores of saes n he arge ask, } τ = s, 1,,s, nt H T, are obaned by ulzng π, whch s nalzed usng randomly seleced polcy =1 parameers. or smplcy of exposon, we assume ha raecores n he source doman have lengh H S and hose n he arge doman have lengh H T ; however, our algorhm s capable of handlng varable-lengh raecores. We are neresed n he seng where daa s scarcer n he arge ask han n he source ask.e., n T n S. Gven raecores from boh he source and arge asks, we flaen he raecores.e., we rea he saes as unordered and hen apply he ask-specfc sae ransformaon o oban wo ses of sae feaure vecors, one for he source ask and one for he arge ask. Specfcally, we creae he followng ses of pons: X S = Φ S s 1S 1 X =,,Φ S s 1S H S, } Φ S s n SS 1,,Φ S s n SS H S Φ s 1 1,,Φ s 1 H T, } Φ s n T 1,,Φ s n T H T. Gven X S R m S H S n S, X R m T H T n T,we can apply he UMA algorhm Wang & Mahadevan 29 wh mnmal modfcaon, as descrbed nex. Unsupervsed Manfold Algnmen UMA The frs sep of applyng UMA o learn he ner-sae mappng s o represen each ransformed sae n boh he source and arge asks n erms of s local geomery. We use he noaon R S x R k+1 k+1 o refer o he marx of parwse Eucldean dsances among he k-neares neghbors of x S X S. Smlarly, R x refers o he equvalen marx of dsances for he k-neares neghbors of x X. The relaons beween local geomeres n X S and X are represened by he marx W R n S H S n T H T wh w, =exp ds R S x ds R S x, R x = [ mn 1 h k!, R x mn R x R x S } and dsance merc h γ 1 R x S, 4 γ 2 R x h ]. We use he noaon h o denoe he h h varan of he k! permuaons of he rows and columns of he npu marx, s he robenus norm, and γ 1 and γ 2 are defned as: r γ 1 = r R T x S R T x S R x R x S h γ 2 = r Rx T h R x S r Rx T h R x h. 256

4 To algn he manfolds, UMA compues he on Laplacan LX S + μγ L = 1 μγ 2 μγ 3 L X + μγ 4 5 wh dagonal marces Γ 1 Γ 4 R n T H T n T H T, where Γ 1, R n S H S n S H S and = w, and Γ 4, = w,. The marces Γ 2 R n S H S n T H T and Γ 3 R n T H T n S H S on he wo manfolds wh Γ 2, = w, and Γ 3, = w,. Addonally, he non-normalzed Laplacans L X S and L X are defned as: L X S = D X S W X S and L X = D X W X, where D X S R n S H S n S H S s a dagonal marx wh D, = X S ws, and, smlarly, D, = X w,. The marces WS and W represen he smlary n he source and arge ask sae spaces respecvely and can be compued smlar o W. To on he manfolds, UMA frs defnes wo marces: τ Z = S τ DS S D = D S. 6 Gven Z and D, UMA compues opmal proecons o reduce he dmensonaly of he on srucure by akng he d mnmum egenvecors ζ 1,,ζ d of he generalzed egenvalue decomposon ZLZ T ζ = λzdz T ζ. The opmal proecons α S and α are hen gven as he frs d 1 and d 2 rows of [ζ 1,,ζ d ], respecvely. Gven he embeddng dscovered by UMA, we can hen defne he ner-sae mappng as: χ S [ ] =α T+ αt S[ ]. 7 The nverse of he ner-sae mappng o proec arge saes o he source ask can be deermned by akng he pseudo-nverse of Eqn. 7, yeldng χ + S [ ] =αt+ S αt [ ]. Inuvely, hs approach algns he mporan regons of he source ask s sae space sampled based on opmal source raecores wh he sae space explored so far n he arge ask. Alhough acons were gnored n consrucng he manfolds, he algned represenaon mplcly capures local sae ranson dynamcs whn each ask snce he saes came from raecores, provdng a mechansm o ransfer raecores beween asks, as we descrbe nex. Polcy Transfer and Improvemen Nex, we dscuss he procedure for nalzng he arge polcy, π. We consder a model-free seng n whch he polcy s lnear over a se of poenally non-lnear sae feaure funcons modulaed by Gaussan nose where he magnude of he nose balances exploraon and exploaon. Specfcally, we can wre he source and arge polces as: π S π s S s = Φ S s S = Φ s T θ S + ɛ S T θ + ɛ, where ɛ S N, Σ S and ɛ N, Σ. τ =1 To nalze π, we frs sample m nal arge raecores D = from he arge ask usng a } m randomly nalzed polcy hese can be newly sampled raecores or smply he ones used o do he nal manfold algnmen sep. Nex, we map he se of nal saes n D o he source ask usng χ + S. We hen run he opmal source polcy sarng from each of hese mapped nal saes o produce a se of m opmal sae raecores n he source ask. nally, he resulng sae raecores are mapped back o he arge ask usng χ S o generae a se of refleced sae- raecores n he arge ask, D = τ } m =1. or clary, we assume ha all raecores are of lengh H; however, hs s no a fundamenal lmaon of our algorhm. We defne he followng ransfer cos funcon: J T S T θ = m p θ =1 τ ˆR τ, τ where ˆR s a cos funcon ha penalzes devaons beween he nal sampled raecores n he arge ask and he refleced opmal raecores: ˆR τ, τ = 1 H H s s = Mnmzng, Eqn. 8 s equvalen o aanng a arge polcy parameerzaon θ such ha π follows he refleced raecores D. urher, Eqn. 8 s n exacly he form requred o apply sandard off-he-shelf polcy graden algorhms o mnmze he ransfer cos. The Manfold Algnmen Cross-Doman Transfer for Polcy Gradens MAXDT-PG framework s dealed 1 n Algorhm 1. Specal Cases Our work can be seen as an exenson of he smpler modelbased case wh a lnear-quadrac regulaor LQR Bemporad e al. 22 polcy, whch s derved and explaned n he onlne appendx 2 accompanyng hs paper. Alhough he assumpons made by he model-based case seem resrcve, he analyss n he appendx covers a wde range of applcaons. These, for example, nclude: a he case n whch a dynamcal model s provded beforehand, or b he case n whch model-based RL algorhms are adoped see Buşonu e al. 21. In he man paper, however, we consder he more general model-free case. Expermens and Resuls To assess MAXDT-PG s performance, we conduced expermens on ransfer boh beween asks n he same doman as well as beween asks n dfferen domans. Also, we suded 1 Lnes 9-11 of Algorhm 1 requre neracon wh he arge doman or a smulaor for acqurng he opmal polcy. Such an assumpon s common o polcy graden mehods, where a each eraon, daa s gahered and used o eravely mprove he polcy. 2 The onlne appendx s avalable on he auhors webses. 257

5 θ, θ Algorhm 1 Manfold Algnmen Cross-Doman Transfer for Polcy Gradens MAXDT- PG Inpus: Source and arge asks T S and T, opmal source polcy πs, # source and arge races ns and nt, # neares neghbors k, # arge rollous zt, nal # of arge saes m. Learn χs : 1: Sample ns opmal source races, τs, and nt random arge races, τ 2: Usng he modfied UMA approach, learn αs and T α o produce χs = αt+ αs [ ] Transfer & Inalze Polcy: 3: Collec m nal arge saes s1 P 4: Proec hese m saes o he source by applyng χ+ S [ ] 5: Apply he opmal source polcy πs on hese proeced S saes o collec DS = τ x, x a Smple Mass θ 3, θ 3 θ 1, θ 1 x, x b Car Pole 2 e11 e21 e31 r 3 e3b e 2B e1 1 Φ B l rol Θ p ch 4 Ψ yaw d Quadroor gure 2: Dynamcal sysems used n he expermens. =1 Three-Lnk Car Pole 3CP: The 3CP dynamcs are descrbed va an egh-dmensonal sae vecor x, x, θ1, θ 1, θ2, θ 2, θ3, θ 3, where x and x descrbe he poson and velocy of he car and θ and θ represen he angle and angular velocy of he h lnk. The sysem s conrolled by applyng a force o he car n he x drecon, wh he goal of balancng he hree poles uprgh. Quadroor QR: The sysem dynamcs were adoped from a smulaor valdaed on real quadroors Bouabdallah 27; Voos & Bou Ammar 21, and are descrbed va hree angles and hree angular veloces n he body frame.e., e1b, e2b, and e3b. The acons conss of four roor orques 1, 2, 3, 4 }. Each ask corresponds o a dfferen quadroor configuraon e.g., dfferen armaure lenghs, ec., and he goal s o sablze he dfferen quadroors. 6: Proec he samples n D S o he arge usng χs [ ] o produce rackng arge races D θ 2, θ 2 x, x c Three-Lnk Car Pole m 7: Compue rackng rewards usng Eqn. 9 8: Use polcy gradens o mnmze Eqn. 8, yeldng θ Improve Polcy: 9: Sar wh θ and sample zt arge rollous 1: ollow polcy gradens e.g., epsodc REINORCE bu usng arge rewards R 11: Reurn opmal arge polcy parameers θt he robusness of he learned mappng by varyng he number of source and arge samples used for ransfer and measurng he resulan arge ask performance. In all cases we compared he performance of MAXDT- PG o sandard polcy graden learners. Our resuls show ha MAXDT- PG was able o: a learn a vald ner-sae mappng wh relavely lle daa from he arge ask, and b effecvely ransfer beween asks from eher he same or dfferen domans. Same-Doman Transfer We firs evaluae MAXDT- PG on same-doman ransfer. Whn each doman, we can oban dfferen asks by varyng he sysem parameers e.g., for he SM sysem we vared mass M, sprng consan K, and dampng consan b as well as he reward funcons. We assessed he performance of usng he ransferred polcy from MAXDT- PG versus sandard polcy gradens by measurng he average reward on he arge ask vs. he amoun of learnng eraons n he arge. We also examned he robusness of MAXDT- PG s performance based on he number of source and arge samples used o learn χs. Rewards were averaged over 5 races colleced from 15 nal saes. Due o space consrans, we repor same-doman ransfer resuls here; deals of he asks and expermenal procedure can be found n he appendx2. gure 3 shows MAXDT- PG s performance usng varyng numbers of source and arge samples o learn χs. These resuls reveal ha ransfer-nalzed polces ouperform sandard polcy graden nalzaon. urher, as he number of samples used o learn χs ncreases, so does boh he nal and final performance n all domans. All nalzaons resul n equal per-eraon compuaonal cos. Therefore, MAXDT- PG boh mproves sample complexy and reduces wall-clock learnng me. Dynamcal Sysem Domans We esed MAXDT- PG and sandard polcy graden learnng on four dynamcal sysems gure 2. On all sysems, he reward funcon was based on wo facors: a penalzng saes far from he goal sae, and b penalzng hgh forces acons o encourage smooh, low-energy movemens. Smple Mass Sprng Damper SM: The goal wh he SM s o conrol he mass a a specfied poson wh zero velocy. The sysem dynamcs are descrbed by wo saevarables ha represen he mass poson and velocy, and a sngle force ha acs on he car n he x drecon. Car Pole CP: The goal s o swng up and hen balance he pole vercally. The sysem dynamcs are descrbed va a four-dmensonal sae vecor x, x, θ, θ, represenng he poson, velocy of he car, and he angle and angular velocy of he pole, respecvely. The acons conss of a force ha acs on he car n he x drecon. 258

6 Average Reward Source 1 Targe Samples 5 Source 5 Targe Samples Source 3 Targe Samples 6 1 Source 1 Targe Samples Sandard Polcy Gradens Ieraons Ieraons Ieraons Ieraons 3 a Smple Mass b Car Pole c Three-Lnk Car Pole d Quadroor gure 3: Same-doman ransfer resuls. All plos share he same legend and vercal axs label. Average Reward Source 1 Targe Samples Source 5 Targe Samples 5 Source 3 Targe Samples 1 Source 1 Targe Samples 315 Sandard Polcy Gradens Ieraons Ieraons Ieraons a Smple Mass o Car Pole 35 b Car Pole o Three-Lnk CP c Car Pole o Quadroor θ r θ * Targe: SM Targe: CP Targe: 3CP Source: SM Source: CP Source: 3CP Procruses Measure d Algnmen Qualy vs Transfer gure 4: Cross-doman ransfer resuls. Plos a c depc arge ask performance, and share he same legend and axs labels. Plo d shows he correlaon beween manfold algnmen qualy Procruses merc and qualy of he ransferred knowledge. Cross-Doman Transfer Nex, we consder he more dffcul problem of crossdoman ransfer. The expermenal seup s dencal o he same-doman case wh he crucal dfference ha he sae and/or acon spaces were dfferen for he source and he arge ask snce he asks were from dfferen domans. We esed hree cross-doman ransfer scenaros: smple mass o car pole, car pole o hree-lnk car pole, and car pole o quadroor. In each case, he source and arge ask have dfferen numbers of sae varables and sysem dynamcs. Deals of hese expermens are avalable n he appendx 2. gure 4 shows he resuls of cross-doman ransfer, demonsrang ha MAXDT-PG can acheve successful ransfer beween dfferen ask domans. These resuls renforce he conclusons of he same-doman ransfer expermens, showng ha a ransfer-nalzed polces ouperform sandard polcy gradens, even beween dfferen ask domans and b nal and fnal performance mproves as more samples are used o learn χ S. We also examned he correlaon beween he qualy of he manfold algnmen, as assessed by he Procruses merc Goldberg & Rov 29, and he qualy of he ransferred knowledge, as measured by he dsance beween he ransferred θ r and he opmal θ parameers gure 4d. On boh measures, smaller values ndcae beer qualy. Each daa pon represens a ransfer scenaro beween wo dfferen asks, from eher SM, CP, or 3CP; we dd no consder quadroor asks due o he requred smulaor me. Alhough we show ha he Procruses measure s posvely correlaed wh ransfer qualy, we hesae o recommend as a predcve measure of ransfer performance. In our approach, he cross-doman mappng s no guaraneed o be orhogonal, and herefore he Procruses measure s no heorecally guaraneed o accuraely measure he qualy of he global embeddng.e., Goldberg and Rov s 29 Corollary 1 s no guaraneed o hold, bu he Procruses measure sll appears correlaed wh ransfer qualy n pracce. We can conclude ha MAXDT-PG s capable of: a auomacally learnng an ner-sae mappng, and b effecvely ransferrng beween dfferen doman sysems. Even when he source and arge asks are hghly dssmlar e.g., car pole o quadroor, MAXDT-PG s capable of successfully provdng arge polcy nalzaons ha ouperform saeof-he-ar polcy graden echnques. Concluson We nroduced MAXDT-PG, a echnque for auonomous ransfer beween polcy graden RL algorhms. MAXDT-PG employs unsupervsed manfold algnmen o learn an nersae mappng, whch s hen used o ransfer samples and nalze he arge ask polcy. MAXDT-PG s performance was evaluaed on four dynamcal sysems, demonsrang ha MAXDT-PG s capable of mprovng boh an agen s nal and fnal performance relave o usng polcy graden algorhms whou ransfer, even across dfferen domans. 259

7 Acknowledgemens Ths research was suppored by ONR gran #N , AOSR gran #A , and NS gran IIS We hank he anonymous revewers for her helpful feedback. References Argall, B. D.; Chernova, S.; Veloso, M.; and Brownng, B. 29. A survey of robo learnng from demonsraon. Robocs and Auonomous Sysems 575: Baneree, B., and Sone, P. 27. General game learnng usng knowledge ransfer. In Proceedngs of he 2h Inernaonal Jon Conference on Arfcal Inellgence, Bemporad, A.; Morar, M.; Dua, V.; and Pskopoulos, E. 22. The explc lnear quadrac regulaor for consraned sysems. Auomaca 381:3 2. Bócs, B.; Csao, L.; and Peers, J Algnmen-based ransfer learnng for robo models. In Proceedngs of he Inernaonal Jon Conference on Neural Neworks IJCNN. Bou Ammar, H.; Taylor, M.; Tuyls, K.; Dressens, K.; and Wess, G Renforcemen learnng ransfer va sparse codng. In Proceedngs of he 11h Conference on Auonomous Agens and Mulagen Sysems AAMAS. Bouabdallah, S. 27. Desgn and Conrol of Quadroors wh Applcaon o Auonomous lyng. Ph.D. Dsseraon, École polyechnque fédérale de Lausanne. Buşonu, L.; Babuška, R.; De Schuer, B.; and Erns, D. 21. Renforcemen Learnng and Dynamc Programmng Usng uncon Approxmaors. Boca Raon, lorda: CRC Press. Goldberg, Y.; and Rov, Y. 29. Local Procruses for manfold embeddng: a measure of embeddng qualy and embeddng algorhms. Machne Learnng 771: Kober, J.; Bagnell, A.; and Peers, J Renforcemen learnng n robocs: a survey. Inernaonal Journal of Robocs Research 3211: Kondars, G., and Baro, A. 27. Buldng porable opons: Skll ransfer n renforcemen learnng. In Proceedngs of he 2h Inernaonal Jon Conference on Arfcal Inellgence, Lu, Y., and Sone, P. 26. Value-funcon-based ransfer for renforcemen learnng usng srucure mappng. In Proceedngs of he 21s Naonal Conference on Arfcal Inellgence, Peers, J., and Schaal, S. 26. Polcy graden mehods for robocs. In Proceedngs of he IEEE/RSJ Inernaonal Conference on Inellgen Robos and Sysems, Peers, J., and Schaal, S. 28. Naural acor-crc. Neurocompung 717-9: Peers, J.; Vayakumar, S.; and Schaal, S. 25. Naural acor-crc. In Proceedngs of he 16h European Conference on Machne Learnng ECML, Sprnger. Suon, R. S.; McAlleser, D. A.; Sngh, S. P.; and Mansour, Y Polcy graden mehods for renforcemen learnng wh funcon approxmaon. Neural Informaon Processng Sysems Taylor, M. E., and Sone, P. 29. Transfer learnng for renforcemen learnng domans: a survey. Journal of Machne Learnng Research 1: Taylor, M. E.; Kuhlmann, G.; and Sone, P. 28. Auonomous ransfer for renforcemen learnng. In Proceedngs of he 7h Inernaonal Jon Conference on Auonomous Agens and Mulagen Sysems AAMAS, Taylor, M. E.; Sone, P.; and Lu, Y. 27. Transfer learnng va ner-ask mappngs for emporal dfference learnng. Journal of Machne Learnng Research 81: Taylor, M. E.; Wheson, S.; and Sone, P. 27. Transfer va ner-ask mappngs n polcy search renforcemen learnng. In Proceedngs of he 6h Inernaonal Jon Conference on Auonomous Agens and Mulagen Sysems. Torrey, L.; Shavlk, J.; Walker, T.; and Macln, R. 28. Relaonal macros for ransfer n renforcemen learnng. In Blockeel, H.; Ramon, J.; Shavlk, J.; and Tadepall, P., eds., Inducve Logc Programmng, volume 4894 of Lecure Noes n Compuer Scence. Sprnger Berln Hedelberg Voos, H., and Bou Ammar, H. 21. Nonlnear rackng and landng conroller for quadroor aeral robos. In Proceedngs of he IEEE Inernaonal Conference on Conrol Applcaons CCA, Wang, C., and Mahadevan, S. 29. Manfold algnmen whou correspondence. In Proceedngs of he 21s Inernaonal Jon Conference on Arfcal Inellgence IJCAI, Morgan Kaufmann. 251