Visual Robot Homing using Sarsa(λ), Whole Image Measure, and Radial Basis Function.

Transcription

1 Vsul Robo Homng usng Srs(λ, Whole Imge Mesure, nd Rdl Bss Funcon. Abdulrhmn Alhhn, Kevn Burn, Sefn Wermer Hybrd Inellgen Sysems Reserch Group, School of Compung nd echnology, Unversy of Sunderlnd, SR6 DD UK Absrc hs pper descrbes model for vsul homng. I uses Srs(λ s s lernng lgorhm, combned wh he Jeffery Dvergence Mesure (JDM s wy of ermnng he sk nd ugmenng he rewrd sgnl. he vsul feures re ken o be he hsogrms dfference of he curren vew nd he sored vews of he gol locon, ken for ll RGB chnnels. A rdl bss funcon lyer cs on hose hsogrms o provde npu for he lner funcon pproxmor. An onpolcy on-lne Srs(λ mehod ws used o rn hree lner neurl neworks one for ech con o pproxme he convlue funcon wh he d of elgbly rces. he resuln neworks re rned o perform vsul robo homng, where hey cheved good resuls n fndng gol locon. hs work demonsres h vsul homng bsed on renforcemen lernng nd rdl bss funcon hs hgh poenl for lernng locl nvgon sks. I. INRODUCION A skll whch plys n negrl role n chevng robo uonomy s he bly o lern o opere n pror unknown envronmens[]. Vsul homng s he c of fndng gol locon by comprng he mge currenly vewed wh sored snpsho mges (normlly ken whle nml or robo s hedng off s home locon. Vsul nvgon s he c of nvgng form one locon o he oher n he envronmen, s effcenly s possble. In hs pper we presen model for vsul homng, whch cn lso be used n locl nvgon, usng renforcemen lernng (RL from now on nd n onlne snpsho comprson echnque. hs snpsho comprson fcles onlne lernng nd execuon n pror unknown envronmens o rech gol locon. Robocs borrows severl conceps from nml homng nd nvgon sreges descrbed n he bologcl lerure [2, 3]. Whle boh vsul homng nd vsul nvgon re reled, hey hve been kep frly pr due o he fc h vsul homng s more nspred by he bology nd due o he fc h vsul nvgon s more generl hn vsul homng. Neverheless, nvgon cn be ccomplshed more drecly by usng locl homng sreges o rech some locon, whou drecly buldng mp or Noe: gol locon nd home locon wll be used nerchngebly n hs pper. usng model of envronmen dynmcs. he lmon s h he lerned sreges o nvge o home s bound o h prculr locon. herfore, f he robo needs o nvge o dfferen locon, should be rned o do so. We rgue h our model cn lso be used for generl nvgon sks due o he fc h cn opere n ny envronmen nd requres no ddonl effor excep showng he robo, onlne or offlne, s gol locon, hen leng rns. Algorhms bsed on he snpsho model [3] propose vrous sreges for fndng feures whn mges nd esblshng correspondence beween hem n order o deermne home drecon. Block mchng, for exmple, kes block of pxels from one mge nd serches for he bes mchng block n noher mge whn fxed serch rdus [4]. he degree of mch beween blocks s usully judged by he Sum of Squred Dfferences (SSD or some oher locl correlon mesure[5]. In our model we wll ke more effecve pproch by comprng bns of hsogrms hrough Rdl Bses Funcon lyer, nd usng mges only ken round he home, nohng more. Renforcemen Lernng hs been used prevously n robocs nvgon nd conrol problems. Severl of he models h used re nspred by bologcl fndngs, e.g. [6]. Alhough successful, some of hose models lck he generly nd/or prccly, nd some re resrced o her envronmen. he model proposed by [7] for exmple depends hevly on objec recognon of lndmrk n he envronmen o cheve he sk. We hve ddressed hs ssue n our model by vodng objec recognon nd usng whole mge mesure echnque nsed, o mesure he dssmlry of curren nd gol vews o denfy wheher he robo reched he gol locon (wh he desred orenon. hs ws possble wh no pror knowledge or consrns regrdng hose mges. By ddng he bove dvnge o he lernng robusness nd generly of RL, coupled wh vsul ses nd rewrds, he model cheved hgh level of robusness, generly, nd pplcbly. Whle envronmen-dynmcs or mp-buldng my be necessry for more complex or nercve forms of nvgon or loclzon, vsul homng bsed on modelfree lernng cn offer n dpve form of locl homng. Alhough he mmede execuon of model-bsed

2 nvgon sysem cn be successful [8, 9], RL echnques hve go he dvnges of model-free sysems.e. here s no knowledge needed pror o operng he robo. I lerns he bes polcy for he envronmen dynmcs. Whle he de of usng snpshos o do robo loclzon s no new [], vsul homng bsed on renforcemen lernng nd rdl bss npu lyer nd whole mge mesure s novel conrbuon of hs pper. We begn by presenng n overvew of our renforcemen lernng conex nd Mrkov Decson Processes (MDP frmework followed by he emporl Dfference (D lernng lgorhm for connuous ses spce. hs s followed by deled descrpon of our model, demonsrng generly nd smplcy of execuon. hen we presen emprcl resuls of robo rechng gol locon vsully n smulon envronmen. II. BACKGROUND OF REINFORCEMEN LEARNING Renforcemen lernng concerns he problem of lernng o predc he sum of rewrds n gen s recevng whle nercng wh s envronmen n order o opmlly execue sk []. Insed of beng gven exmples of he desred behvor, he lernng gen mus fnd ou - usng s envronmen feedbck nd usng grdul explorve cons - how o c bes o execue sk. Usully hs feedbck s mnml sgnl of rewrd or punshmen nduced n some wy n he envronmen. hs sgnl s clled he renforcemen sgnl. In ny envronmen here exss se of ses h represen he suons h he gen cn fce (or recognze. hose ses defne he se spce denoed by S, whch cn be fne or nfne nd connuous. he cons re hose smple cves he gen s ble o do n cern se. he se of hose cons defne he cons spce A. hose cons cn lso be fne or nfne. he envronmen normlly recs or responds o ny con ken by he gen by reurnng sgnl ndcng or renforcng how good or bd hs con ws for he sk. I s clled he rewrd sgnl or he renforcemen sgnl. he dynmcs of n envronmen re he se of probbly dsrbuons h dsngush s nernl properes. hose re mnly he se rnson funcon nd he rewrd funcon. he se rnson funcon s probbly dsrbuon defned on he se spce h specfes he probbly of movng form se s me o noher se s' me + fer pplyng con : P Pr{ s = s s = s, } s s = + = he rewrd funcon s defned s he expeced rewrd reurned by he envronmen for ech se fer pplyng cern con: R s s = E{ r s = s, s s + =, = } where r s he cul rewrd reurned by he envronmen nd fully observed by he gen. As wh mos renforcemen work, we wll resrn ourselves o he Mrkovn envronmens. A Mrkov decson process (MDP s defned by uple ( S, A, P s s, Rss, γ, where γ [,] s dscoun re prmeer, nd where he Mrkov propery s ssfed [9, ]. A rjecory of experence s sequence s,, r2,s2, 2, r3,... where he gen n s kes con hen receves rewrd r 2 nd rnsonng o s 2 before kng 2, ec. A polcy specfes (probblsclly or deermnsclly he con h needs o be ken for ech dfferen se. : S A [,], ( s, =. where ( s, s he probbly of selecng con when n gen s n se s. A deermnsc polcy s mppng beween ses nd cons : S A. he ulme gol of renforcemen lernng mehods (lgorhms s o lern n opmum polcy h, when followed, mxmzes he ccumuled rewrds expeced o be gned by he gen durng nercon wh s envronmen. hs s normlly reched hrough esmng he expeced sum n some form snce model of he envronmen s normlly no vlble nd undesred o be requremen. Even n mehods h ssume model of he envronmen dynmcs o be known, such s Dynmc Progrmmng mehods, he expecon sll needs o be esmed due o he boosrppng chrcersc of such mehod. By boosrppng we men buldng on n own nl esmon o rech beer esmon closer o he rel vlue []. he dscouned sum of rewrds me sep s clled he reurn R where: R = r r γ r = k= γ γ r ( k + k+ Expeced ccumuled rewrds for cern polcy cn be expressed n wo forms: he vlue funcon V (s nd he con-vlue funcon Q ( s,. A vlue funcon for polcy s defned s: V ( s : S R. V specfes he expeced reurn (sum of rewrds r from he srng se s nd onwrds. Obvously ech polcy hs dfferen vlue funcon, hence he upper superscrp. = [ = ] = k V ( s E R s s E γ r + k + s = s (2 k = he cenrl de of RL s o ry o lern n esme of he vlue funcon of he doped polcy dependng on he nercon beween he gen nd s envronmen. In oher words, o predc he vlue funcon of he gen's MDP polcy. An essenl propery of he vlue funcon cn be deduced from he nrnsc recurson posses: = + k V ( s E r + γ γ r + k + 2 s = s k = ( s, P R + γv ( s (3 = ss ss s [ ] he con-vlue funcon s defned s

3 Q ( s, : S A R, where: = k Q ( s, E γ r + k + s = s, = (4 k = For clry, we wll presen below he mn resuls for he vlue funcon, hen we wll shf o he con-vlue funcon when presenng our model. III. OWARDS OUR MODEL Our work uses echnques developed for he problem of onlne on-polcy evluon, where n pproxme convlue funcon s mnned nd mproved fer ech me sep of followng he polcy. In prculr we re neresed n lner Q-funcon pproxmor h uses emporl Dfference lernng (D [2] snce D lernng cn be gurneed o converge wh ny lner funcon pproxmor nd suble sep sze [3]. For he connuous cse nd non-lner funcon pproxmon, convergence s no gurneed [4] lhough some models hve been presened wh good resuls [5] In hs work we focus on presenng model h lerns n pproxmon of polcy s con-vlue funcon from smple rjecores of experence followng h polcy. A mehod for solvng hs problem s core componen of our vsul robo homng model. In prculr, mnnng n onlne esme of he Q-funcon cn be combned wh generlzed polcy mprovemen (GPI o lern conroller []. For prculr vlue funcon V le he D error me be defned s: δ ( V r V ( s θ = + + γ V ( s (5 δ ( V θ hen, E [ δ ( V ] = = predcon+ predcon, h s, he men D error for he polcy s rue vlue funcon mus be zero. We re neresed n pproxmng V usng lner funcon pproxmor. In prculr, suppose we hve funcon whch gves feure n represenon of he se spce φ : S R. We re neresed n n pproxmed vlue funcon of he form n V = s θ θ R re he prmeers of he vlue θ ; funcon. Becuse he polcy s rue vlue funcon my no be n our spce of lner funcons, we wn o fnd se of prmeers h pproxmes he rue funcon. One possble pproch s o use he observed D error on smple rjecores of experence o gude he pproxmon. he sndrd one-sep D mehod for vlue funcon pproxmon s D(. he bsc de of D( s o djus he predced vlue of se o reduce he D error. Gven some new experence uple ( s,, r+, s +, he upde wh lner funcon pproxmon s: θ = θ + + α u ( θ (6 u θ = δ ( V s (7 ( θ V s he esmed vlue wh respec o θ θ nd α s he lernng re. he vecor u ( θ s lke grden esme h specfes how o chnge he predced vlue of s o reduce he observed D error. We wll cll u ( θ he D upde me. Afer updng he prmeer vecor, he experence uple s removed form memory. IV. HE PROPOSED VISUAL HOMING SARSA MODEL In hs secon we descrbe he proposed model. In he smples perspecve, ny renforcemen lernng model, (or ny MDP model n generl, consss of elemens nd experence gned bou hose elemens. he envronmen dynmcs encoded n he uple ( S, A, P ss, R ss, γ descrbes he bsc elemens of he model, whle he nercon beween he robo nd he envronmen consues he gned experence. hs experence s normlly encoded n he lernng prmeers usng some lernng mehod h mnly lerns vlue funcon. For conrol, rechng n * opml polcy cn be done hrough polcy mprovemen. We frs begn by descrbng he mn elemens, hen we descrbe he lernng rules nd lgorhm, nd conclude hs secon wh he overll model srucure. A. Bsc Elemens of he Sysem, he Se Spce: Snce we re consderng vsul homng, s nurl o choose he vson s he mn medum o dsngush beween dfferen suons. Hence, we ssume s he mge ech me sep h represens he curren se, nd he se spce S s he se of ll he mges h cn be possbly ken for ny locon (wh specfc orenon n he envronmen. hs complex se spce hs wo problems. Frs, ech se s of hgh dmensonly,.e. ech se s represened by lrge number of pxel componens. Second, hs se spce s huge nd polcy cnno be lerned drecly for ech se. Insed, feure represenon of he ses s used o reduce he hgh dmensonly of he mges se spce nd o gn he dvnges of codng [6]. hs feure represenon of se spce s ssumed o reserve he dsncveness of ses, hence cn reduce he hgh-dmensonly problem bu we re sll fced by he nrcbly problem. herefore, generlzon echnque s needed n order o ccommode he nrcbly of se spce. More precsely, generlzon s needed n order o pproxme he vlue for se h hs never been vsed before, hrough prevous vss o smlr ses. A nurl wy o do so s o use funcon pproxmon echnque such s neurl nework. We would lke o encode n hose feures mplcly how dfferen he curren mge vew s from hose of he gol. hs vsul clue should gude he process of fndng he gol locon. he problem s h hs pproch does no gve drec dsnce ndcon. We wll no ssume h he gol locon s lwys n he robo's feld of vew, bu by comprng he curren vew wh he gol vew we combne

4 he properes of dsncveness, dsnce nd orenon n one represenon. B. Defnng he gol locon: Snce he home locon cn be pproched from dfferen drecons, he wy s represened should ccommode hs fc. herefore, home (or gol locon s defned by m snpshos clled he sored vews. he few snpshos (normlly m 3 of he home locon re ken he very sr, ech from fxed dsnce bu from dfferen ngle. he dsnce should be compble wh he scle of he envronmen nd he chrcerscs of he home locon. hs llows for he hghes dsncveness of he locon whou loosng nfo or nvolvng unneeded nformon. hese snpshos re he only requremen of he sysem o lern o rech s home locon srng from ny poson n he envronmen (ncludng hose from whch cnno see he home from,.e. he robo should be ble o rech hdden gol locon. C. he Feures Vecors: We ke hsogrm of ech chnnel of he curren vew nd compre wh hose of he sored vews hrough rdl bss funcon (RBF lyer. hs gves us he feure n spce Φ : S R represenon (8 whch s used wh he Srs(λ lgorhm, s we shll see ler. 2 h ( s ( c h ( v( c, φ ( s ( c, = exp( (8 2 2σ he ndex s for he me sep, j snds for he j h sored vew, nd c s he ndex of he chnnel where we used he RGB represenon of mges. So v( c, s he mge of chnnel c of he j h sored vew, h ( v( c, s he bn of he mge v ( c,, nd h ( s (c s bn of he chnnel c of he curren ( vew. Of course he number of bns hs n effec on he performnce of hs mesure nd hence on he model, nd wll be suded n he expermenl secon. D. he Acon Spce: he se of cons s A = [Lef_Forwrd, Rgh_Forwrd, Go Forwrd], where he wo dfferenl wheel speeds were se o fxed vlues so h we hve counble se of cons. E. Dssmlry Mesure nd he ermnon Condon: We need wy o deermne how close he curren poson s o he gol locon, hs s done hrough mesurng how dssmlr s he curren vew o ech sored vew of he gol locon. One cn use ny of he dssmlry mesures dscussed exensvely n he nformon rerevl feld [, 7]. In prculr we re neresed n he Jeffery Dvergence Mesure, gven by (9. ( H, K 2h 2k JDM = h log + k log (9 h + k h + k Where H nd K re wo mges o be compred, h nd k re he number of elemens belong o bn of he hsogrms of H nd K, respecvely. Fg. ( shows smple vew of robo's cmer, pr (b shows he chnges h ook plce n JDM mesuremens when urnng wy from hs locon Jeffrey Fg.. Exmple of he JDM behvour relve o he robo roonl moon JDM hs been successfully used wh omn-dreconl cmer o perform robo loclzon []. We used norml cmer, however, o be ble o dsngush he robo s orenon whch s crucl o our nvgonl sk. hs s o vod he dsdvnge of orenon-nsensvy of omn-dreconl cmer whch s desrble for loclzon bu undesrble for nvgon. We wll denoe JDM ( c, s beng he Jeffery Dvergence Mesure beween he curren vew nd he sored vew j ccordng o he chnnel c, nd we denoe o be he verge dssmlry beween he curren JDM ( vew nd he sored vew j on ll of he chnnels: JDM ( = JDM ( c, C ( c ( (b We se our ermnon se o be he curren vew for whch one of s JDM ( c, wh he m sored vews s less hn cern hresholdψ,.e. he vew h mches well wh one of he gol vews. If mn( JDM ( c, <ψ ermne Epsode. c, j he wy o se hs envronmen-scle-specfc hreshold s dscussed n he expermenl secon. F. he Rewrd Funcon: he rewrd funcon R consss of hree prs: s s -he mn pr s he cos whch s se o - for ech sep ken by he robo whou rechng he home locon (rechng ermnon se. he rewrd sgnl cn be ugmened by noher wo sgnl o nsure hgher performnce lhough he model works regrdless of her nvolvemen. hose re: -Approchng he gol rewrd: s he mxmum reducon n dssmlry beween he curren sep nd he prevous sep. If hs dfference s decresng mens h he robo s cully movng n he rgh drecon owrds he home locon. Whle f s ncresng mens he oppose. We cll hs sgnl he dfferenl dssmlry sgnl nd s defned s:

5 JDM = mx( JDM ( JDM ( ( j - he Poson sgnl s he nverse of he curren dssmlry. JDM hus, s he curren locon dffers less, from he home locon, hs rewrd wll ncrese. r = cos + JDM + (2 JDM Of course he prevous wo rewrd componen wll only be consdered f he dssmlres of boh seps flls under cern hreshold ψ o ensure h he robo s pprochng he home locon. hs hreshold s envronmen- sclespecfc, nd s nroduced merely o enhnce he performnce. he overll srucure of he model s shown n Fg. 2. JDM r _ c Curren Srs(λ Feure vecor of ech φ (s hsogrm bn θ Q+ ( φ ( s, Curren Imge Sored Vews Imges C B m RBF of ech feure wh reference vew G. he Elgbly rce: An elgbly rce consues mechnsm for emporl cred ssgnmen. I mrks he memory prmeers ssoced wh he con s elgble for undergong lernng chnges []. Dependng on our pplcon, he elgbly rce for con s he dscouned sum of he feure vecors for he mges h he robo hs seen so fr, fer pplyng hs con. he elgbly rce for oher cons whch hs no been ken whle n he curren se s smply s prevous rce bu dscouned,.e. hose cons re now less responsble for he cred: γλ e ( + s f = e ( (3 γλe ( oherwse where λ s he dscoun re for he elgbly rces e H. he Lernng Mehod: he remnng s he lernng lgorhm. Our lgorhm s n on-polcy boosrppng Srs(λ [] wh lner pproxmon of he Q con-vlue funcon. Srs(λ s n B m s Curren se Q Hsogrm of ech chnnel = feures ( s, + Robo Conrol Polcy Q-funcon pproxmon ( s, Q( φ ( s, Fg. 2. he vrous componen of he proposed model. lgorhm h uses D(λ for conrol. I lerns on-lne hrough nercon wh smulon sofwre h feeds wh he robo vsul sensors. he lgorhm coded s conroller reurns he chosen con o be ken by he robo, nd updes s polcy hrough updng s se of prmeers used o pproxme he con-vlue funcon Q. hree lner neworks re used o pproxme he con-vlue funcon for he hree cons. ( ( ( θ ( ( = ( θ,, θ,, θ n =,.. A he curren mge ws pssed hrough n RBF lyer whch gves he feure vecor φ ( s = ( φ,, φ,, φn. he robo ws lef o run hrough severl epsodes. Afer ech epsode he lernng re ws decresed, nd he polcy ws mproved furher hrough GPI. he overll lgorhm s h of he Srs(λ conrol lgorhm [] nd s summrzed n Fg. 3. Inlzon θ ( = 2 Repe for ech epsode e ( = = : A s Inl robo vew, Genere usng smplng of probbly ( s, Repe (for ech sep of epsode ke con, Observe r +, s +, Genere usng smplng of probbly ( s = : A + [ r + γ s θ( s θ( ] δ γλe ( + s f = e ( γλe ( oherwse θ( θ( + α ep e ( δ s s unl mn( JDM ( < ψ j + + unl epsode == fnl_epsode,. Fg. 3. Lner on-polcy grden-descen Srs(λ conrol, wh RBF feures lgorhm for lner con-vlue funcon pproxmon nd Polcy Improvemen. he pproxme Q s mplcly funcon of θ he lernng re ws he sme used by Boyn [8]: n(fnl_epsode + α ep = α (4 n(fnl_epsode + epsode I. he polcy used o Genere Acons: A combnon of ε-greedy polcy nd Gbbs sof-mx [] polcy s used o pck n con nd o srke he blnce beween exploron nd exploon. Usng ε-greedy probbly llows exploron o be ncresed s needed by nlly seng ε o hgh vlue hen decresng hrough epsodes. Gbbs sof-mx probbly, +

6 Gbbs ( (, φ ( s = A j = exp [ φ ( s θ ( ( ] [ φ ( s θ ( ( j ] exp, (5 helped n ncresng he chnces of pckng he con wh he hghes vlue when he dfferences beween he vlues of nd he remnng cons s lrge,.e. helped n ncresng he chnces of pckng he con wh he hghes Q-vlue when he robo s sure h s he rgh one. [ s θ ( ( ] ε ε + f = rg mx A (6 Pr(, s = ε oherwse A ε + Gbbs(, s = Gbbs(, s + Pr(, s (7 J. he Neurl Nework Lyers: From neurl nework pon of vew, when consderng he RBF lyer ogeher wh he compeve lyer, one cn relze h hs rchecure s smlr o Probblsc Neurl Nework (PNN h clcules he probbly of pckng up cern con condonl o he gven gol. We wll cll he neurl nework used n our model he RBF-Q-D Nework (nd lgorhm becuse we used he RBF lyer for feure exrcon nd hen lner lyer wh Srs(λ lgorhm nd he dssmlry mesure. Fg. 3 shows smplfcon of our model wh s lyers. K. he Lner Neworks nd Feures Dmensons : he prmeers hve he sme dmenson s he feure spce whch s n = C B m ; where C = 3s he number of chnnels, B s he number of bns per mge nd m s he number of sored vews for he gol locon. Snce we use n RGB mges wh vlues n he rnge of [, 255] for ech pxel, he dmenson of he feure spce s gven by: 256 n C m (8 b where b s he bn s sze. Dfferen bn szes gve dfferen dmensons, whch n urn gve dfferen number of pproxmon prmeersθ. he equly s no complee due o he fc h he precse number of bns s gong o be 256 B = round ( +. b Noe h σ of he feures hs been chosen hrough connuous upde of he sum of he feures vecor colleced n ll he me seps so fr. / 2 2 σ. ( (, ( (, = h s + c j h v c j (9 hs llowed for mnnng beer ncremenl esmon of ech feure vrnce nd hence beer performnce. Afer enough exploron of he envronmen hs vlue s lmos sble nd chnges o re mnmzed. I hs been observed h he vrnce of hs nernl prmeer hs dropped fer epsode o neglgble vlue, whch mens resuls re relble for epsode> nd h he neurl neworks re lernng he vlue funcon for lmos he sme ses h re gong o be encounered n he fuure. L. Imporn enhncemens nd Lmons: One problem of unnecessry wnderng remns. Mnly s cused by consequen conflcng posve nd negve rewrds gven by he envronmen due o pprochng he gol nd wnderng round whou rechng. Smply sever punshmen ws ppled for he prculr cse when he robo goes from posve rewrdng o negve punshmens n wo successve seps. V. EXPERIMENAL RESULS he model ws ppled usng smuled Kheper [9] robo n Webos [2] smulon sofwre. he Kheper s mnure rel robo, 7 mm dmeer nd 3 mm hegh, nd s provded wh 8 nfr-red sensors for recve behvour, s well s colour cmer exenson. A.5x m 2 smuled envronmen hs been used s es bed for our model. he sk s o lern o nvge from ny locon n he envronmen o home locon (whou usng ny specfc objec or lndmrk. For rnng, he robo lwys srs from he sme locon, where cnno see he rge locon, nd he end se s he rge locon. rge locons Kheper robo n s srng locon Fg. 4. Snpshos of he relsc smuled envronmen. Fg. 4 shows he envronmen used. A cone, bll nd V re ncluded o dd more exure o he gol locon,.e. o enrch nd mke dfferen from he oher envronmen locons. We reemphsze h no objec recognon echnque ws used, only he JDM. he conroller wren s combnon of C++ code nd Mlb Engne code. he robo srs by kng (m= 3 snpshos for he gol locon. I hen goes hrough specfc number (5 of epsodes. he robo srs wh rndom polcy, nd fnshes n epsode when reches he desred locon. A. he Prccl Sengs of he Model Prmeers: For our pplcon we hve chosen he feure spce prmeers o be b=3, m=3 hence

7 n = 3 ( round(256 / = 774. λ ws se o he vlue of.8 dependng on he sudes [, 2] h referred o he rnge of [.7.8] s he pek of he performnce of he D(λ-lernng. he dscoun consn ws se oγ =,.e. here s no dscoun hrough me. ψ, ψ re purely emprcl nd were se o.7 nd 2 respecvely. B. Seng he Exploon vs. Exploron: Snce con spce s fne, nd o vod flucuon nd overshoong n he robo behvour, low wheel speeds were doped for hese cons. hs n urn requred seng he exploron o relvely hgh re (lmos 5% durng he erly epsodes. I ws hen dropped grdully hrough epsodes, n order o mke sure h mos of he poenl phs re suffcenly vsed. Seng exploron hgh lso helps n decresng he number of epsodes needed before rechng n ccepble performnce. hs explns he exponenl ppernce of he dfferen lernng curves dscussed below. he model performnce hs been suded for smll number of sored vews (m=3 o show he robusness of he model. One cn enhnce ccurcy by ncresng he dmenson spce bu would hve o rde-off speed of convergence nd execuon. he mos nurl wy o ncrese he se spce dmenson s by ncresng he number of hsogrm's bns consdered. However, o concenre on he pure effec of chngng m nd elmne he ncrese n se dmenson due o he ncrese n m (8, one cn se m=b hen chnge boh m nd b ogeher. hs could fx he dmenson of he feure spce nd consequenly he sze of he pproxmor, nd show he cul effec of chngng he number of vews m. C. Sudyng he Model Performnce: Fg. 5, shows he effec of lernng verged over 8 rls, ech wh 5 epsodes. All of he rls successfully converged. Dvergence occurred only when seng he lernng re o hgh vlue, or when exploron ws quckly decresed. he reson h we needed low lernng re s h we use Gbbs probbly dsrbuon for he exploron/exploon blnce. hs exponenlly formed probbly cn go quckly o nfny f cre s no ken when ssgnng s exponens. he fc h we hve relvely lrge se spce dmenson ws he mjor fcor n hs suon. Pr ( shows he mos mporn spec of ny renforcemen lernng model; he reurn vlues of ech epsode, convergng opmlly. Afer ll, he mn purpose of he RL-bsed model s o opmlly ncrese he sum of he receved rewrds. he reurn vlues (mosly negve hve ncresed nurlly hrough epsodes due o he mprovemen kng plce from epsode o epsode. hs s done v mprovng he doped polcy mplcly; by movng o beer esmes nd decresng exploron from epsode o he oher. he ccurcy of he con-vlue funcon esmes s grdully/ervely ncresng usng he lernng prmeer θ. Fg. 5 (b shows he decrese h ook plce n he number of seps needed o cheve he sk. hs norml decrese s n ccordnce wh pr ( nd becuse of he cos pr of he rewrd funcon. In fc, we decresed he dfference beween ψ nd ψ, so h he oher wo prs of he rewrd formul hve mnml effec on he model convergence. Fg. 5 (c depcs he chnges n he lernng prmeers hemselves, snds for he componen ndex of he lernng vecor. Mos mporn s h he hree prs hve n exponenllke shpe showng he hgh speed of convergence hs model hs reched. hs s hghly desrble n renforcemen lernng model due o s domnn convergence slowness problem []. In fc, one mjor conrbuon of hs work s he hgh performnce reched wh lle experence usng complex vsul npu. Fg. 6, depcs performnce nd nernl prmeers llusrons. Fg. 6, ( shows he lernng re decrese hrough he epsodes whch ws used hroughou he rls. Pr (b shows he decrese enforced on he exploron re ε whle pr (c shows he overll percenge of explorve con nd explove cons. Roues ken by he robo n hree epsodes (erly, mddle, nd fnl for one of he rls re shown n prs (d-(f. VI. DISCUSSION AND CONCLUSION b Fg. 5. Lernng Curves verged over 8 rls. c We hve ckled he polcy mprovemen for Srs(λ sysems combned wh JDM nd RBF. hs s novel o models nroduced n he lerure due o he wy we ppled renforcemen lernng usng neuro-dynmc progrmmng mehods lke Srs(λ. Below we se some of he dvnges of hs model:

8 c e Fg. 6. Lernng performnce nd smple roue for smple rl. Smplcy of lernng: he robo cn lern o perform s vsul nvgon sk n very smple wy whou long process of mp buldng. 2 Lmed sorge of nformon s requred n he form of m sored vews. 3 No pre or mnul processng s requred. No pror knowledge bou he envronmen s needed.e. no lndmrks re needed n hose vews. 4 An mporn dvnge of our model over MDP explc model-bsed pproches s h bducon of robo s solved drecly.e. he robo cn fnd s wy nd recover fer hs been dsplced from s curren poson nd pu n olly dfferen poson. o rse he dfferenbly of he vews, however, hey should be rch wh colours ec. (.e. good moun of nformon. hrough he lernng robusness nd generly of RL robos, coupled wh vsul ses nd rewrds, he sysem cheved hgh level of robusness, generly, nd pplcbly. hs combnon envely proved o work very well for our nvgon problem. Fuure work ncludes crryng ou more exensve expermens over our model by ryng dfferen confgurons usng (8, boh n erms of more vews o be consdered s well s dfferen bns szes nd dfferen envronmens. Fuure work cn lso nclude usng off-polcy nsed of he on-polcy mehod o ccommode for wo behvours lyers used by he gen. b d f [2] A. M. Anderson, "A model for lndmrk lernng n he honey-bee," Journl of Comprve Physology A vol. 4, pp , 977. [3] B. A. Crwrgh nd. S. Colle, "Lndmrk mps for honeybees," Bologcl Cybernecs, vol. 57, pp , 987. [4] A. Vrdy nd F. Oppcher, "A scle nvrn locl mge descrpor for vsul homng," n Bommec neurl lernng for nellgen robos., G. Plm nd S. Wermer, Eds.: Sprnger, 25 [5] M. Szenher, "Vsul Homng wh Lerned Gol Dsnce Informon," presened Proceedngs of he 3rd Inernonl Symposum on Auonomous Mnrobos for Reserch nd Edunmen (AMRE 25, Awr-Sp, Fuku, Jpn, 25. [6] D. Sheynkhovch, R. Chvrrg,. Srossln, nd W. Gersner, "Spl Represenon nd Nvgon n Bo-nspred Robo," n Bommec Neurl Lernng for Inellgen Robos, S. Wermer, M. Elshw, nd G. Plm, Eds.: Sprnger, 25, pp [7] C. Weber, D. Muse, M. Elshw, nd S. Wermer, "A cmer-drecon dependen vsul-moor coordne rnsformon for vsully guded neurl robo," Knowledge-Bsed Sysems, Scence Drec, Elsever vol. 9, pp , 26. [8] S. hrun, Y. Lu, D. Koller, A. Ng, Z. Ghhrmn, nd H. Durrn- Whye, "Smulneous loclzon nd mppng wh sprse exended nformon flers," Inernonl Journl of Robocs Reserch, vol. 23, pp , 24. [9] S. hrun, W. Burgrd, nd D. Fox, Probblsc Robocs. Cmbrdge, Msschuses; London, Englnd: he MI Press, 25. [] I. Ulrch nd I. Nourbkhsh, "Appernce-Bsed Plce Recognon for opologcl Loclzon," presened IEEE Inernonl Conference on Robocs nd Auomon, Sn Frncsco, CA, 2. [] R. S. Suon nd A. Bro, Renforcemen Lernng, n nroducon. Cmbrdge, Msschuses: MI Press, 998. [2] R. S. Suon, "Lernng o predc by he mehods of emporl dfferences," Mchne Lernng, vol. 3, pp. 9 44, 988. [3] J. N. sskls nd B. Vn Roy, "An nlyss of emporl-dfference lernng wh funcon pproxmon," IEEE rnscons on Auomc Conrol vol. 42, pp , 997. [4] J. A. Boyn, "echncl upde: Les-squres emporl dfference lernng.," Mchne Lernng vol. 49., pp , 22. [5] C. Gske, D. Weergreen, nd A. Zelnsky, "Q-Lernng n Connuous Se nd Acon Spces," presened Ausrln Jon Conference on Arfcl Inellgence, Ausrl, 999. [6] P. Sone, R. S. Suon, nd G. Kuhlmnn, "Renforcemen lernng for robocup soccer keepwy," Inernonl Socey for Adpve Behvor vol. 3, pp , 25. [7] Y. Rubner nd e l., "he Erh Mover's Dsnce s Merc for Imge Rerevl," Inernonl Journl of Compuer Vson, vol. 4, pp. 99-2, 2. [8] J. A. Boyn, "Les-squres emporl dfference lernng.," presened In Proceedngs of he Sxeenh Inernonl Conference on Mchne Lernng, Sn Frncsco, CA, 999. [9] D. Floreno nd F. Mondd, "Hrdwre soluons for evoluonry robocs," presened Frs Europen Workshop on Evoluonry Robocs, Berln, 998. [2] O. Mchel, " Webos: Professonl Moble Robo Smulon," Inernonl Journl of Advnced Roboc Sysems, vol., pp , 24. [2] L. C. Brd, "Resdul Algorhms: Renforcemen Lernng wh Funcon Approxmon," presened Inernonl Conference on Mchne Lernng, proceedngs of he welfh Inernonl Conference, Sn Frncsco, CA, 995. REFERENCES [] U. Nehmzow, Moble robocs: A Prccl Inroducon: Sprnger- Verlg, 2.