Reinforcement Learning for a New Piano Mover s Problem

Transcription

1 Renforcemen Lernng for New Pno Mover s Problem Yuko ISHIWAKA Hkode Nonl College of Technology, Hkode, Hokkdo, Jpn Tomohro YOSHIDA Murorn Insue of Technology, Murorn, Hokkdo, Jpn nd Yuknor KAKAZU Reserch Group of Complex Sysems Engneerng, Hokkdo Unversy, Spporo, Hokkdo, Jpn ABSTRACT We emp o cheve corporve behvor of uonomous decenrlzed gens consruced v Q-Lernng, whch s ype of renforcemen lernng. As such, n he presen pper, we exmne he pno mover s problem. We propose mul-gen rchecure h hs rnng gen, lernng gens nd nermede gen. Lernng gens re heerogeneous nd cn communce wh ech oher. The movemen of n objec wh hree knds of gen depends on he composon of he cons of he lernng gens. By lernng s own shpe hrough he lernng gens, vodnce of obscles by he objec s expeced. We smule he proposed mehod n wo-dmensonl connuous world. Resuls obned n he presen nvesgon revel he effecveness of he proposed mehod. Keywords pno mover s problem, renforcemen lernng, herrchy gen sysem 1. Inroducon Orgnl Pno mover s problem ws nroduced by Schwrz nd Shrrs [1] for mhemcl model. The Pno Mover s Problem s reed s problem whch conrols he ude of n objec by erng roons nd prllel rnslons n order o move he objec ousde of room. The problem s esy o mgne how dffcul o solve becuse usully people hve been fced wh smlr problem when you ry o move sof n smll room or move grnd pno no room nd so on. Schwrz nd Shrrs proved o clcule he opml ph mhemclly for he cse n whch he geomeres of he spce h he objec psses s se. However, n he rel world, s mosly mpossble o clcule excly her envronmen nd he objecs, for exmple he sof s now 13.5cm fr from he ble nd f s moved o 3cm o wre, s 15cm fr from he chr nd so on. The envronmen s dynmcs. There re mny pproches for moon plnnng problem o solve hs problem n rel world. Confguron spce mehod nd poenl feld mehod re mjor pproches. The Confguron spce mehod (C-spce) res robo s pon whou physcl sze n he spce, nd ny obscles nd free spces known. Lozno-Perez [] generes he c-spce obscles usng Mnkowsk sum of he robo nd envronmen. Dors,L. nd ec.[3] used rserzed c-spce pproch o pln for wo-lnk rm. Lengyel.J nd ec. [4] rserzed confguron spce obscles no seres of bmp slces, nd dynmc progrmmng o cree cree nvgon funcon. Khb.O nd Mre [5] proposed poenl feld mehods frs. The obscles were represened s zero level surfces of sclr vlued nlyc funcons. However hs mehod hs one mjor problem whch s spurous locl mnm, especlly for concve robos. Yoko e l. [6] solved hs problem usng he vbrng poenl mehod. In her sudy, he shpe of he objec ws recngle, clled he AGV (uonomous guded vehcle). We propose new pno mover s problem. The new pno mover s problem ncludes fndng ph problem n he complex spce nd n ddon he ude mus be conrolled uonomously. Generlly hese wo problems,.e. fndng ph problem nd conrollng ude problem, conrdc ech oher o solve hem smulneously. In order o fnd ph, he globl lernng s needed bu o conrol he ude, he locl lernng s necessry for collson vodnce. We vew hs conrdc problem s geomerc mbguous. Wh he geomerc mbguous mens ncludes no mched evlued sgnls splly. To solve hs mbguy problem we propose heren herrchy gen sysem whch hs hree knds of gens, rnng gen, lernng gens nd n nermede gen. Trnng gen hs no percepon bou s own shpe hen cnno be ppled o mhemcl mehod. Lernng gens s lso unknown bou he shpe, bu grsp vcnly by lernng obscle vodnce. Inermede gen ry o mch he evlued sgnls beween rnng gen nd lernng gen. Ech gen mkes s own decson for kng con, so usully he ISSN: SYSTEMICS, CYBERNETICS AND INFORMATICS VOLUME 3 - NUMBER 4 85

2 xs of spce f(,s+ s)=f*(,s) f*(,s): rue vlue for lerner Ambuy splly shown ph bu cnno pss f(,s- s)=f*(,s) Fg.1 Defnon of mbguy splly decsons of he rnng gen nd lernng gens re dfferen from ech oher. In oher words here s hgh possbles for rnng gen o show he wrong ph whch lernng gens cnno pss becuse of he sze. Inermede gen s necessry for solvng hs knd of problem. We propose heren heerogeneous mul-gen sysem n whch homogeneous mul-gens reed s lernng gens, one rnng gen nd one nermede gen re consruced. Ech lernng gen kes smple cons, nd he rnng gen s reed s n objec. Q-Lernng [7] s performed for ech lernng gen, whch consequenly lerns obscle vodnce. The TD-mehod [8] s used for he rnng gen, whch lerns he ph from sr o fnsh. The smple rule s ppled o nermede gen. The proposed sysem s form of herrchc renforcemen lernng.. The defnon of mbguy In hs pper splly mbguy s shown (geomery). Fg.1 shows he geomerc mbguy. In Fg.1 f*(,s) mens he rue vlue for lerners. Δ nd Δs men he fne dfferences of me nd se respecvely. The mbguy s defned s he dfferences,.e. ±Δ or s± Δs. Fg. shows n exmple he geomerc mbguy. Trnng gen orders o pss he nrrow ph o lerner nd rnng gen dose no grsp he lerner s shpe excly. Lernng gens ry o vod he obscles bu he drecon of ken con s no sme s shown by rnng gen. I cuses ded lock. In hs knd of unmched evlued sgnls cse, nermede gen decded he new con by observng boh rnng gen nd lernng gens evlued sgnls. The behvors of nermede sgnls re mporn. 3. Proposed Sysem Three knds of gen re proposed s follows. The rnng gen ndces he enre objec of oulne, lernng gens re consruced nsde of he rnng gen nd he nermede gen mches beween evlued sgnls from rnng gen nd ken con by lernng gens. Fg.3 shows he oulne of our sysem. Fg. The mbguy resrced splly (geomery) The dvnges of hs mehod re 1) correspondence o he vrn envronmen, ) lhough ech gen requres smple rchecure, complced movemen s possble, 3) fuls of gens re ddressed, nd 4) pplcon o oher problems (no only pno mover s or moon conrol problems) ccordng o sepre heerogeneous gens s possble. For exmple, lernng gens lern obscle vodnce, bu hey lern lso suspenson, rollng nd comng nd gong. These gens smply emp o vod wlls nd do no lern how o fnd he ph. If he rewrds re se fnely, hen he ph cn be lerned. However, n hs cse he rewrd seng s focused nd should be rese s he problem (even he envronmen) s chnged. In order o vod hs rewrd-seng problem, we propose he rnng gen, whch lerns he ph, nd shows he drecon o he lernng gens. In hs mehod he rewrd cn be se smply. In he followng secon, we descrbe ech gen n del. 3.1 The rchecure of n objec Le he shpe of he objec be recngulr, nd le he lengh of long boundry nd shor boundry be nd b respecvely (>b). The mgnude of vecor ndcng n con ken s f, nd he mss s M. Le he nl coordnes of he cener-of-grvy be (X,Y ), where =. The movemen of n objec cn be rewrd Fg.3 observon Sensors Acon selecor (Q-Lernng) Lernng gen Acuor Envronmen con Acon Selecor Lernng gen 1 con Trnng gen Lernng gen n Inermede gen con Relon shps of hree knds of gen for pno mover s problem. 86 SYSTEMICS, CYBERNETICS AND INFORMATICS VOLUME 3 - NUMBER 4 ISSN:

3 f θ=45 y Agen Agen 1 Agen 3 Agen b Tn 1 b α= x < he ses of rgh boom gen> sensor1 : (no response) sensor : (no response) sensor3 : (no response) sensor4 : (no response) sensor5 : (no response) upper lef gen : 1 (wh response) upper rgh gen : ( no response) b lf :1 ( h Fg. 4 An exmple of physcl clculon. resolved no prllel rnslon componen nd roon componen. The componen vecors of force re obned respecvely s follows: For x componen, 3 3 π π F x = f cos π + f cos π + f cos + f cos =, For y componen, 3 3 π π F y = f sn π + f sn π + f sn + f sn =, Then, we obn he coordnes fer movemen n me s X = X + = X Y = Y + = Y. In hs cse, he objec hs no prllel rnslon componen becuse he cener-of-grvy of he objec hs no moved. The momen-of-force M_P for he emphss of gen s clculed s follows: M _ P = π cos 4 j b π sn 4 k π π = sn + b cos k. 4 4 The momen-of-ner I round he cener of he objec s ( b ) 1 I = M +. 3 The equon of moon for roon round he cener-of-grvy s d θ M _ P = I. d The equon of moon bou θ cn hen be solved s follows: Fg.5 An exmple of se ses for n gen π π sn + b cos 4 4 θ =. 1 M ( + b ) 3 The coordnes of Agen me (x, y ) re hen x y = X + = Y + + b + b cos sn ( θ α ) ( θ α ) 1 b, whereα = n. Snce he objec s rgd body, s movemen cn be clculed usng he relve movemen beween he cener-of-grvy nd pon no on he body. We herefore om he oher gens clculon. 3. Lernng gens Q-lernng, whch s renforcemen lernng, s consruced n ech lernng gen. The se of ech lernng gen s gven by sensors nformon of fve drecons {s,s1,,s4}, nd one compressed nformon s gven by communcons. The rnge of ech sensor s dvded no sx sges o obscles, nd ech lernng gen cn ke cons n four drecons, s shown n Fg.4. Fg. 5 shows n exmple of se ses for n gen. Ech gen cn obn he sensor nformon (fve drecons) usng boh her own sensors nd he sensors of oher gens ( or 1) hrough communcon. Therefore, lernng gens lern obscle vodnce by cooperng wh ech oher. Updng Q-vlues n lernng gens s defned n he followng equon: Q ( s ( ), o ( )) (1 α) Q ( ( ), ( )) + + mx ( ( + 1), ( + 1)) s o α r γ Q s o o ' O,where s() s he se me, o() s he ken con e me, α s he sep-sze prmeer, γ s he dscoun-re ISSN: SYSTEMICS, CYBERNETICS AND INFORMATICS VOLUME 3 - NUMBER 4 87

4 Inernl gen mches beween evlued sgnls from rnng gen nd con ken by lernng gens. Fg. 6 shows he sregy of nernl gen. A sregy of n nernl gen s employed here. I s o selec he con for whch he Q-vlue s mxmum mong he combnons h show he drecon decded by he rnng gen. The ndced drecon s lso descrbed n Fg.6. Fg.6 Inernl gen selec he drecon mong he blue rrows. prmeer nd r s he rewrd from he envronmen. If r s obned s penly, hen boh lernng nd rnng gens cn lern obscle vodnce. The funcon mx selecs he mxmum Q-vlue con from he fne con se O. When lernng gens ke cons, f he rnng gen colldes wh n obscle or does no move ll, hen he funcon mx becomes, mx o ' O Q ( s ( + 1), o ( + 1)) = Q 3.3 Trnng Agen ( s ( ), o ( )). The shpe of rnng gen s ssumed s recngle, nd n he recngle of ech corner four lernng gens re pu n poson respecvely. A rnng gen s only one. The rnng gen lerns he ph from gven sr pon o gol, nd shows he drecon o he gol locon o he lernng gens. The rnng gen does no hve s own sensors, so he movemen depends on he lernng gens. We used TD-lernng for he rnng gen. The se of rnng gen s he x-y coordne. The rnng gen cn selec n con from combnon of he con drecons of he four lernng gens, whch gves ol of 56 possble cons. Updng V-vlues s performed usng he followng equon: [ r + γv ( s ) V ( s )] V ( s ) V ( s ) + α, + 1 We employed TD-lernng becuse Q-lenng depends on he dmensonly of he course, nd s mpossble o mplemen due o he se ses,.e. he cul coordnes of he envronmen, nd he con ses from physcl clculon. Here, we use renforcemen lernng for boh lernng gens nd he rnng gen. However, for he rnng gen, severl ypes of lernng sysems cn be ppled. If s esy o gve he ph roughly, he ph cn be gven drecly o he rnng gen. Therefore, he uonomous robo cn follow he ph whle vodng obscles nellgenly. In hs cse, even he lernng sysem for n rnng gen my no be necessry. 3.4 Inernl gen 4. Expermens Fg. 7 shows he rjecores for dfferen envronmens of sze 8 x 8. Lernng gens nd rnng gen re lernng smulneously. Fg. 8 uses smlr envronmen bu he gol poson s dfferen. In Fg. 8 lernng gens mus swch bck ner gol. The rjecores re shown respecvely. Lower smll fgure shows he expnson of ppernce of swch bck. Fg.9 shows he resuls of dfferen envronmen. The common prmeers for ll smulons re s follows: for lernng gens; sensor rnge = 3, sensbly = 6 levels, nd he movemen lengh s 4 per sep, for Q-lernng; sep-sze prmeer α =.1, dscoun re γ =.9, nl Q-vlues =., nd penly r =-1/(mx sep number), for he rnng gen; sze = ( x 4), nd mss M = 1, for TD lernng; sep-sze prmeer α e =.1, dscoun re γ e =.9, nl V-vlue=., nd rewrd r e =1 when he gens rech he gol. The mx sep number s 5 nd he epsode s 1. The polces of boh renforcemen lernng re greedy lgorhms. 6. Concluson We propose new pno mover s problem whch ncludes boh ph fndng problem nd conrollng ude problem. Generlly hese wo problems re ncongruous wh ech oher. One of hem s needed globl lernng nd he oher s needed locl lernng. We reed he problem s problem no mched evlued sgnls splly (geomerc mbguous). In order o solve hs new pno mover s problem, we sugges herrchy gen sysem whch hs hree knds of gen,.e., rnng gen, lernng gens nd nermede gen. Trnng gen hs no percepon bou s own shpe hen cnno be ppled o mhemcl mehod. Lernng gens s lso unknown bou he shpe, bu grsp vcnly by lernng obscle vodnce. In hs problem here s hgh possbles for rnng gen o show he wrong ph whch lernng gens cnno pss becuse of he sze. Inermede gen s necessry for solvng hs knd of problem. From he vrous expermens s shown he robusness of gen sysem rchecure, s compred beween he 88 SYSTEMICS, CYBERNETICS AND INFORMATICS VOLUME 3 - NUMBER 4 ISSN:

5 Fg.7 The rjecores of rnng gen. Sr pon s mddle of he mouse nd gol poson s shown n red squred rgh upper sde. Envronmen sze s 8 x 8. Lower fgure shows zoomed of he complex behvor,.e. swch bck n he bove envronmen. Fg.8 An exmple of he rjecores of rnng. Sr pon s mddle of he mouse nd gol poson s shown n red squred lef upper sde n he puernodon. Envronmen sze s 8 x 8. Lower fgure shows zoomed of he complex behvor,.e. swch bck n he bove envronmen. cse of chngng only rnng gen, nd he cse of chngng only nernl gen respecvely. As resul he problem cn be solved even hough he sysem hs chnged. Accordng hese expermens he esness of chngng sysem nd robusness re shown. Smulneous lernng by wo ypes of gen,.e. lernng gens nd rnng gen, s prculrly effecve. However, he problem of our lgorhms s lso ppered. The lernng of he rnng gen s domnn over h of he lernng gens, so nermede gen pples he effcency of he lernng of he lernng gens s dmnshed. In order o vod hs problem, we mus nvesge dpve nernl gen. In he fuure, he expermenl envronmen wll be exended o hree dmensons. 7. References [1] Jcob T. Schwrz nd Mch Shrr : On he Pno Movers Problem:Ⅱ.Generl Technques for Compung Topologcl Properes of Rel Algebrc Mnfolds, Plnnng, Geomery, nd Complexy of Robo Moon, pp , 1983 [] Lozno-Prez,T. nd M.A.Wesly, "An Alogorhm for plnnng Collson-Free Phs Among Polyhedrl Obscles", Communcons of he ACM,, pp56-57,1979 [3] Dors,L. nd K.Troo.,"Opml ph Plnnng wh Sx Degrees of Freedom", Arfcl Inellgence,31, pp95-353,1987. [4] Jed Lengyel nd Mrk Recher nd Bruce R. Donld nd Donld P. Greenberg:"Rel-Tme Robo Moon Plnnng Usng Rserzng Compuer Grphcs Hrdwre",journl of Compuer Grphcs, vol.4, No.4, pp37-335,199 [5] Khb,O. nd J.Le Mre,"Dynmc Conrol of Mnpulors Operng n Complex Envronmen", Proceedngs Thrd Inernonl CISM-IFToMM Symposum,Sepember 1978, pp [6] Yoko H., Mzuno T., Tk M nd Kkzu Y.: Obscle Avodnce Usng Vbrng Poenl Mehod (Self-Orgnzon n Nrrow Ph, Journl of Robocs nd Mechroncs, Vol.8 No.4, 1996 [7] Wkns, C.J.H. nd Dyn, P.: Techncl noe: Q-lernng, Mchne Lernng, Vol.8,pp.55-68,199 [8] Suon R. nd A.G. Bro: Renforcemen Lernng, MIT Press, Cmbrdge, MA,1998 ISSN: SYSTEMICS, CYBERNETICS AND INFORMATICS VOLUME 3 - NUMBER 4 89

6 Fg.9 An exmple of he rjecores of n rnng gen usng Exp.3 fer lernng n room. The sze of envronmen s 8 x 8. Sr pon s he lef boom nd gol s shown n red squred lef upper sde. There s only one ex door per room from sr o gol. 9 SYSTEMICS, CYBERNETICS AND INFORMATICS VOLUME 3 - NUMBER 4 ISSN: