We are IntechOpen, the world s leading publisher of Open Access books Built by scientists, for scientists. International authors and editors

Transcription

1 W ar InchOpn, h world s lading publishr of Opn Accss books Buil by sciniss, for sciniss 3, , M Opn accss books availabl Inrnaional auhors and diors Downloads Our auhors ar among h 151 Counris dlivrd o OP 1% mos cid sciniss 12.2% Conribuors from op 500 univrsiis Slcion of our books indxd in h Book Ciaion Indx in Wb of Scinc Cor Collcion (BKCI) Inrsd in publishing wih us? Conac book.dparmn@inchopn.com Numbrs displayd abov ar basd on las daa collcd. For mor informaion visi

2 14 Novl Framwork of Robo Forc Conrol Using Rinforcmn Larning Byungchan Kim 1 and Shinsuk Park 2 1 Cnr for Cogniiv Roboics Rsarch, Kora Insiu of Scinc and chnology 2 Dparmn of Mchanical Enginring, Kora Univrsiy Kora 1. Inroducion Ovr h pas dcads, roboic chnologis hav advancd rmarkably and hav bn provn o b succssful, spcially in h fild of manufacuring. In manufacuring, convnional posiion-conrolld robos prform simpl rpad asks in saic nvironmns. In rcn yars, hr ar incrasing nds for robo sysms in many aras ha involv physical conacs wih human-populad nvironmns. Convnional roboic sysms, howvr, hav bn inffciv in conac asks. Conrary o robos, humans cop wih h problms wih dynamic nvironmns by h aid of xclln adapaion and larning abiliy. In his sns, robo forc conrol sragy inspird by human moor conrol would b a promising approach. hr hav bn svral sudis on biologically-inspird moor larning. Cohn al. suggsd impdanc larning sragy in a conac ask by using associaiv sarch nwork (Cohn al., 1991). hy applid his approach o wall-following ask. Anohr sudy on moor larning invsigad a moor larning mhod for a musculosklal arm modl in fr spac moion using rinforcmn larning (Izawa al., 2002). hs sudis, howvr, ar limid o rahr simpl problms. In ohr sudis, arificial nural nwork modls wr usd for impdanc larning in conac asks (Jung al., 2001; suji al., 2004). On of h noicabl works by suji al. suggsd on-lin virual impdanc larning mhod by xploiing visual informaion. Dspi of is usfulnss, howvr, nural nwork-basd larning involvs havy compuaional load and may lad o local opimum soluions asily. h purpos of his sudy is o prsn a novl framwork of forc conrol for roboic conac asks. o dvlop appropria moor skills for various conac asks, his sudy mploys h following mhodologis. Firs, our robo conrol sragy mploys impdanc conrol basd on a human moor conrol hory - h quilibrium poin conrol modl. h quilibrium poin conrol modl suggss ha h cnral nrvous sysm uilizs h springlik propry of h nuromuscular sysm in coordinaing muli-dof human limb movmns (Flash, 1987). Undr h quilibrium poin conrol schm, forc can b conrolld sparaly by a sris of quilibrium poins and modulad siffnss (or mor gnrally impdanc) a h joins, so h conrol schm can bcom simplifid considrably. Scond, as h larning framwork, rinforcmn larning (RL) is mployd o opimiz h prformanc of conac ask. RL can handl an opimizaion problm in an

3 260 Robo Manipulaors unknown nvironmn by making squnial dcision policis ha maximiz xrnal rward (Suon al., 1998). Whil RL is widly usd in machin larning, i is no compuaionally fficin sinc i is basically a Mon-Carlo-basd simaion mhod wih havy calculaion burdn and larg varianc of sampls. For nhancing h larning prformanc, wo approachs ar usually mployd o drmin policy gradin. On approach provids h baslin for gradin simaor for rducing varianc (Prs al., 2006), and h ohr suggss Baysian upda rul for simaing gradin (Engl al., 2003). his sudy mploys h formr approach for consrucing h RL algorihm. In his work, pisodic naural acor-criic algorihm basd on h RLS filr was implmnd for RL algorihm. Episodic Naural Acor-Criic mhod proposd by Prs al. is known ffciv in high-dimnsional coninuous sa/acion sysm problms and can provid opimum closs soluion (Prs al., 2005). A RLS filr is usd wih h Naural Acor-Criic algorihm o furhr rduc compuaional burdn as in h work of Park al. (Park al., 2005). Finally, diffrn ask goals or prformanc indics ar slcd dpnding on h characrisics of ach ask. In his work, h prformanc indics for wo conac asks wr chosn o b opimizd: poin-o-poin movmn in an unknown forc fild, and caching a flying ball. h prformanc of h asks was sd hrough dynamic simulaions. his papr is organizd as follows. Scion 2 inroducs h quilibrium poin conrol modl basd impdanc conrol mhods. In scion 3, w dscrib h dails of moor skill larning basd on rinforcmn larning. Finally, simulaion rsuls and discussion of h rsuls ar prsnd. 2. Impdanc conrol basd on quilibrium poin conrol modl Mchanical impdanc of a robo arm plays an imporan rol in h dynamic inracion bwn h robo arm and is nvironmn in conac. Impdanc conrol is a widly-adopd conrol mhod o xcu roboic conac asks by rgulaing is mchanical impdanc which characrizs h dynamic bhavior of h robo a h por of inracion wih is nvironmn. h impdanc conrol law may b dscribd as follows (Asada al., 1986): [ ] τ = JqKxx ( ) ( ) + Bx (1) acuaor C d C Whr τ rprsns h join orqu xrd by h acuaors, and h currn and acuaor dsird posiions of h nd-ffcor ar dnod by vcors x and x d, rspcivly. Marics K C and B C ar siffnss and damping marics in Carsian spac. his form of impdanc conrol is analogous o h quilibrium poin conrol, which suggss ha h rsuling orqu by h muscls is givn by h dviaions of h insananous hand posiion from is corrsponding quilibrium posiion. h quilibrium poin conrol modl proposs ha h muscls and nural conrol circuis hav spring-lik propris, and h cnral nrvous sysm may gnra a sris of quilibrium poins for a limb, and h spring-lik propris of h nuromuscular sysm will nd o driv h moion along a rajcory ha follows hs inrmdia quilibrium posurs (Park al., 2004; Hogan, 1985). Fig. 1 illusras h concp of h quilibrium poin conrol. Impdanc conrol is an xnsion of h quilibrium poin conrol in h conx of roboics, whr roboic conrol is achivd by imposing h nd-ffcor dynamic bhavior dscribd by mchanical impdanc.

4 Novl Framwork of Robo Forc Conrol Using Rinforcmn Larning 261 Figur 1. Concpual modl of quilibrium poin conrol hypohsis For impdanc conrol of a wo-link manipulaor, siffnss marix K C is formd as follows: C K K K (2) = K 21 K 22 Siffnss marix K C can b dcomposd using singular valu dcomposiion:, whr λ 0 Σ and 1 = 0 λ 2 V Κ = VΣV C (3) cos( θ) sin( θ) = sin( θ) cos( θ) In quaion (3), orhogonal marix V is composd of h ignvcors of siffnss marix K C, and h diagonal lmns of diagonal marix Σ consiss of h ignvalus of siffnss marix K C. Siffnss marix K C can b graphically rprsnd by h siffnss llips (Lipkin al., 1992). As shown in Fig. 2, h ignvcors and ignvalus of siffnss marix corrspond o h dircions and lnghs of principal axs of h siffnss llips, rspcivly. h characrisics of siffnss marix K C ar drmind by hr paramrs of is corrsponding siffnss llips: h magniud (h ara of llips: 2λ 1 λ 2 ), shap (h lngh raio of major and minor axs: λ 1 /λ 2 ), and orinaion (h dircions of principal axs: ). By rgulaing h hr paramrs, all h lmns of siffnss marix K C can b drmind. In his sudy, h siffnss marix in Carsian spac is assumd o b symmric and posiiv dfini. his provids a sufficin condiion for saic sabiliy of h manipulaor whn i inracs wih a passiv nvironmn (Kazrooni al., 1986). I is also assumd ha damping marix B C is approximaly proporional o siffnss marix K C. h raio B C /K C is chosn o b a consan of 0.05 as in h work of Won for human arm movmn (Won, 1993).

5 262 Robo Manipulaors Figur 2. A graphical rprsnaion of h nd-ffcor s siffnss in Carsian spac. h lnghs 1 and 2 of principal axs and rlaiv angl rprsn h magniud and h orinaion of h nd-ffcors siffnss, rspcivly For rajcory planning, i is assumd ha h rajcory of quilibrium poin for h ndffcor, which is also calld h virual rajcory, has a minimum- jrk vlociy profil for smooh movmn of h robo arm (Flash al., 1985). h virual rajcory is calculad from h sar poin x i o h final poin x f as follows: x ( ) = x+ ( x x)(10( ) 15( ) + 6( ) ) (4) i f i f f f, whr is a currn im and f is h duraion of movmn. 3. Moor Skill Larning Sragy A wo-link roboic manipulaor for wo-dimnsional conac asks was modld as shown in Fig. 3. h roboic manipulaor is conrolld using h impdanc conrol mhod basd on h quilibrium poin conrol hypohsis as dscribd in Scion 2. h siffnss and damping of h manipulaor ar modulad during a conac ask, whil h rajcory of quilibrium poin is givn for h ask. h manipulaor larns h impdanc modulaion sragy for a spcific ask hrough rinforcmn larning. h sa vcor is composd of h join angls and vlociis a h shouldr and lbow joins. h acion vcor changs h hr paramrs of siffnss llips: h magniud (h ara of llips), shap (h lngh raio of major and minor axs), and orinaion (h dircion of principal axs). his scion dscribs h larning mhod basd on rinforcmn larning for conrolling ask impdanc of h wo-link manipulaor in prforming wo-dimnsional conac asks.

6 Novl Framwork of Robo Forc Conrol Using Rinforcmn Larning 263 Figur 3. wo-link roboic manipulaor (L 1 and L 2 : Lngh of h link, M 1 and M 2 : Mass of h link, I 1 and I 2 : Inria of h link) 3.1 Rinforcmn Larning h main componns of RL ar h dcision makr, h agn, and h inracion wih h xrnal nvironmn. In h inracion procss, h agn slcs acion a and rcivs nvironmnal sa s and scalar-valud rward r as a rsul of h acion a discr im. h rward r is a funcion ha indicas h acion prformanc. h agn srivs o maximiz rward r by modulaing policy π(s,a ) ha chooss h acion for a givn sa s. h RL algorihm aims o maximiz h oal sum of fuur rwards or h xpcd rurn, rahr han h prsn rward. A discound sum of rwards during on pisod (h squnc of sps o achiv h goal from h sar sa) is widly usd as h xpcd rurn: R k = γ i= 0 r i ++ 1 π k V () s = Eπ γ ri ++ 1s= s i= 0 (5) Hr, is h discouning facor (0 1), and V π (s) is h valu funcion ha rprsns an xpcd sum of rwards. h upda rul of valu funcion V π (s) is givn as follows: ( 1 ) V π ( s) V π ( s) + αr+ γ V π ( s ) V π + ( s ) (6) π π In quaion (6), h rm rv + γ ( s+ 1) V ( s ) is calld h mporal Diffrnc (D) rror. h D rror indicas whhr acion a a sa s is good or no. his updad rul is rpad o dcras h D rror so ha valu funcion V π (s ) convrgs o h maximum poin.

7 264 Robo Manipulaors 3.2 RLS-basd pisodic Naural Acor-Criic algorihm For many roboic problms, RL schms ar rquird o dal wih coninuous sa/acion spac sinc h larning mhods basd on discr spac ar no applicabl o high dimnsional sysms. High-dimnsional coninuous sa/acion sysm problms, howvr, ar mor complicad o solv han discr sa/acion spac problms. Whil Naural Acor-Criic (NAC) algorihm is known o b an ffciv approach for solving coninuous sa/acion sysm problms, his algorihm rquirs high compuaional burdn in calculaing invrs marics. o rliv h compuaional burdn, Park al. suggsd modifid NAC algorihm combind wih RLS filr. h RLS filr is usd for adapiv signal filring du o is fas convrgnc spd and low compuaional burdn whil h possibiliy of divrgnc is known o b rahr low (Xu al., 2002). Sinc his filr is dsignd for infinily rpad ask wih no final sa (non-pisodic ask), his approach is unabl o dal wih pisodic asks. his work dvlops a novl NAC algorihm combind wih RLS filr for pisodic asks. W namd his algorihm h RLS-basd NAC (pisodic Naural Acor-Criic) algorihm. h RLS-basd NAC algorihm has wo spara mmory srucurs: h acor srucur and h criic srucurs. h acor srucur drmins policis ha slc acions a ach sa, and h criic srucur criicizs h slcd acion of h acor srucur whhr h acion is good or no. In h acor srucur, h policy a sa s in pisod is paramrizd as π(a s )=p(a s, ), and policy paramr vcor is iraivly updad afr finishing on pisod by h following upda rul: α J ( ) (7), whr J ( ) is h objciv funcion o b opimizd (valu funcion V π (s)), and J( ) rprsns h gradin of objciv funcion J ( ). Prs al. drivd h gradin of h objciv funcion basd on h naural gradin mhod originally proposd by Amari (Amari, 1998). hy suggsd a simplr upda rul by inroducing h naural gradin vcor w as follows: α J ( ) + αw (8), whr dnos h larning ra (0 1). In h criic srucur, h las-squars (LS) D-Q(1) algorihm is usd for minimizing h D rror which rprsns h dviaion bwn h xpcd rurn and h currn prdicion valu (Boyan, 1999). By considring h Bllman quaion in driving LSD-Q(1) algorihm, h acion-valu funcion Q π (s, a) can b formulad as follows (Suon al., 1998): Q π a s π (, sa) = P R a + γv ( ) ss ss s π { ( s) γv ( s+ 1) s sa, a} = Er + = = π (9) Equaion (9) can b approximad as Q π (s, a) r(s )+ V(s +1 ) (10)

8 Novl Framwork of Robo Forc Conrol Using Rinforcmn Larning 265 whr V(s +1 ) approximas valu funcion V π (s +1 ) as iraiv larning is rpad. Prs al. inroducd advanag valu funcion A π (s, a)=q π (s, a)-v π (s) and assumd ha h funcion can b approximad using h compaibl funcion approximaion A π ( sa, ) = log π( as ) w (Prs al., 2003). Wih his assumpion, w can rarrang h discound summaion of (10) for on pisod rajcory wih N sas lik blow: N π N π π γa sa 0 = γ Q sa 0 V s = = (, ) ( (, ) ( )) N N γ π 0 γ r 0 γv + 1 V = as w = sa + s s = log ( ) ( (, ) ( ) ( )) N N+ 1 = γr(, ) + γ V( N+ 1) = 0 N N γ π 0 V 0 γr = as w + s = sa = 0 log ( ) ( ) (, ) sa s V ( s0) whr h rm N+1 V(s N+1 ) is ngligibl for is small ffc. By ling V(s 0 ) h produc of 1- by-1 criic paramr vcor v and 1-by-1 faur vcor [1], w can formula h following rgrssion quaion: N = 0 (11) N w γ [ log π( as ),1] (, ) = γr v sa (12) = 0 Hr, naural gradin vcor w is usd for updaing policy paramr vcor θ (quaion (8)), and valu funcion paramr v is usd for approximaing valu funcion V π (s ). By ling N γ = 0 φ = [ log π( as ), 1], = [ w, v ], and rarrang quaion (12) as a las-squar problm as follow: whr N r = γ r( sa, ), w can = 0 G = b (13) G = φφ and b = φ r. Marix G for pisod is updad o yild h soluion vcor using h following upda rul: G G + δi PG = 1 = Pb, whr is a posiiv scalar consan, and I is an idniy marix. By adding h rm I in (14), marix G bcoms diagonally-dominan and non-singular, and hus invribl (Srang, 1988). As h upda progrsss, h ffc of prurbaion is diminishd. Equaion (14) is h convnional form of criic upda rul of NAC algorihm. h main diffrnc bwn h convnional NAC algorihm and h RLS-basd NAC algorihm is h way marix G and soluion vcor χ ar updad. h RLS-basd NAC algorihm mploys h upda rul of RLS filr. h ky faur of RLS algorihm is o xploi G -1-1, which alrady xiss, for h simaion of G -1. Rahr han calculaing P (14)

9 266 Robo Manipulaors (invrs marix of G ) using h convnional mhod for marix invrsion, w usd h RLS filr-basd upda rul as follows: (Moon al., 2000): P k = P 1 β β + φ P 1φ P φ = β + φ P φ 1 1 P φφp = + k ( r φ ) 1 1 (15) Whr forging facor (0 1) is usd o accumula h pas informaion in a discoun mannr. By using h RLS-filr basd upda rul, h invrs marix can b calculad wihou oo much compuaional burdn. his is rpad unil h soluion vcor convrgs. I should b nod, howvr, ha marix G should b obaind using (14) for h firs pisod (pisod 1). h nir procdur of h RLS-basd pisodic Naural Acor-Criic algorihm is summarizd in abl 1. Iniializ ach paramr vcor: = G0P0b0s0,,,, 0 = = = = for ach pisod, Run simulaor: for ach sp, ak acion a, from sochasic policy π, hn, obsrv nx sa s +1, rward r. nd Upda Criic srucur: if firs upda, Upda criic informaion marics, following h iniial upda rul in (13) (14). ls Upda criic informaion marics, following h rcursiv las-squars upda rul in (15). nd Upda Acor srucur: Upda policy paramr vcor following h rul in (8). rpa unil convrg abl 1. RLS-basd pisodic Naural Acor-Criic algorihm 3.3 Sochasic Acion Slcion As discussd in scion 2, h characrisics of siffnss llips can b changd by modulaing hr paramrs: h magniud, shap, and orinaion. For prforming conac asks, policy π is dsignd o plan h chang ra of h magniud, shap, and orinaion of siffnss llips a ach sa by aking acions ( a = [ a, a, a ] ). mag shap orin hrough h squnc of an pisod, policy π drmins hos acions. h goal of h

10 Novl Framwork of Robo Forc Conrol Using Rinforcmn Larning 267 larning algorihm is o find h rajcory of h opimal siffnss llips during h pisod. Fig. 4 illusras h chang of siffnss llips corrsponding o ach acion. Policy π is in h form of Gaussian dnsiy funcion. In h criic srucur, compaibl funcion approximaion log π ( as ) w is drivd from h sochasic policy π basd on h algorihm suggsd by Williams (Williams, 1992). Policy π for ach componn of acion vcor a can b dscribd as follows: 2 1 ( a µ ) π ( as ) = Ν( a µ, σ) = xp( ) (16) 2 σ 2π 2σ Sinc h sochasic acion slcion is dpndn on h condiions of Gaussian dnsiy funcion, h acion policy can b dsignd by conrolling h man and h varianc σ of quaion (16). hs variabls ar dfind as: µ = ξ s 1 σ = ξ xp( η ) Sinc h sochasic acion slcion is dpndn on h condiions of Gaussian dnsiy funcion, h acion policy can b dsignd by conrolling h man and h varianc σ of quaion (16). hs variabls ar dfind as: µ = ξ s σ = ξ xp( η ) As can b sn in h quaion, man is a linar combinaion of h vcor and sa vcor s wih man scaling facorξ. Varianc σ is in h form of h sigmoid funcion wih h posiiv scalar paramr and varianc scaling facorξ. As a wo-link manipulaor modl is usd in his sudy, h componns of sa vcor s includ h join angls and s = xxxx,,,, 1, whr h fifh componn of 1 is vlociis of shouldr and lbow joins ( [ ] a bias facor). hrfor, naural gradin vcor w (and hus policy paramr vcor ) is composd of 18 componns as follows: w = [,,, η, η, η ] mag shap orin mag shap orin, whr, mag, shap, and orin ar 5-by-1 vcors and mag, shap, and orin ar paramrs corrsponding o h hr componns of acion vcor ( a = [ a, a, a ] ). mag shap orin (17) 4. Conac ask Applicaions In his scion, h mhod dvlopd in h prvious scion is applid o wo conac asks: poin-o-poin movmn in an unknown forc fild, and caching a flying ball. h wo-link manipulaor dvlopd in Scion 3 was usd o prform h asks in wo-dimnsional

11 268 Robo Manipulaors spac. h dynamic simulaor is consrucd by MSC.ADAMS2005, and h conrol algorihm is implmnd using Malab/Simulink (Mahworks, Inc.). Figur 4. Som variaion xampls of siffnss llips. (a) h magniud (ara of llips) is changd (b) h shap (lngh raio of major and minor axs) is changd (c) h orinaion (dircion of major axis) is changd (solid lin: siffnss llips a im, dashd lin: siffnss llips a im +1)

12 Novl Framwork of Robo Forc Conrol Using Rinforcmn Larning Poin-o-Poin Movmn in an Unknown Forc Fild In his ask, h wo-link roboic manipulaor maks a poin-o-poin movmn in an unknown forc fild. h vlociy-dpndn forc is givn as: F viscous 10 0 x = 0 10 y h movmn of h ndpoin of h manipulaor is impdd by h forc proporional o is vlociy, as i movs o h goal posiion in h forc fild. his condiion is idnical o ha of biomchanics sudy of Kang al. (Kang al., 2007). In hir sudy, h subjc movs ons hand from a poin o anohr poin in a vlociy-dpndn forc fild, which is sam as quaion (18), whr h forc fild was applid by an spcial apparaus (KINARM, BKIN chnology). For his ask, h prformanc indics ar chosn as follows: 1. h roo man squar of h diffrnc bwn h dsird posiion (virual rajcory) and h acual posiion of h ndpoin ( ) 2. h magniud of h im ra of orqu vcor of wo arm joins ( 2 2 τ = τ + τ ) rms 1 2 h orqu ra rprsns powr consumpion, which can also b inrprd as mabolic coss for human arm movmn (Franklin al., 2004; Uno al., 1989). By combining h wo prformanc indics, wo diffrn rwards ar formulad for on pisod as follows: rward1 = κ ( ) N 1 = 1 rms N N ( κ 1 rms ) w ( κ = τ = 1 ) rward 2 = w ( ) , whr w 1 and w 2 ar h wighing facors, and 1 and 2 ar consans. h rward is a wighd linar combinaion of im ingrals of wo prformanc indics. h larning paramrs wr chosn as follows: = 0.05, = 0.99, = h chang limis for acion ar s as [-10, 10] dgrs for h orinaion, [-2, 2] for h major/minor raio, and [-200π, 200π] for h ara of siffnss llips. h iniial llips bfor larning was s o b circular wih h ara of 2500π. h sam physical propris as in (Kang al., 2007) wr chosn for dynamic simulaions (abl 2). Lngh(m) Mass(Kg) Inria(Kg m 2 ) Link Link abl 2. Physical propris of wo link arm modl Fig. 5 shows h chang of siffnss llips rajcory bfor and afr larning. Bfor larning, h ndpoin of h manipulaor was no vn abl o rach h goal posiion (Fig. 5 (a)). Figs. 5 (b) and (c) compar h siffnss llips rajcoris afr larning using wo diffrn rwards (rward1 and rward2). As can b sn in h figurs, for boh rwards h major axis of siffnss llips was dircd o h goal posiion o ovrcom rsisanc of viscous forc fild. (18)

13 270 Robo Manipulaors Fig. 6 compars h ffcs of wo rwards on h changs of wo prformanc indics ( and τ ) as larning iras. Whil h choic of rward dos no affc h im ingral of rms, h im ingral of τ was supprssd considrably by using rward2 in larning. h rsuls of dynamic simulaions ar comparabl wih h biomchanics sudy of Kang al.. h rsuls of hir sudy suggs ha h human acivly modulas h major axis oward h dircion of h xrnal forc agains h moion, which is in accordanc wih our rsuls. rms Figur 5. Siffnss llips rajcoris. (dod lin: virual rajcory, solid lin: acual rajcory). (a) Bfor larning. (b) Afr larning (rward1). (c) Afr larning (rward2) Figur 6. Larning ffcs of prformanc indics (avrag of 10 larning rial). (a) posiion rror. (b) orqu ra 4.2 Caching a Flying Ball In his ask, h wo-link roboic arm cachs a flying ball illusrad in Fig. 7. h simulaion was prformd using h physical propris of h arm as lisd in abl 2. h main issus

14 Novl Framwork of Robo Forc Conrol Using Rinforcmn Larning 271 in ball-caching ask would b how o dc h ball rajcory and how o rduc h impulsiv forc bwn h ball and h nd-ffcor. his work focuss on h lar and assums ha h ball rajcory is known in advanc. Figur 7. Caching a flying ball Whn a human cachs a ball, on movs ons arm backward o rduc h impulsiv conac forc. By considring h human ball-caching, h ask is modld as follow: A ball is hrown o h nd-ffcor of h robo arm. h im for h ball o rach h nd-ffcor is approximaly 0.8sc. Afr h ball is hrown, h arm sars o mov following h parabolic orbi of h flying ball. Whil h nd-ffcor is moving, h ball is caugh and hn movs o h goal posiion oghr. h robo is s o cach h ball whn h ndffcor s moving a is highs spd o rduc h impulsiv conac forc bwn h ball and h nd-ffcor. h impulsiv forc can also b rducd by modulaing h siffnss llips during h conac. h larning paramrs wr chosn as follows: = 0.05, = 0.99, = h chang limis for acion ar s as [-10, 10] dgrs for h orinaion, [-2, 2] for h major/minor raio, and [-200π, 200π] for h ara of siffnss llips. h iniial llips bfor larning was s o b circular wih h ara of π. For his ask, h conac forc is chosn as h prformanc indx: F = F + F 2 2 conac x y h rward o b maximizd is h impuls (im ingral of conac forc) during conac: rward = F conac N κ = 1 whr is a consan. Fig. 8 illusras h chang of siffnss during conac afr larning. As can b sn in h figur, h siffnss is und sof in h dircion of ball rajcory, whil h siffnss normal o h rajcory is much highr. Fig. 9 shows h chang of h impuls as larning coninus. As can b sn in h figur, h impuls was rducd considrably afr larning.

15 272 Robo Manipulaors Figur 8. Caching a flying ball Figur 9. Caching a flying ball 5. Conclusions Safy in roboic conac asks has bcom on of h imporan issus as robos sprad hir applicaions o dynamic, human-populad nvironmns. h drminaion of impdanc

16 Novl Framwork of Robo Forc Conrol Using Rinforcmn Larning 273 conrol paramrs for a spcific conac ask would b h ky faur in nhancing h robo prformanc. his sudy proposs a novl moor larning framwork for drmining impdanc paramrs rquird for various conac asks. As a larning framwork, w mployd rinforcmn larning o opimiz h prformanc of conac ask. W hav dmonsrad ha h proposd framwork nhancs conac asks, such as door-opning, poin-o-poin movmn, and ball-caching. In our fuur works w will xnd our mhod o apply i o ach a srvic robo ha is rquird o prform mor ralisic asks in hr-dimnsional spac. Also, w ar currnly invsigaing a larning mhod o dvlop moor schmaa ha combin h inrnal modls of conac asks wih h acor-criic algorihm dvlopd in his sudy. 6. Rfrncs Amari, S. (1998). Naural gradin works fficinly in larning, Nural Compuaion, Vol. 10, No. 2, pp , ISSN Asada, H. & Sloin, J-J. E. (1986). Robo Analysis and Conrol, John Wily & Sons, Inc., ISBN Boyan. J. (1999). Las-squars mporal diffrnc larning, Procding of h 16h Inrnaional Confrnc on Machin Larning, pp Cohn, M. & Flash,. (1991). Larning impdanc paramrs for robo conrol using associaiv sarch nwork, IEEE ransacions on Roboics and Auomaion, Vol. 7, Issu. 3, pp , ISSN X Engl, Y.; Mannor, S. & Mir, R. (2003). Bays ms bllman: h gaussian procss approach o mporal diffrnc larning, Procding of h 20h Inrnaional Confrnc on Machin Larning, pp Flash,. & Hogan, N. (1985). h coordinaion of arm movmns: an xprimnally confirmd mahmaical modl, Journal of Nuroscinc, Vol. 5, No. 7, pp , ISSN Flash,. (1987). h conrol of hand quilibrium rajcoris in muli-join arm movmn, Biological Cybrnics, Vol. 57, No. 4-5, pp , ISSN Franklin, D. W.; So, U.; Kawao, M. & Milnr,. E. (2004). Impdanc conrol balancs sabiliy wih mabolically cosly muscl acivaion, Journal of Nurophysiology, Vol. 92, pp , ISSN Hogan, N. (1985). Impdanc conrol: An approach o manipulaion: par I. hory, par II. implmnaion, par III. applicaion, ASME Journal of Dynamic Sysm, Masurmn, and Conrol, Vol. 107, No. 1, pp. 1-24, ISSN Izawa, J.; Kondo,. & Io, K. (2002). Biological robo arm moion hrough rinforcmn larning, Procdings of h IEEE Inrnaional Confrnc on Roboics and Auomaion, Vol. 4, pp , ISBN Jung, S.; Yim, S. B. & Hsia,. C. (2001). Exprimnal sudis of nural nwork impdanc forc conrol of robo manipulaor, Procdings of h IEEE Inrnaional Confrnc on Roboics and Auomaion, Vol. 4, pp , ISBN Kang, B.; Kim, B.; Park, S. & Kim, H. (2007). Modling of arificial nural nwork for h prdicion of h muli-join siffnss in dynamic condiion, Procding of IEEE/RSJ Inrnaional Confrnc on Inllign Robos and Sysms, pp , ISBN , San Digo, CA, USA

17 274 Robo Manipulaors Kazrooni, H.; Houp, P. K. & Shridan,. B. (1986). h fundamnal concps of robus complian moion for robo manipulaors, Procdings of h IEEE Inrnaional Confrnc on Roboics and Auomaion, Vol. 3, pp , MN, USA Lipkin, H. & Parson,. (1992). Gnralizd cnr of complianc and siffnss, Procdings of h IEEE Inrnaional Confrnc on Roboics and Auomaion, Vol. 2, pp , ISBN , Nic, Franc Moon,. K. & Sirling, W. C. (2000). Mahmaical Mhods and Algorihm for Signal Procssing, Prnic Hall, Uppr Saddl Rivr, NJ, ISBN Park, J.; Kim, J. & Kang, D. (2005). An rls-basd naural acor-criic algorihm for locomoion of a wo-linkd robo arm, Procding of Inrnaional Confrnc on Compuaional Inllignc and Scuriy, Par I, LNAI, Vol. 3801, pp , ISSN Park, S. & Shridan,. B. (2004). Enhancd human-machin inrfac in braking, IEEE ransacions on Sysms, Man, and Cybrnics, - Par A: Sysms and Humans, Vol. 34, No. 5, pp , ISSN Prs, J.; Vijayakumar, S. & Schaal, S. (2005). Naural acor-criic, Procding of h 16h Europan Confrnc on Machin Larning, LNCS, Vol. 3720, pp , ISSN Prs J. & Schaal, S. (2006). Policy gradin mhods for roboics, Procding of IEEE/RSJ Inrnaional Confrnc on Inllign Robos and Sysms, pp , ISBN X, Bijing, China Srang, G. (1988). Linar Algbra and Is Applicaions, Harcour Brac & Company Suon, R. S. & Baro, A. G. (1998). Rinforcmn Larning: An Inroducion, MI Prss, ISBN suji,.; rauchi, M. & anaka, Y. (2004). Onlin larning of virual impdanc paramrs in non-conac impdanc conrol using nural nworks, IEEE ransacions on Sysms, Man, and Cybrnics, - Par B: Cybrnics, Vol. 34, Issu 5, pp , ISSN Uno, Y.; Suzuki, R. & Kawao, M. (1989). Formaion and conrol of opimal rajcory in human muli-join arm movmn: minimum orqu chang modl, Biological Cybrnics, Vol. 61, No. 2, pp , ISSN Williams, R. J. (1992). Simpl saisical gradin-following algorihms for conncionis rinforcmn larning, Machin Larning, Vol. 8, No. 3-4, pp ISSN Won, J. (1993). h Conrol of Consraind and Parially Consraind Arm Movmn, S. M. hsis, Dparmn of Mchanical Enginring, MI, Cambridg, MA, Xu, X.; H, H. & Hu, D. (2002). Efficin rinforcmn larning using rcursiv las-squars mhods, Journal of Arificial Inllign Rsarch, Vol. 16, pp

18 Robo Manipulaors Edid by Marco Cccarlli ISBN Hard covr, 546 pags Publishr Inch Publishd onlin 01, Spmbr, 2008 Publishd in prin diion Spmbr, 2008 In his book w hav groupd conribuions in 28 chaprs from svral auhors all around h world on h svral aspcs and challngs of rsarch and applicaions of robos wih h aim o show h rcn advancs and problms ha sill nd o b considrd for fuur improvmns of robo succss in worldwid frams. Each chapr addrsss a spcific ara of modling, dsign, and applicaion of robos bu wih an y o giv an ingrad viw of wha mak a robo a uniqu modrn sysm for many diffrn uss and fuur ponial applicaions. Main anion has bn focusd on dsign issus as hough challnging for improving capabiliis and furhr possibiliis of robos for nw and old applicaions, as sn from oday chnologis and rsarch programs. hus, gra anion has bn addrssd o conrol aspcs ha ar srongly volving also as funcion of h improvmns in robo modling, snsors, srvo-powr sysms, and informaics. Bu vn ohr aspcs ar considrd as of fundamnal challng boh in dsign and us of robos wih improvd prformanc and capabiliis, lik for xampl kinmaic dsign, dynamics, vision ingraion. How o rfrnc In ordr o corrcly rfrnc his scholarly work, fl fr o copy and pas h following: Byungchan Kim and Shinsuk Park (2008). Novl Framwork of Robo Forc Conrol Using Rinforcmn Larning, Robo Manipulaors, Marco Cccarlli (Ed.), ISBN: , Inch, Availabl from: hp:// cmn_larning Inch Europ Univrsiy Campus SP Ri Slavka Krauzka 83/A Rijka, Croaia Phon: +385 (51) Fax: +385 (51) Inch China Uni 405, Offic Block, Hol Equaorial Shanghai No.65, Yan An Road (Ws), Shanghai, , China Phon: Fax: