Learning Goal Seeking and Obstacle Avoidance using the FQL Algorithm

Transcription

1 Proceedngs of he World Congress on Engneerng and Compuer Scence 2007 WCECS 2007, Ocober 24-26, 2007, San Francsco, USA Learnng Goal Seekng and Obsacle Avodance usng he FQL Algorhm Abdelkarm Souc, Hacene Rezne UER Auomaque EMP BP 7 M EMP Bordj-El-Bahr, 6 Alger emal:aks752005@yahoo.fr emal:rezne_hacene_emp@yahoo.fr Absrac In hs arcle, we are neresed n he reacve behavours navgaon ranng of a moble robo n an unknown envronmen. The mehod we wll sugges ensures navgaon n unknown envronmens wh presence off dfferen obsacles shape and consss n brngng he robo n a goal poson, avodng obsacles and releasng from he gh corners and deadlock obsacles shape. Ths s dffcul o do manually n he case of FIS wh a large rule base. In hs framework, we use he renforcemen learnng algorhm called Fuzzy Q-learnng, based on emporal dfference predcon mehod. The applcaon was esed n our expermenal PIONEER II plaform. Key-words Moble robo navgaon, reacve navgaon, fuzzy conrol, renforcemen learnng, FACL Fuzzy Acor-Crc learnng, FQL Fuzzy Q-learnng. I. INTRODUCTION In hs arcle, we propose a renforcemen ranng mehod where he apprence explores acvely s envronmen. I apples varous acons n order o dscover he saes causng he emsson of rewards and punshmens. The agen mus fnd he acon whch mus carry ou when s n a gven suaon mus learn how o choose he opmal acons o acheve he fxed goal. The envronmen can punsh or reward he sysem accordng o he appled acons. Each me ha an agen apples an acon, a crc gves hm a reward or a penaly o ndcae f he resulng sae s desrable or no [], [6]. The ask of he agen s o learn usng hese rewards he connuaon of acons whch ges he greaes cumulave reward. Moble robocs s a prvleged applcaon feld of he renforcemen learnng [4], [0], []. Ths esablshed fac s relaed o he growng place whch akes, snce a few years. The goal s hen o consder behavour as a mappng funcon beween sensor and effecors. The ranng n robocs consss of he auomac modfcaon of he behavour of he robo o mprove n s envronmen. The behavours are synheszed sarng from he smple defnon of objecves hrough a renforcemen funcon. The consdered approach of he robo navgaon usng fuzzy nference as apprence s ready o negrae ceran errors n he nformaon abou he sysem. For example, wh fuzzy logc we can process vague daa. The percepon of he envronmen by ulrasounds sensors and he renforcemen ranng hus prove o be parcularly well adaped one o he oher [2], [5], [7], and [3]. I s very dffcul o deermnae correc conclusons manually n a large rule base FIS o ensure he releasng from gh corner and deadlock obsacles, even when we use a graden descen learnng mehod or a poenal-feld echnque due o he local-mnmum problem. In such suaons he robo wll be blocked. Behavours made up of a fuson of a «goal seekng» and of an «obsacle avodance» ssues are presened. The mehod we wll sugges ensures navgaon n unknown envronmens wh presence off dfferen obsacles shape, he behavours wll be realsed wh SIF whose conclusons are deermned by renforcemen ranng mehods. The algorhms are wren usng he Malab sofware afer negrang n a Smulnk block he funcons of percepon, localzaon and morcy of he robo. The applcaon was esed n our expermenal plaform PIONEER II. II. FQL ALGORITHM The seleced apprence s a zero-order Takag-Sugeno FIS due o s smplcy, unversal approxmaor characerscs, generalzaon capacy and s real me applcaons. The npu varables have a rangular and rapezodal membershp funcons. The SIF hus consss of N rules of he followng form [6]: If suaon hen Y= u[, ] wh q[,] Y= u[, 2] wh q[,2] Y= u[, j] wh q[,j] The FQL algorhm s a compac verson of he FACL [7] [4] and s an adapaon of Q-Learnng proposed by Wakns [2] [3] for a fuzzy apprence ype, each rule R of he apprence has: a se of dscree acons U dencal for all rules. a vecor of parameers q ndcang he qualy of he varous dscree acons avalable ISBN: WCECS 2007

2 Proceedngs of he World Congress on Engneerng and Compuer Scence 2007 WCECS 2007, Ocober 24-26, 2007, San Francsco, USA The characerscs of he npu varables membershp funcons (number, poson) are fxed. The number of rules s also fxed. Thus, he only modfable characerscs of he apprence are he conclusons y wch s he elecon of an acon u[,j] among J acons avalable n he rule R. Indeed he FQL represen an adapave verson of he VI Value Ieraon n whch he funcon Q(S,U) s no calculaed for all he saes bu only approxmaed on he curren sae [7]. Le us remember ha hs funcon Q (S, U) represens he prmary renforcemen expeced f he acon U s appled n he sae S plus he dscouned sum of he secondary renforcemens of all he possble successors saes, f he apprence follows an opmal polcy hereafer. The apprence nfers hs value Q from he marx Q presens n he Fuzzy Acor-Crc learnng FACL (he acor) [7] [4]: Q ( S, U ) = q ( U ) α R ( S ), () R A Where U represens he local acon eleced n he rule R a he me sep, U he nferred global acon and α S ) R ( he ruh degrees of he rule R. Accordng o he VI algorhm, hs funcon s defned by: Q ( S, U ) = R( S, U ) + γ PS S ( U ) V ( S ), (2) S S Expresson () s a fuzzy approxmaon of he expresson (2). Where he opmal evaluaon funcon s defned by he values Q correspondng o he opmal acons: V ( S) = MaxQ( S, U ). (3) U U In he case of our FIS apprence, we propose o defne hs funcon by he bes acon of each rule, ha s o say: Q ( S ) = ( Max( q ( U )) α R ( S ), (4) R A U U Where Q ( S ) represen he -opmal evaluaon funcon. Then he same dffculy as for he FACL appears: he absence of a MDP model allowng he calculaon of he expresson (2). The mehod s hen dencal [7] [4] and consss n approxmang hs expresson only wh he value avalable a he me sep +: ( = Q S, U ) r γ Q ( S ), (5) The approxmaon error of he apprence hen s smply defned by: ~ ε + = r + + γq ( S+ ) Q ( S, U ) (6) Where ~ ε + represen TD error. The updae of he parameers q modellng he funcon Q s hen srcly dencal o ha defned for he acor n FACL algorhm [7] [4], wh an nroducon of ranng raes, because here hese parameers are no only used o mplemen he compeon beween acons bu also approxmae he funcon Q. we obans hen : q = q + ~ + ϑ ε + α R, R (7) The above expresson shows ha a posve error TD mples ha he acon wch has been jus appled s beer han he -opmal acon. I s necessary o ncrease he qualy of he acon beng appled. Recprocally, f he TD error s negave, s necessary o decrease he qualy of he acon beng appled because led he sysem n a sae whose evaluaon s lower han ha expeced. A. Elgbly Traces The acor races mplemenaon rses drecly from he ncremenal verson of he TD. The race for he acor consss n memorzng he acons appled n he saes. We use a shor erm memory of he ruh values of each rule accordng o each acon avalable n hese rules. Le e ( U ) be he race value of he acon U n he rule R a he sep [7]: γλ. e ( U ) + φ,( U = U ), e ( U ) = (8) γλ. e ( U ), else where λ s he proxmy facor of he acor and φ s he percepon (ruh degrees). Thus, he updaes of he parameers prese by (7) for all he rules and acons become: q T = q + ~ + ε+ e, ϑ R (9) B. Descrpon of he FQL execuon procedure The execuon n a me sep can be dvded no sx prncpal sages. Le + be he sep of curren me; he apprence hen has appled he acon U eleced n he prevous me sep and on he oher hand has receved he prmary renforcemen r + for he ranson from he sae S o S +. Afer he calculaon of he ruh values of he rules, he sx sages are as follows [7]: - Calculaon of he -opmal evaluaon funcon of he curren sae: ( + R A U U R + 2- TD Error calculaon: Q S ) = ( Max( q ( U )) α ( S ), (0) ISBN: WCECS 2007

3 Proceedngs of he World Congress on Engneerng and Compuer Scence 2007 WCECS 2007, Ocober 24-26, 2007, San Francsco, USA ~ ε r + Q ( S ) Q ( S, U ) () + = + γ + 3- Updae of ranng raes correspondng o all rules and acons n hree sages: In order o accelerae he ranng speed and o avod nsably, we adop a heursc adapve ranng rae [7]. The mplemenaon of hs heursc s carred ou by he means of Dela Bar Dela rule: - ~ δ ( U ) = ε e ( U ) - ϑ + ϑ ( U) + k f δ δ > + ( U) = ϑ ( U)( ψ ) f δ δ ϑ ( U) else. - δ = ( ψ ) δ + ψδ where n our case: + U 0, < 0, (2) - ϑ ( ) s he ranng rae of he acon U n he me sep for he rule R. - ~ δ ( U ) = ε + e ( U ) represen he varaon brough o he parameer q n whch we propose o negrae he elgbly race drecly and no smply he paral dervave. - δ = ψ ) δ + ψδ represen he exponenal average. ( Ths rule hen ncreases lnearly he ranng raes n order o preven ha do no become oo large, and decreasng n an exponenal way o ensure ha decrease quckly. 4- Tranng he acor by updang he marx q same as (9): T = q + ~ + ε + e, q ϑ R (3) 5- Now remans he choce of he acon o be appled n he sae S +. We consder he case of connuous ype acons, whch s nferred from he varous acons eleced n each rule. + ( S+ ) = EleconU ( q+ ). R ( S + ), U R A + U α U, Where Elecon s defned by: Elecon U( q + ) = ArgMax U U ( q + ( U) +η ( U) + ρ ( U)) (4) And η (U ) erm of random exploraon, ρ (U ) erm of dreced exploraon. The races elgbly updae for he acor s gven by he wo followng formulas: e + γλ. e ( U ) + φ+,( U ( U ) = γλ. e ( U ), else = U + ), (5) C. Proposed Exploraon/Exploaon The acons elecon sraegy whch we used s a combnaon of dreced and random exploraon by he mean of an exploraon erm uses a random par and a erm based on a couner [7]. The global elecon funcon, appled o each rule, s hen defned by: EleconU ( q) = ArgMaxU U ( U) +η ( U) + ρ( U)), (6) Where U represens he se of avalable dscree acons n each rules, and q he assocaed vecor qualy. The erm of random exploraon η s n fac a random values vecor. I corresponds o a vecor ψ of values sampled accordng o an exponenal law, sandardzed n order o ake no accoun he q quales sze scale: Where f Max( U)) = Mn( U )) s f ( U) = s Max q U Mn q U p ( ( ( )) ( ( ))), else Max( ψ ) (7) η( U ) = s ( U ). ψ, (8) f s p represens he maxmum sze of he nose relave o he quales amplude, s f s he correspondng normalsaon facor. Used alone, hs erm of exploraon allows a random choce of acons when all quales are dencal and allows n he oher case, o es only he acons of whch qualy sasfy: U) U ) s p ( Max( U)) Mn( U))) (9) Ths erm hen allow us o avod he choce of bad acons [7]. The erm of dreced exploraon ρ perm he es of acons no havng appled ofen. I s hus necessary o memorze he mes number he acon was eleced. Ths erm s defned n he θ followng way: ρ ( U ) = (20) ( U ) e n Where θ represens a posve facor whch perm o equlbrae he dreced exploraon, n (U ) he number of acon U applcaons a he sep of me. In he case of acons of he connuous ype, n (U ) corresponds o a number of applcaons of he dscree acon U n he consdered rule. Le U he dscree acon eleced a he sep me n he rule R, he updae of hs varable s deermned by he followng equaon: n ( U ) = n ( U ) +, R A, U = U (2) 6- I s agan necessary o calculae he -opmal evaluaon funcon of he curren sae bu hs me wh he recenly updaed parameers: = + R A + Q ( S, U ) q ( U ) α ( S ), (22) + R + ISBN: WCECS 2007

4 Proceedngs of he World Congress on Engneerng and Compuer Scence 2007 WCECS 2007, Ocober 24-26, 2007, San Francsco, USA Ths value wll be used for he TD error calculaon n he nex me sep. III. EXPERIMENTAL PLATFORM Poneer P2-dx Fg. wh s modes sze lends self well o navgaon n he gh corners and encumbered spaces such as he laboraores and small offces. I has wo drvng wheels and a casor wheel. To deec obsacles, he robo s equpped wh a se of ulrasounds sensors. I suppors egh ulrasonc sensors placed n s fron Fg. 2. The range of measuremens of hese sensors les beween 0cm as mnmal range and 5m as maxmum range. The sofware Saphra/Ara [8], [9] allows he conrol of he robo (C/C++ pragrammng). We have négraed funcons of Saphra (percepon, localzaon and morcy) n Smulnk usng API (S-Funcon), Thus we can benef from MaLab compung power and smplcy of Smulnk o es our algorhms and o conrol he robo Poneer II. Fgure : Poneer II Robo Phoo Fgure 3: membershp funcons forθ Rb (a) and ρ b (b) From he heursc naure of FQL algorhm, we carred ou several ess o deermne he values of he parameers whch accelerae he ranng speed and o oban good performances for he apprence. Afer a seres of expermens, we found he followng values: θ = 50, λ = 0.9,, sp = 0. and γ = 0. 9 Avalable acons for all rules, are as follow {-20 /s, -0 /s, -5 /s, 0 /s, +5 /s, +0 /s, +20 /s} for he roaon velocy and {0 mm/s, 50 mm/s, 350mm/s} for ranslaon, whch gves 2 possble acons n each fuzzy rule. The renforcemen funcon s defned as follows: - If he robo s far from he goal, he renforcemen s equal o f ( θ Rb. & θ < 0) & V=0 Rb f (- < θ Rb <+ ) & V 0 0 f ( θ Rb. & θ = 0) & V=0 Rb - else - If he robo s close o he goal, he renforcemen s equal o f ( θ Rb. & θ < 0) & V=0 Rb - else Fgure (4) shows he rajecory of he robo durng he ranng and valdaon phase. Fgure 2: Poson of he ulrasonc sensors used n he robo Poneer II. IV. EXPERIMENTATION The ask of he robo consss n sarng from a sarng pon achevng a fxed goal whle avodng he sac obsacles of convex or concave ype. I s realsed by he fuson of wo elemenary behavours «goal seekng» and «obsacle avodance ". A. The «goal seekng» behavour For he «goal seekng» behavour, we consder hree membershp funcons for he npu θ Rb (robo-goal angle), and wo membershp funcons for he npu ρ b (robo-goal dsance) (Fg. 3). The knowledge base consss of sx fuzzy rules. The FIS conroller has wo oupu for he acor, roaon speed Vro and he ranslaon speed V. Fgure 4: Trajecory of he robo durng and afer ranng B. The «obsacle avodance» behavor For hs behavour, we have deermned he ranslaon speed of he robo proporonal o he dsance from he fronal obsacles, wh a maxmum value of 350 mm/s Fg.5. Fgure 5: ranslaon speed V ISBN: WCECS 2007

5 Proceedngs of he World Congress on Engneerng and Compuer Scence 2007 WCECS 2007, Ocober 24-26, 2007, San Francsco, USA The npus of he fuzzy conroller for hs behavour are he mnmal dsances provded by he four ses of sonar {mn(d 90,d 50 ), mn(d 30,d 0 ), mn(g 0,g 30 ), mn(g 90,g 50 )}, wh respecvely hree membershp funcons for each one Fg. 6. P M L P M L (c) (d) Fgure 8: Tme evoluon of he robo afer ranng (a) (b) Fgure 6 : npu membershp funcons for he fuzzy conroller Vro (a) :{mn(d 90,d 50 ), mn(g 90,g 50 )} (b) :{mn(d 0,d 30 ), mn(g 0,g 30 )}, The se of he acon U common o all rules consss of fve acons {-20 /s, -0 /s, 0 /s, +0 /s, +20 /s}. The renforcemen funcon s defned as follows: + f mn {mn(d 90,d 50 ),mn(d 30,d 0 )} < mn {mn(g 0,g 30 ),mn(g 90,g 50 )} & Vro > 0 + f mn {mn(d 90,d 50 ),mn(d 30,d 0 )}> mn {mn(g 0,g 30 ),mn(g 90,g 50 )} & Vro < 0 - oherwse The fgure (7) shows he evoluon of he robo durng he ranng phase. Fgure 7: Trajecores of he robo durng and afer ranng On he followng fgure, we represen he me evoluon of he robo. I shows ha he robo s able o be released from he gh corners and deadlock obsacles shape. C. Fuson of he wo behavors The goal of he wo elemenary behavors fuson s o allow he robo navgaon n envronmens composed by fxed obsacles of convex or concave ype and o acheve a fxed goal whle ensurng s safey, whch s a fundamenal pon n reacve navgaon. The suggesed soluon consss n consderng he whole npu varables of he wo behavours «goal seekng» and «obsacle avodance» assocaed wh dsrbued renforcemen funcon by he means of a weghng coeffcen beween he wo behavours (0.7 for obsacles avodance and 0.3 for he goal seekng). The fuzzy conroller npu are sx who are he mnmal dsances provded by he four ses of sonar {mn(d 90,d 50 ), mn(d 30,d 0 ), mn(g 0,g 30 ), mn(g 90,g 50 )}, wh respecvely hree membershp funcons for he sde ses and wo membershp funcons for he fronal ses, o whch we add he npu θ Rb wh hree membershp funcons, and ρ b wh wo membershp funcons. The rules base hus consss of 26 fuzzy rules; manually s dffcul o ensure concluson coherence. The ranslaon speed of he robo s proporonal o he dsance from he fronal obsacles. The FQL algorhm parameers are slghly modfed as follows: θ =20 & sp =0.9 Fgure (9) represens he ype of rajecory obaned durng he ranng phase. The robo manages o avod he obsacles and o acheve he goal assgned n envronmens encumbered by mananng a reasonable dsance beween he obsacle and s wh sde dmensons. Fgure (0) llusraes he sasfyng behavour of he robo afer ranng. The varous rajecores obaned for he same envronmen and he same arrval and sarng pons are due prmarly o he problems of he percepon of he envronmen by he robo and are relaed o he phenomena of sonar readngs, hese dfferences confrm a he same me he effecveness of he ranng algorhm. (a) (b) Fgure 9: Trajecores of he robo n ranng phase ISBN: WCECS 2007

6 Proceedngs of he World Congress on Engneerng and Compuer Scence 2007 WCECS 2007, Ocober 24-26, 2007, San Francsco, USA (a) (c) (d) Fgure 0: Varous rajecores of he robo afer ranng The followng fgure llusraes he sasfyng behavour of he real robo whch evolves/moves n an unknown envronmen and whch manages o acheve he fxed goal. (b) [] S. Babvey, O. Momahan, M. R. Meybod, Mul Moble Robo Usng Dsrbued Value funcon Renforcemen Learnng, IEEE, Inernaonal Conference on Robocs &Auomaon, Sepember [2] H.R. Beom, H. S. Cho, A Sensor-Based Navgaon for a Moble Robo Usng Fuzzy Logc and Renforcemen Learnng, IEEE, Transacons on Sysems Man and Cybernecs, vol.25, NO.3,pp , March 995. [3] G. Fara, R. A.F Romero Incorporang Fuzzy Logc o Renforcemen Learnng, IEEE, pp ,brazl,2000. [4] T. Fuj, Y. Ara, H. Asama, I. Endo, Mullayered Renforcemen Learnng For Complcaed collson Avodance problems, Proceedngs of IEEE, Inernaonal Conference on Robocs &Auomaon, pp ,may 998. [5] T. Fukuda, Y. Hasegawa, K. Shmojma, F. Sao, Renforcemen Learnng Mehod For Generang Fuzzy Conroller, Deparmen of Mcro sysem Engneerng, Nagoya Unversy, pp , Japan, IEEE, 995. [6] P. Y. Glorennec, Renforcemen Learnng: An Overvew, INSA de Rennes ESIT 2000, Aachen, pp.7-35, Germany, Sepember [7] L. Jouffe, Apprenssage de Sysème D nférence Floue par des Méhodes de Renforcemen, Thèse de Docora, IRISA, Unversé de Rennes I, Jun 997. [8] K. G. KONOLIGE Saphra Référence, Edon Doxygen, Novembre [9] K. G. KONOLIGE Saphra Robo Conrol Archecure, Edon SRI Inernaonal, Avrl, 2002 [0] W. D. Smar, L. P. Kaelblng, Effecve Renforcemen Learnng For Moble Robos, Proceedngs of he IEEE, Inernaonal Conference on Robocs &Auomaon, pp , May [] R.S. Suon, A. Baro, Renforcemen Learnng, Bradford Book, 998. [2] C. J. C. H. Wakns, Learnng from delayed rewards, Ph.D. hess, Cambrdge Unversy, 989. [3] C. Wakns, P. Dayan, Q-Learnng Machne Learnng, pp , 992. [4] AK. Souc H.Rezne Behavour Navgaon Learnng Usng FACL Algorhm, paper acceped n ICINCO 07, Angers France, Ma Fgure : Trajecory of he real robo afer ranng V. CONCLUSION FQL Algorhm makes possble o nroduce generalzaon no saes and acons space, and a conclusons adapaon ncremenally of a zero order Sugeno ype FIS and hs only by he means of he neracons beween he apprence and hs envronmen. The renforcemen funcon consues he measure of he performance of he requred behavour soluon. Wh he FQL algorhm we can solve decson problems n Fuzzy Inference Sysem conclusons. Also, fuson of he behavours «goal seekng» and «obsacle avodance» s presened by usng a combned renforcemen funcon. The smulaon and expermenaon resuls n varous envronmens are sasfacory. VI. REFERENCES ISBN: WCECS 2007