Goal Seeking of Mobile Robot Using Fuzzy Actor Critic Learning Algorithm

Transcription

1 Goal Seekng of Moble Robo Usng Fuzzy Acor Crc Learnng Algorhm F. Lachekhab, M. Tadjne Absrac In hs paper, we presen a sudy of a basc behavor of moble robo, whch s goal seekng. Frsly, we use he heurscs approaches o develop a conroller based on fuzzy logc o conrol he robo and brng from an nal poson o a fnal poson. Secondly, we use he learnng echnque -Fuzzy acor crc learnng algorhm (FACL)-. The renforcemen learnng FACL has o selec he acons avalable n each fuzzy rule. The man advanage of he proposed mehod s he auomac consrucon of hese rules. So far, smulaon has shown ha he conroller s able o perform successful navgaon ask n known envronmens, and has smooh acon and exceponally good robusness. I. INTRODUCTION Few decades ago, a specal effor had been made n he felds of research and ndusry o buld auonomous moble robos wh mnmal human nervenon [8]. However, when he envronmen becomes more complex, s essenal ha he robo would be equpped wh ably of decson-makng o reac o hazards ha can hwar hs movemens. The navgaon resoluon problem of a moble robo has been largely a subjec of research and resuls of arfcal nellgence. We have chosen o use a reasonng based on he use of fuzzy logc, where mprecse nformaon can be processed n smlar way as has been done by human. I allows avodng he complex phase of mahemacal modelng of sysems o command. I provdes more decsonmakng and nformaon processng n much reduced compuaon mes. Due o s performance, we chose o use hs ool n he proposed approach module. The work presened n hs arcle focuses on he developmen of a conrol sysem for a moble robo based on fuzzy logc. The overall goal s o make he robo learn a smple behavor as gong owards a goal. For hs, n he frs phase, he navgaon module was made wh a sandard fuzzy nference sysem where paramerc and srucural characerscs have been deermned by a human exper (heurscally). In he second phase, we use a renforcemen learnng mehod whch s based on he acve exploraon of he envronmen n order o dscover he saes causng he emsson of rewards and punshmens. In hs conex, we use he Fuzzy Acor-Crc Learnng (FACL) algorhm based on he predcon mehod of emporary dfferences (TD). I allows he selecon of he bes acon from a se of avalable acons n each fuzzy rule. Ths paper s organzed as follows: In Secon II, he robo model s gven. In Secon III he heursc mehod and he resuls are presened. In Secon IV he renforcemen learnng mehod s nroduced hen he funcon of hs module (goal seekng) s verfed by smulaon. In Secon V real expermens on he Poneer 2P robo are gven. Fnally he paper s concluded n secon VI. II. EXPERIMENTAL PLATFORM Poneer 2P-DX s a small moble robo shown n Fg. 1. I conans all he basc componens for navgaon n a real envronmen, mosly nended for use n ndoor envronmens on hard and fla surfaces. 8 ulrasonc sonar Fgure 1. Image of he robo Poneer 2P-DX. Poneer 2 has 8 ulrasonc sonar ransducers o provde objecs deecon, each sensor S (for =1...8) gves a dsance d from he robo o he obsacle n s feld of vew of 20. For obsacle deecon and avodance purposes, he sensors are dvded no 4 sensor groups G (for =1 3). Saphra sofware s a robo conrol sysem developed a he Inernaonal Arfcal Inellgence Laboraory SRI. I s he nerface for robos Poneer Amgobo, PeopleBo and some oher knds of robos [4]. From he verson 8.x, several funcons of Saphra have been moved compleely o ARIA "AcvMeda Robo Inerface for Applcaons" Saphra and Ara sofware s wren n C ++.In our sudy, due o he smplcy offered by he oolbox Smulnk of MATLAB and hs ably of negraon of he developed program n C ++, we use a block dagram o acheve our smulaons. F. Lachekhab s wh he Unversy of Mhamed Bougara, Boumerdes BP3500. Algera. She s wh Laboraory of Auomac and Appled Sgnal Processng. ( e-mal: lachekhab_f@yahoo.fr). M.Tadjne s wh Polyechnc Naonal School, Algers, Algera. He s wh Process Conrol Laboraory (e-mal: adjne@yahoo.fr).

2 III. HEURISTIC METHOD In hs secon, we desgn he conroller heurscally. Furhermore he membershp funcons of npus and oupus and he fuzzy rules are deermned by human experse. A. Prncple of conrol Ths behavor allows he acon "convergence oward he goal" by he robo. For ha he conrol uses he knowledge of curren robo poson and he defnon of a poson relave o acheve (he goal). The purpose s o conrol he movemens of he robo so can reach he desnaon. I s clearly undersood ha hs behavor can only operae n uncongesed envronmens where no obsacles mpede he progress of he robo [7]. The defnon of he arge poson s done hrough wo npu varables " θ " and " ρ b ", he polar coordnaes of hs pon expressed n he coordnae sysem of he robo n Fg. 2. he varable " θ " represens he angle beween he robo velocy vecor and he vecor Goal-Robo, he varable " ρ b " s he dsance Robo-Goal. The wo oupu varables are he roaon speed robo Vro_AB and V_AB lnear speed ranslaon. Fgure 3. The conroller npu varables «convergence oward he goal». The developed fuzzy conroller s a zero order Sugeno ype, each oupu varable s paroned no hree fuzzy ses. The numercal values of conclusons for he ranslaonal speed expressed n (mm/s): ME=150; Z=0; FA=300; and for he speed of roaon expressed n ( /s): TR=-15, ZE=0, TL=15. The nference able 1, represens he fuzzy rule base for he speed of ranslaon, and he nference able (2) s he bass of he rules of he roaonal speed. Labels are: ME (medum), Z (zero), RA (fas) for he ranslaonal speed, and TR (urn rgh), ZE (zero), TL (urn lef) for he roaonal speed. TABLE I. FUZZY RULE BASE FOR THE TRANSLATIONAL SPEED θ ρ b N ZE P N ME Z ME F FA FA FA Fgure 2. Represenaon of conroller npu varables. B. Defnon of membershp funcons The unverse of dscourses of he npu varable θ and ρ b of he frs fuzzy conroller examned are respecvely decomposed no hree and wo fuzzy ses. Ths srong fuzzy paron s smple and provdes a concse rule-based and easy o nerpre (06 fuzzy rules). The robo-goal angle varable θ s paroned no hree fuzzy subses Fg. 3 : N (Negave), ZE (Zero), P (Posve) whle he varable ρ b s paroned no wo fuzzy subses N (near) and F (far). The oupu varables are V_AB as ranslaonal speed and Vro_AB as roaon speed. TABLE II. θ ρ b FUZZY RULE BASE FOR THE SPEED OF ROTATION N ZE P N TR ZE TL F TR ZE TL

3 C. Resuls of he expermen Fg. 4, s a screen copy showng he pah aked by he robo. The fuzzy conroller wh sx fuzzy rules has gven sasfacory resuls. A Fgure 4. Robo pahs for he arges pons A (2920, 1920) and B (2920, -1920) In he followng we presen a smulaon of several sub goals wh hs same conroller. In hs smulaon we defne sub goals. The robo sars from he nal poson and jons sub goals. We noe ha he robo dd no ake he shores way o go o he second subgoal as llusraed n Fg. 5. Fgure 5. Robo pah for he arge pon (2920, 1920) and (3000,-4000) IV. REINFORCEMENT LEARNING METHOD In hs secon, we use he learnng mehod o develop a conroller of behavour convergence oward he goal. Ths learnng mehod s based on he renforcemen usng fuzzy nference sysem called Fuzzy-acor -crc-learnng [5]. A. Fuzzy acor crc -learnng In he renforcemen learnng paradgm an agen receves a scalar reward value called renforcemen from s envronmen. Ths can be Boolean (rue, false) or fuzzy (bad, far, very good..) ec B A sequence of conrol acons s ofen execued before recevng any nformaon on he whole sequence. Therefore, s dffcul o evaluae he conrbuon of one ndvdual acon. Ths Cred Assgnmen problem has been wdely suded snce he poneerng work of Baro, Suon[1], and Anderson he whole mehodology s called Temporal dfference Mehod [2] and conans a famly of algorhms. Recenly, Wakns proposed a new algorhm of hs famly; Acor crc learnng, hs algorhm, s a form of compeve learnng whch provdes agens wh he capably of learnng o ac opmally by evaluang he consequences of acons. Acor crc-learnng keeps a funcon, whch aemps o esmae he dscouned fuure renforcemen for akng acons from gven saes. Ths funcon s a mappng from sae-acon pars o predced renforcemen. In he fuzzy acor crc learnng, he seleced agen s a SIF of Takag-Sugeno ype of zero order because of s smplcy, s characersc of unversal approxmae, s capacy of generalzaon and s applcaons n real me [6]. Is npu varables membershp funcons are a rangular and rapezodal, and has wo oupus. The SIF s consss of N rules of he form: If suaon hen Y1 s A1 and Y2= [I, 1] wh q[i,1]=0 Y2= [I, 2] wh q[i,2]=0 Crc Y2= [I, j] wh q[i, j]=0 Acor In he algorhm FACL each rule has: A se of dscree acons U dencal for all he rules. A vecor of parameers q ndcang he qualy of he varous dscree acons avalable and used n he defnon of he curren polcy. The premse par s mposed by he operaor (he npu varables membershp funcons are fxed characerscs). The conclusons are chosen by renforcemen learnng algorhm among a whole of dscree acons avalable for each fuzzy rule. B. Descrpon of he execuon procedure In hs learnng mehod he dea consss on usng he marx q o mplemen no only he local polces wh he rules, bu also o represen he evaluaon funcon of he polc-he global -opmal-. One obans hen he FACL, adapaon of AC-Learnng for an apprence of he ype SIF. The execuon on a sep of me can be dvded no sx prncpal sages [3]. The sep of curren me s +1; he apprence execued he eleced acon o he sep of me prevous and receved he prmary educaon renforcemen for he ranson from he sae S o S +1. Afer he deermnaon of he values of ruh of he rules α R, he sx sages are as follows:

4 1- The frs sage of he algorhm consss o deermne he funcon of -opmal evaluaon of he curren sae usng he marx q: T V ( S 1 ) v = φ (1) And updae of he races of elgbly: φ φ γρφ (2) Deermnaon of he error TD: ~ ε + 1 = r γ. V ( S+ 1) V ( S ) (3) 3- Updae he raes of ranng correspondng o he parameers of he crc for all he rules n hree sages: -δ = ~ ε + 1. φ β + k f δ 1. δ > 0, - β + 1 = β (1 ψ ) f δ 1. δ < 0, (4) β else. - δ = 1 ψ ) δ + ψδ ( 1 Where n our case - β s he rae of ranng of he crc for he rule wh he sep of me -δ ψ δ ψ represen he exponenal = ( 1 ) + δ 1 average. 4- The updae of he marx q and he vecorv : v = ~ φ (5) + 1 v + ε + 1. β + 1. q q ~ e T = ε (6) 5- Recalculaon of he evaluaon funcons of he curren sae by he crc, bu hs me wh he newly updaed parameers: T V 1 ( S + 1 ) v = φ (7) Ths value wll be used o calculae he error TD n he nex sep of me. 6- The acon should now be chosen o be appled o he sae S+1.In he case of connuous acons; he crsp acon s deermnaed from he varous acons eleced for each rule: U α U, (8) + 1( S + 1 ) = EleconU ( q + 1 ). R ( S + 1 ), U R A + 1 Where Elecon s defned by: Elecon U ( q + U U 1 1) = ArgMax ( q + ( U ) +η ( U ) + ρ ( U )), Then we updae races of elgbly φ = φ + γλφ + 1., (9) e γλ. e ( U ) = γλ. e + 1( U ( U ) + φ + 1, ( U ), oherwse = U + 1 ), (10) C. Deals of he expermen The lack of he Need for a prelmnary model of he process and negraon experse whch characerze he heursc fuzzy conrol, responds perfecly o our goals [9]. However, hs frs approach has also demonsraed he lms of human experse, n parcular he dffculy of assessng he neracons. Indeed, n he desgn of FIS a choce s carred ou relavely crude and he developmen of a fuzzy conroller can be long and dffcul especally f he number of fuzzy rules s mporan [10]. Therefore, he fuzzy nference sysems desgned from only he human experse ofen have very average performance. Moreover, learnng mehods have heorecally and praccally proven her ably o address hese problems. For hs we use a renforcemen learnng algorhm n order o deermne he conrol of ranslaonal speed and roaon robo speed (he conclusons of he FIS). 1. The apprence The Apprence s a Sugeno fuzzy conroller zero order n whch he conclusons of he ranslaonal speed, he roaon speed and he conclusons of crcal are nally se o zero. The npus of he fuzzy conroller " ρ b " and " θ " are defned respecvely by wo and hree membershp funcons Fg. 3. For he se of acons we consder a se of acons U common o all fuzzy rules, formed of seven values for he speed of roaon (a1= -15( /s), a2 = -10( /s), a3 = -5( /s), a4 = 0( /s), a5 = 5 ( /s), a6= 10( /s), a7 = 15( /s), ) and hree values for he ranslaonal speed (a 1 = 0(mm/s), a 2 = 100(mm/s), a 2 = 300 (mm/s)). The fuzzy conroller can generae connuous acon beween -15 and 15 /s for he speed of roaon, and beween 0 and 300 mm/s for he ranslaonal speed, by nerpolaon of hese dscree acons. The basc rule s as follows: s N) and ( ρb s P) hen (Vro s A1) (Vs B1) (C_ Vro s v1) s N) and ( ρb s L) hen (Vro s A2) (Vs B2) (C_ Vro s v2) s ZE) and ( ρb s P) hen (Vro s A3) ) (Vs B3) (C_ Vro s v3) s ZE) and ( ρb s L) hen (Vro s A4) (Vs B4) (C_ Vro s v4 ) s P) and ( ρb s P) hen (Vro s A5) (Vs B5)(C_ Vro s v5) s P) and ( ρb s L) hen (Vro s A6) (Vs B6) (C_ Vro s v6).

5 2. Renforcemen funcon The acor n our case consss of he par of acons: roaon speed and ranslaon speed. The renforcemen funcon s defned o allow he roaon of he robo n he drecon of he objec pon and drec convergence o he goal. If he robo s far o he goal 1 for ( θ. θ & < 0) 1 for (-1 < θ <+1 ) 0 for ( θ. & θ = 0) -1 else If he robo s near hen he goal Fgure 7. Robo pah for he arge pon (2920, 1920) and (3000,-4000) 10 for ( θ. & θ < 0) & f V=0-1 else 3. Condons of expermenaon The condons of he expermen for he case of he algorhm FACL are as follows: An adapve learnng rae β. A dscoun rae γ (o evaluae he nfluence of he long-erm predcon of he evaluaon funcon) An exploraon and exploaon SP, θ Traces of elgbly: proxmy he crcal facor λ. Table 3 ndcaes he values of parameers of he algorhm FACL ncluded n hese smulaons. Fgure 8. Robo pah for several arges pons. We noe ha afer learnng he robo comes o properly reach sub-goals. Fg. 8, shows rajecores of he robo. TABLE III. TABLE OF USED PARAMETERS. γ λ ' λ ψ ϕ θ Sp Fg. 6, shows rajecores of he robo durng and afer he learnng phase. Afer an mporan exploraon phase, he robo moves owards he arge pon. Fgure 9. Robo pah for he arge pon. Learnng phase Valdaon phase Fgure 6. Robo rajecores before and afer learnng by FACL algorhm «pon goal (2920, 1920)». Fgure 10. Rrenforcemen funcon.

6 Fgure 11. Error TD. V. REAL EXPERIMENTS ON THE PIONEER II ROBOT Usng qualy marx values obaned by he algorhm FACL afer valdaon, we performed a real es on he moble robo Poneer II. Fg. 12, represens he movemens of he robo n an open envronmen. Sarng from an nal poson (0,0), he robo performs a roaon on se n he drecon of an assgned goal (X b, Y b ) = ( ) and moves wh a speed of 150 mm / s. Then he robo begns o decrease s speed once approaches he pon goal and fnally sops on. Fgure 12. Behavor "convergence oward a goal" of he real robo VI. CONCLUSION In hs paper, we have smulaed he behavor of "convergence oward he goal" of a moble robo usng heursc approaches and he renforcemen learnng algorhm FACL. The expermens show ha wh a proper choce of parameers nfluencng learnng (learnng rae, dscoun facor, choce of renforcemen funcon, polcy exploraon and exploaon) he conroller of sx fuzzy rules oban fas convergence owards he goal. These resuls also show he generalzaon capacy offered by hs learnng algorhm unlke o he heurscs mehod of FIS developng. Fnally, he laes resuls show ha FACL algorhm gves sasfacory resuls, however, non-adaped choce of he parameers of hs algorhm resuls degradaon n performance. REFERENCES [1] R. S. Suon, D. Mcalleser, S. Sngh and Y. Mansour, Polcy Graden Mehods for Renforcemen Learnng wh Funcon Approxmaon, In Advances n Neural Informaon Processng Sysems, Volume 12, [2] P. Y. Glorennec, Fuzzy Q-Learnng and Dynamcal Fuzzy Q- Learnng, IEEE, Proc of he Thrd Inernaonal Conference on Fuzzy Sysem, pp , 1994 [3] L. Jouffe and P.Y. Glorennec, Comparson Beween Connecons and Fuzzy Q-Learnng, Proc of Izuka 96, Fourh Inernaonal Conference on Sof compung, Lzuka, Fukuoka, Japan, Sepember, pp , 1996.

7 [4] K. G. Konolge, Saphra Robo Conrol Archecure, Edon SRI Inernaonal, Avrl, 2002 [5] C. Ye, N. H. C. Yung, D. Wang, A Fuzzy Conroller wh Supervsed Learnng Asssed Renforcemen Learnng Algorhm for Obsacle Avodance, IEEE, Transacons on Sysems Man and Cybernecs -Par B Cybernecs, vol.33, NO.1, pp.17-27, February [6] M,Zucker. and J. A,Bagnell. Renforcemen plannng: RL for opmal planners. In IEEE Inernaonal Conference on Robocs and Auomaon (ICRA) [7] R. S, Suon and A.Koop, and D,Slver., he role of rackng n saonary envronmens, In Inernaonal Conference on Machne Learnng (ICML) [8] C,Lakhmss and M,Boumehraz : Desgnng of Goal Seekng and Obsacle Avodance Behavors for a Moble Robo Usng fuzzy Technques,J. Auomaon & Sysems Engneerng : [9] L. M. Zamsen, A. A. Arroyo, E. M. Schwarz, S. Keen, B. C. Suon, and G. Gandh, "Koolo : Pah Plannng usng Renforcemen Learnng on a Real Robo Plaform FCRAR, 19h Florda Conference on Recen Advances n Robocs, Mam, Florda, May 25-26, [10] C,Lakhmss and M,Boumehraz: Usng Q-Learnng and Fuzzy Q-Learnng Algorhms for Moble Robo Navgaon n Unknown Envronmen, Inernaonal Journal of Scence and Engneerng Invesgaons. vol. 1, ssue 2, March 2012.