An Intelligent System for Parking Trailer using Reinforcement Learning and Type 2 fuzzy Logic

Transcription

1 Jourl of Elecrcl d Corol Egeerg A Iellge Sysem for Prg Trler usg eforceme Lerg d Type 2 fuzzy Logc Morez Shrf, 2 Mohmmd ez Alm, 3 Seyed Abolfzl Foor,2,3 Deprme of Elecrcl Egeerg,Islmc Azd Uversy obd brch morezshrf@gmlcom,rezs68@yhoocom,foor@yhoocm Absrc: I exmples of reforceme lerg where se spce s couous, seems mpossble o use referece bles o sore vlue-co I hese problems mehod s requred for vlue esmo for ech se-co pr The pus o hs esmo sysem re chrcerscs of se vrbles whch reflec he sus of ge he evrome The sysem c be eher ler of oler For ech member se of cos of ge, here exss esmo sysem whch deermes se vlue for he co O he oher hd, mos rel world problems, us s he se spce s couous, so s he co spce for ge I hese cses, ype 2 ype 2 fuzzy sysems my provde useful soluo seleco of fl co from co spce I hs pper we ed o combe reforceme lerg lgorhm wh fuzzfed cos d se spce log wh ler esmo sysem o ellge sysems for prg Trlers cses where boh se d co spces re couous Flly, he successful performce of he proposed lgorhm s show hrough smulos o rler prg problem Keyword: eforceme Lerg; Type 2 ype 2 fuzzy Sysems; Trler Prg Problem; SASA Algorhm I BINTODUCTION Lerg s he bly o mprove behvor bsed o prevous expereces d observos Mche lerg ws proposed rfcl ellgece order o cree lerer mches d herefore hgher levels of flexbly d ellgece If equpped wh lerg ools, mche c cosly mprove s performce d crese s effcecy eforceme lerg s wde re of reserch mche lerg [-22] I reforceme lerg, ge emps o mprove s behvor bsed o rl d error d usg s expereces Ages c observe evrome chrcerscs whch form he se spce for lerer The, durg ech ervl, ge flueces he evrome hrough some operos Therefore, he ex ervl, dffere pus re fed o he ge bsed o her prevous cos I ddo o hese ew pus, ech ge receves reforceme sgl clled rewrd whch shows he ppropreess of he prevous co ewrd c ssume posve or egve vlues regrdg he ppropreess of he prevous co I lmed dscree spces, vlue of ech co for ech se s sored referece ble clled Q Tble [, 2, 5] Ech row of hs ble represes se whle ech colum correspods o co Age mes decsos bsed o hs ble d s polces However, lrger d more complex evrome where se spce s couous, s vrully mpossble o use referece bles The problem becomes more serous whe co se s lso couous d o coed o lmed umber of cos Oe wy o geerlze ses couous spce d o produce couous ses of cos s o employ ype 2 ype 2 fuzzy ferece sysems [23] I ype 2 ype 2 fuzzy evromes, umber of membershp fucos or ype 2 ype 2 fuzzy ses re defed over he rge of ech vrble; ech vrble s he descrbed ccordg o s membershp o y of hese ype 2 ype 2 fuzzy ses [6, 7, 8, 9] Type 2 ype 2 fuzzy rules llow he sysem o perform hum-le ferece uder ucery [23] I hs pper, we emp o employ SASA lgorhm well-ow lgorhm reforceme lerg [] combed wh ype 2 fuzzy logc for cses where boh se d co spces re couous The, we ry o mprove sysem effcecy usg esmo sysems for vlue-co fucos Flly, he sysem s ppled o ellge Trler prg problem The problem s defed s desgg uomc coroller for prg Trler wh forwrd d bcwrd movemes [24] Here, we ssume h Trler drver s o exper d hs/her owledge c o be used corol d smulos; rher, he ellge ge should ler how o pr he Trler bsed o rl d error The pper s orgzed s follows Seco II descrbes he proposed lgorhm; Trler prg problem s roduced Seco III where he proposed lgorhm s used o fd soluo o he problem; Seco IV provdes smulo resuls; d flly, he pper cocludes wh Seco V d some suggesos II BTHE POPOSED ALOITHM I hs chper we roduce ype 2 fuzzy Se-Aco-ewrd-Se-Aco SASA lgorhm wh ler vlue esmo Frs, we defe ype 2 fuzzy ses he problem evrome d over rge of cos for ges I hs cse, ech rule s reled o rego se spce correspodg o co he co se Ech co he rule hs vlue Q whch s used by he ge o selec prculr co I oher words, he ble Q s exeded wy h ech row pos o se of ses sed of oly oe se; s well, ech colum represe s se of cos These ses re ype 2 fuzzy ses whch cover he whole se d co spce Ths leds o defo of oe eleme s Q ~ s, ~ where boh s~ d ~ re ype 2 fuzzy ses Vol 3 No 5, 23 PP 3-9 wwwoeceorg/ Amerc V-Kg Scefc Publshg 3

2 Jourl of Elecrcl d Corol Egeerg As meoed erler bou reforceme lerg lgorhms, hs eleme represes spce-co vlue whch s esmed by ler esmor he proposed lgorhm usg weghed combo of se vrbles Ech row of he ble s form of ype 2 fuzzy rule f x 2 Ad: m s A d x2 s A2 d d x s s B wh w w x w x or 2 s B wh w w x w x or m s B wh w w x w x A he Q S, B = w w x w x, S =< A, A,, A > 2 Where <x, x2,, x > re se vrbles; membershp fuco for he h vrble he se spce he h rule; d B m s he membershp fuco for he m h co Furhermore, he eleme Q S, B correspods o he se S d he co B whch re boh ype 2 fuzzy ses Accordg o wh we hve so fr, he umber of rules equls he mulplco of umber of membershp fucos for ll ses For rel vlues, d for he pu vecor x = x,, x T he oupu y he Tg-Sugeo ype 2 fuzzy sysem usg Mmd mulplco ferece ege for rules wll be s follows : Where Φ s y = = Φ x y x = Φ x Φ x = µ x = Where μ refers o he h membershp fuco he h rule y x s he ceer of he seleced se of cos he h rule somemes referred o s h locl co I he proposed lgorhm se-co vlue s deermed s follows : Q S, B = w w x w x As meoed before, B represes he membershp fuco for he co seleced bsed o he ge s polces he h rule Illy, ll w s re zero Suppose h he ge performs d co d es he se x Accordg o x s membershp fuco d ccordg o he co seleced bsed o he rule d polces locl co, deermes he bsc co seleced by he ge Equo 4 preses how he bsc co s chose: = φ x Where s he bsc co he sep d represes he locl co he sep chose ccordg o he ge s A s polcy he rule I ddo, φ x s bsc ype 2 fuzzy fuco defed s below: φ x = Φ x = Φ x Q x, Whch shows he vlue of he se x for he co s clculed s follows : Q x, = Q s, φ x Accordg o SASA, he vlue of ech se mus be upded ow The upde equos re T rge Q s,, ~ = Q s αε φ x ε = r γq x, Q x, Now Q s, mus be upded for =,2,, For dog so, MSE error s defed s e s, = rge Q s, Q s, 2 E s, = 5e s, Now, we c upde Q s, prmeers bsed o hs error d usg opmzo mehods Such opmzo lgorhm my be of seepes desce ype, LSE, or ellge opmzo mehods The lgorhm wll be descrbed dels he followg secos For ech vrble volved creg se, severl ype 2 fuzzy ses re defed over s rge The, premses for ype 2 fuzzy rules re creed bsed o hese ype 2 fuzzy ses The wegh vecor w s he gve l vlue s he umber of rules d represes he umber of se vrble I s cler h for m cos here wll be m vlue of w The followg seps re repeed for ech epsode : = d he process begs wh he l se x Accordg o he ge s polces, locl co s seleced ech rule The, he bsc co s seleced ccordg o 4 For he sep he epsode do he followg: 2- Perform he co ; go o he se x ;d receve he rewrd r 2-2 -Use 6 o deerme he se-co vlues Q x, d Q x, for dog so, you eed whch s deermed ccordg o he ge s polces 2-3 Use 7 o deerme he rge for ech co seleced ech rule 2-4 derve error fuco form of 8 for ech co seleced ech rule d upde w usg opmzo lgorhm 2-5 x = x, = Vol 3 No 5, 23 PP 3-9 wwwoeceorg/ Amerc V-Kg Scefc Publshg 4

3 Jourl of Elecrcl d Corol Egeerg Ed, f x s fl se; oherwse = d go o 2 [25] Fuzzy Srs Lerg Fuzzy Srs lerg FSL s Fuzzy mehod bsed o Srs lerg [], for evromes wh couous opero d se Srs mehods show he vlue of he c of se of s s Q s, d upded formul s s Q s, Q s, α [ r Q s, Q s, ] γ Whereα s he lerg re γ forgeg re d r s he wrd whch he evrome fer he c of gve o fcor se of s A zero-order TSK Fuzzy sysem wh rules hve he followg form [3]: f x s L d d x s L : The wh vlue w or or wh vlue w s= x ** x s -dmesol pu vecor L = L ** L re se of Ipu Fuzzy membershp fucos, m s dscree se of cos for ech rule, d w re he -h Fuco cdde d Approxme vlues for he -h rule, respecvelyfsl ms o regule w ole o ge he bes polcy The possbly of choosg s obed he -h co -h rule se of s bsed o he polcy of sof mx [ 3] exp µ s w / T p = m exp µ s w / T = 2 µ s Is Fre esy ormlzed of -h rule se of s d T> s he emperure fcor The cos chose by ech rule d s vlue wh w d re show, respecvely, d her vlues 2 d 3 s clculed s [, 9]: s µ s = s, = s w = = 3 Q ˆ µ 4 Therefore he ol weghed dscree fuco s seleced by he rules Performg he c of goes he evrome o ex se of s d he fcor e he wrd sgl r The ex co clculed bsed o curre weghed w Thus The -h ule weghs upded by followg formul [ 3] : whch w ˆ, α Q s µ s f = = oherwse Q Is he error fuco h obed by 6: Qˆ s, = r γ Qˆ, ˆ s Q s, 2 - FSL mul-ge MAFSL 5 6 I hs pper we follow he commo prcce for operg mul-ge sysems The m des e from [3] d [4] Cosderg h he mxmum vlue of he fuco mode cse s he sme he followg V s = mxqs, upded formul for V d Q ses s follow: Q s, = µ wq s * = Wq = Q s, α * r δ = V s Q s, * s µ V s Wv = V r δv s = µ * = Wv = α * _ * V s µ s 8 I ule Number, Number of operg, Oe of he possble cs, represes he me sep, Wv Iferor o he rules of sysem reled o V d Wq Iferor o he rules of re chose sysem reled o Q, r mou of wrd d c of rule-h Lower vlues of V d Q ech hve sepre phse Ipu sysem s s follows Ipu of fuzzy sysems re se-spce dmesos I he rg phse, he rules re se lower rules rezero-order TSK-ype Fgure : fuzzy Membershp fucos The rules of he sysem mulpled by he umber of pu membershp fucos Ech represes fuzzy rule Q feror o he rules relg o sysems cog he eger vlues correspodg o ech fuzzy se mples h 7 Vol 3 No 5, 23 PP 3-9 wwwoeceorg/ Amerc V-Kg Scefc Publshg 5

4 Jourl of Elecrcl d Corol Egeerg he wegh of ech opero Ech se of operg rules d s depede of oher fcors Iferor o he rules of he sysem V of egers, ech of whch correspods o he vlues of he fuzzy rule s expressed By prog Mode d clculg he fuzzy mode, fuzzy Q d V for he sme poso erely solve he problem of curse of dmeso d he umber of dmesos eve Compred o [4] re lso sgfcly Oe of he ssues h hve sgfc mpc decresg szes, he proper defo of he se I he ex seco we wll elbore o hs The followg ble shows he pseudo-code descrpo of he lgorhm s gve Ths ble shows Pseudo-code lgorhm MAFSL For y ge we hve sme co: Ilze Q d V by Zero 2loop ule Q vlues coverge 3observe se S 4selec co from ech rule by usg 2 5choose fl co by 6 equo 6observe S2, rewrd d oher ges cos 7upde Q d V use 7 d 8 equos 8ed loop The rler s corolled by chgg θ Oly bcwrd moveme s llowed here I ech sep, he rler moves We ssume h suffce spce s prese bewee he rler d prg spo, d herefore, he vercl poso y s o requred s se vrble for our purpose Φ Φ c Φ Φc < * Φ Φ Φ c Φ Φ > Fgure 2 eloshp bewee ruc d rler gles Problem cosrs: The rler hs cos velocy of V The legh of he rler s L c L s TABLE I L c Fgure 3 cosrs of he rler prg problem Ls IV 3BDEFININ THE TYPE 2 FUZZY SETS TYPE 2 FUZZY ULES DEFINED FO THE CONTOL POBLEM v A Problem Defo III 2BTAILE PAKIN POBLEM Desgg opml phwy for bcwrd moveme of rler hrough umber of fxed d movg obscles s mog mos complced problems egeerg Fcors such s ype, shpe, d re of moveme s well s me lmos for chevg he rge doc my roduce furher complco o he problem Bcwrd moveme of rler o doc s oler corol problem Usg he coveol corol mehods, mhemcl model for he sysem c be obed, d he oler corol heory my be employed o desg of he coroller A lerve o hs s o desg coroller whch smules hum behvor The ler s used he prese pper We ssume h experece rler drver s vlble; ddo, we c mesure dffere posos of he rler d correspodg drver s cos o move he rler bcwrd Fgure shows he rler d he lodg prg doc x, y φ Fgure se vrbles volved ellge bcwrd moveme of he rler φ c θ Vol 3 No 5, 23 PP 3-9 wwwoeceorg/ Amerc V-Kg Scefc Publshg 6 Se Corol Cosrs Equos of Moo Prmeers Φ Dels of Pl x, y Φ Φ c θ x > Φ s 9 7 θ 7 A = r cos θ B = A cos Φc[ ] Φ x[ ] = x[ ] B y[ ] = y[ ] B Coordes of he ceer rer of he rler Agle of rler wh x-xs Agle of he cb wh x-xs Seerg gle of he fro wheels relve o cb oreo The lodg doc s x = The gle bewee he cb d rler c' exceed 9 Lm of seerg of he fro wheel [ ] cos Φ[ ] s Φ[ ] r s θ Φc[ ] = Φc[ ] rcs Ls Lc A s Φc[ ] Φ[ ] Φ[ ] = Φ[ ] rcs Ls Is he dused o respec Φc[ ] he cosr o Φ Φ s dsce fro wheel moves r 3m per me sep legh of he rler, from L s 4m rer o pvo legh of he cb, from L c 6m pvo o xle

5 Jourl of Elecrcl d Corol Egeerg Fgures 5 hrough 8 show he membershp fucos defed MATLAB: I hs problem, ech se s represeed by < x,α, β > To fuzzfy he ses, fve ype 2 fuzzy ses re defed for x whle seve ype 2 fuzzy ses re defed forα d hree ype 2 fuzzy ses re defed for β Seve ype 2 fuzzy ses re cosdered for seer wheel gle s he co Fgure 4 membershp fucos for poso of he rler Fgure 5 membershp fucos for he gle α = Φ degrees Fgure 6 membershp fucos for he gle β = Φ c Fgure 7 membershp fucos for he gle θ A 6BDeermg sr pos for epsodes Sr pos mus be uformly dsrbued over se spce To cheve hs, x s vred bewee d wh he sep ; hs resuls vlues for x For ech vlue of x, α vres bewee -9 d 27 wh he sep ; hs produces 36 vlues forα d β vres bewee -9 d 9 wh he sep ; hs produces 8 vlues for β Therefore, here re 648 sr pos for epsodes B 7BAssgme of rewrds For rewrdg, wo rges re cosdered for he vrbles x d gle The llowble rge lly cludes he whole rge e[, ] for x, [-9, 27]d [-9,9] for gles The, he rges re reduced owrd he rge proporol o he crese umber of epsodes If he ge goes o predeermed rge fer performg co, wll receve posve rewrd whch s proporol o he sze of hs rge or he epsode umber; f he ge ex he rge, however, receves he sme rewrd bu s egve mou The mxmum rewrd s h of he fl epsode, whle he mmum rewrd s - There re wo excepos rewrdg ech epsode echg he rge resuls he grd prze e Exg he llowed rge brgs - pushme If he ge goes o he deermed rge s resul of co, he epsode wll ed The procedure coues ul he ge goes o he deermed rge or ex he llowble rge Fgure 8 membershp fucos for he dreco ype 2 fuzzy ses for ses d cos; vrble x se spce; vrble α se spce; vrble β se spce d seer wheel gle θ co 3-2 Trler prg problem s reforceme lerg problem The followgs re he mpor oes o cosder he proposed lgorhm Fuzzfco of cos d ses V 4BDETEMININ THE PAAMETES AND SIMULATION ESULTS I smulos, seleced vlues for α, γ, d rg re seepes desce re 5, 9, d respecvely Mxmum umber of cos ech epsode s Tble I shows me vlues for 5 rus wh dffere umber of epsodes The vrce vlues re lmos zero The frs colum shows he umber of epsodes, whle he secod colum lss umber of ses whch resuled o fl se s percege of 72 ses For esg, we creed rdom ses he llowble rge d es hem usg he Q ble The resuls re show he hrd colum There re umber of ses he se spce whch do o resul rge se d produce resuls ousde he llowble rge hrough umber of cos The forh d he ffh colum show he smulo resuls fer elmo of hese ses smll olerce s ccepble rechg he rge; Δφ=5, Δx=5 Vol 3 No 5, 23 PP 3-9 wwwoeceorg/ Amerc V-Kg Scefc Publshg 7

6 Jourl of Elecrcl d Corol Egeerg TABLE II PECENTAE OF STATES, OUT OF TOTAL TEST STATES, ESULTIN IN TAET STATE 9 8 Number of epsode VI 5BCONCLUSION To es he proposed vgo mehod, smulo s performed usg MATLAB dffere codos for he poso of he rler d movg d fxed obscles We red o mprove he rler moveme hrough chgg or weghg he ype 2 fuzzy rules Due o he low ose he slled sesors comprso o he mmum rge of corol pus mxmum of o 2 cm compred o m, here s o eed o cosder he mesureme ose whle modelg he coroller I he followg les, he moveme of he rler whle fcg obscles movg wh cos velocy s revewed The sg * s used o deoe he moveme d speed of he rler The spce bewee he srs represes he chges he rler speed The proposed lgorhm s ppled for suo cossg of wo fxed obscles d oe movg wh cos speed however, here s o lmo o he umber of fxed d movg obscles The lgorhm my be ppled for greer umber of obscles movg wh dffere veloces As c be see, he ype 2 fuzzy coroller operes well vrey of codos The followg shows he progrm resuls for dffere codos Number of movg d fxed c be cresed he smulo Fgure 9 shows he resuls obed from MATLAB smulo I hs pper, we proposed mehod for lerg complex evromes where boh ses d co spces re couous The lgorhm frs fuzzfes he vlue-co ble; he he desred co s deermed bsed o he se of he pu d ype 2 fuzzy ferece; flly, ype 2 fuzzy SAA d seepes desce re used o upde weghs he ble A mpor feure of hs lgorhm s dpbly of s ype 2 fuzzy ble whch resuls hgh effcecy s see smulo resuls I ddo, he lgorhm does o eed o ler whch decso he decso mg process s he opmum decso; rher opmzes he decso mg hrough rl d error Ths mes he sysem depede of exper owledge for decso mg complex sysems Flly, he proposed lgorhm ws used for fdg soluo o Trler prg problem usg reforceme lerg for he frs me A mpor ssue solvg he problem s how o rewrd cos whch ws ddressed by heursc process I he fl seco, we preseed smulo resuls o prove he proposed mehod successful Fgure 9 Trecores deermed for dffere l ses EFEENCES [] Suo d A Bro, eforceme Lerg: A Iroduco Cmbrdge, MA: MIT Press, 998 [2] L P Kelblg, M L Lm d A W Moore, eforceme lerg: A survey, J Arf Iell es, vol 4, pp , 996 [3] D P Berses d J N Tssls, Neuro-Dymc Progrmmg Belmo, MA: Ahe Scefc, 996 [4] Suo, Lerg o predc by he mehods of emporl dfferece, Mch Ler, vol 3, o, pp 9 44, Aug 988 [5] C Ws d P Dy, Q-lerg, Mch Ler, vol 8, o 3/4, pp , 992 [6] D Vegerov, N Bmbos, d H Bere, A ype 2 fuzzy reforceme lerg pproch o power corol wreless rsmers, IEEE TrsSys, M, Cyber B, Cyber, vol 35, o 4, pp , Aug 25 [7] H Beom d H S Cho, A sesor-bsed vgo for moble robo usg ype 2 fuzzy logc d reforceme lerg, IEEE Trs Sys,M, Cyber, vol 25, o 3, pp , Mr 995 [8] C I Coolly, Hrmoc fucos d collso probbles, I Job es, vol 6, o 4, pp , Aug 997 [9] W D Smr d L P Kelblg, Effecve reforceme lerg for moble robos, Proc IEEE I Cof obo Auom, 22,pp [] T Kodo d K Io, A reforceme lerg wh evoluory se recrume sregy for uoomous moble robos corol, oboauo Sys, vol 46, o 2, pp 24, Feb 24 [] M Werg d J Schmdhuber, HQ-lerg, Adp Behv, vol 6, o 2, pp , 997 [2] A Bro d S Mhev, ece dvces herrchcl reforceme lerg, Dscre Eve Dy Sys: Theory Appl, vol 3, o4, pp 4 77, Oc 23 [3] Suo, D Precup, d S Sgh, Bewee MDPs d sem-mdps: A frmewor for emporl bsrco reforceme lerg, ArfIell, vol 2, o, pp 8 2, Aug 999 Vol 3 No 5, 23 PP 3-9 wwwoeceorg/ Amerc V-Kg Scefc Publshg 8

7 Jourl of Elecrcl d Corol Egeerg [4] T Deerch, Herrchcl reforceme lerg wh he MAXQ vlue fuco decomposo, J Arf Iell es, vol 3, pp , 2 [5] Theochrous, Herrchcl lerg d plg prlly observble Mrov decso processes, PhD dssero, Mchg Se Uv, Es Lsg, MI, 22 [6] A J Smh, Applcos of he self-orgsg mp o reforceme lerg, Neurl New, vol 5, o 8/9, pp 7 24, Oc 22 [7] P Y loreec d L Jouffe, Type 2 fuzzy Q-lerg, Proc 6h IEEE I Cof Type 2 fuzzy Sys, 997, pp [8] S Tzfess d gos, Type 2 fuzzy reforceme lerg corol for complce ss of roboc mpulors, IEEE Trs Sys,M, Cyber B, Cyber, vol 32, o, pp 7 3, Feb 22 [9] M J Er d C Deg, Ole ug of ype 2 fuzzy ferece sysems usg dymc ype 2 fuzzy Q-lerg, IEEE Trs Sys, M, Cyber B,Cyber, vol 34, o 3, pp , Ju 24 [2] C L Che, H X L, d D Y Dog, Hybrd corol for obo Nvgo: A herrchcl Q-lerg lgorhm, IEEE obo Auom Mg, vol 5, o 2, pp 37 47, Ju 28 [2] S Wheso d P Soe, Evoluory fuco pproxmo for reforceme lerg, J Mch Ler es, vol 7, pp , Dec26 [22] M Ky d Alh, A ovel pproch o mulge reforceme lerg: Ulzg OLAP mg he lerg process, IEEE TrsSys, M, Cyber C, Appl ev, vol 35, o 4, pp , Nov25 [23] J S Jg, C T Su d E Mzu, Neuro-Type 2 fuzzy d Sof Compug, Eglewood Clffs, NJ, Prece-Hll, 997 [24] Wg, Le-X, A Course ype 2 fuzzy sysem d corol, Prece Hll PT,pp57-6,997 [25]Zhr Moe, Vhd seyyed ghomshe, Mohmmd Teshe lb, h Ir Type 2 fuzzy Sysem Coferece pp July Vol 3 No 5, 23 PP 3-9 wwwoeceorg/ Amerc V-Kg Scefc Publshg 9