Connectionist Classifier System Based on Accuracy in Autonomous Agent Control

Transcription

1 Connecionis Classifier Syse Based on Accuracy in Auonoous Agen Conrol A S Vasilyev Decision Suppor Syses Group Riga Technical Universiy /4 Meza sree Riga LV-48 Lavia E-ail: serven@apollolv Absrac In his paper a new connecionis classifier syse based on accuracy is proposed which uses a layer of copeiive arificial neurons for decision-aking New algorihs of his syse learning in uli-sep probles are developed where he correc resul becoes known only a a cerain syse s sep The connecionis syse is eployed o conrol an auonoous agen in discree environens Inroducion Ipeuous developen of arificial neural neworks akes i possible o ransfer any ideas fro his area ino adjacen areas Furher we will ry o invesigae an opporuniy of apping learning classifier syses (LCS) [] ino arificial neural neworks (ANN) and o propose learning algorihs for a hybrid connecionis classifier syse (CLCS) [2] inended o solve uli-sep probles An LCS is a self-learning syse based on synacically siple rules (classifiers) Nuerous classifiers can be acivaed a he sae ie herefore he inforaion is being processed in parallel Each classifier has is srengh ha shows is curren usefulness in he syse The srengh varies as experience is accuulaed 2 Srucure The opology of a hybrid connecionis classifier syse is based on he srucure proposed in [3] assuing ha binary essages A sg = { } in he LCS are replaced by bipolar essages A sg = {- +} ie corresponds o - Due o his assupion a LCS can be represened as a copeiive wo-layer arificial neural nework Each classifier is represened by a neuron in he hidden layer Each bi of he inpu essage is associaed wih an inpu eleen of he nework Oupu layer neurons are associaed wih he oupu effecors (acions) Weighs are bipolar (eiher + or -) Connecions beween inpus and neurons of he hidden layer wih weigh + correspond o in he classifier condiion Connecions beween inpus and neurons of he hidden layer wih weigh - correspond o in A cond The absence of connecions corresponds o # (don care)

2 3 Learning As i was arked above CLCS consiss of wo layers of neurons: he hidden layer and he oupu layer These layers are copeiive and are updaed by using reinforceen learning There are wo ypes of reinforceen signal in CLCS: r and < r2 < The signal r oves o he hidden layer and r 2 - o he oupu one r r 2 a x x a 2 x n w nj w j - a Inpus x n ={-+} Hidden layer Each node represens a classifier w nj ={+-} f(ne)={} Oupu layer f(ne)=(-) Oupus Fig Connecionis learning classifier syse One of he basic coponens of he CLCS oupu layer is he learning rae c Is adapabiliy iproves a learning process A he error increasing he learning rae c increases by a pre-se facor γ incr ie c= γ incrc A he error decreasing c decreases by a pre-se facor γ decr ie c= γ c decr 3 Hidden layer learning In os cases arificial neural nework learning is based on he learning wih a eacher (supervised) and wihou a eacher (unsupervised) In urn LCS uses reinforceen learning approach Unlike ANN where he weighs are updaed during learning he learning of he CLCS hidden layer consiss in updaing classifiers srenghs Only acivaed classifiers srenghs are updaed The hidden layer learning is carried ou by any reinforceen learning algorihs ie bucke brigade or Q-Learning 32 Accuulaion oupu layer learning For he uli-sep proble he reward r becoes known only a ie l herefore in his ease i is beer o use he reinforceen ype of learning for exaple a eporal-difference (TD)

3 learning However TD learning is used for predicion of reward r and can be applied for a criic learning However our ask is o each he conroller For solving his proble we will describe a new approach for learning copeiive ANNs I uses he conceps of winner akes all and eporal difference learning The learning is based on he preise ha one of he neurons in he layer has he axiu response o inpu x This neuron is declared he winner w x = ax ( w x) i= 2 p i As a resul of his winning even a sep k he weigh vecor appears w = [ w w w ] 2 n k k ou = f ( w x) where f is a coninuously differeniable funcion The gradien is: ou = f ( w x) x Weighs are updaed for he winning neuron only as follows: w w w + s l k= c ( l k ) λ wou where s = sgn(r) is he coefficien (eiher - or +) which shows negaive or posiive reinforceen; c is he learning rae < c ; λ is he absolue value of he reinforceen signal r λ = r < λ When λ weighs are no updaed ie w ; for λ = he weigh changes a every sep are idenical A he beginning all won neurons and inpu vecors x are sored The weigh updaing occurs when he reward r a ie l has been obained The closer he reward (he sep l) he ore updaes in weighs are The accuulaion algorih of he oupu layer learning looks as follows:

4 Iniialize weigh vecors wi i = 2 p wih rando values () 2 While no end do (Main cycle) 2 For k = l do (Save each inpu vecor x and index of won neuron ) xk x wx= ax ( wx) i= 2 p k 22 For i = l do (Updae epy able of weighs when reinforceen r becoes known) = scλ f w x x ( ) ( l i) i i i i 23 For i = l do (Updae weighs w using able ) w = i i i As here is daa accuulaion and changes occur only when he signal r is known a large eory space for soring eporal variables is necessary There are required inpus l eleens for soring all inpus x during a sequence l This accuulaion is possible only for raher shor sequences This disadvanage can be bypassed by using bucke brigade learning 32 Bucke Brigade oupu layer learning This ype of learning does no ake any accuulaion The winner gives a par of is weighs o he previous won neuron The las neuron in he sequence l receives a reinforceen signal fro he environen < r < A neuron in he oupu layer has he axiu oupu signal This is declared he winner w x = ax ( w x) i= 2 p i The weigh vecor of he won neuron a curren sep k is w = [ w w w ] k 2 n k The arix A of diension p p consiss of eleens a = i j and a = x i = j ij ij i The ieraive learning rule reads: d = cw Α k k

5 w = d w = d x + k k w k where c is he learning rae < c ; d is he vecor of changes of he won neuron; d is he nor of vecor (scalar) k Weighs change of he las winner (acivaed) a sep l is: w = rx l l The bucke brigade algorih of he oupu neural layer learning is as follows: Iniialize weigh vecors wi i = 2 p wih rando values () 2 While no end do (Main cycle) 2 For k = l do (Updae weighs w of he previous k and curren k won neurons) w x = ax ( w x) k k i k i= 2 p d = cw Α k w = d k w = d x k k + w k 22 (Now we have reinforceen signal r updae las won neuron) w = rx l l k This learning is siilar o he classical classifier syses learning algorih excep ha here are no axes In he case of success no only he las acivaed neuron is encouraged bu also he previous acivaed neurons obain soe fracion of he reinforceen signal Noe ha he learning facor c is siilar o he bid aucion facor c bid in he classical bucke brigade algorih 4 Geneic algorihs and covering procedure Since each classifier is conneced o each acion and has he paraeers showing srenghs of hese connecions (weighs) i is necessary o change he weighs of a classifier a is odificaion or replaceen by a new one For his purpose we will ake he following addiions o he procedures of geneic algorihs (GA) and covering: Selecion operaor insead of weakly adaped classifier k chooses a ore adaped one i fro populaion P All weighs of he new classifier are copied insead of old weighs

6 -2 w = w for j = 2 i j k j Crossover operaor crosses condiional pars of classifiers i and k Weighs of classifiers are heir averaged weighs wi j+ wk j wi j= wk j= for j = 2 2 Covering procedure creaes a new classifier which aches he given inpu essage insead of he weak one Weighs of he new classifier are iniialized randoly: wi j= rand for j = 2 5 Connecionis classifier syse XCS Currenly here are various LCS ypes (eg XCS [4]) For exaple XCS can be easily apped ino CXCS (Fig 2) Fig 2 Connecionis XCS For XCS ransforaion ino a connecionis classifier syse i is necessary o subsiue a se of acivaed copeiive neurons for acivaion se H Consider he behaviour of an agen (Ania) which operaes in he discree environens Maze5 and Maze (Fig 3) using a classifier syse CXCS The ain objecive of he agen is o collec food (F) by aking he leas possible nuber of seps for i (5 seps axiu) The agen can ove in eigh direcions o he non-occupied posiions The agen has sensors by eans of which i obains he vecors of local percepions An agen s iniial posiion every ie is chosen randoly

7 Fig 3 Markov environen Maze5 (a) and non-markov Maze (b) All he graphs given below are averaged over runs The paraeers used in ACS and XCS are defaul [4 5] Addiional paraeers for CXCS used in he experiens: ( r ) = - a posiive reward (food is found) for he hidden layer received fro he pos 2 environen; ( r ) = 6 - a negaive reward (food is no found) for he hidden layer received fro he neg 2 environen; c = 6 - hidden layer s learning rae; γ = 5 - increase facor of he learning rae c; incr γ = 7 - decrease facor of he learning rae c; decr c = - iniu learning rae c; in c = 9 - axiu learning rae c ax Le us copare he perforance of a connecionis syse XCS (CXCS) (see Fig2) wih siple XCS and ACS in he Ania ask Seps o food XCS CXCS ACS Opiu Trials Fig 4 Classifier syses perforance in Markov environen Maze5 Populaion size ACS size XCS size CXCS size Trials Fig 5 Variaion of classifiers populaion size in Maze5 The CXCS shows uch beer resuls in Maze5 in coparing wih he XCS However as well as XCS i canno find an opiu For exaple a he 2-h rial (Fig 4) a CXCS finds he goal objec (food) wihin 2 seps on he average while an XCS finds i in 32 seps The bes resuls are achieved by an ACS The populaion size of ACS XCS and CXCS classifier syses is ie-varian (Fig 5)

8 Seps o food XCS CXCS ACS Opiu Trials Fig 6 Classifier syses perforance in non- Markov environen Maze Populaion size ACS size XCS size CXCS size Trials Fig 7 Variaion of classifier populaion size in Maze Learning classifier syses used in he experiens do no possess eory [6] and acion chunking procedure Tha is why in non-markov environen Maze (Fig 6 Fig 7) hey canno find an opiu since for navigaion o be efficien i is no enough o have only local percepion inforaion 7 Conclusion In his paper he possibiliy of using copeiive neural neworks in classifier syses was sudied Two ehods of CXCS learning have been developed ha are based on reinforceen learning: accuulaion and bucke brigade ype These copeiive ANN learning ehods can also be applied independenly of he CXCS o he probles where he exac sequence of acions is no known however i is known wheher his sequence resuls in a posiive reward or no The ain CXCS advanage is ha during he learning process no only classifiers srenghs are updaed bu also acions are adjused In he case of unsuccessful acions he values of he paraeers responsible for acivaion of hese acions decrease This increases he probabiliy of choosing anoher acion for he sae siuaion As a resul he nuber of classifiers necessary for CXCS is less han ha for LCS Apar fro ha CXCS is less dependen on he geneic algorih procedure A coparison of siple classifier syses and connecionis classifier syses was carried ou in he agen navigaion ask (Ania) in Markov and non-markov environens However in soe cases he CXCS does no show an opial perforance References [] J H Holland Adapaion in Naural and Arificial Syses Ann Arbor: The Universiy of Michigan Press 975

9 [2] A S Vasilyev Synergeic Approach in Adapive Syses Maser s Thesis Transpor and Telecounicaion Insiue Riga Lavia supervised by Prof Borisov AN 22 [3] R E Sih & H B Gribbs Is a classifier syse a ype of neural nework? Evoluionary Copuaion 2() pp [4] 4 Wilson SW Classifier based on accuracy Evoluion Copuaion 3(2) pp [5] W Solzann An inroducion o anicipaory classifier syses PL Lanzi W Solzann and SW Wilson (Eds): LCS 99 LNAI 83 pp [6] 6 Lanzi P L Adding Meory o XCS In Proceedings of he IEEE Conference on Evoluionary Copuaion (ICES98) IEEE Press 998