New conditioning model for robots

Transcription

1 ESANN 011 proceedngs, European Symposum on Arfcal Neural Newors, Compuaonal Inellgence and Machne Learnng. Bruges (Belgum), 7-9 Aprl 011, 6doc.com publ., ISBN Avalable from hp:// New condonng model for robos Jean Marc Salo IMS laboraory, ENSC-IPB, Bordeaux Unversy 146 Rue Léo Sagna Bordeaux Cedex France Absrac. We presen a neural newor for he predcon of rewards n a condonng model. I s based on wo nosy-or and one nosy-and nodes and updae rules nspred from BANNER echnque. In specfc cases, we show ha he compuaon s smlar o Rescorla-Wagner's equaon, whch nspred many compuaonal models n he doman of condonng. 1 Condonng The objecve of hs wor s o defne he behavors of robos accordng o he heory of condonng. Anmal condonng occurs when a smulus s repeaedly presened before a reward or a punshmen. Pavlov' expermens wh dogs are well nown and classcal or operan condonng have been suded for a long me [3]. However, despe several decades of wors on condonng models, s sll dffcul o esablsh all he rules ha govern he properes of condonng. Mos approaches are based on he orgnal model proposed by Rescorla and Wagner [5] n whch condonng s characerzed by assocave srenghs. Modfcaon of he assocave srengh of a smulus afer a new ral s gven by equaon (1). The ncrease s proporonal o he salence of (parameer α) and he effcency of condonng (parameer β). λ s he maxmum srengh and V Toal s he sum of all assocave srengh of he presen smul. n+ 1 n n V = V + α β ( λ VToal ) (1) The assocave srengh of a gven smulus can be nerpreed as he degree o whch a reward s predced. Anoher mporan model has been proposed by Klopf wh furher consderaons by Grossberg [1], []. In hese models, smul were represened by neurons and assocave srenghs were deermned by synapc weghs. Oher auhors followed he same prncples [1], [5], [6]. Suon and Baro also proposed a model of classcal condonng based on well nown renforcemen learnng echnques (TD model, 1987 and 1990 [7]). Despe all hese wors, all models suffer from specfc drawbacs. We propose n hs paper a new predcon sysem for condonng and s applcaon n robocs. I s based on a specfc neural newor archecure correspondng o he nodes of a Bayesan newor. The mehod s explaned n Secon. Some resuls are hen presened n Secon 3. In Secon 4, we descrbe a smple applcaon of our model wh robos. We fnally conclude wh he perspecves of hs wor. 147

2 ESANN 011 proceedngs, European Symposum on Arfcal Neural Newors, Compuaonal Inellgence and Machne Learnng. Bruges (Belgum), 7-9 Aprl 011, 6doc.com publ., ISBN Avalable from hp:// Proposed model.1 Descrpon of he newor Smul are defned as percepual evens n he represenaon of robos. In a smplfed world, here s only one reward (or only one uncondoned smulus, whch predcs he reward wh probably 1) and he objecve s o deermne f one or several observed smul predc he reward even n he nex few seconds or evenually f hey nhb. Our neural newor s defned by hree levels (see Fg. 1). In he frs level, he oupu of he neurons corresponds o he observaon of he dfferen smul. In he second level, he oupu of he neurons correspond o he probably of obanng a rue value for a hdden Boolean varable. There are wo hdden varables. Each of hem corresponds o a Nosy-Or node of he condonal probables of he frs level (see nex secon). The frs one can be consdered as he probably of rggerng he real bu hdden cause of he reward even and he second one corresponds o he probably of rggerng a hdden nhbory mechansm ha prevens he acon of he cause and he observaon of he reward. Tha second varable s necessary o allow nhbory condonng [4]. Inhbory condonng ndeed occurs f he condoned response s always observed afer he deecon of a gven smulus (reward expeced) bu s never observed f Y s presen a he same me. If a sngle nosy-or node had been used, he reward would have been expeced f one of he causal mechansms had been presen. Tha specfc problem has already been denfed n prevous wor [6]. The neuronal oupu of he hrd level corresponds o he probably of observng anoher even. I s defned as he probably of presence of he cause wh absence of he nhbory mechansm. Inhbory condonng s hus easly negraed n he model. In general, he las even s he reward, bu no necessarly.. Neural compuaon Fg. 1: Neural newor model. Pearl proposed he Nosy-Or o smplfy he problem of updang many condonal probables n a Bayesan newor [4]. We propose o mplemen hs mehod as follows. Frs of all, we defne P(H 1 ) as he condonal probably of obanng H 1 =rue durng a lmed perod of me (for nsance 5 seconds) followng he 148

3 ESANN 011 proceedngs, European Symposum on Arfcal Neural Newors, Compuaonal Inellgence and Machne Learnng. Bruges (Belgum), 7-9 Aprl 011, 6doc.com publ., ISBN Avalable from hp:// observaon of even. In he neural newor, synapc weghs correspond o condonal probables. For nsance he condonal probably P(H 1 ) s he synapc wegh beween node P() and P(H 1 ). If 1.. n are possble smul evens predcng H 1 =rue, n a Nosy-Or node s suffcen o have an esmae of all P(Y ) o compue P(Y 1.. n ) [4]. For he second level, he oupu of he neuron s herefore deermned by he oupu of he neurons of he frs level and her assocave weghs accordng o equaon (). P 1 ) 1 (1 P( H1 ) P( )) () = A smlar equaon holds for he compuaon of P(H ). The dfference only appears durng he updaes of he synapc weghs (e.g. he condonal probables). A he las level of he newor, P(Reward) s compued accordng o equaon (3). P(Re ward ) = P( H1)(1 P( H )) (3).3 Updae rules As s suggesed n he BANNER mehod proposed by Ramachandran, he updae of he condonal probables can be performed accordng o a rule ha mnmzes he mean square error [8]. However, snce we do no wan o sop he robo durng learnng, we propose o ae no accoun each error and o updae he probables n an ncremenal way. Le us consder he error for a gven ral. There are wo cases. If he reward s observed afer a gven se of smul, he global error s 1-P(Reward) and f no reward s observed he global error s P(Reward). Snce he compuaon of P(Reward) nvolves a mulplcaon beween P(H 1 ) and P(H ), he propagaon of he error o he second level s rval. If s an observed smulus, he updae of he condonal probables P(H 1 ) and P(H ) can be easly performed usng a graden descen echnque, whch s very smlar o he one proposed n BANNER, see equaons (4) and (5). Err( P( H1)) = Pcorr 1) P( H1) (4) P corr (H 1 ) s he correc value of P(H 1 ) consderng only he curren observaons. I s equal o 1 f he reward has been observed and 0 oherwse. Err( P 1) P = + + 1( H1 ) P 1 ) α P ) 1 P H ) = P ) + (1 P ) Vu( )) + ( α (5) where s he me accordng o a gven dscrezaon, α s a learnng rae smaller han 1, Vu( ) s a logcal varable equal o 1 f has been observed and 0 oherwse. There s an ncremen or a decremen of he probably accordng o he sgn of he error. A smlar equaon holds for P(H ) wh a symmerc error. The problem s ha we are dealng wh probables and even hough he updae rule aes no accoun he paral dervave of he error and evoluaes n he approprae slope here s no guaranee ha he probably would reman n he range [0..1] dependng on he value of he learnng parameer. We herefore propose ang no accoun he 149

4 ESANN 011 proceedngs, European Symposum on Arfcal Neural Newors, Compuaonal Inellgence and Machne Learnng. Bruges (Belgum), 7-9 Aprl 011, 6doc.com publ., ISBN Avalable from hp:// maxmum modfcaon of he probably. If he updae rule ncreases he probably, he maxmum local error s 1-P(H 1 ) and f s decreasng, s equal o P(H 1 ). We propose o mulply he rgh erm by hs maxmum local error so ha he updae s always a fracon of he maxmum allowed modfcaon. The new equaons are herefore (6)(7) and (8)(9), respecvely, when he reward s observed and when s no. P = ) P 1 ) α P 1 (1 1 ) Vu( )) (6) P + 1 ) = P ) α P ) (1 P ) Vu( )) (7) P 11 ) = P 1 ) α 3P 1 ) (1 P 1 ) Vu( )) (8) + = + + ) P ) (1 P 4 P 1 α ) Vu( )) (9) Noe ha n equaons (6) and (9) he new erm does no explcly appear because s now ncluded n he produc. Remar: We suggesed ha a predced reward even would be expeced durng a fxed perod of me afer a smulus even. We herefore have o memorze all evens durng ha perod. However, f a reward s observed a a gven me, he races of he smulus and he reward evens mgh sll be presen a me +1, + and so on dependng on me dscrezaon. If hs s he case, we have o apply equaons (6) and (7) a every sep. Ths s n fac an neresng propery because he ncrease of he condonal probably s nversely proporonal o he nerval beween he smulus and he reward and ha propery has been observed n he doman of anmal condonng. Conversely, f he reward even s absen durng all he fxed perod, equaons (8) and (9) are appled only once and he race of he smulus even s forgoen. Fg. : Condonng expermen: durng he frs 60 rals a smulus even occurs and afer 3 seconds a reward s presened. The probably of reward afer ha smulus s equal o P(H1). Then durng he nex 60 rals, he smulus s sll presened bu no he reward. The probably of reward afer ha smulus 150

5 ESANN 011 proceedngs, European Symposum on Arfcal Neural Newors, Compuaonal Inellgence and Machne Learnng. Bruges (Belgum), 7-9 Aprl 011, 6doc.com publ., ISBN Avalable from hp:// decreases below 0.5. There s exncon of condonng. Fnally, durng he las 60 rals, he smulus and he reward are presened (reacquson of condonng). 3 Resuls Some resuls are presened Fg.. The followng parameers have been used: α1=α=0.00, α3=α4=0.008, me dscrezaon: 0.s, fxed perod: 5s. I s mporan o noe ha smlar curves would be obaned wh dfferen parameers. If we consder ha condonng s acqured f he probably of he reward exceeds 0.5, our expermen llusraes a condonng followed by s exncon and fnally s reacquson a a faser rae. Oher condonng expermens are correcly descrbed wh he proposed model. Le us consder he blocng effec. Condonng wh a gven smulus Y s bloced or aes a long me f Y always follows a smulus and s already a smulus ha predcs he reward. In equaon (6), P(H 1 /Y) does no ncrease much because P(H 1 /) s close o 1 and he produc s proporonal o 1- P(H 1 /). Laen nhbon s characerzed by an nhbon of condonng f he smulus has already been observed whou he reward. In our model laen nhbon s observed because P(H ) ncreases before P(H 1 ). Our model can accoun for many condonng properes. More mporanly, he man advanage of our model s ha s no necessary o rese he parameers before a new expermen. The condonng wh a gven smulus can be exngushed, reacqured, combned wh anoher smulus a all me. The model can herefore be used n robocs n real me. 4 Applcaon n robocs We mplemened our model n Java for Lego Mndsorms robos. The robo s a smple rover equpped wh ulrasonc sensors for measurng dsance o an obsacle and a RFID sensor. There are hree acon modes. In he normal mode, he robo randomly chooses an acon among several ones, such as "go forward 0 cm", "urn 45 eas", "urn 45 wes". If an obsacle s deeced a a dsance less han 0 cm, he robo swch on a red lgh and eners a reacve mode. I was seconds, hen maes a 180 urn and reurn o he normal mode. The obsacle plays he role of an uncondoned smulus. In he real world, smul are ypcally sounds or vsual feaures and he problem of smulus recognon s nown o be hard. A RFID sensor has been used o smplfy neracons wh he robo. If an emer s presened n fron of he sensor, s rado frequency s mmedaely denfed wh 100% cerany. In our expermens smul are dfferen emers. Each me a smulus s deeced (and denfed), s race s memorzed durng 5 seconds. Meanwhle, f an obsacle s deeced, he condonal probably of observng an obsacle afer he deecon of ha smulus s updaed accordng o our model. We adaped he learnng raes such ha condonng s funconal afer 3 smlar suaons. As a consequence, he fourh me he same smulus s deeced, he robo eners a condonng mode, swch on a red lgh and goes foward durng 5 seconds wang for he deecon of he obsacle. If s deeced before he end of he 5 seconds, eners a reacve mode. If he obsacle s no deeced reurns o he normal mode. Expermens have also been conduced wh several smul. Exncon, reacquson, laen nhbon, blocng and 151

6 ESANN 011 proceedngs, European Symposum on Arfcal Neural Newors, Compuaonal Inellgence and Machne Learnng. Bruges (Belgum), 7-9 Aprl 011, 6doc.com publ., ISBN Avalable from hp:// nhbory condonng (obsacle never expeced afer a gven smulus) have been successfully observed. 5 Concluson Our model has been brefly descrbed. There are many oher neresng resuls ha could be dscussed. The applcaon of our model o he real world of robos s currenly nvesgaed. A promsng perspecve s he use of he model for he ranng of robos le he ranng of anmals o do smple ass such as s down, jump, go and ae an obje and so on. Ths s possble f he model s used for operan condonng. Snce condonal probables can ae no accoun any even, s easy o consder he acons of he robos as possble evens ha would predc rewards or punshmens. References [1] Balenus C. and Morén J., Compuaonal models of classcal condonng: a comparave sudy, n Mayer, J.-A., Robla, H. L., Wlson, S. W., and Blumberg, B. (Eds.), From Anmals o Anmas 5. Cambrdge, MA: MIT Press (1998). [] Commons, M. L., Grossberg, S., and Saddon J.E.R. (edors), Neural Newor Models of Condonng and Acon. Hllsdale, NJ: Lawrence Erlbaum Assocaes (1991).11. Klopf A., A neuronal model of classcal condonng, Psychobology, 16,, (1988). [3] Pavlov, I.P., Condoned Reflexes: An Invesgaon of he Physologcal Acvy of he Cerebral Corex (ranslaed by G. V. Anrep). London: Oxford Unversy Press (197). [4] Pearl, J. Probablsc Reasonng n Inellgen Sysems: Newors of Plausble Inference. San Maeo, CA: Morgan Kaufmann Publshers, Inc. (1988). [5] Rescorla R.A. and Wagner A.R., A heory of Pavlovan condonng: Varaons n he effecveness of renforcemen and nonrenforcemen, In Blac, A. H., & Proasy, W. F. (Eds.), Classcal condonng II: Curren research and heory, 64-99, New Yor: Appleon- Cenury-Crofs (197). [6] Salo J.M., "Nosy-Or nodes for condonng models", Lecure Noes n Arfcal Inellgence, Sprnger, from he conference "Smulaon of Adapve Behavors", (SAB 010), Pars, 4-8 Augus, 010. [7] Suon R.S. and Baro A.G., A emporal-dfference model of classcal condonng, Proceedngs of he 9h Annual Conference of he Cognve Scence Socey, (1987). [8] Sowmya Ramachandran, "Theory Renemen of Bayesan Newors wh Hdden Varables, PhD dsseraon, The Unversy of Texas a Ausn,