Reinforcement Learning in Associative Memory

Transcription

1 Renforcemen Learnng n Assocave Memory Shaojuan Zhu and Dan Hammersrom Cener for Bologcally Inspred Informaon Engneerng, ECE Deparmen OGI School of Scence and Engneerng a Oregon Healh & Scence Unversy {zhusj, srom}@ece.og.edu Absr- A renforcemen learnng based assocave memory srucure (RLAM) s proposed. In hs srucure, a one-layer feed forward Palm [] model s appled o he neworks. Insead of bach ranng, an on-lne learnng mehod s used o consruc he memory. The neworks are raned nervely accordng o renforcemen learnng, whch s bologcally plausble. The epermen resuls show ha he neworks converge and generalze well. I. INTRODUCTION Assocave memory appears o be one of he mos mporan funcons n many cognve processes, such as recognon, predcon, plannng, ec. In a saccade, for eample, he sensory nformaon obaned by an eye s ransformed no movemen nformaon, whch conrols he eyeball muscles o mplemen he saccade. The sysem ha deermnes he parameers of he saccade can be modeled as an assocave memory ha s raned o assocae ceran useful npu/oupu mappngs by prevous eperences. Many bran srucures can ofen be modeled as assocave memores. Assocave memory s ually a sysem ha sores mappngs from npu paerns o oupu paerns, so ha when one s encouner a he npu, he oher can be effcenly recalled. More mporanly, when a paern s n he neghborhood of he npu paern beng sored, an oupu paern correspondng o ha npu can be assocavely recalled, even f he npu paern s ncomplee or nosy. In heero-assocaon, he npu and oupu paern are dfferen. Many cognve processes mplemen heeroassocaon, such as ransformng sensory nformaon no muscle commands. In auo-assocaon, he npu and oupu vecors are n he same space, n hs case, paern compleon occurs. The Palm nework [, 2] s one of he useful assocave nework models. I s smlar o he nework developed by llshaw s group[3], and has large nformaon capacy and converges relably. e have done smulaons of large neworks on boh PC clusers and NASA s super-compuers [4]. The resuls showed ha he Palm neworks are robus and scale reasonably well. However, n hese models he mappngs were no raned nervely. All he npu and oupu paerns were randomly generaed a one me, and a wegh mar s compued from he ranng se. In real applcaons, especally for a bologcally nspred sysem, ncremenal learnng va neron wh a real envronmen s very mporan. One mporan famly of algorhms, renforcemen learnng [5], has been used by sysems ha learn from neron wh her envronmen. The essence of renforcemen learnng s ha s ral-and-error learnng, where here s no supervsory feedback provded o he sysem, only a sngle reward sgnal s avalable. The RLAM sysem dscussed s par of a larger roboc sysem ha we are developng for NASA. Ths frs verson of RLAM wll be used as a very rough smulaon of he pareal core. I s beleved ha he pareal lobe s nvolved n vsual corcal processng by combnng eye poson sgnals wh renoopc vsual npu o generae headcenered represenaons of he vsual npu [6, 7]. Our goal for RLAM was o creae an assocave memory srucure ha duplcaes aspecs of pareal funcon by learnng mappngs wh a renforcemen based learnng mechansm. I s possble o mplemen renforcemen learnng va a smple look-up able, where he agen chooses an by searchng he look-up able. I s possble o buld localzed assocaon models ha are very smlar o a look-up able. However, for he ask ha has a very large sae space or on space, raversng every sae and on s no possble. A more general srucure s needed and preferably one ha allows reasonable generalzaon so ha he agen can make good use of s eperences. Ths requres an assocaon model ha uses dsrbued represenaons and sorage. In hs paper we nroduce a renforcemen learnng assocave nework. The nework s based on he Palm model, where a renforcemen learnng based Hebban learnng rule s used for he sorage process, and k-wnnerake-all algorhm s appled o mplemen he rereval process. II. PALM ASSOCIATE NETORKS In he Palm model, he se of mappngs o be sored s S = µ, y µ µ =,2,, M, where µ s he npu denoed as: { ( ) } paern, and y µ s he correspondng oupu paern. Boh µ and y µ are sparsely encoded bnary vecors. In he ranng procedure, each par of he mappng s presened o he nework. A clpped Hebban learnng rule s appled o generae he wegh mar: Bologcal compung for robo navgaon and conrol, NASA, PI: Marwan Jabr, Co-PIs: Chrs Assad, Dan Hammersrom, Msha Pavel, Terrence Sejnowsk, and Olver Coenen.

2 M T = [ y µ ( µ ) ], µ = where s Boolean OR operaon, and s ouer produc. In he recall procedure, a paern ˆ s presened o he nework npu. The npu ˆ can be he nosy versons of he ranng vecor. The oupu vecor ha s rereved by he nework wh wegh can be calculaed by: yˆ = f ( ˆ θ ), (2) where represens he nner produc, θ s a global hreshold and f () s a bnary-valued ransfer funcon. To se he hreshold, he k wnners ake all rule (k-ta) was proposed by Palm. k s he number of ve nodes n an oupu vecor. All he oupu neurons have o compee for hemselves so ha only k elemens n he oupu vecor are allowed o be. The hreshold s se adapvely o he value where only hose nodes ha have he k mamum values can be se as. By keepng k small, he npu and oupu vecors are hen sparsely encoded, n addon, he use of bnary synapc weghs add o he smplcy of he nework and do no affec recall performance. Palm shows ha he nformaon conen n bs per synapse s hgher han he radonal conen addressable memory, and he mamum nformaon capacy occurs when he wegh mar s half full. Anoher charersc of Palm nework s ha, he rereval procedure generally converges whn one or wo eraons. e have done smulaons of large neworks on our Beowulf cluser and he NASA super-compuers. Also, we have a spkng verson of he basc Palm nework. Palm neworks performance scales reasonably well o he nework sze. The assocave memory s faul oleran and can do bes mach recall such ha when a nosy or ncomplee npu s gven, he assocave memory wll assume he closes mach npu and produce he oupu vecor assocaed wh ha npu. In he case of auo-assocaon, he closes ranng vecor s reurned. The concep of bes-mach mples ha here s some merc defned over he vecor space he memory operaes n. For he purposes of hs paper we wll assume a basc Eucldean dsance. However, one can easly magne such neworks operang n envronmens where more comple represenaons would be requred. For some applcaons, f he npu and oupu mappngs are known, and boh npu and oupu paerns can be represened by bnary sparse vecors, buldng a Palm nework s sraghforward. However, he Palm model we are usng does no perform nerve ranng. Ineron beween he sysem and s envronmen s crucal. A robo can perform some smple asks easly. Bu for more comple asks, where flebly or generalzaon s requred, nerve learnng s ofen he mos effcen approach. In hs case, can he robo adap o he unknown envronmen? Can learn new sklls o accomplsh a new ask? () Renforcemen learnng s a mechansm where a robo learns o know he envronmen and esablsh he mappngs, based on a smple reward sgnal. In hs paper we demonsrae a verson of Palm s nework ha s raned va renforcemen learnng, n whch he npu and oupu mappngs are esablshed ncremenally. e wll sll use he same Palm rereval algorhm. In he ne wo secons we wll frs eamne he ssue of ranng a Palm nework ncremenally, and second, he use of hs ncremenal updae model n renforcemen learnng. III. INCREMENTAL LEARNING In he orgnal Palm model, he assocave nework s consruced n a bach mode, where all he npu and oupu mappngs are used o generae he compleed wegh mar before any rereval s done. The nework weghs are only updaed afer all of he npu and oupu mappngs are presened. Bach learnng s also called off-lne ranng. Conrary o he off-lne ranng s he on-lne ranng, where learnng occurs ncremenally for each npu vecor presened o he nework. Incremenal learnng allows he assocaon memory o adap o he envronmen. For ncremenal learnng, he conrbuon of each npu/oupu par s summed up, and he adapaon s noed as: µ µ = R(, y ), (3) where R (, y ) s he local synapc rule ha deermnes he amoun of wegh change based on he assocaon of npu µ o oupu y µ. For bach learnng, f oo many vecors are sored, he nerference among vecors can become a severe problem, and somemes, he caasrophc forgeng occurs. Usng ncremenal learnng, he learnng sysem may predc s oupu based on s prevous eperences, and can adjus s parameers o be beer adapve o s envronmen. So he sysem may remember he newer paerns a he prce of forgeng he older ones. Bayesan Confdence Propagaon Neural Neworks (BCPNN), proposed by Anders Lansner [8], perform hs knd of ncremenal palmpses learnng. For he me beng we are assumng a saonary envronmen and are no consderng more comple BCPNN lke learnng rules a hs me. Afer bach learnng, he assocave memory appromaely parons he npu space no Vorono regons, each of whch corresponds o one npu/oupu vecor mappng. A Vorono regon of a vecor V s he unon of all vecors V o whch s he closes one: V = V : V V < V V, j, (4) { } j where V V s he Eucldean dsance. 2 In bach ranng, f all he npu paerns are equally lkely, afer ranng, all he ranng vecors wll have appromaely

3 equal wegh and he Vorono boundares are equdsan beween any par of neghborng vecors n he space. Snce he vecors are he same lengh, he Vorono regon ss on an N-dmensonal sphere. For ncremenal learnng o work, we have o nroduce he concep of weghed Vorono regons. Defnon: le S denoe a se of aror vecors such ha each vecor V has assgned a posve, fne nfluence ω ( ). In hs case, he dsance of an arbrary vecor V from V s gven by V V ( ω( ) / ω( )). The weghed Vorono regon for S s a subdvson of he npu space such ha each vecor V n S s assocaed wh a regon conssng of all vecors V n he regon for whch V s he neares vecor usng a weghed dsance merc. In ncremenal learnng, f a vecor V s presened frequenly, s nfluence ω ( ) on s neghbors wll become larger, and s correspondng Vorono regon wll ge larger. Our RLAM nework hen ulzes hs aspec of assocaon memory wh renforcemen learnng echnques o drve he ncremenal learnng. IV. REINFORCEMENT LEAERNING I s beleved ha many bologcal neural sysems perform some knd of renforcemen learnng [5]. The agen and he envronmen are he wo componens of he learnng sysem, and he learnng akes place va a seres of seps. A each sep, he agen akes an on, f he on s successful, he agen ges a reward, oherwse, no reward s gven. The agen s goal s o mamze he reward ges. Afer each sep, he agen updaes s oupu selecon polcy accordng o he reward, and wll ncremenally mprove a performng he ask. Afer eploaon and eploraon, he agen generally fnds an opmum polcy for ha ask so as o ge he mamum reward. One smple way o do renforcemen learnng s Q- learnng. Q learnng assumes ha here s some sysem ha can be n varous saes, S. Furhermore he agen can ake several ons, a, ha change he sae. The goal of Q- learnng s o develop a polcy ha generaes ceran ons n ceran saes ha mamzes he reward. In Q-learnng, Q s a, he value for he sae S and on a par, (, ) represens he agen s curren knowledge abou epeced long erm award for akng on a n sae S. For each sae, here may be several Q-values correspondng o each on, he agen wll generally ake he on ha has he mamum Q-value ( eploaon ). The goal of learnng s o reach a long-erm mamum reward from he envronmen. The updae of he Q-value s: Q( s, a ) Q( s, a ) + α[ r + γ ma Q( s, a) Q( s, a )]. (5) + + a Q-learnng s one of he mos popular forms of renforcemen learnng. However, may converge que slowly o a good polcy. The resul of he Q-learnng s essenally a look-up able such ha, for any sae, he correspondng possble ons and her relave value fucons are lsed. Snce s dffcul o eplore all possble ons from all possble saes (even once, le alone several mes), we wan our learnng sysem o be able o generalze effecvely, whch allows he effcen sorage of learned nformaon and broad ulzaon of he knowledge beween smlar saes o ons. Many researchers have been usng neural neworks o solve he generalzaon problem. For eample, he nodes n he nework can be raned o respond o a ceran se of saes, and he neworks can be raned o sore a varey of mappngs, ncludng: sae polcy, sae value funcons, sae+on reward, ec. As menoned before, an assocave memory sores npu/oupu mappngs. The renforcemen learnng s also a procedure of esablshng mappngs. e wll use he Q-learnng echnque n our ncremenal learnng Palm nework., whch mplemens he wo mappngs: S a and S Q. Boh neworks are onelayer assocaon neworks wh no hdden layers. V. THE LEARNING MODEL e choose he grd-world ask o compare he renforcemen learnng based Palm nework wh he able drven Q-learnng sysem. In our epermens, a grd-world of 5 5 s dscussed. Each grd corresponds o a sae, and n each sae, 4 dfferen ons are avalable: ake one sep n one of four drecons, norh, souh, eas or wes. Each on s a sep, and he sep lengh s se o. The agen sars a a random poson n he grd, and he ask n our grd-world epermen s o fnd he goal ha s locaed a he cener of he grd. Ths ask can be eended o a more general ask n whch he agen sars a a random grd pon and he goal s also locaed a a random pon n he grd. Anoher varaon s o add barrers whch mus be raversed (lke maze learnng). For he more general ask, we only need o map he curren grd-world sae space o a larger sae space, where he relave poson beween he goal and he agen s ncluded. In he more general grd-world sae space, he agen s poson s always cenered a he sae space. e have epermened on he general grd-world ask, and movng he smaller space o a larger space s farly sraghforward, so o smplfy he dscusson, we only dscuss he grd-world ask where he goal s always se a he cener of he grd 2. 2 The grd world eample, hough very smplsc, fs he general ask of learnng o perform vsual coordnaon as requred by our robo sysem.

4 In he ask, when he agen fnds he goal, a new epoch begns and he agen s sared a a new random sae. The reward s eher or 0, correspondng o fndng or mssng he goal respecvely. The saes are represened by a, 5 5, wo-dmensonal array of neurons. For each sae ha s no a he edge of he grd, he 9 neurons n a 3 3 feld surroundng he neuron are fred. For some saes ha are a he edge or he corner of he grd, a mnmum of 4 neurons may be vaed. e consruced wo neworks: and. mplemens he mappngs from he sae S o he on a, and mplemens he mappngs from he sae S o he value funcon Q. The npus o boh neworks are he sae. The oupu of he nework s he on vecor, whch has only 4 elemens, correspondng o he 4 drecons an on may ake. The on vecor can have only ve elemen. The oupu from he nework s he Q-value represened by he elemen s value. s nalzed o random values, and all he elemens n are nalzed wh a very small posve value 0. o se up connecons beween all he saes and he oupu value neuron. The on vecor s calculaed by equaon (6), whch s he same rule used n he Palm rereval process. In he on vecor, here s always one and only one neuron ha s ve. If, by equaon (6), here s more han one ve neuron, he agen randomly chooses one on from all he possble opons for ha sae. Ths means ha more han one on s possble for ha sae. I also njecs a ceran degree of randomness no he on selecon whch s necessary for he nework o adequaely eplore s envronmen ( eploraon ). The wegh adapaon for and he wegh adapaon for s shown n equaon (2), s shown n equaon (3). In he equaons, represens he nner produc: a f ( S θ ) = (6) (7) ( S, a ) S + Q = S (8) crc Q = S (9) + crc + * γ + Q = r + Q (0) ε = Q Q () * = ε S (2) (, ) crc = αε S crc crc j (3), j In he above algorhm, a me sep, he agen s a sae S. The on a s compued from and he agen akes he on, enerng a new sae S +. The value funcon for sae S and S + s obaned from equaons (8) and (9) respecvely. The error s calculaed by equaon (), whch n urn conrols he adapaon of wegh and. Afer learnng, boh and are normalzed by he use of he Sgmod funcon, so he wegh values are consraned o be beween 0 and. Ths forces a slgh nhbory effec on he wegh values. The weghs hen requre mulple bs o represen. e have no ye eplored he mnmum precson requred for effecve ncremenal learnng. I also reduces he effec ha Hebban learnng for connually ncreasng he weghs ndefnely. Snce we are assumng a sysem wh saonary probablsc behavor, whch s manfes by he goal and s reward sgnal beng fed n me, no addonal subrve or decay mechansm s requred. The values can also be hough of as represenng he confdence or probably of he srengh beween he npu and oupu neurons. VI. RESULTS e compared he resuls from he renforcemen learnng based assocave memory wh ha from he radonal Q- learnng sysem. Usng he Renforcemen Learnng based Assocave Memory (RLAM) srucure, he learnng process converged more rapdly han he Q-learnng Look-up Table (QLT) sysem, and RLAM s ably o generalze was much beer han QLT. In he below ermnology, each on s a sep, and an epoch s he seres of seps ha he agen akes o accomplsh he ask. In Fg., afer 3000 ranng seps, he RLAM acheves a successful rereval rae of 98%, whch means ha, for 98% of all he grd posons, he agen can successfully accomplsh he ask. hle n he QLT sysem, afer 3000 seps, he sysem aans % successful rereval rae, only afer 40K seps of rals, can he agen acheve 97% successful rereval rae. From Table I, o acheve a successful rereval rae of 98%, he agen has been raned by 37 epochs, and no all he grd coordnaes were vsed by he agen. Bu he agen sll has learned enough knowledge o be able o erapolae successfully o he unvsed saes he agen has acqured he generalzaon ably, an mporan componen of an assocaon memory mplemenaon. In he QLT sysem, he agen was raned for 245 epochs, whch means ha, on average, each grd pon was vsed a leas 9 mes (245 (5 5)). Snce he look-up able does no generalze well, he agen has o raverse every sae several mes, unl has enough eperence abou he envronmen o operae successfully.

5 Successful Rereval Rae Number of Seps (K) 30 RLAM QLT Fg.. Rereval performance versus oal number of ranng seps TABLE I Number of Seps Number of Epochs Successful Rereval Rae RLAM 3K 37 98% QLT 40K % In he RLAM sysem, afer ranng, nework shows a herarchcal organzaon. e llusrae he wegh marces by gray-level mages wh 0 represenng black and represenng whe. In Fg 2, each mage corresponds o he wegh mar ha connecs o one of he four neurons: (a) llusraes he weghs o neuron whch s responsble for gong souh, (b) o neuron 2 for gong norh, (c) o neuron 3 for gong eas and (d) o neuron 4 for gong wes. e can see from Fg 2 ha each oupu neuron assocaes wh dfferen groups of saes. 50 compared wh he radonal Q-learnng sysem (QLT) on he speed and effcency of he learnng, and s generalzaon durng rereval. RLAM s more bologcal n ha, he sysem s adaped ncremenally by he neron beween he agen and he envronmen. Also, he memory of RLAM converges more quckly and generalzes beer han he QLT. I s very lkely ha we could acheve comparable resuls by usng more radonal error-drven neural nework models such as Back-Propagaon or SVMs. However, n our roboc sysem, we have a requremen for rapd learnng, whch he assocaon memory based sysem provdes. The grd problem shown here does no requre he memory o creae comple represenaons. The ne sep n hs work s o pu RLAM no a real applcaon wh real-world daa and assess s ably o creae he necessary represenaons under hese condons. Early resuls are encouragng. e use wo assocave memores o smulae wo mappngs. Boh neworks are one-layer nework, wh no hdden layer. Ths provdes a smple srucure for he assocave memory desgn. The RLAM proves o be sable, and he conjecure ha ncremenal learnng based Palm neworks appromaes he weghed Vorono regons works n he above smple eample. Bu heorecally, how he aror basn of he sysem s formed when more and more saes are raned s sll under research, and he ably of RLAM o represen he mappngs n a more comple problem mus be eamned. ACKNOLEDGEMENTS Ths work was suppored n par by NASA Conrs NCC and NCC REFERENCES (a) (b) (c) (d) Fg. 2. Pseudo gray-level mages of weghs correspondng o four oupu neurons, wh represenng whe and 0 represenng black. Each mage llusraes he synapc srengh of he wegh. Before ranng, he npu/oupu mappngs are random, and he weghs dsplay chaos pseudo mages. hen more ranng eraons ake place, he nework wll gradually ge self-organzed. In Fg. 2, when he agen lands a he op saes of he grd, he movng souh on wll domnae among he 4 neurons, so he agen wll move souh. If for some saes, here are more han wo neurons are promnen, he agen can randomly choose one drecon. For he fnal goal, here s no much dfference beween choosng whch on o ake, and n real world problem, here s usually more han one opon of comparable value. VII. CONCLUSION e have presened a renforcemen learnng based verson (RLAM) of he Palm assocave memory, and have [] Palm, G., On Assocave Memory. Bologcal Cybernecs, 980: p [2] Palm, G., e al., Neural Assocave Memores, n Assocave Processng and Processors, E.A.K.a.C.C. eems, Edor. 997, IEEE Compuer Socey: Los Alamos, CA. p [3] Buckngham, J. and D. llshaw, Performance Charerscs of he Assocave Ne. Nework: Compuaon n Neural Sysems, : p [4] Zhu, S. and D. Hammersrom. Smulaons of Assocave Neural Neworks. n ICONIP' [5] Suon, R.S. and A.G. Baro, Renforcemen Learnng: An Inroducon. 998, Cambrdge, MA: MIT Press. [6] Goodman, S.J. and R.A. Andersen. Algorhm programmed by a neural nework model for coordnae ransformaon. n IJCNN [7] Buneo, C.A., e al., Drec vsuomoor ransformaons for reachng. Naure, : p [8] Sandberg, A., e al., A palmpses memory based on an ncremenal Bayesan learnng rule. Neurocompung, : p