Convergence Results in an Associative Memory Model

Transcription

1 NeualNetwoks, Vol. 1, pp , /88 $ Pited i the USA. All ights eseved. Copyight (c) 1988 Pegao Pess plc ORIGINAL CONTRIBUTION Covegece Results i a Associative Meoy Model JANOS KOML0S AND RAMAMOHAN PATURI Uivesity of Califoia, Sa Diego (Received Septebe 1987; evised ad accepted Febuay 1988 ) Abstact--This pape pesets igoous atheatical poofs fo soe obseved covegece pheoea i a associative eoy odel itoduced by Hopfield (based o Hebbia ules)fo stoig a ube of ado -bit pattes. The capability of the odel to coect a liea ube of ado eos i a bit patte has bee established ealie, but the existece of a lage doai of attactio (coectig a liea ube of abitay eos) has ot bee poved. We peset poofs fo the followig: Whe, the ube of pattes stoed, is less tha /(4 log ), the fudaetal eoies have a doai of attactio of adius o with p = 0.024, ad the algoith coveges i tie 0 (log log ). Whe = a (with c~ sall), all pattes withi a distace pfo a fudaetal eoy ed up, i costat tie, withi a distace e fo the fudaetal eoy; whee ~ is about e -1/4~ We also exted soewhat Newa "s desciptio of the "eegy ladscape," ad pove the existece of a expoetial ube of stable states (extaeous eoies) with covegece popeties siila to those of the fudaetal eoies. Keywods--Neual etwoks, Associative eoy, Cotet addessable eoy, Dyaical systes, Spi-glass odel, Rado quadatic fos, Leaig algoiths, Theshold decodig. 1. INTRODUCTION Udestadig of the eoy i biological systes is a ipotat poble. A ipotat chaacteistic of such a eoy is its associative atue, that is, the ability to ecall, give patial ifoatio. Followig McCulloch ad Pitts (1943 ), we ca expect to odel eoy as a itecoected syste of euos with biay activity. Sice biological eoy is highly distibuted (see Lashley, 1960), it would be easoable to assue that itecoectios aog the euos collectively ecode ifoatio. We also expect the biological systes to evolve usig a siple leaig ule. Oe such atual leaig ule was suggested by Hebb (1949) which essetially says that the syaptic itecoectio stegths eflect the coelated activity at the coespodig euos. Models of eoy based o these ideas have bee studied by vaious eseaches (see Hito & Adeso Ackowledgeet--We thak the efeees of this pape fo thei helpful coets egadig histoical atecedets. Requests fo epits should be set to Raaoha Patui, Mail Code C-014, Depatet of Copute Sciece & Egieeig, Uivesity of Califoia, Sa Diego, La Jolla, CA (1981); Kohoe (1984); Kohoe (1987) fo histoy ad efeeces), ad oe ecetly by Hopfield (1982). I fact, a closely elated odel is studied by Little (1974) ad Little ad Shaw (1978). The odel used by Hopfield cosists of a syste of fully itecoected euos o liea theshold eleets whee the itecoectios have cetai weights. Each euo i the syste ca be i oe of two states ad the state of the etie syste ca be epeseted by a -diesioal vecto, whee is the ube of euos i the syste, ad the copoets of the vecto deote the states of the coespodig euos. The euos update thei states based o a liea fo coputed by the weights of thei itecoectios ad the cuet state of the syste. This syste of itecoected euos ca exhibit soe popeties of eoy, if the weights ae chose appopiately. Fo exaple, we obseve stable states fo which o out-of-the-state tasitios occu. We also obseve that stable states have a attactig popety, that is, whe the syste is iitiated i a state close to a stable state, it will ed up i the stable state afte a successio of state tasitios. The stable states togethe with thei attactig behavio ca be thought of havig the associative, cotet-addessable o eo-

2 /. ~,otbs act R Patut~ coectig popety of a eoy. I paticula, we ca egad the stable states as the ifoatio stoed ad thei ability to attact eaby vectos as the ability to ecall, give distoted o patial ifoatio. Such a syste will be useful as eoy if. gave a set of vectos to be stoed, the syste, by a siple leaig ule, ceates the equied itecoectios such that the give vectos appea as stable states with a attactig popety. The study of this odel is iteestig ot oly i the cotext of odels of eoy but also i the cotext of the statistical echaics of disodeed agetic systes, fo exaple, ifiite-age spi glasses. Kikpatick ad Sheigto (1978) ad Ait, Gutfeud. ad Sopolisky ( ) studied this ifiiteage spi glass odel usig the oigoous eplica ethod. We discuss soe of thei esults i the coig sectio. I the ext sectio, we give a pecise desciptio of the odel ad foulate the atheatical pobles associated with it. Ou pape is stictly techical. We do ot ited to povide a systeatic itoductio to the subject, explai potetial use, liitatios, o ecet efieets of the odel: these have bee doe i seveal othe papes. 2. THE MODEL We have a syste of fully itecoected euos. The weight of the itecoectio fo the ith euo to the jth euo is deoted by w~j. We assue that w u = wji, that is the itecoectios ae syetic. The weights ae coveietly epeseted as a syetic atix W, with zeos i the diagoal. Each euo ca be i oe of two states, say, + 1 o -1. The state of the etie syste is deoted by a -diesioal vecto x, whee the ith copoet x~ ofx is the state of the ith euo. We will use the wods state ad vecto itechageably. Coputatio at each euo follows a liea theshold decisio. Each euo coputes a weighted su of the states of the othe euos. If this su exceeds a cetai theshold associated with the euo, the ew state of the euo is set to be + 1. Othewise, it is set to - 1. I this pape we assue, without loss of geeality, that the theshold associated with each euo is zeo. Theefoe, ifx is the cuet state of the syste, the ew state of the ith euo is siply the sig of the weighted su ~ wijxj whee the fuctio sig is defied by sig(w) = {+l. ifw>o othewise The syste ca opeate i oe of two odes: sychoous o asychoous. I the sychoous ode. the euos siultaeously chage thei states at discete tie itevals. Thus, afte oe sychoous step, the state x is chaged to the state sig(wx). I the asychoous ode. the idividual euos chagc thei state oe by oe, i a uspecified ~o ado ode of successio. (We assue, of couse, that evey euo has its tu tie ad agai I eithe case. we say that a state v eaches a state t- if the syste, whe stated i state v. will be i state x afte a successxo of state tasos. We defie that a state x is stahh:' if x -stg(fox). Note that the otio of stable states Ls idepedet of the ode (sychoous o asychoous ~ of opeauo of the syste. We defie E = -½xll~3c = --~ ~ wijx,xi as the eegy of the syste at state x. ~the quadatic fo - ½ x TWx defies the "eegy ladscape" of the syste. This cocept is ipotat if we ote that the eegy of the syste deceases at evey asychoous ove. This always guaatees that a stable state is eached evetually i the asychoous ode of opeatio, ad all stable states ae local iia of the eegy ladscape. (Ideed, a state x is a local iiu fo the eegy fuctio if Qi = Qi(x) = ~ w~j.~,x ~ O,Jo all i, which j--i is satisfied by stable states. ) We defie the adius of attactio of a stable state x as the lagest value of p such that evey vecto y at a distace ot oe tha p fo x evetually eaches the state x. ( Fo coveiece of otatio, we will teat p as a itege, ad oit details elated to oudig. ) I othe wods, the syste ca coect abitay p eos ade i the stable state x. Hee. ad thoughout the pape, "distace" efes to the Haig distace, which is the ube of copoets i which the two vectos diffe. Give a set of vectos v ~. v 2... v" to be stoed. a siple leaig ule is used to deteie the weights of the itecoectios aog the euos. The goal is to costuct the weights wij i such a way that the vectos v' ae stable with a sufficietly lage adius of attactio. Fo oe vecto v. the atual choice of weights is the oute poduct atix, wi; = vivj, sice v is a eigevecto of this atix with a lage eigevalue, ad all othe eigevalues ae 0 (ak 1 atix). Thus, sig(wx) aps evey vecto withi /2 fo v ito v. To stoe ay vectos, we siply take the su of the coespodig oute poducts with the hope that the syste will eebe these vectos with soe adius of attactio. Thus, ou atix W of weights is defied by w~= ~, vivj' ' (i 4~ j) t=l

3 Covegece Results 241 Note that the syste does ot eebe the idividual vectos vt, but oly the weights wij which basically epeset coelatio aog the vectos. If a ew vecto is to be stoed, the syste "leas" by addig the ew oute poduct to the existig weight atix W. The vectos thus peseted to the syste will be called fudaetal eoies. We ow copleted the desciptio of the odel. Fo the syste to fuctio as a eoy, we equie that evey fudaetal eoy is stable o at least thee is a stable state at a sall distace ~. It is also desiable to have soe positive adius of attactio aoud the fudaetal eoies. Whe the ube of fudaetal eoies is lage, that is, the syste leaed too ay vectos, oe ca expect soe "fadig" of eoy. The it is still easoable to expect soe kid of eo coectio popety like the followig: if the syste is stated at a state withi a distace 0 fo a fudaetal eoy, it will each a stable state withi a distace of ~ fo the fudaetal eoy afte a sequece of state tasitios. (Note that whe e = 0, this eas that the fudaetal eoy is stable with a adius of attactio p. ) Give ay set of fudaetal eoies, we would like to stoe the i the syste. But this equieet is soewhat at odds with the equieet of eo-coectio. We caot expect to stoe vectos which ae too close to each othe. I fact, it is easy to costuct, fo ay >_ 4, fou vectos fa fo each othe, which caot be stable i ay syste. A easoable iial equieet i this coectio is that we would like to stoe alost all sets of vectos. Theefoe, we will take a set of ado vectos as ou set of fudaetal eoies. Notatios Fo vectos x, xi will deote the ith copoet ofx. If x ad y ae vectos of the sae diesio, the Haig distace d(x, y) betwee x ad y is defied as the ube of copoets i which they diffe. The scala poduct (x, y) of two vectos is defied as (x, y) = E xiyi. i The o of a vecto x is defied as llxll = (E x~) '/2 i We wite [] = { 1, 2... }. P(A ) efes to the pobability of the evet A. E stads fo expected value. Fo oegative p _< 1, we defie the etopy fuctio h(p) = -p log p - (1 - p)log(1 - p). Fo oegative p ad p' such that o + p' -< 1, we defie the etopy fuctio h(p, p') = -p log p - p' log p' - (1 - p - p')log(l - p - p') (log stads fo atual logaith ). c l, c2, ae sall positive absolute costats. 3. PREVIOUS RESULTS I this sectio, we peset the elevat pevious atheatical esults egadig the associative eoy odel Stable States ad Eegy Ladscape It is a basic questio to fid out fo what values of the fudaetal eoies will be stable with high pobability. (Recall that a vecto x is stable if x = sig(wx). A alteative teiology is "fixed poit." ) Basic questios about the absolute stability of the global patte foatio is dyaical systes which ae geealizatios of the associative eoy odel have bee studied by Gossbeg (1982) ad Cohe ad Gossbeg (1983) usig Liapuov fuctios. McEliece, Pose, Rodeich, ad Vekatesh (1987) poved the followig ipotat esults egadig stable states: If < - -, the (with pobability ea 1) all 4 log fudaetal eoies will be stable. I f - - < < - -, the still ost fuda- 4 log 2 log etal eoies will be stable. Whe is lage tha c/log, i paticula, whe = a, the fudaetal eoies ae ot etievable exactly, but oe still ay fid stable states i thei viciity. This is suggested by the "eegy ladscape" esults of Newa (1988). I paticula, Newa poves that, fo all fudaetal eoies, all the vectos which ae exactly at a distace of ~ fo the fudaetal eoy have eegy i excess of at least # 2 above the eegy level of the fudaetal vecto. These esults of Newa ae valid whe a _< Ait, Gutfeud, ad Sopolisky (1985, 1987) studied this associative eoy odel ad exteded it to iclude tepeatue. The tepeatue T = 0 coespods to the associative eoy odel discussed i this pape. They use the oigoous eplica ethod (Kikpatick & Sheigto, 1978) to ivestigate the odel. The eplica ethod siplifies coputatios usig cetai additioal idepedece assuptios ad it is kow to give icoect esults whe T = 0. Soe of thei elevat esults ae give i the followig. Whe the ube of stoed pattes is such that a = / is costat, as becoes vey lage, Ait, Gutfeud, ad Sopolisky ( 1985 ) obseved that the

4 242 / h%l/6,~ ad R Patut syste povides vey effective etieval of eoy tb cetai age of a. Whe a < ac ~ 0.14, oe ca fid stable states which ae vey close to the oigial stoed pattes at a distace of less tha As a deceases to 0, this distace deceases as e -1/2". Hece. if etieval of patte with a sall pecetage of eo is toleated, the stoage capacity of the etwok is a,. If oe equies etieval with a vaishig factio of eos as -~ oo, the a ust vaish with iceasig. Fially, etieval of pattes fee of eos occus whe ~ < 1/(2 log ). They also oted that the etieval of eoy is stable agaist a sall aout of oise i the weights w~j, but the axiu allowed level of oise deceases to 0 as a appoaches a,. They deostated, by ueical siulatios, the existece of a citical value a, ~ 0.14, below which eoy etieval is still possible ad athe efficiet. Ait, Gutfeud, ad Sopolisky also obseved. that, fo sufficietly sall ct. thee ae expoetially ay stable states (extaeous eoies) which ae "liea cobiatios" of stoed pattes. Gade (1986) coputed a expoetial (i ) uppe boud o the ube of such extaeous states. We will late pove a expoetial lowe boud o the ube of extaeous eoies Rado Eos It is also ipotat to fid out the extet of eo coectio aoud the fudaetal eoies. Oe atual poble is to ivestigate the behaviou of the syste i the pesece of ado eos. This questio has bee studied by McEliece, Pose, Rodeich, ad Vekatesh (1987). They pove the followig: If < /(4 log ), the a give vecto withi distace p (p < ½) fo a fudaetal eoy will be attacted to it i oe sychoous step of the syste with high pobability. I the asychoous case. a liea ube of steps suffice. This eas that fo ost choices of < /(4 log ) fudaetal eoies, the syste ca coect ost pattes of at ost p eos i oe sychoous step (o liea ube of asychoous steps). The above esults leave ope the questio of etieval i the case of abitay eo pattes. I the ext sectio, we look at this questio Wost Case Eos I ost applicatios, coectig ado eos ay be satisfactoy. Yet, it is iteestig to fid out if stable fudaetal eoies ca attact all the vectos withi a distace of a fo soe positive costat. Fo this, oe caot ely o siulatios sice siulatios (due to the pohibitively lage ube of eo pattes) ca oly eveal the behavio of the syste ~ the pesece of ado eos~ I fact, the followig obsevatios clealy deostate that ado eos ad wost case eos behave i etiely diffeet ways. Thee caot be a oe sychoous-step covegece i the pesece of abitay p eos, ot eve fo abitay ~ eos. Note that oe sychoous-step covegece is kow fo ado eos as etioed i the pevious sectio. Oe caot have a adius of attactio o ea :~. 0 > ~ is aleady ipossible, eve whe is vey. sall. Cotast this agai with the case of ado eos. This obsevatio was ade ealie by Motgoey ad Vijaya Kua ( 1986)_ Thus, oe ca oly hope fo gadual covegece ad a sall adius of attactio i the case of wost case eos. It would be iteestig to lid the axiu adius of attactio v~- We cal~ oly show thal t < 00 < I this pape, we establish the existece of a positive adius of attactio p aoud the fudaetal eoies whe < /(4 log ). Both sychoous ad asychoous odes of opeatio wilt povide the stated eo coectio. Moeove: it oly takes O( log log ) sychoous-steps fi.w covegece. Futhe, whe = (~. fo soe costat c~. we pove eo coectio fo o eos to ~ eos. ( Hee 0 is a costat, ad e ca be ade abitaily sall by choosig a sufficietly sall a. I I the asychoous ode. if oe stats withi a bits of a fudaetal vecto v. the syste will covege to aothe state withi ~ bits of v. 4. PRESENT WORK I the followig we peset a bief suay of ou esults. This suay gives a idea of the "'eo-coectig behavio" aoud the fudaetal eoies Suay as, aa, Pa, Pa, Pb ae absolute costats. If _< a~. ad if the syste is stated withi a distace of p~ fo a fudaetal eoy, the. i about log (/) sychoous steps, it will ed up withi a distace e -/4 fo the fudaetal eoy, that is, it will evetually get withi a distace e -~/4'~ of the fudaetal eoy ad eais withi that distace. Whe < /(4 log ), the syste will covege to the fudaetal eoy i O(log log ) sychoous steps. I the asychoous case, if ~ ota, ad if the syste is stated withi a distace of Pa fo a fudaetal eoy, the it will covege to a stable

5 Covegece Results 243 state withi a distace of e -/4 fo the fudaetal eoy. I paticula, whe < /(4 log ), the syste will covege to the fudaetal eoy. t Fo ay fudaetal eoy v, the axiu eegy of ay state withi a distace of pa fo v is less tha the iiu eegy of ay state at a distace of Ob fo v, ad thee ae o stable states i the auli defied by the adii pb ad e -/a ceteed at the fudaetal eoies. The followig subsectio cotais a pecise stateet of ou esults Results The followig Mai Lea gives a quatitative pictue of the eo-coectig behavio of the odel. This pictue is basically as follows. Wite a = /. Thee ae costats as ad ps such that fo a _< as the followig holds with pobability ea 1. If the syste is stated at ay state at a distace o (p <- ps) fo a fudaetal eoy (0 "eos"), the it will coect ost of the p eos i oe sychoous step, ad be at a uch salle distace p' fo the fudaetal eoy. This p' is about p3 ifp > a, ad about ap 2 ifp < a. A epeated applicatio of the sychoous opeatio will esult i a double expoetial shikage of eo, util thee ae oly about e 1/4~ eos left. Afte that, the syste will eve depat fathe tha this distace. ( I paticula, thee ae o eos left, whe a < 1/ (4 log ), i.e., we have sychoous covegece. ) The Mai Lea iplies that the eegy fuctio has o local iia i the auli defied by the adii ps ad e-1/4~ aoud ay fudaetal eoy. This will help us establish asychoous covegece. I all the followig stateets, pobability 1 - o(1) deotes pobability appoachig 1 as ~ ~. Mai Lea (Oe-Step Eo-Coectio). Thee is a as, ad fo evey a <_ as thee ae two ubes e( a) < k (a) with the followig popeties. 1. Ma) is iceasig to a costat po as a teds to O. 2. As a teds to O, ~(a) is deceasig as e 1/4, 3. The followig holds with pobability 1 - o(1): Fo all p ~ ( ~ ( a ), k (a)) ad fo all fudaetal eoies x, if y is such that d(y, x) = o, the xili(y) > 0 fo all but at ost f(p ) of the idices i whee f(p ) ca be chose as f(p) = ax{e -1/4~, c,ph(p)(a + h(p))}. Repeated applicatio of the Mai Lea yields the followig eo-coectio theoe which says that O(log(/)) sychoous steps ae sufficiet to big dow the eos to ~(a). Theoe 1 (Eo-Coectio Theoe). The followig holds with pobability 1 - o(1). Fo all a < as ad fo all fudaetal eoies x, if the syste is stated at a vecto y such that d(y, x) < k(a), the, i O(i { log( / ), log log } ) sychoous steps, it will ed up withi a distace of ~ (a).fo x ad stays withi this distace. I paticula, we establish a doai of attactio whe < /(4 log ). The adius of this doai of attactio is po. Theoe 2 (Sychoous Doai of Attactio). The followig holds with pobability 1 - o(1). If < /(4 log ), x is ay fudaetal eoy, ady is such that d(y, x) <_ po, the the syste stated i state y will covege to x withi O(log log ) sychoous steps. I fact, this O(log log ) covegece ca be cosideed vey fast. We aleady idicated that oe-step sychoous covegece is ot possible eve with O(l~a) abitay eos. The idea behid this obsevatio is the followig: Oe-step covegece would ea, fo exaple, xili(y) > 0 fo all y close to x. By chagig the jth bit, we chage the quatity xili(y) ad with 2wijxiyj, which is of the ode f. Sice x ili( x ) = O( ), by chagig appopiate cf/a bits of x, we ca ake xili(y) < O. As a esult of the eo-coectio behavio i the aulus defied by ~(a) ad k(a), we ca coclude that thee ae o stable states i this egio. Theoe 3. The followig holds with pobability l - o(1) fo all a < ao. Thee ae o stable states i the auli defied by the adii ~(a) ad k(a) aoud the fudaetal eoies. Reak: Actually, we get that thee ae ot eve local iia of the eegy fuctio i the aulus defied by k(a) ad ~(a). Covegece esults i the asychoous case, equie the existece of high eegy baies aoud the fudaetal eoies. This will esue that thee is o escape too fa away fo a fudaetal eoy. These baies togethe with the esult that thee ae o stable states uless oe gets withi ~ (a) distace fo a fudaetal eoy, esue evetual covegece to withi ~(a) distace fo the fudaetal eoies. Fist, we peset the esults elated to the high eegy baies. The followig theoe establishes uppe ad lowe bouds o the eegy of ay state i the viciity of a fudaetal eoy. Theoe 4 (Eegy Levels). The followig holds with pobability 1 - o(1). Let 0 < o <- ½ ad = a.

6 - Ee 244.~ &ot6s ad R Patu~ ff X is a fudaetal eoy ad y is such that d(y, x) = o, the whee E(y) - E(x) - 2o(1 pi ~ <- b ~ /~ = 6(a, 0) < 2[(h(p) - A/) --V2a[h(p)*A/)] of~ 0) whee A is such that A - log teds to ifiity. We say that b2 is a baie fo b~ if, fo all fudaetal eoies x. 5. PROOFS I this sectio, we peset the poot~ of ou theoes. Befoe that. we develop soe otatio ad pobability tools. Let v L. v z... v be adoly selected tudaetal eoies. The leaig ule gives us the weights Let ~., ax / E(y) : d(y, x) < b~ < i { E(y) : d(y, x) = b2 ~.. -1 Theoe 4 gives the followig eegy baie esult. which is a geealizatio of Newa's esult { 1988 ). Theoe 5. Thee exists a theshold 0t,h ad two positiw, costats bl < b2 such that. with pobability 1 o( 1 ). fo all < ath, b2 is a baie fo b~. I fact, ay pai b~ ad b2 ae good fo which 2bl(1 - bl) -~ 6(a, bl) < 2b2(1 - b2) ~(Ot, b2~ whee 6( a, p) is as give Theoe 4. To establish covegece i the asychoous case. we will select ao < 0ts such that ),(a~) is a baie fo ~(aa). We wite o~ = e(a~) ad p2 - X(0t~). (Moe pecisely, sice ~,(a~) ay be too lage, oe should choose p2 = ai { bz, ),(aa)}. ad assue that 02 is a baie fo 0 ~. ) This iplies, i the asychoous ode, that if a < a~ the ay state y withi a distace of o~ fo a fudaetal eoy x will covege to a state withi a distace of e(a) fo x. Ideed. by Theoe 3. thee ae o stable states i the aulus defied by X(0t) ad ~(0t), ad X(0t) > ),(0t~) >- t02. But the syste was stated withi p~ of the fudaetal eoy, ad, sice 02 is a baie fo o~, it caot escape fo the eighbohood with adius p2. Thus. we get the followig esult. Theoe 6 (Asychoous Covegece Theoe). The followig holds with pobability 1 - o(1) fo all < Ota. Fo all fudaetal eoies x, if y is such that d(y, x ) < pi, the the vecto y will covege to a vecto withi a distace of ~ (0t) fo x. I paticula, we get a asychoous doai of attactio whe < /(4 log ). Theoe 7. The followig holds with pobability 1 - o(1) fo all < /(4 log ). Fo all fudaetal eoies x, if y is such that d(y, x) <- ot, the the vecto y will covege to x. We defie the -diesioal vectos u = ~t';... vt') fo i (eadig the atix of the vectos v coluwise ). We ow have Q,(x) = Z (xiu', xju' ) = ~ ~,zc. Z x iu 1 I~t t whee ( u'. u j ) deotes the scala poduct of the vectos u' ad u j. I the ext subsectio, we peset cetai lage deviatio theoes. The followig subsectios cotai the poofs of ou esults Lage Deviatio Theoes Just like i Newa's pape (t988). ost of ou poofs will be based o lage deviatio theoes to boud the pobabilities i questio. Let X be a ado vaiable. The fuctio Rx(t) tx is called the oet geeatig fuctio of X. The ost ipotat popeties of Rx(t) ae listed hee. Fact 1. 1[" X ad Y ae idepedet, the Rx+y(t) = Rx(t)Ry(t). I paticula, ifx~, X2... ~', ae idepedet ad idetically distibuted (i.i.d.) ado vaiables ad S, = ~ Xi, the I = I Rs.(t) = [Rx,(t)]". Fact 2. ( Cheoff's boud). Fo c > # = EX, P(S > c) ~ [ift~oe-c'r(t)] " whee R( t) = Ee txl. This siply follows fo the followig iequality, kow as Makov's iequality: If Y is a o-egative ado vaiable, the fo ay y > 0, P(Y >- y) EY Y Let X~, )(2... X~ be idepedet -1 ado vaiable, with P(Xi = 1 ) = P( X i = - 1) = t. As befoe,

7 Covegece Results S = E X,., ad, fo a set I _~ [ ], St = E X~. ( Recall i=l i@l that [] = { 1, 2... }.) Fact 3. Ee,s"/~<- e t2/2 --~ < t < +oo 1 Fact 4. Ee ts2"/" < <- t < ½ Vl - 2t 1 Fact 5. Ee ts's'/ < - lfi~_ t 2' 1 < t < 1, whee I ad J ae disjoit sets. I the last thee iequalities, we used the followig obsevatios: if all coefficiets i the Taylo seies expasio aoud 0 of a fuctiof(x) ae o-egative, the whee ~ is stadad oal. This follows fo the wellkow iequalities S2"k < EZk. E --~- _ (We assue absolute covegece of the Taylo seies to itegable fuctios with espect to the above easues. ) I the followig, we give estiates of the scala poduct of sus of ado vectos. Let u l, u 2,..., be idepedet ad uifoly distibuted + 1 ado vectos of diesio. The fist lea estiates the o of a su of ado vectos. Lea 1. Let be a itege. P[]IZ uj][ : >- (1 + 3)] <_ e -/2)pl(6) j=l whee Pl (6) is defied as pt(3) = 3 - log(1 + 3). The fuctio p~ has the popety pt (z + ~z) > z fo z >0. Coollay 1. P[II Z uj[i2 >-(1 +2z+2~z)] j=l <e-~" Thus, with pobability 1 - e-a,fo all I, [ II = p, II Z uql 2 < (1 + 2z + 2V-zz)l/I iel whee z = h(p)/a + A/. Cosequetly, with pobability 1 - o(1), fo all x ad1, Ill = p, O < p <_ ½, I, II ~ x,uql 2 < c=p(~ + h(p)) ~ ie1 I paticula, with pobability 1 - o( 1 ),fo all x ad tl E x,u'll ~ < c+ ~ i~l Poof: Let A be the evet II E u j II 2 ~ (1 + 6). j=l A iplies that, fo all t > 0, et/lle;-tuql = > ett[++) By usig Makov's iequality, we get that P(A ) < e-++)ee udlxv''uql:. Note that i the sus (Eu~),=u)+u uf d=l the tes ae idepedet ad uifoly distibuted _+1 ado vaiables, ad diffeet sus ae idepedet of each othe. Hece we get that P(A) < e-'<l++)"(ee'/"("',... ~)~)". By usig the oet geeatig fuctio iequality Fact 4, we get that P(A) _< e -+~) (Vl - 2t) -- e-/2[2t(l+6)+log(l-2t)] The expoet i the above expessio achieves its 6 iiu i the age [0, ½) whe t (1 + 6) Hece, the lea follows. (The popety of the fuctio Pl etioed i the lea is stadad calculus. ) The fist pat of the coollay follows fo the lea ad the fact that the ube of sets I to coside is a < e h(p)17. (This facto akes the diffeece p betwee ado eos ad wost case eos.) I the secod pat, we have to ultiply by a additioal facto 2 p fo the choice ofxi. Next, the followig lea estiates the scala poduct of two vecto sus. Lea 2. Let ad s be iteges. The, +s P((~ u i, ~ i=1 j=+l whee P2 (6) is give by u j)> b~s) < e -/zp~<+) ( vl ) p2(b) = (Vl ) + log 232

8 246 / ~,ol6s ad R t~atu~ The fuctio p~ has the popety p2( 2 + Vz) > zjo z >0. Coollay 2. P(( ~, u ~, ~ +s u ~) >- (z + ~z)v~s) < e-:" i=1 j=+l Thus, with pobability l - e -a, fo all pais of disjoit sets I, J, I II = o', I JI -- o", (~ u', ~ u ~) < (z + l~z)~ll IJI i~l j~j whee z = h(p', p")/a + A/. Cosequetly, with pobability 1 - o(1), fo all x ad fo all pais of disjoit sets I, J, [ I[ = p', I Jl =p",o<p',p" <_ ½, t( Z u ~, Z u')l < c~7~o'p"h(o', p")(,~ + h(t;, t;')) 2. i~l jgj Poof: Let A deote the evet (~ u', Z u J) i=l j~-+[ > 6f-s. A iplies that, fo all t > 0. et/77j~s(x,~., u,, Zj-,+~,*, e) > (?ta. ~.s' P(A)_< P(qK, IK] :o, l, il! : ~,'~ such that vi 6_- I. x~l(y) ~ O~ Note that vi E I, x,l,( y) ~ 0 plies L' :v,l,(y) _< 0. But,.viLi(.V) = ~ xili(x) -- z.. ~" -vi(l~(. ' ) I.Ay)t IU t lg I 1.1 tp [ I~ l [P& K Fo the sake of coveiece, let the fudaetal vecto x = v ~. Hece, the fist copoets of all the vectos x~u' ae t. Let 6' deote the - l-diesioal vecto obtaied fo x, u' afte eovig the fist copoet. Clealy, t7 ae idepedet ad uifoly distibuted _1 ado vectos of diesio 1 (fo siplicity, we will ot chage to -- 1 i the foulas ). We ow have i~ 1 t& I j~[] A'jU J ) - i 1 By Makov's iequality, we have, P(A) < e-taee ~/{s(~'' u'. 27-~,,el ;E l ]~i[]- I Xjlt 3 ) + ~ ~" Xtlti) 2 -- l[ ;+: ;' By usig Fact 5, we get = e_~a(eej'ia(~,'., u'. 27-~, ~)),,. P(A) -< e -ta- 1 _ : e_/212t~+log(t_t~)] (VI t2) The expoet i the above expessio will be if]- + 4~ 2-1 ial i the age [0, l)whet = 2b. Hece, the lea follows. Cobiig Coollay 1 ad Coollay 2, we get Coollay 3. With pobability 1 - o(1 ), jo all x ad fo all pais of (ot ecessaily disjoit) sets I, J, II1 t : p, IJ[ = p ", O<p'-p" < <-- ), t l(z u', Z d)l i~i j~j < c~[vp'p"h(p")(a + h(p")) + o'(a + h(p'))l Poofs of Ou Results Poof of Mai Lea. Fo a give fudaetal vecto x, we will copute the pobability that thee exists a y such that d(y, x) = o ad that thee ae oe tha f(p) idices i such that x~li(y) < O. Let A be this evet. Let Kbe the set of idices i which x ad y diffe ad let IKI -- o. Letp' =f(p). Clealy, we have = ( ) II,- T~ ~- 12. Thus. Z xili(y)=( ) It + TI + '1~- 21"3 ig l +2 IV1KIwhee iei l~[[- ] T: = (~ t~'~z: ad t,, ) x,u', ~ x,u l. We ow estiate fo below each of the above tes idividually. Fo Coollay 2, we get Tl > -{zl- ;(-l V~zl)VIIl( -- I/I) with pobability 1 - e -a, whee z~ = h(p')/a + A/. (We used the fact that the ie poduct has a syetic distibutio. ) T2 is obviously o-egative. To estiate T3, we use Coollay 3. #~ A T3 < c~[vp'ph(p)(a + h(p)) + v'(a ~ h(p'))]2 Hece. if - 2c~[Vp'ph(p)(a + h(pd, + o'(a + h(p'))] > 0 the the pobability P (] a fudaetal eoy x a such that the evet A holds) is uppe bouded by e. This pobability is sall if A - log is lage. It is ot had to see that

9 Covegece Results p' :f(p) = ax{e -~/4", clph(p)(a + h(p))} will satisfy the last iequality. (The oly case that eeds caeful aalysis is whe p' = 1/.) We defie the fuctio (o') = e-~/4". It is easy to see that thee ae positive O's ad p~ such thatf(p) < p fo ~ (O') < p <- ps, O" < O'o. Choose a iceasig ~ ( O" ), p~ _< X(o') < ½ such thatf(p) < p fo ~(o') < p < X(o'), O'~ O' s. Refieets: A little oe caeful aalysis of the above equatio (sepaatig the case p' < k fo o' > k) shows the followig oe detailed pictue: Whe k+l <-- k+221og the ube of eos left is at ost k. At the othe ed, whe h (p) > a, oe gets a bette boud by estiatig T3 with the poduct of the os of the two factos i the scala poduct. We get T3 < cvpp'(a + h(p))(a + h(p')) z that leads to the followig oe detailed eo coectio p' < e -c/(ph(p)) as log as h(p') > O" holds. Afte this icedibly fast eo coectio, the followig slowe (but still double expoetial) fuctio takes ove: p' = ax{e -1/4", caph(p)} Poof of Theoe 1: The theoe easily follows fo the Mai Lea ad the defiitios of M O') ad ~(o'). Poof of Theoe 2: The Mai Lea shows that whe a < 1/(4 log ), ~(a) < 1, that is, thee ae o eos left. The adius of attactio is po = ~(o'o). Poof of Theoe 4: Fo the sake of coveiece, assue that the fudaetal eoy x = v j. Let y be such that d(y, x) = o ad let Kbe the set of coodiates i which y ad x diffe. It is easy to see that E(y) - E(x) = 2( ~ XkU k, ~ xkuk). ke K kei~ Sice x = v 1, we get E(y) - E(x) = 2p(1 - p) 2 + 2( ~ ilk, ~ ak) ke K whee ~7' is obtaied fo xiu i by eovig the fist copoet. Note that the OLs ae idepedet ad uifoly distibuted 1 ado vectos of diesio -1. Fo Coollay 2 we get k~k [E(y) - E(x) - 2p(1 - p)21 <2(z+ 247 V~z)VIKJ( - IKJ) with pobability 1 - e -~, whee z -- h(p)/a + A/. Choose agai a A uch beyod log. (The exact equatio is ]E(y) - E(x) - 2p(1 - p)2] < 2abVp(l - p) 2 with pobability < e -"/2p2 ~), which is equivalet to Newa's equatio. ) 6. EXTRANEOUS MEMORIES I this sectio, we will establish the existece of a expoetial ube of stable states, ad exted soe of ou pevious esults to these stable states Stability Note that ou covegece poofs wee based o the fact that the gadiet at the fudaetal eoies was lage. Moe pecisely, all Qi ae lage whe < c/log, ad still ost Qi ae lage whe = c. We obseve that this popety is sufficiet to exted ou poofs. Hece, we itoduce the otio off-stable vectos. Give 0 < fl < 1, a vecto x is f-stable ifqi(x) >- fi fo all i. Note that a f-stable vecto is ot oly stable but also a deep eegy iiu. I fact, f-stable vectos have a lage doai of attactio. Futheoe, all the fudaetal vectos ae f-stable whe < c/log. Whe = O' fo soe costat O', we caot establish the existece off-stable vectos. A weake otio of stability (valid fo all fudaetal eoies)will be itoduced, ad used to deive soe eo-coectig popeties. Fo 0 </3 < 1 ad 0 _-< ~ < 1, we defie that a vecto is (/3, E)-stable if fo all but at ost ~ idices i, Qi(x) > fi. Eve though ([3, ~)-stable vectos ae ot stable theselves, we will late see that thee ae stable states i thei close viciity. Theoe 8. Let 0 < 13 < 1. The, thee ex&ts Co = Co(13) such that, with pobability 1 - o(1), if < co~log, the all fudaetal eoies ae/3- stable. (co ca be ade abitay close to ~ by choosig a sall eough/3. ) Theoe 9. Let 0 < f < 1 ad ~ > O. The thee exists ao = O'0(/3, ~) such that, with pobability 1 - o(1),/f <- O'o, the, all fudaetal eoies ae (f, ~)- stable. I additio to the fudaetal eoies, thee ae a expoetial ube of othe stable vectos. Soe ae thee accidetally (tue extaeous eoies), but

10 248 c'" of the ae thee fo a easo. The whole odel is based o a liea ethod, so oe is ot supised to see that it eebes ot oly idividual vectos, but the whole subspace geeated by the. (This has bee obseved by seveal eseaches befoe. ) Give vectos v ', v 2,..., we defie S(v 1, v 2, ") = {sig(~ div;) : di = +_I }. Theoe 10. Fo evey positive ~, thee & a a* = a * ( ~ ) such that, with pobability 1 - o(1), if <_ a*, the oe tha half of the 2 "liea cobiatios" of the fudaetal eoies (i.e., eleets of S( v~... v,,)) ae (0.5, ~)-stable. Fo the poof of this theoe, we eed the followig siple lea. Lea 3. Let Yi,j be idepedet ad uifoly distibuted + 1 ado vaiables. The, e ~ i=l I~2,.jI <d!~- I <(4V-d) t'. /=I Poof: Let Yi = I Z Yi,/I, ad wite D = df - 1. j= I N Thee exist oe tha ~- i's such that Y; < 2D. Thus, the pobability i questio is bouded by 2N[P(Yi < 2D)] ~v/2 sv/2 ] Poof of Theoe 10: We show fist that fo ay specific choice of the coefficiets d;, the vecto x = sig( Y~ d;vi) is (0.5, O-stable with pobability l i=l - o(l). Give ay specific d~, we ca eplace the vectos v' with the vectos Nil) i sice the scala poducts (W, u k ) ae ivaiat ude the sig chages of the vectos v'. Hece, without loss of geeality, we take d; = 1 fo all i. Let x = sig( E v;). Wite ~i = x~u; = sig( Z i=l X u~) u'. ~ is obtaied by flippig ove the copoets of u; if u; has oe -l's tha +l's. Othewise, W equals u ;. Let I deote the set of idices i such that (x,u', E xju j) < 0.5, ad let us wite ~ = [ II/. The, we have (~x~u i, ~ a ~)- ill <0.5llI. i~l j~l j=l! Ac,t6.s a R. Palu: The vectos if' wilt be witte as u' + 2z: whee z: is a (0, 1)-vecto defied as follows. Let d = (1, u ~) whee 1 is a -diesioal vecto all of whose copoets ae equal to 1. Ifd ~ 0, the z' is a 0 vecto. Othewise, we will adoly select { d! of the idices j whee u ~ is - 1, ad set z; to ~ fo these j ad 0 elsewhee. It is clea that zt; ad u ~ + 2z; have the sae distibutio. Futheoe. the pobability p that a copoet of z' is l. is appoxiately V We ow have ";'" ('t' + -~")) t~ ] = I tel : We estiate the two tes S, = (>2 xiu', ~.'," ~ t~l i:: I &, = (~x;u', E :) IG 1 f 1 sepaately. We estiate Sz i the followig way. & = (Z x;u', E Z z ~) 16- /,, = 1 + (~ x;u', ~ (z" "- Ez~)) = Szl + $22. iel /=1 Sice x,u' is the flipped ove vesio of u;; it follows that s2, = p Z (x,u', 1) = p Z t E u;i. ie l i~:: l i=! It follows fo Lea 3 that, with pobability 1-0(1), fo all 1II, S2j > ce2li]: = ce3 2, We also have that I Sz,.I < lie x,u'lt lie (z'--ezj)lt ie1 j~-i EIIZ (z j- EzJ)l/2 = X E[X (~- Ez~)] 2 j=l k=l j=t = Z ZE(~-Eza*) 2_< ~E(z 2=p. k=t t=l ~,=1 i=1 tt Thus, sice p --~ 0, 11E (z j - Ez:)H= = o(:) with j=l pobability 1 - o(1). Fo this ad fo Coollay 1, we get that $22 ad o( 2) with pobability 1 ~ o(1): To estiate St, we use the iequality I&l ~ IIZ x~uql ~Zu~t[. Agai, fo Coollay 1, we get tha~ with pobanlity 1 2 o(1), foallxadfoall iil, ts, I <-e~-"-fly- = C~GH 2.

11 Covegece Results Copaig S~ ad S2j, we get that as log as a < c"e 6, the ube of idices i fo which Q~(x) < 0.5, is at ost C. Hece, with pobability 1 - o(1), x is a (0.5, e)- stable vecto. It follows that, with pobability 1 - o(1), ost of the 2 "liea cobiatios" of the fudaetal vectos ae (0.5, E)-stable which poves the theoe. Reak: I fact, the poof shows that the above set of liea cobiatios ca be exteded fo + 1 liea cobiatios to all liea cobiatios, that is, to the set { sig( Z divi) } with eal coefficiets di. This would i ipove the lowe boud c to e ~i{2'} Covegece I this subsectio, we show that the stability itoduced above guaatees covegece popeties siila to those of the fudaetal eoies. Theoe 11. The Mai Lea, ad Theoes 1-7 eai valid if we eplace fudaetal eoies by 13-stable vectos, but the ueical quatities ivolved chage as follows: The.fuctio f(p ) i the Mai Lea is eplaced by.f(p) = c2(/3)ph(p)(ee + h(p)) (i.e., ~(c~) becoes 0). (O, as we eaked afte the poof of the Mai Lea, oe ca take fo f(p ) the salle of the two quatities ax{a, e -(/(ph(p))} ad c(/3)ph(p)(e~ + h(p)).) Thus, Theoes 2 ad 7 (sychoous ad asychoous covegece) hold eve if = a with a costat a. The coditio = 0 ( / log ) is ot ecessay ay oe sice we assued that x is ~3-stable. All costats ivolved i these theoes will deped o/3. The boud i Theoe 4 chages to I E(y) - E(x)[ _< cvo(a + h(p)) 2. Ideed, the oly te that chages i the poof of the Mai Lea is T~, but this is ow assued to be geate tha/3[ I[. The othe theoes ae coollaies. The poof of the aalogue of Theoe 4 follows the sae patte. We stat with the idetity E(y) - E(x) = 2( ~ XkU k, ~ XkU k) k~ K ke_i(" ad the estiate the scala poduct by the legths of the vectos usig Coollay 1. The followig theoe follows fo the above ad the defiitio of (/3, )-stable vectos. Theoe 12. The Mai Lea, ad Theoes 1, 3, 4, 5, 6 eai valid if we eplace fudaetal eoies by (/3, ~ ) -stable vectos, c ( a ) by ~, ad f( p ) by f( p ) + E. 249 Coollay 4. If x is a (/3, e )-stable vecto, the thee is a stable state withi a distace of~.fo x. Coollay 5. Thee ae costats aexp > 0 ad c > 1 such that, with pobability 1 - o(1), (I' <- OCexp, the the ube of stable states is oe tha c. Ideed, it is easy to see that fo a fixed ~ < ~, the pobability that two of the vectos i S(v 1, v 2, ) ae at a distace 2~ o less, is expoetially sall (i ). Statig with the 2-1 vectos etioed i Theoe 10, ad usig a geedy algoith, oe ca select c of these vectos such that ay two of the ae at a distace lage tha 2~. Fo each of the, select a stable vecto withi a distace ~. Reak: If oe is satisfied with a expoetial covegece (tie log ) istead of double expoetial covegece (tie log log ), the the followig uch siple poof could be give fo the covegece to/3- stable vectos. It is based o a lea that establishes a Lipschitz type popety fo the gadiet of the eegy fuctio. Lea 4 Gadiet Lea. Let 0 < [3 < 1. The thee exist positive so ([3), po (~) such that the followig holds with pobability 1 - o(1)fo all a <_ c~o(/3), p <- po(/3). Fo all x ad y, if d(y, x) <_ p, the the ube of i such that I Li(x) - Li(y) l >- fi is at ost O' whee o' = c(/3)p(e~ + h(o)) < p/2 Poof: Ideed, if K deotes the set of idices whee x ad y diffe, the LI(x)-Li(y) = 2(u i, ~ XkU k) Thus, if I stads fo the set whee the, by Coollay 1, k~k Li(x) - Li(y) >-- [3 II]S < ~ (Li(x) - Li(y)) ie1 =2(~u ie1 i, ~ XkUk)--2]INK] k~k -< 211 ~ u'll fl Z xkukll i~: l k~ K <-- 2Vc2p(o~ + h(p))vczp'(c~ + h(p')) 2 < p'[3 2 (a cotadictio) if p'> (c/132)p(c~ + h(p)) < 0/2 assuig P ad a ae sall i tes of 13. Thus, ifx is /3-stable, the it has the eo coectig popety of the Mai Lea with p' < c(/3)o(a + h(o)). Ifx is (/3, i

12 250./ )~h),~ ad R. Pat~ )-stable, the it also has the eo coectig popety, but with P' above eplaced by p' + e. REFERENCES Ait, D. J., Gutfeud, G., & Sopolisky, H. (1985). Spi-glass odels of eual etwoks. Physical Review A, 32, O18. Ait, D. J., Gutfeud, G., & Sopolisky, H. (1987). Statistical echaics of eual etwoks ea satuatio. Aals of Physics. 173, Cohe, M. A., & Gossbeg, S. (1983). Absolute stability of global patte foatio ad paallel eoy stoage by copetitive eual etwoks. IEEE Tasactios o Systes, Ma, ad Cybeetics, 13, Gade, E. (1986). Stuctue of etastable states i the Hopfield odel. Joual of Physics A, 19, L 1047-L Gossbeg, S. (1982). Studies of id ad bai. Bosto: Reidel. Hebb, D. O. (1949). The ogaizatio of behavio. New Yok: Wiley. Hito, G. E., & Adeso, J. A. (Eds.). (1981). Paallel odels' ~i/ associative eoy. Hillsdale, N J: Elbau. Hopfield, J. J. (1982). Neual etwoks ad physical systes with eeget collective coputatioal abilities. Poceedigs of the Natioal Acadey of Scieces of the Uited States of Aeica, 79, Kikpatick. S.. & Sheigto, D. (1978 I. Ifiite-aged odels ~ spi-glasses. PhysicalReview B. 17, Kohoe. T. (1984). Self-ogaizatio ad a ~sociative eoo; Ne,~ Yok: Spige-Vetag. Kohoe. T. ( 1987 ), State of the at i eual coputig. 1EI E F,'~ Iteatioal Co['eece o Neual Net~ks L Lashley, Y. S. (1960). The euophysiotogy ~:![ Lashle); Selected pape~. ~/K S. Lashley. New Yok: McGaw-Hill Little. W. A (1974). 1"he existece of pesistet states i the bai Matheatical Bioscieces 19. I 01 - l 1Q Little. W. A.. & Shaw. G. L. (1978). Aalyuc stu~) of the eoy stoage capacity of a eual etwok. Matheatical Bioscieces 39, McCulloch. W. S.. & Pitts, W. H. ~ A logical calculus fo ideas iaet i evous activity. Bulleti ~l Matheatical Biophys~ l~ McEliece. R. J.. Pose. E. C.. Rodeich. E R.. & Vekatesh. S. ~ ~. The capacity of the Hopfield associative eoy. IEEE l)asactios o Ifoatio Fheo): Motgoey, B. L.. & Vijaya Kua. B. V. K f 1986 ~. Evaluatio of the use of the Hopfield eual etwok odel as a eaest-eighbo algoith. Applied Optics, Newa. C. M. (1988/. Meoy capacity i eual etwok odels: Rigoous lowe bouds. Neual Netwoks. I