Stationary processes and equilibrium states in non-symmetric neural networks

Transcription

1 IVESTIGACIÓ REVISTA MEXICAA DE FÍSICA 48 (4) AGOSTO 2002 Statonary processes an equlbrum states n non-symmetrc neural networks A. Castellanos Departamento e Físca, Unversa e Sonora Apo. Post. 1626, Hermosllo 83000, Son. Méxco. acastell@fsca.uson.mx L. Vana Depto. e Físca Teórca, CCMC - UAM Apo. Post. 2681, Ensenaa B.C., Méxco. laura@ccmc.unam.mx Recbo el 9 e novembre e 2001; aceptao el 4 e abrl e 2002 Statonary processes an equlbrum states are scusse n fnte separable recurrent neural networks sequental ynamcs an away from saturaton. We escrbe thermal fluctuatons of the ynamcal orer parameters orgnate as fnte sze effects of orer O 1/2 by means of ther corresponng Fokker-Planck equaton, an fn ther tme epenent probablty strbuton. We ntrouce the concept of extene entropy of fluctuatons n orer to fn a general conton to characterze statonary states n eural etworks non symmetrc nteractons. Dvergence an rotatonal of the probablty current n the space of fluctuatons are also use to fferentate between statonary an equlbrum states. Beses, algebrac contons are foun to know when statonary states can exst. The results are llustrate by analyzng a neural network a macroscopc ynamcal fxe pont but not satsfyng etale balance at mcroscopc level. Keywors: Statstcal physcs; thermoynamcs an nonlnear ynamcal systems; neural networks; fuzzy logc; artfcal ntellgence; stochastc processes; probablty theory; stochastc processes an statstcs. Se scuten los procesos estaconaros y los estaos e equlbro en rees neuronales recurrentes, fntas y separables, con námca secuencal y muy lejos e la saturacón. Por meo e la corresponente ecuacón e Fokker-Planck, escrbmos las fluctuacones térmcas e la námca e los parámetros e oren, orgnaas como efectos e tamaño fnto e oren ( 1/2 ) y encontramos la strbucón e probabla epenente el tempo. Introucmos el concepto e entropía extena e las fluctuacones para encontrar una concón general que caracterce los estaos estaconaros en Rees euronales con nteraccones no smétrcas. Tambén se utlzan la vergenca y el rotaconal e la corrente e probabla en el espaco e fluctuacones para ferencar entre estaos estaconaros y e equlbro. Aemás, se encuentran las concones algebracas para saber cuano pueen exstr los estaos estaconaros. Los resultaos son lustraos meante el análss e una re neuronal con un punto fjo en la námca macroscópca pero que no satsface el balance etallao a nvel mcroscópco. Descrptores: Físca estaístca; termonámca y sstemas námcos no lneales; rees neuronales; lógca fusa; ntelgenca artfcal; procesos estocástcos; teoría e la probabla; procesos estocástcos y estaístca. PACS: 05.; Mh; Ey; r 1. Introucton Early works n statstcal physcs of eural etworks () were concentrate n the mean fel stuy of equlbrum propertes of nfnte systems symmetrc nteractons (for a revew see Ref. 1). However, one of the basc characterstcs of s ther hgh number of barrers an valleys n the energy space, whch makes the stuy of the ynamcs essental. In the case of non symmetrc nteractons, ynamcal technques are n fact the only tool avalable, as n ths kn of systems etale balance oes not hol an the usual statstcal concepts such as free energy cannot even be efne. On the other han, moels of the mean fel type, have prectons val for nfntely large systems ( ). Takng the thermoynamcal lmt has prove to be a useful approxmaton to stuy gases, lqus an sols, where the number of elements s of the orer of ; however, the ncluson of fnte sze effects s clearly mportant when conserng fnte systems lke the bran havng neurons workng n small groups whose sze ranges from a few thousans to 10 5 neurons. These fnte sze effects have been shown to play an mportant role n both symmetrc an non symmetrc 2, 3], even affectng, n some partcular cases, the macroscopc behavour of some systems 4]. Although conserng fnte sze effects an non symmetrc nteractons s not enough to obtan a goo escrpton of bologcal, t s clearly a frst an mportant step n ths recton; other characterstcs shoul be nclue nto the moels (see Ref. 5 for a goo revew) such as spatal structure 6 9, 5], as well as short an long-range connectvty 10, 11]. However these conseratons are out of the scope of ths paper. Untl now, most work n asymmetrc neural networks has been concentrate n the stuy of sequental retreval of embee patterns 12], an n the storage capacty of those networks 13], for nfnte systems. We beleve the stuy of the stochastc behavour resultng as a fnte sze effect n both symmetrc an non symmetrc wll contrbute towars a better unerstanng of fferent aspects of the ynamcs of an therefore t eserves specal attenton.

2 STATIOARYPROCESSESADEQUILIBRIUMSTATESIO-SYMMETRICEURALETWORKS 311 The objectve of ths work s to stuy the fference between statonary an equlbrum states from the analyss of the fluctuatons orgnate as fnte sze effects. Our formalsm escrbes the ynamcs of to frst non trval orer n the system sze 15, 16], by separatng systematcally the = from the fnte contrbutons n the Kramers- Moyal expanson 17] an keepng the frst non trval orer n (1/). In ths way t s obtane a Fokker-Planck Equaton (FPE) whch can be wrtten n terms of the fluctuatons of the ynamcal orer parameters, from whose soluton t can be constructe the probablty strbuton for these fluctuatons. Ths formalsm oes not requre symmetry of the nteracton matrx, as oppose to other methos 14]; so ynamcal flow agrams for ether symmetrc or non-symmetrc nteractons can be obtane by solvng numercally the etermnstc mean fel equatons for the fluctuaton moments. In the partcular case of symmetrc nteractons the approach to equlbrum s warrante by knowng Lyapunov functons. We know that any system n thermoynamcal equlbrum must obey etale balance. However, non symmetrc nteractons can never obey etale balance; ths s reflecte on the behavour as the lack of compensaton between postve an negatve ranom fluctuatons n the orer parameters, n a way that nuces a rotaton of the probablty current n the space of the fluctuaton varables. In ths sense, the vergence an rotatonal of the probablty current are useful tools to characterze the statonary an equlbrum states. In aton to ths geometrc pcture, we obtan the algebrac contons uner whch statonary states can exst. We also ntrouce the concept of extene entropy à la Boltzmann, for asymmetrc, to characterze the statonary states. The lmtng (maxmum) value of ths functon allows us to obtan a conton that has to be satsfe by a n a statonary state; we fn out that ths conton s met, n partcular, by any symmetrc n equlbrum, so we can see the equlbrum state as a partcular case of a statonary process. These results are llustrate by conserng a pecular system prevously analyze: a a macroscopc fxe pont but unable to satsfy etale balance at the mcroscopc level 16]. The paper s organze as follows: n Sec. 2, t s presente an overvew of the general formalsm (for etals see Refs. 15, 16); n Sec. 3, t s ntrouce the concept of extene entropy n orer to scuss statstcal equlbrum an statonary processes; n Sec. 4, the lack of etale balance an ts relaton to the rotaton of the probablty current are examne to obtan a geometrc pcture of ths phenomenon an the algebrac contons for the exstence of statonary states; fnally, n Sec. 5, t s scusse a a fxe pont but out etale balance as an example to llustrate the prevous sectons. Our work can be apple to stuy fnte sze effects n any kn of fnte, separable recurrent sequental ynamcs an away from saturaton, prove that transference functons are sgmoal. Ths nclues moels ether symmetrc or asymmetrc nteractons, an even moels where synapses are state epenent 18, 19]. The analyss of synchronous upatng 19, 20] s outse the lmts of ths theory. 2. Fnte sze effects. General formalsm Let us conser a system compose by a large, but fnte, number of nterconnecte neurons σ { 1, 1}, so the vector σ(t) = (σ 1 (t), σ 2 (t),..., σ (t)) escrbes the state of the system at a gven tme. Each varable σ (t) evolves n tme by escrbng sequental stochastc algnment of the spns uner the acton of an external fel gven by J j = 1 h (σ) = j J j σ j +θ, p ξ µ A µν ξj ν (1 δ j ), µν=1 J j s the (separable) synaptc strength of the connecton gong from neuron j to neuron, θ s a response threshol, an the auto nteractons J are exclue. These nteractons store a fnte number p of quenche bnary patterns {ξ µ } chosen at ranom µ = 1,..., p an the notaton ξ = (ξ 1,..., ξp ) { 1, 1}p, or alternatvely, ξ µ = (ξ µ 1,..., ξµ ) { 1, 1}. An, as we can see, nteractons o not nee to be symmetrcal. The mcro-ynamcs of the system s efne by a master equaton for the probablty ensty p t (σ), gven by t p t(σ) = {w (F σ)p t (F σ) w (σ)p t (σ)}, (1) w (σ) = σ tanh(βh (σ))], where w (σ) s the temperature (T = β 1 ) epenent transton probablty σ (t) σ (t), an F s an operator that flps the th spn. If we efne the probablty ensty for the pattern overlaps m as P t (m) σ p t(σ) δ m m(σ)] the usual efnton for the overlaps m(σ) = (m 1 (σ),..., m p (σ)), m µ = 1 ξ µ σ the master Eq. (1) can be rewrtten n terms of the macroscopc ynamcal varables m(σ) by nsertng δm m(σ)] nto Eq. (1), an makng a Taylor expanson n powers of the vector 2σ ξ /. If we expan the resultng Kramers-Moyal equaton n powers of (1/) an keep the two leang orers 23], we obtan a FPE val on fnte tme-scales (.e. not scalng ) for the stochastc vector m(σ). We can now wrte m(σ) as the sum of a etermnstc part m (t) whch correspons to the nfnte system, an the leang orer stochastc contrbuton q(t)/ vanshng as m = m (t) + 1 q(t) +...

3 312 A.CASTELLAOSADL.VIAA we can now separate the ynamcal equatons for these two orer parameters. In ths way, m (t) s the etermnstc soluton to the Louvlle equaton gven by F µ q; t] = K µ m (t)] + ν L µν m (t)]q ν (t), (4) t m (t) = ξ tanh β ξ Am (t)+θ] ξ,θ m (t), (2) where we efne ξ,θ as an average over an ensemble of characterze by fferent realzatons of patterns. We can now obtan a FPE for the remanng stochastc part q(σ) resultng from the fnte sze effects as follows: t P t(q) = µ q µ {P t (q) F µ q; t]} + µν 2 q µ q ν {P t (q) D µν q; t]}. (3) In ths equaton the flow term F µ q; t] s gven by { K µ x] = lm ξ µ tanh βξ Ax+θ] ξ,θ } 1 ξ µ tanh β ξ Ax+θ ], (5) L µν x] = δ µν β λ ξ µ ξ λ 1 tanh 2 βξ Ax+θ] ] ξ,θ A λν. (6) It s mportant to notce that K µ (t) makes a frozen correcton to the flux, that perssts even at T = 0 an epens on the specfc pattern confguraton {ξ µ }. On the other han, the ffuson term D µν t] of the FPE, Eq. (3), s foun to be symmetrc, an gven by D µν t] = δ µν e t lm 1 ξ µ ξν σ (0) tanh βξ Am(t)+θ ] t 0 s e s t ξ µ ξ ν tanh βξ Am (s)+θ] tanh βξ Am (t)+θ] ξ,θ. (7) It s mportant to notce that the flux s lnear n q(t) an the ffuson term s nepenent from q(t), therefore the process can be entfe as a tme epenent Ornsten-Uhlenbeck process, whose formal soluton s Gaussan: P t (q) = exp { 1 2 q q t] Ξ 1 (t) q q t ] } (8) (2π) p 2 et Ξ (t) Ξ, the tme epenent correlaton matrx for the fluctuatons Ξ µν = q µ q ν t q µ t q ν t, where < > t s the average at tme t over fferent evolutons, for a gven pattern confguraton {ξ µ }. Therefore, the process can be fully characterze by ts two frst statstcal moments, whch are foun to evolve n tme accorng to the followng etermnstc equatons t q t + F µ q; t] t = 0 (9) t Ξ(t) = L(t)Ξ(t) Ξ(t)LT (t) + 2D(t). (10) In orer to arrve to explct expressons for the flux n Eq. (4) an fusson n Eq. (7) terms, for any partcular system consere, we nee to choose nepenently rawn unbase pattern components ξ µ { 1, 1} ( equal probabltes) an nepenently rawn threshols θ (from some probablty strbuton W (θ)). Afterwars, by solvng equatons (9) an (10) n terms of the flow an fusson we can construct the probablty strbuton for the fluctuatons of the ynamcal orer parameters (8) aroun mean fel trajectores gven by Eq. (2). These stochastc fluctuatons epen on the actual realzaton of store patterns, through the frozen correcton term n Eq. (5) an ther sze s of orer O ( 1/2). 3. Statstcal equlbrum an statonary processes A physcal system s sa to be n a statonary state when the correlatons of the stochastc moments of q µ, become nepenent of tme translatons, that s f q µ (t 1 + τ) q µ (t 2 + τ)...q µ (t n + τ) = q µ (t 1 ) q µ (t 2 )...q µ (t n ) for any tme τ > 0, an µ = 1,...p 21]; ths conton s satsfe n partcular by any system n thermoynamcal equlbrum. However, thermoynamcal equlbrum requres atonally the presence of etale balance, whch means that at mcroscopcal level the probablty for a rect process

4 STATIOARYPROCESSESADEQUILIBRIUMSTATESIO-SYMMETRICEURALETWORKS 313 s equal to the probablty of the reverse process, so there s no net probablty current between any two mcroscopc states of the system. The equlbrum state has strong stablty propertes: any perturbaton actng on t wll e out fast; ths behavour s reflecte n the secon law of thermoynamcs, accorng to whch equlbrum s reache n an rreversble fashon an correspons to a maxmum entropy at constant energy, or mnmum free energy at constant temperature. However, all ths scheme s no longer useful for systems non-symmetrc nteractons, as n ths case t s not possble to efne an energy functon an etale balance no longer hols, so t s not possble to talk about equlbrum. However, t s observe that these kn of systems stll evolves from less probable to more probable states, an ths evoluton leas them towars more prvlege parts of the space of states. Therefore, for a system not satsfyng etale balance we can efne an extene entropy 21] as follows: S(t) = H(t) H(t) qp t (q) lnp t (q) (11) ( Boltzmann s constant k B 1). Ths extene entropy s an extensve quantty, whch, accorng to the H-theorem, ncreases monotoncally untl t reaches ts maxmum value, prove t exsts a statonary state; an ths result s val for any system satsfyng, or not, etale balance. The contrbuton to the extene entropy, ue to fnte sze fluctuatons, at tme t s obtane by substtutng (8) nto the prevous expresson (11). In ths way we fn S q (t) = ln P t (q) q;t = 1 ln {et Ξ (t)]} + C, 2 where C s a constant. Therefore, the ncrease n entropy between two nstants t 1 an t 2 such that t 1 < t 2, s gven by S = ln { } 1 et Ξ (t2 2 )] 0, et Ξ (t 1 )] whch mples et Ξ (t 2 )] et Ξ (t 1 )]. If we combne the Wronsk entty t ln {et B]} = T r B 1 t B] Eq. (10) escrbng the tme evoluton of the secon moments, we fn t {ln et Ξ (t)]} = 2 Tr D Ξ 1 L ], whch mples that n any relaxaton process t s satsfe Tr D Ξ 1] Tr L], where the equal sgn apples when the system reaches a statonary state. Therefore, Boltzmann s H-theorem tells us that f a statonary state exsts, t satsfes the expresson T r D Ξ 1] = T r L]. We can compare ths result to the stronger conton D Ξ 1 = L, satsfe by symmetrc systems n an equlbrum state (therefore, satsfyng etale balance) 16]. 4. on etale balance, rotatng probablty current an the exstence of statonary states In ths secton t s gven another way to characterze the fference between statonary an equlbrum states, n the framework of. We also fn the contons neee for the exstence of a statonary state. The frst objectve s reache f we analyze the probablty current n the space of fluctuaton varables by rewrtng the FPE as a contnuty equaton for the probablty flux J t (q) as follows: t P t(q) + J t (q) = 0, J t (q) = J mec t (q) + J s t (q), (12) where we have separate the probablty flux n two terms, one relate to the flux, an the other to the ffuson; for an Ornsten-Uhlenbeck process, lke ths one, these are gven by J mec t (q) = P t (q) F q, t], J s t (q) = P t (q)dξ 1 q(t) q(t) ]. (13) Van Kampen 22] has note that the frst term, resultng from the presence of an external fel, can be calle purely mechancal as t has the characterstcs of a Louvlle flux, whle the secon term s purely ffusve as t s relate to an rreversble ffusve process. From expresson n Eq. (12), t shoul be clear that when the vergence of the total probablty current J t (q) = J mec t (q) + J s t (q) s zero the system s n a statonary state. If we make q /t = 0, Ξ/t = 0 n Eqs. (9-10) for long tmes, we obtan F q] = 0 = q = L 1 K (14) L Ξ + Ξ L = 2D (15) The frst of Eqs. (14) shows that, n general, the mechancal probablty current vanshes. On the other han, Eq. (15) s known as the fluctuaton-sspaton theorem (FDT), an t shows that the correlaton Ξ µν may be n a statonary state as a consequence of the compensaton between ampng, represente by the convecton matrx L (6) an ffuson, represente by D (7). In ths lmt, we also have L = I β D A. Ths state s an equlbrum state f, atonally, the probablty current s rrotatonal, that s, f J µ /q ν = J ν /q µ, for all µ, ν. But ths happens only for A µν = A νµ, whch mples symmetrc nteractons, as expecte. On the contrary, rotatonal probablty current s present when asymmetrc nteractons appear an etale balance cannot exst. Beses, t P (q) = 0 mples that J t (q) = 0, so that, for symmetrc A, there wll be etale balance n the probablty current whenever there s a compensaton between the mechancal an the sspatve currents, that s: J µ = ( J mec µ + J s ) µ = 0. q µ q µ

5 314 A.CASTELLAOSADL.VIAA In partcular, the conton A µν = A νµ s satsfe by the use of the Hebb rule or a varaton of t where each pattern {ξ µ } s gven a fferent weght n the learnng prescrpton 24]. Therefore, vergence an rotatonal operatng on the probablty current, J t (q), gve us two contons to characterze statonary an equlbrum states, respectvely, n the framework of. To complete ths scusson from the formal pont of vew, we nee to present the algebrac contons for the exstence of statonary states. Obvously, the frst conton s that the convecton matrx shoul be nonsngular. But n aton, we nee to fn the formal soluton to the autocorrelaton matrx Ξ. We start from the FDT: L Ξ + Ξ L = 2 D an by choosng the egenvectors { λ n } of the matrx L L as the bass to represent our matrxes. Then we get L L = λ n λ n, an multplyng (15) by L on the left, an by L on the rght, we obtan λ n L L Ξ L λ m + λ n L Ξ L L λ m = λ n 2 L D L λ m. From Eq. (6) we know that f the nteracton matrx satsfes A µν = ± A νµ, then the convecton matrx L s also symmetrc or antsymmetrc, respectvely. In ths case we can use that λ n L L = ±λ n λ n, rearrangng an multplyng by L 1 from the rght we arrve to the soluton Ξ n,n = λ n L D L G n L 1 λ n G n = 2 p m=1 λ m λ n ±λ n + λ m. Then we fn, as expecte, that f the nteracton matrx A µν s symmetrc, then the soluton exsts. On the other han, for an antsymmetrc nteracton matrx t s a conton for havng a statonary state that all egenvalues {λ n } shoul be fferent. For other types of nteractons t s not possble to know, n general, f the soluton to the FDT exsts (15), so each partcular problem must be analyze separately, as we wll o t n the next secton by analyzng a system prevously stue 16]. 5. Stuy of a system a fxe pont but out etale balance In ths secton we wll stuy the ynamcs of a partcular that, even though ue to ts structure t s unable to satsfy etale balance, t has a stable fxe pont n the lmt:. In ths neural network, p bnary patterns {ξ µ } = {+1, 1}, ranomly chosen equal probablty for each of ther two values, have been store accorng to a (p p) asymmetrc nteracton matrx gven by A µν = δ µν +ε δ µ1 δ ν2, so ε can be seen as a parameter breakng the symmetry. As we wll corroborate later, n the nfnte verson t only matters the probablty strbuton P (ξ µ ), whle n the fnte case t s also relevant the partcular pattern realzaton {ξ µ } (measure by the overlap between pars of patterns). As we mentone before, the tme evoluton of the etermnstc part of the ynamcal orer parameter, m µ(t), can be obtane by numercal soluton of the Louvlle equaton, Eq. (2). If ε = 0 the characterstcs of the system are as follows: there s complete symmetry between all the Louvlle equatons for the parameters m µ (2), the nfnte system has 2p fxe ponts relate to pure states (only one orer parameter s fferent to zero) gven by m µ = ±m β, µ = 1,..., p whose values are temperature epenent an are gven by the soluton of m β = tanh (βm β ). Beses, there are other unesrable stable states relate to mxtures of the orer parameters (more than one of them are smultaneously fferent to zero), smaller basns of attracton 25]. The system evolves towars one or another of the fxe ponts, epenng on the ntal contons. In the case ε 0 the symmetry between patterns s lost, an the shape of the basns an the placng of the attractors epen on the value of ε. The convecton matrx s gven by L µν = δ µν β ξ µ ξ ν 1 tanh 2 β (ξ Am + θ) ] ξ,θ +β ε δ ν2 ξ µ ξ 1 1 tanh 2 β (ξ Am + θ) ] ξ,θ an ts etermnant s fferent from zero, so the soluton to q = L 1 K exsts. For smplcty, we wll analyze the statonary states for p = 2; as generalzaton to other values of p s straghtforwar an oes not a new nsght nto the physcs of the problem. In ths case, at T = 0 for 0 < ε < 1 there exst four attractors, two of them relate to pattern {ξ 1 } an two others havng a bg overlap pattern {ξ 2 } an a small, but stll macroscopc overlap pattern {ξ 1 }; for ε > 1 there exst only the two attractors relate to pattern 1. For hgher temperatures an analytcal expresson has been foun for ε c (T ) 16], the crtcal value of ε where stablty s lost. The FDT (15) can be summarze nto three relevant equatons to be wrtten as SΦ = Λ, Φ = (Ξ 11, Ξ 12, Ξ 22 ) an Λ = (D 11, D 12, D 22 ). For the uner conseraton, S s nonsngular whenever (L 11 ) 2 = (L 22 ) 2 L 12 L 21. Snce ths conton s met, we can conclue that the statonary state exsts. Then the FPE can be wrtten as J = 0 to establsh that compensaton between ampng an ffuson occurs accorng to the analyss wrtten before. It s just a matter of algebra to stuy the fnal sze effects close to the statonary state m = m β (1, 0). Accorng to Eq. (9), at the statonary state, m µ = m β δ µ1, the average flux F s equal to zero an the value of the average fluctuatons q epens on the explct confguraton of patterns an t s weghte by the overlap between patterns: R 12 = (1/ ) ξ1 ξ2. The fxe pont m s an stable attractor when the overlap between patterns s non negatve (R 12 0), whle for the case R 12 < 0, the state of the system escapes from ths basn of attracton an evolves towars m = m β ( 1, 0) 4]; ths

6 STATIOARYPROCESSESADEQUILIBRIUMSTATESIO-SYMMETRICEURALETWORKS 315 s an mportant fnte sze effect, snce escapng s mpossble n the nfnte case (R 0). The autocorrelaton matrx s gven by the next expresson ] 1 Ξ µν = H m (T ) {δ µν 1 + δ µ1 δ ν1 2 β2 ε 2 Hm 2 (T ) + 1 } 2 βεh m (T ) δ µ1 δ ν2 + δ µ2 δ ν1 ], n whch H m (T ) = 1 (m ) 2 1 β1 (m ) 2 ], H m (0) = 0, H m (1) =, so the contrbuton to the extene entropy, ue to fnte sze fluctuatons, s gven by S q ( ) = 1 2 ln {et Ξ ]} +C = H m (T ) ] 4 β2 ε 2 Hm 2 (T ). Ths expresson allows to see that the entropy assocate to the statonary state of the system not satsfyng etale balance, S est, can be wrtten as follows S est = S eq ln + 1 ] 4 β2 ε 2 Hm 2 (T ), where S eq = (1/2) ln H m (t)]+c. The rotaton ( of the) probablty current: J (q) = P (q) βε 1 m 2 β, prouces ths atonal entropy. 6. Dscusson an conclusons In general terms, n the scentfc lterature there exsts some confuson about the terms statonarty an equlbrum an sometmes they are even treate as synonymous. In ths work we scusse the conceptual fferences between these two terms from a formal pont of vew an prove a set of tools that can be use to llustrate ths fference n terms of the analyss of fnte sze effects. For ths purpose t was ntrouce the concept of Extene Entropy à la Boltzmann, an t was shown that n any process leang towars a statonary state t s satsfe the relatonshp Tr D Ξ 1] Tr L]. Ths conton s less restrctve than the conton satsfe by a process leang to an equlbrum state, namely D Ξ 1 L, whch can only be reache by symmetrc systems satsfyng etale balance. We also showe how the lack of etale balance s relate to the rotaton of the probablty current for the fnte sze fluctuatons q(t). For the case of antsymmetrc, whch are often use to retreve patterns sequentally 12], we emonstrate that they can reach a statonary state only f ts convecton matrx s nonsngular an the egenvalues of { λ n } are all fferent. We emonstrate our results by examnng the behavour of a pecular system whose nfnte verson has fxe ponts even though these states o not satsfy etale balance, an therefore, t cannot reach an equlbrum state. We emonstrate how, f we conser fnte sze effects, these fxe ponts transform nto the wer category of attractors of the ynamcs whch keep the system movng aroun but never reachng them, as the probablty strbuton of the fnte sze fluctuatons has a rotatonal fferent from zero. Our theory escrbes fnte sze effects aroun the orer parameter values whch are exact n the thermoynamcal lmt, an therefore help us to fn out how the fnte sze of the sample mofes the macroscopc behavour of the system. These fluctuatons, orgnate by the fnte sze of the system, are specfc for each partcular system an epen on the specfc set of store patterns. As we mentone before, ths theory can be apple n prncple, to stuy fnte sze effects n any kn of fnte, separable recurrent sequental ynamcs an away from saturaton. However, we shoul keep n mn that ths theory s correct only n the low storage regme as n the ervaton of the theory t s essental the assumpton that the overlap between any par of store patterns (chosen at ranom) goes to zero n the thermoynamcal lmt. In the case of systems parallel upatng a theory to escrbe fnte sze effects has not yet been evelope. Acknowlegments Ths work was partally supporte by grant DGAPA I (UAM). 1. A. C. C. Coolen an D. Sherrngton, n Mathematcal Approaches to eural etworks, e. Taylor JG, (orth-hollan, Amsteram, 1993) B. M. Forrest, J. Phys. A21 (1988) G. A. Kohrng, J. Phys. A: Math. Gen. 23 (1990) L. Vana, A. Castellanos an A. A. C. Coolen, Lect. otes n Comp. Sc (1999) E. Domany, J. L. van Hemmen, an K. Schulten, (e), Moels of eural etworks II, (Berln: Sprnger, 1994). 6. A. Cannng an E. Garner J. Phys. A: Math. Gen. 21 (1988) A. J. oest Europhys. Lett. 6 (1988) E. Domany, W. Knzel an R. Mer J. Phys. A: Math. Gen. 22 (1989) 2081.

7 316 A.CASTELLAOSADL.VIAA 9. A. C. C. Coolen an L. Vana J. Phys. A: Math. Gen S. Skantzos an A. C. C. Coolen, J. Phys. A: Math. Gen. 33 (2000) S. Skantzos an A. C. C. Coolen, J. Phys. A: Math. Gen. 33 (2000) H. shmor, T. akamura an M. Shno, Phys. Rev. A41 (1990) P. Peretto, J. Phys. France 49 (1988) J. M. G. Amaro, an J. F. Pérez, Europhys. Lett. 5 (1988) A. Castellanos, PhD Thess CICESE, Méxco (1998). 16. A. Castellanos, A.C.C. Coolen, L. Vana J. Phys. A31 (1998) A. C. C. Coolen an Th. W. Rujgrok, Phys. Rev. A. 38 (1988) I. J. Matus, an P. Pérez, Phys. Rev. A41 (1990) F. Zertuche, R. López-Peña an H. Waelbroeck, J. Phys. A27 (1994) O. Berner, Europhys. Lett. 16 (1991) G. Van Kampen, Can. J. Phys. 39 (1961) 551; Av. Chem. Phys. 34 (1976) G. Van Kampen, Stochastc Processes n Physcs an Chemstry (2n. e. orth-hollan 1992), p Accorng to Pawula s theorem, a consstent probablstc theory s obtane n only two cases a) f only the frst non trval orer n (1/) s kept, or b) f t s mantane the whole Kramers-Moyal expanson. 24. L. Vana, J. Phys. 49 (1988) L. Vana an A. C. C. Coolen, J. Phys. I 3 (1993) 777.