Local Minima of a Quadratic Binary Functional with Quasi-Hebbian Connection Matrix. Yakov Karandashev, Boris Kryzhanovsky and Leonid Litinskii

Transcription

1 Loal inima of a Quadrati Binar Funtional with Quasi-Hebbian Connetion atrix Yaov Karandashev, Boris Krzhanovs and Leonid Litinsii Sientifi Researh Institute for Sstem Analsis Russian Aadem of Sienes (osow), a_rad_wsem@mailru, rzhanov@mailru, litin@mailru The loal minima of a quadrati funtional depending on binar variables are disussed An arbitrar onnetion matrix an be presented in the form of quasi-hebbian expansion where eah pattern is supplied with its own individual weight For suh matries statistial phsis methods allow one to derive an equation desribing loal minima of the funtional A model where onl one weight differs from other ones is disussed in details In this ase the above-mention equation an be solved analtiall Obtained results are onfirmed b omputer simulations I Introdution Let us examine the problem of minimization of quadrati funtional depending on N binar variables S i = ±, i =,,, N : { } N S E( S ) = J S S min () ij i j N i, j= This problem arises in different sientifi fields starting with phsis of magneti materials and neural networs up to analsis of results of phsial experiments and logistis The state of the sstem as a whole is given b N- dimensional vetor S = ( S, S,, S N ) with binar oordinates S i These vetors will be alled onfiguration vetors or simpl onfigurations The funtional E = E( S ), whih has to be minimized, will be alled the energ of the state Without loss of generalit the onnetion matrix = ( J ij ) ( J = J ) sine the substitution J J + J ) / does not hange the value of E ij ji ij ( ji ji J an be onsidered as a smmetri one In the general ase pratiall nothing is nown neither about the number of loal minima of the funtional (), nor about their struture Onl in some ases there are more or less lear ideas about the energ surfae of the funtional These ases are the Sherrington-Kirpatri model of spin glass [] and high-smmetrial neural networ of Hopfield s tpe [] In both ases haraters of onnetion matries were of great importane Statistial phsis methods were effiientl used for analsis of the energ surfae in the Hopfield model [3]- [5] In these papers was used the Hebb onnetion matrix It is a orrelation matrix onstruted from a set of ξ = ( ξ, ξ,, ξ ) = ±, =,,, : given onfigurations, { } N ξ i J ij = = { ξ } In the theor of neural networs onfigurations are alled patterns We also use this notation When statistial phsis methods are applied, the result is as follows If patterns ξ are randomized and independent and their number < 4 N, in the viinit of eah pattern there is neessaril a loal minimum of the funtional () In other words, the neural networ wors as an assoiative memor For the first time this result was obtained in [3], [4] Up to now it was supposed that statistial phsis methods are appliable onl for the Hebb onnetion matrix However, in the papers [6]-[8] it was shown that an smmetri matrix an be presented as quasi-hebbian expansion in unorrelated onfigurations ξ : ξ i ξ j J ij = r ξ ξ () i = In Eq () weights r are determined from the ondition of non-orrelatedness of onfigurations ξ, and the number is determined b the aura of approximation of the original matrix J The representation () is alled the quasi-hebbian sine all weights r are different We hope that this representation would allow one to use statistial phsis methods for analsis of an arbitrar onnetion matrix First obtained results are presented in this publiation In Setion II with the aid of statistial phsis methods the prinipal equation for a onnetion matrix of the tpe () is derived Determination of onditions under whih the equation has a solution provides information about loal j

2 minima of the funtional () As an example the lassial Hopfield model (all the weights are r ) is examined in details In Setion III we analze the ase when onl one weight differs from the others, whih are identiall equal: r r = r3 = = r = This model ase an be analzed analtiall up to the end It was found that a single weight that differs from substantiall affets the properties of loal minima Computer simulations onfirm this result Some onlusions and remars are given in Setion IV II Priniple Equation and Hopfield odel Priniple equation Here an d in what follows we analze the funtional () with the quasi-hebbian onnetion matrix () Let S = ( S, S,, S N ), where S i = ± define a state of the sstem The energ of the state an be presented in the form where m ( S) is the overlap of the state S with the pattern ξ : E( S) = r m( S ), (3) = N N m ( S) = Siξi N Statistial phsis methods allow one to obtain equations for the overlap of a loal minimum of the funtional () with the patterns ξ After solving these equations it is possible to understand under whih onditions the overlap of a loal minimum with the -th pattern is of the order of ( m ~ ) In other words: under whih onditions the loal minimum oinides (or nearl oinides) with the -th pattern Supposing the dimensionalit of the problem to be ver large ( N >> ), let us set that the number of patterns is proportional to N : = α N In the theor of neural networs the oeffiient of proportionalit α is alled the load parameter Let us suppose that in the viinit of the pattern ξ there is a loal minimum of the funtional Re peating alulations perf ormed in [3]-[5] for the Hopfield model, we obtain the sstem of equations i= rm m = erf, σ r σ =, N ( C r ) (4) rm exp C = σ π σ Here m is the overlap of the loal minimum with the -th pattern In the ase of equal weights ( r ) this sstem redues to the well-nown sstem for the Hopfield model (see Eqs (7)-(73) in [5]) Let us introdue an auxiliar variable priniple equation = r m / σ Exluding σ and C from the sstem (4), we obtain the where γ = r = α γ ϕ (5) e π ( r r ) π erf e and (6) ϕ = The funtio n γ = γ ( ) dereases monotoniall, and t he funtion ϕ = ϕ( ) inreases monotoniall from its minimal value ϕ ( ) = In what follows these funtions are frequentl used For simpliit sometimes we omit their arguments, an d use the notations γ and ϕ for these funtions Let us fix the values of external parameters: α = N, r and If is a solution of Eq (5), N >>, / { } the overlap of the loal minimum with the -th p attern is equal to m = erf (7)

3 With the aid of the value of one other important har ateristi σ = speaing, rm an be alulated Roughl σ is the weighted sum of squared overlaps of the loal minimum from the viinit of with all patterns exept the -th one The expression for σ has the form erf r σ = (8) An important role plas the analsis of onditions under whih Eq (5) an be solved For example, it is useful to determine the ritial value of the load parameterα for whih the solution of Eq (5) still exists, but for α that are larger than α there is no solution of Eq(5) All harateristis orresponding to α are mared off with the same subsript, namel the are, m and σ The Hopfield model: r Sine all patterns are equivalent, in Eqs (4), (5) the subsript an be omitted It is eas to see that in this ase Eq (5) has the form ξ α = γ ϕ ( ) (9) The plot of right-hand side of Eq (9) is shown in Fig We see that for all α less than a ritial valueα, there are two solutions: α and α ( α < α ) We are interested in the larger solutio n α The solution α is a spurious one, and it has to be omitted 4 α α α 3 4 Fig Graphial solution of Eq (9) for the Hopfield model is shown Solid line is the graph of the funtion in the right-hand side of Eq(9) Dashed line orresponds to the value of load parameter α = 4 In general, when Eq (5) has some solutions, the solution with maximal overlap m orresponds to the minimal value of the free energ Consequentl, it is neessar to pi out the solution that is loated as muh as possible to the right on the absissa axis In what follows we use this rule Let us return to our analsis of the plot in Fig When α inreases the solution α shifts to the left In the same time the overlap of the loal minimum with the pattern also dereases In other words, the loal minimum little b little moves awa from the pattern Inreasing α we reah the ritial valueα The ritial value α is determined b maximum of the right-hand side of Eq (9) It is not diffiult to alulate it s value: α 38 This well-nown result related to the ritial value of the load paramet er firstl was obtained in [3], [4] The expression in the right-hand side of Eq (9) reahes its maximum value at the point 5 Equation (7) allows one to alulate the ritial value of the over lap of the loal minimum with the pattern: m 967 We see that even for the maximal load parameter α the loal minimum is ver lose to the pattern If the load parameter α is less than α, the overlap of the loal minimum with the pattern even loser to Sine in this ase an arbitrar pattern is onsidered, we an sa that whileα < α there is a loal minimum of the funtional () in the viinit of eah pattern It an be shown that the depths of all these loal minima are approximatel t he same [5] What happens when the parameter α exeeds the ritial valueα? In this ase there is no solution of Eq (9) As the sa, a breadown of the solution happens Detailed analsis [3]-[5] shows that the overlap dereases

4 abruptl almost to zero value In terms of statistial phsis this means that in our sstem the first ind transition taes plae The energ surfae of the funtional () hanges: loal minima in viinities of patterns ease to exist III The Case of One Differing Pattern Interesting is the ase when all weights r exept one are equal to, and onl one weight differs from other Without loss of generalit we an write r =, r = r3 = = r = () The first weight an be both larger and less than one It might seem that the differene from the lassial Hopfield model has to be small: enormous number of patterns with the same weight taes part in the formation of the onnetion matrix and onl one pattern provides a different ontribution Intuition suggests that the influene of a single pattern with the individual weight has to be negligible small omparing with ontribution of infinitel large number of patterns with the same weight r = However, this is not the ase One pattern with an individual weight an substantiall affet the distribution of loal minima Let us examine separatel what happens with loal minimum in the viinit of the pattern with the individual weight, and what happen with loal minima near other patterns whose weights are equal to Pattern with an individual weight: r = For this pattern equation (5) has the form ( ) = () α γ ϕ () If is a solution of Eq (), the overlap of the loal minimum with the first pattern is m = erf( ) The super sript () emphasizes that we deal with the overlap of the loal minimum with the pattern number The point of breadown () ( ), where the right-hand side of Eq() reahes its maximum, is the solution of the equation ϕ( ) = + () After finding () () () () ( ) the ritial harateristis m = erf( ) and α an be alulated It turns out that for > Eq() has a nontrivial solution onl if 3 When is larger than 3, this equation has onl trivial solution: () > 3 ( ) This means that the overlap of the loal minimum with the pattern vanishes: () () m = erf( ) So, when > 3 in the sstem the phase transition disappears This is an unexpeted result We do not understand the reason wh the phase transition disappears (We have for 3 : σ = and π () α = ( ) π ) In Figa we present grap hs of right-hand side of Eq () for three different values of the weight oeffiient: =, = and = 3 The ase of = orresponds to the lassial Hopfield model We see that inreasing of the oeffiient above on the one hand is aompanied b an inrease of the ritial value α () ( ), and on the other hand it is aom panied b stead removal of the breapoint () ( ) toward As a result, when inreases the overlap of the loal minimum with the pattern m () ( ) in the ritial point dereases Generall speaing, the derease of the overlap, aompaning an inrease of the weight oeffiient, is ver unusual Let us point out that this behavior relates onl to the ritial values () ( ( ) and m ) ( ), ie to their values at the breadown point Absolutel other situation taes plae if α < α Indeed, Figa shows intersetions between a straight line parallel to absissa axis and the graphs representing the right-hand side of Eq () for different values of Absissa values of eah intersetion are solutions of the equation We see that when inreases the point of intersetion shifts to the right, ie in the diretion of its greater values This means that when inreases the overlap m () ( ) of the loal minimum with the pattern also inreases This behavior of the overlap is in agreement with the ommon sense: the greater the weight of the pattern, the greater its influene Then the overlap of this pattern with the loal minimum has to be greater Now let us examine the interval [,] of the weight Analsis of Eqs (), () shows that when dereases from to the maximum point () () ( ) steadil shifts to the right, and the maximum value α steadil dereases (see the behavior of maxima of the urves in Figb) In other words, when the weight oeffiient dereases the ritial value of the overlap m () ( ) tends to Note, when < the funtion ϕ( ) vanishes ()

5 neessaril B ( ) we denote the point where this funtio n equals zero The behavior of the urve to the left from ( ) does not interesting for us, sine in this region Eq () has onl spurious solutions α3 5 5 = 3 a) α = = 8 b) 5 = = = Fig The graphs of the right-hand side of Eq () for different values of a) ; the dashed straight line intersets the gr aphs orresponding to different values of ; b) ; minima of eah graph are equal to zero Graphs showing the behavior of ritial harateristis m () ( ) and Fig3 α () ( ) are presented on two left panels of α a) α ) () m b) () m d) Fig 3 Critial values of the load parameter α (two upper panels) and overlap of the loal minimum with the pattern m (two lower panels) as funtions of the weight oeffiient The urves on the left panels (a and b) orrespond to the pattern with individual weight r = ; the urves on the right panels ( and d) orrespond to patterns with the same weight =, r

6 Patterns with the same weight: =, Let us examine how the overlap of the loal minimum r with one of the patterns whose weight oeffiient is equal to depends on the value Sine all these patterns are equivalent, we hoose the pattern with number With the supersript () we mar off the harateristis () () () m,, α that are interesting for us Now Eq (5) has the form: L ( ) = α, where ( ϕ( ) ) ( ϕ( ) ) ( ) ( ) γ ( ) L ( ) = ( ε) ϕ( ) + ε ϕ( ) and ε = (3) When, the quantit ε tends to zero However, we annot simpl tae ε =, sine (at least for > ) for some value of the denominator of L ( ) neessaril vanishes Then it is impossible to anel idential funtions in the numerator and denomin ator of L ( ) So, we analze Eq (3) for a small, but finite value of ε and then we tend it to zero This wa of analsis is orret When, the funtion ϕ( ) nowhere vanishes, and we an simpl tae ε = In this ase the ( expression for L ( ) turns into γ ( ) ϕ( ) ) Then Eq (3) transforms in Eq (9) that orresponds to the Hopfield model Consequentl, until the value of belongs to the interval (,] the following is true:, ( ) α 38, m () ( ) m 967 (4) () () ( ) 5 α Let us exam ine the region > In this ase the funtion ϕ( ) vanishes at the point ( ) The value of ( ) is determined b equation: Out of the small viinit of the point ( ) the parameter ε in Eq ϕ( ( )) = (5) (3) an be tended to zero At that the expression for the funtion L ( ) is as in the ase of the Hopfield model: L= ( ) γ ( ) ϕ( ) At the point ( ) ( ) itself for small, but finite value of ε, the funtion L ( ) is equal to zero: L ( ( )) = If is from the viinit of the point ( ) and it tends to ( ), for an finite value of ε the urve L ( ) quil drops to zero Thus, for an finite value of ε the graph of the funtion L ( ) pratiall everwhere oinides with the urve ( ) γ ( ) ϕ ( ) that orrespond s to the Hopfield model And in the viinit of the point ( ) the urve L ( ) has a narrow dip up to zero whose width is proportional to the value of ε As long as the weight < ϕ( ) 5568, the point ( ) is at the left of For this ase in Fig 4a we show the urve L ( ) The maximu m of the urve L ( ) orresponds to the ritial point 5 and it does not depend on Consequentl, the equalities (4) are justified not onl for [,], but in the wider interval < ϕ( ) 5568 α α 4 a ) b) ( ) ( ) Fig 4 The graph of the funtion L( ) from Eq (3), when ε 5 = (solid line): a) when = < ϕ ), the point ( ) is on the left of 5 ; b) when = > ϕ( ), the point ( ) is on the right of shows the differene from the lassial Hopfield model 3 ( 5 Dashed line

7 On the ontrar, for the values of the weight > 5568 the p oint ( ) is on the right of For this ase an example of the urve L ( ) is shown in Fig 4b T he maximum point of the urve L ( ), whih is interesting for us, oinides with the pea of the urve that is slightl on the right of the point ( ) From ontinuit onditions it is ε for > we have e vident that when ε this pea shifts to the point ( ) Consequentl, when 5568 () ( ) ( ) > 5, m ( ) erf( ( )) 967 () >, ( ) () () ( ) ( α ) = e < α = 38 (6) π Let us summarize the obtained results for patterns with the same weights: r =, Firstl, as far as the value of is less t han ϕ ( ) = 5568, all harateristis of loal minima do not depe nd on and ex atl oinide with harateristis of the Hopfield model Seondl, as soon as the value of exeeds ϕ ( ) = 5568 the situation hanges In this ase the brea-down point of the solution depends on and oinides with ( ) that is the solution of Eq (5) Note, when inreases the ritial load parameter α ( ) dereases and the overlap of the loal minim um with the pattern in reases and graduall tends to (see Eq (6)) The graphs showing the behavior () of the ritial harateristis ( ) and α () ( ) are presented at the two right panels in Fig3 m 3 Computer simulations The obtained results were verified with the aid of omputer simulat ions For a given value of N the load parameter α was fixed Then = α N randomized patterns were generated, and the were used to onstrut the onnetion matrix with the aid of Eq () The weights were defined b Eq () When hoosing the weight oeffiient we proeeded from following reasons Let α be a fixed value of the load parameter Then we found th e v alue of the weight ( α ) for whih the given α was a ritial one (this an be done when solving Eqs () and () simultaneousl) We also defined the brea-down point of the solution ( α ) If onstruting the onnetion matrix with the weight that is equal to ( α ), the mean value of the overlap of the loal minimum with the pattern has to be lose to m( α) = erf( ( α)) If the weight is less than ( α ) the mean value of the overlap with the pattern has to be lose to If the weight is larger than ( α ), the mean value of the overlap has to be larger than m ( α ) Under further inrease of the mean overlap has to tend to To verif the theor relating to the pattern with the individual weig ht r =, three experiments has been done In all experiments the dimensionalit of the problem was N = For eah load parameter α the mean value of overlap (with the single pattern) < m > was alulated with the aid of ensemble averaging Ensemble onsisted of different matries For testing three values of α were hosen Let us list them together with values of ( α ) and m ( α ): ) α =, ( α) 944, m ( α) 97; ) α = 38, ( α) 5, m ( α) 99 ; 3) α = 3 ; for this value of α there is no jump of the overlap, but beginning from ( α) 37, the overlap has to inrease smoothl In Fig5 the graphs for all three values of the load parameter are presented Theoretial harateristis m () ( ) are shown b dashed lines The results of ompute r simulations are given b solid lines with marers < m > α = α = 38 α = 3 Fig 5 Theor and experiments for the pattern with the individual weight r = The graphs orrespond to three different values of the load parameter α =, 38 and 3 Solid lines are the results of experiments; dashed lines show theor

8 For α = and α = 38 on the experimental urves we learl see expeted jumps of mean overlap that have plae in the viinities of ritial values of the weights ( α) 944 and ( α) 5, respetivel At the left side the values of the mean overlap are not equal to zero First, this an be due to not suffiientl large dimensionalit of the problem The point is that all theoretial results are related t o the ase N For our omputer simulations we used ver large, but finite dimensionalit N Seond, the theor is orret when the mean overlap is lose to (but not to ) In this region of values of we have rather good agreement of the theor with omputer simulations For the third load parameter α = 3 we expeted a smooth inreasing of the mean overlap < m > from to Indeed, the last right solid urve inreases smoothl without an jumps However, aording our theor the inreasing ought to start beginning from 3, while the experimental urve differs from zero muh earlier This disrepan between our theor and experiment an be explained in the same wa as it has been done in the end of the previous paragraph To verif our theor relating to patterns with equal weights r, we used the same proedure For the load parameter α = and several dimensionalities N ( N = 3, and 3) we alulated the mean overlap < m > of the loal minimum with the nearest pattern for different weights We averaged both over patterns of the given matrix and over randomized matries onstruted for the given value of Aording to the theor, for α = breadown of the overlap < m > has to tae plae when 7 In Fig 6 this plae is mared b the right dashed straight line with the label If N and reall were infinitel large just in this plae th e breadown of < m > had to tae plae In Fig 6 the real dependen of < m > on that was observed in our experiments was shown b three solid lines orresponding to different dimensionalities N < m > N = 3 N = N = 3 Fig 6 Theor and experiments for patterns with the same weights r when α = Solid lines show the results of experiments for three values of dimensionalit N Notieable differene between the theor and omputer simulations must not onfuse us Apparentl this differene is due to finite dimensionalities of experimental onnetion matries Earlier omputer verifiations of lassial theoreti results faed just the same problems [], [] As a wa out the authors of these papers extrapolated their experimental results into the region of ver large dimensionalities N Note, wh en N inreases the experimental urve in Fig 6 tends to theoretial step-funtion, whih is indiated with the aid of dashed line A orretion due to finite dimensionalit of the problem an be taen into aount if we insert the expliit expression ε = in Eq (3) Then for N = 3 we have = α N = 36 When this value of is used in Eq (3), we obtain that the breadown of the overlap < m > has to tae plae not in the viinit of 7, but muh earlier when 7 The orresponding dashed line with the label = 36 is shown in Fig 6 Its loation notieabl better orrelates with experimental urves 4 Depths of loal minima Let us find out how the depth of loal minimum depends on the value of B the depth of the loal minimum we impl the modulus of its energ (3), E For different matries the energies an be ompared onl if the same sale is used for alulation We normalized elements of eah matrix dividing them b

9 the square root from the dispersion of matrix elements: σ J = + αn is the standard deviation of quasi-hebbian matrix elements of the form () for the weights (); mean value of matrix elements is equal to Then for the loal minimum depth we obtain the expression E = r m σ = N J Let us examine the loal minimum in the viinit of the pattern ξ with the weight r = For this minimum we obtain aording to Eqs (5)-() E = ( m + σ α), σ = γϕ (7) σ J Here α is the load parameter, m and σ are alulated aording to Eqs (7) and (8), γ and ϕ are given b the expressions (6) Analsis of Eq (7) and numerial alulations show that when inreases the depth of the minimum inreases monotoniall While is less than a ritial value ( α ) the loal minimum is ver far from the pattern ( m ) and its depth is equal to zero When > (α ), minimum quil approahes to the pattern ( m ) and its depth inreases in a jump-lie wa After that E inreases as / + αn In Fig 7 fo r different load parameters α the dependene of E on is shown When beomes ve r large ( > α N ) depth of the minimum asmptotiall tends to It ould be thought th at for suh large the matrix J is onstruted with the aid of onl one pattern ξ the E 37 5 Fig 7 The dependene of the depth of the loal minimum from the viinit of the pattern ξ on The load parameter α hanges from to E Fig 8 The dependene of the depth of the loal minimum from the viinit of one of the patterns ξ ( r = ) on The load parameter α hanges from to 37 Now let us examine loal minima near patterns with the same weight r = ( ) First, independent of the value of the weight, these loal minima exist onl when α α (see Eqs (4), (6)) The expression for the depth of the -th minimum has the form: E = ( m + σ α ), σ = γϕ (8) σ J Analsis of Eq (8) shows that when inreases, the depth of the loal minimum dereases (see Fig 8) When beomes equal to some ritial value ( α ), the depth of the minimum reahes its minimal value E = E (α ) : E( α) = αα α N For further inrease of the overlap m step-wise beom es equal to zero: minimum abruptl goes awa from the pattern ξ, and its depth E drops to zero From the point of view of the assoiative memor this means that for > ( α ) the memor of the networ is destro ed IV Disussion and Conlusions Loal minima of the funtional () are fixed points of an assoiative neural networ In this paper we examined minima loalized near patterns, whih were used to onstrut the onnetion matrix All obtained results

10 an be interpreted in terms of neural networs onepts, whih are the storage apait, patterns restoration and so on From this point of view two results are of speial interest First, notie that if 3 for the pattern with individual weight r = the phase transition of the first ind () disappears We reall that when inreases the ritial value of the load parameter α also inreases, the overlap of the loal minimum with the pattern m steadil dereases Beginning from = 3 the overlap in the ritial point () vanishes (see graphs on the left panels in Fig 3) This means that the phase transition eases to exist As a rule disappearane of phase transition signifies substantial strutural hanges in the sstem Seond, it is interesting how harateristis of loal minima loated near patterns with weight oeffiients r = ( ) depend on the value of We reall that at first inrease in does not affet these loal minima at all While < 5568 neither the overlap of the loal minimum with the pattern, nor its ritial harateristis depend on (see Eq (4)) It seems rather reasonable sine the number of patterns with weight oeffiients r = is ver large ( >> ) Therefore, as would be expeted, the influene of onl one pattern with the weight is negligible small However, as soon as exeeds the ritial value 5568 (whih is not so large) its influene on the huge number of loal minima from the viinities of patterns with weight oeffiients r = beomes rather notieable (see Eqs (5)) Let us emphasize that all the results are justified b omputer simulations In onlusion we note that Hebbian onnetion matrix with different weight oeffiients r was also examined previousl (see, for example [], [3]) In [3] basi equations for this onnetion matrix were obtained The oinide with our sstem of equations (4) However, then the authors simplified their equations and onl the standard Hopfield model was analzed We see that even rather little differenes in values of weights lead to new and nontrivial results We hope to use the same approah for analsis of loal minima of the quadrati funtional () with an arbitrar onnetion matrix (see Introdution and papers [6]-[8]) The wor was supported b the program of Russian Aadem of Sienes Information tehnologies and analsis of omplex sstems (projet 7) and in part b Russian Basi Researh Foundation (grant ) Referenes [] D Sherrington, S Kirpatri Solvable model of spin-glass Phs Rev Letters (975) v35, p [] LB Litinsii High-smmetr Hopfield-tpe neural networs Theoretial and athematial Phsis (999) v8, pp 7-7 [ 3] D Amit, H Gutfreund and H Sompolins Storing Infinite Numbers of Patterns in a Spin-Glass odel of Neural Networs Phs Rev Letters, 55 (985), [4] D Amit, H Gutfreund and H Sompolins Statistial ehanis of Neural Networs Near Saturation Annals of Phsis, 73 (987), 3-67 [5] J Hertz, A Krogh, R Palmer Introdution to the Theor of Neural Computation Addison-Wesle, NY, 99 [6] BV Krzhanovs, B agomedov, AL iaelian Binar optimization: A relation between the depth of a loal minimum and the probabilit of its detetion in the Generalized Hopfield odel Dolad athematis, vol7, N3, pp (5) [7] BV Krzhanovs Expansion of a matrix in terms of external produts of onfiguration vetors Optial emor & Neural Networs (Information Optis) (7) v 6(4), pp87-99 [8] BV Krzhanovs, B agomedov, AB Fonarev On the probabilit of finding loal minima in optimization problems Proeedings of International Joint Conferene on Neural Networs IJCNN-6, pp , Vanouver, Canada [9] RJ Baxter Exatl solved models in statistial mehanis, London: Aademi Press In, 98 [] G A Kohring A High Preision Stud of the Hopfield odel in the Phase of Broen Replia Smmetr Journal of Statistial Phsis, 59, 77 (99) [] AA Frolov, D Huse, IP uraviev Informational effiien of sparsel enoded Hopfield-lie autoassoiative memor Optial emor & Neural Networs (Information Opti) (4) v, #3, pp77-97 Allerton Press, In, NY [] JL van Hemmen, G Keller and R Kuhn Forgetful emories Europhsis Letters, vol 5, pp , 988 [3] JL van Hemmen, R Kuhn Colletive Phenomena in Neural Networs In: odels of Neural Networs, E Doman, JL van Hemmen and K Shulten (Eds), Berlin: Springer-Verlag, 99