T sg. α c (0)= T=1/β. α c (T ) α=p/n

Size: px

Start display at page:

Download "T sg. α c (0)= T=1/β. α c (T ) α=p/n"

Kevin Bruce
6 years ago
Views:

1 Taejon, Korea, vol. 2 of 2, pp. 779{784, Nov. 2. Capacity Analysis of Bidirectional Associative Memory Toshiyuki Tanaka y, Shinsuke Kakiya y, and Yoshiyuki Kabashima z ygraduate School of Engineering, Tokyo Metropolitan University Hachioji Tokyo Japan zinterdisciplinary Graduate School of Science and Engineering, Tokyo Institute of Technology Yokohama Japan Abstract Macroscopic properties of the bidirectional associative memory (BAM) are studied in a framework of statistical physics. The relative capacity, which is the relative number of pattern pairs being able to be memorized and retrieved by the BAM while allowing nite fraction of retrieval error, is evaluated using replica method. The obtained capacity for a system with N units in each of two layers is :998N, which can be regarded as BAM's counterpart of the result :38N for the autocorrelation associative memory, or the Hopeld model, evaluated by Amit, Gutfreund, and Sompolinsky. Introduction The bidirectional associative memory (BAM) [] is a variation of associative memory neural networks. The principal function of associative memories is to store and retrieve multiple patterns in a distributed manner. The storage capacity, which represents how many patterns can be stored in a network, is one of the fundamental performance measures to evaluate the ability of the network. There have been two distinct denitions of the storage capacity in the literature: One is the so-called absolute capacity, which refers to the upper limit of the number of memorized patterns when one requires perfect recall, i.e., no retrieval error is allowed. The other is the so-called relative capacity, in which one allows a nite amount of retrieval error rate. The objective of this paper is to evaluate the relative capacity of BAM. The autocorrelation associative memory (AAM), sometimes termed as the Hopeld model, is a basic model of associative memory neural networks, and its storage capacity has been extensively investigated. The absolute capacity of AAM is known to scale as O(N= log N), where N is the size of the model [2, 3, 4]. There have been two approaches to the relative capacity of AAM. One is from statistical physics [5], which evaluates the capacity as :38N, while a dierent value :6N is given by the other approach using statistical neurodynamics [4]. (Some renements have been done for each of these approaches: See [6, 7, 8] for the statistical physics approach, and [9] for the statistical neurodynamics approach.) BAM can be regarded as a kind of AAM, whose connections are systematically removed []. This observation suggests that methods for analyzing the properties of AAM can be applied also for analyzing the properties of BAM. In fact, the absolute capacity of BAM has been analyzed along this line, and the same order of the capacity O(N= log N) has been reported by Haines and Hecht-Nielsen []. As for the relative capacity, however, to our knowledge, there are so few results reported in the literature. Amari [2] derived the macroscopic time evolution equations for BAM based on the statistical neurodynamics, but he did not discuss the relative capacity issue. Yanai et al. [] mentioned the value of the relative capacity of BAM ( :22N) evaluated by following Amari's argument. Leung et al. [3] evaluated a lower bound of the relative capacity of BAM based on a large-deviation-type argument. As far as we know, this is the rst study on the relative capacity of BAM based on the statistical physics approach. 2 Model In order to apply the statistical physics approach to the capacity analysis of BAM, we consider a version of BAM in which each unit is updated stochastically and asynchronously, just as in the case of AAM. BAM consists of two layers of neural units (see Figure ). Let and be the numbers of units these layers have. The state of unit is represented by a binary value, and the state of BAM is given by the pair fs; ~sg, where s 2 f?; g and ~s 2 f?; g are the states of the layers. Let w ij be the connection weight between unit i in the rst layer and unit j in the second layer. Unit i in the rst layer changes its state s i based on the input h i fed into it, according to the following stochastic state updating rule: Prob[s i := ] = e hi e hi + e?hi ; ()

2 s s 2 s 3 s W = (w ij ) ~s ~s 2 ~s 3 ~s Layer Layer 2 Figure : Structure of BAM where denotes the \inverse temperature," which controls the degree of stochasticity of the updating. The limit! corresponds to deterministic updating of states. The input h i is the weighted sum of the state of the second layer, i.e., h i = j= w ij ~s j : (2) Similar rules are applied to unit j in the second layer as well, i.e., and e ~ h j Prob[~s j := ] = ; (3) e ~ h j + e? h ~ j ~h j = i= w ij s i : (4) We assume that each unit is updated asynchronously with others, although it seems common in most of previous studies to assume the synchronous update over each layer and alternative between the layers. The system described so far has the following equilibrium distribution of the state: p(s; ~s) =? e?h(s; ~s) (5) where the Hamiltonian H(s; ~s) of the system is de- ned as X H(s; ~s) w ij s i ~s j ; (6) i; j and is the partition function normalizing p(s; ~s). We consider a situation in which p pattern pairs f( ; ~ )g, = ; : : : ; p, are memorized in BAM. Here, 2 f?; g and ~ 2 f?; g represents binary random patterns with the components i and ~ j being realizations of independent and identically-distributed (iid) binary variables following Prob[ i = ] = Prob[ ~ j = ] = =2. Hereafter, we focus on the Hebbian learning, so that synaptic weight w ij is given by w ij = N px = i ~ j : (7) Since we follow the statistical physics approach, we will consider properties of the model in the \thermodynamic" limit N! while keeping c; ~c = O(). Just like AAM, we can expect that the relative capacity of BAM should be of the same order as N, so that we will consider the case where the memory rate p=n is of the order O() in the limit N!. The principal interest lies in whether or not there is an equilibrium state corresponding to the retrieval of a pattern pair. We assume that only one pattern pair is being retrieved, and that the rst pattern pair ( ; ~ ), for the sake of simplicity, is nominated for retrieval. The overlaps m i= i s i; ~m j= ~ j ~s j (8) serve as the macroscopic measure of how well the nominated pattern pairs are retrieved. jmj; j ~mj holds, and if s and ~s are close to and ~, m and ~m become close to, respectively. An equilibrium state with nonvanishing m and ~m is the retrieval state of the model. We follow the statistical physics approach to explore the existence of the retrieval state. 3 Replica Analysis We have followed the conventional replica approach to analyze the equilibrium states of BAM. For details of the approach, see, for example, [4]. We take the following two assumptions:. The macroscopic properties of the model are self-averaging with respect to the randomness of the patterns in the thermodynamic limit, i.e., we can analyze the properties of the model by evaluating the pattern-averaged free energy hhlog ii instead of the (sample) free energy log, the latter of which is the relevant quantity in describing the properties of a single sample system. 2. The macroscopic quantities satisfy the replica symmetric (RS) ansatz. In the replica approach, the pattern-averaged free energy hhlog ii is evaluated in the thermodynamic limit by the saddle-point approximation, and RS ansatz means that we place some restrictions on the form of the saddle-point solutions: see [4] for details.

3 The rst assumption is clearly supported by experimental observation, in which one can nd that uctuations of macroscopic quantities, such as the overlaps, vanish as the system size N is increased. The second assumption, the RS ansatz, cannot be checked in such a direct way. For AAM, the validity of the RS ansatz has been established except for a region of very low temperature (large ). More rigorous treatments considering replica-symmetrybreaking (RSB) [6, 7] has shown, however, that the results need very small corrections even when the RS ansatz is not valid [4]. We can therefore expect that the RS ansatz is a reasonable assumption also for BAM. Under these assumptions, we obtained the following result. Proposition In the thermodynamic limit, the overlaps m and ~m of the model at thermal equilibrium at the inverse temperature satisfy the following saddle-point equations: ~ ~ Dz tanh ( p rz + ~c ~m) Dz tanh ( p ~rz + cm) Dz tanh 2 ( p rz + ~c ~m) Dz tanh 2 ( p ~rz + cm) r = ~c[~q + 2 c~cq(? ~q) 2 ] [? 2 c~c(? q)(? ~q)] 2 ~r = c[q + 2 c~c(? q) 2 ~q] [? 2 c~c(? q)(? ~q)] 2 (9) where fq; ~q; r; ~rg are the variables to be determined simultaneously with m and ~m by these equations, and Dz p (dz= 2)e?z2 =2 is the Gaussian measure. This is the main result of this paper. The physical interpretations of the variables fm; ~m; q; ~q; r; ~rg are the following: ~ ~ r = i= j= i hs ii ~ j h~s ji hs i i 2 i= h~s j i 2 j= px (m ) 2 > T=/β α c (T ) α=p/n T sg α c ()=.998 Figure 2: Phase diagram for BAM with c = ~c = where m ~r = i= px ( ~m ) 2 ; () > i s i; ~m j= ~ j ~s j; () and hi denotes the average over stochasticity arising from the stochastic nature of the state updating rule (Equations () and (3)). On the other hand, hhii represents the average over memorized pattern pairs f =; :::; P ; ~ =; ::: ;P g. The resulting saddle-point equations (9) have a strong similarity to those obtained for AAM by Amit, Gutfreund, and Sompolinsky [5]. The latter are r = Dz tanh ( p rz + m) Dz tanh 2 ( p rz + m) q [? (? q)] 2 : (2) The main dierence lies in r and ~r, which represent the variance of the noise components in the inputs h i and ~ h j. This may aect the quantitative properties of the models, including the relative capacity. 4 Numerical Results and Discussion One can solve the saddle-point equations (9) numerically with respect to the variables fm; ~m; q; ~q; r; ~rg. A solution with m ~m 6= corresponds to the retrieval state. We investigated the conditions under which the retrieval state exists. Figure 2 shows the result for the case c = ~c =. The curve = c (T ) represents the relative capacity, the maximum memory rate with which the retrieval state exists at the temperature T = =. The retrieval state exists in the lower-left, shaded region. At zero temperature

4 (! +), the relative capacity of BAM with c = ~c = is evaluated as c () = :998N. We have also performed a series of numerical experiments of BAM at zero temperature with varying the system size N, and have found, via the nite-size-scaling analysis [8], that the relative capacity of BAM with c = ~c = is :2N. The analytically-obtained value :998N is in good agreement with, but slightly lower than, the experimentally-obtained result :2N. This small discrepancy can be ascribed to failure of the RS ansatz, as discussed below. For AAM at zero temperature, one can notice a similar discrepancy between the analytically-obtained result for the relative capacity (:38N [5]) and the experimentallyobtained one (:4N [8]). It has been known for AAM that the retrieval state at zero temperature does not satisfy the RS ansatz [5]. In order to investigate this possibility for the current system, we have checked the de Almeida- Thouless instability [5], which signals the rst bifurcation from the RS solution to the solutions without replica symmetry. Evaluating the eigenvalues of the Hessian of the replicated free energy around the RS solution, we found that only the socalled \replicon" modes become possibly unstable as long as we focus on the attractor states of iteration dynamics used to solve the saddle-point equations (9). In contrast with the case of AAM, the eigenvalues are not degenerated but split into two values for BAM due to the existence of inter-layer interaction. By considering stability condition for these modes, two critical temperatures are obtained for each value of. However, only one temperature is physically relevant among them because the other one is always negative. The relevant temperature corresponds to the conventional AT line, below which the RS solution becomes unstable. The obtained AT line is shown in Figure 3. From this gure, it is found that the RS solution is unstable, and therefore the RS ansatz is no longer valid, in suciently low temperatures in the vicinity of the capacity = P=N :998. We can point out further evidence which supports the observed discrepancy, on the basis of similarity about the properties of the RS solution between AAM and BAM. In Figures 2 and 3 one can observe that, at very low temperature, the capacity of BAM slightly increases as the temperature rises, i.e., d c (T )=dt >. This \re-entrant" phenomenon is also observed in the AAM [6]. For AAM, it is known that the re-entrant phenomenon is an artifact arising from the RS ansatz, and it tends to disappear in analyses taking RSB into consideration, resulting in a slight increase of the relative capacity [6, 7]. We therefore expect that the situation should be the same for BAM, because T=/β m AT line α α c (T ) Figure 3: AT line for BAM with c = ~c = Figure 4: Overlap m of the retrieval state for BAM with c = ~c = and T = α of the formal similarity between BAM and AAM. This implies that the theoretical prediction about the relative capacity is slightly larger than :998N, which is in excellent agreement with the experimental result. Another curve shown in Figure 2 is the transition temperature T = T sg, at which the solution bifurcates from ~ (above the curve; the socalled \paramagnetic" phase) to ~q 6= (below; the \spin-glass" phase). In both of the two phases m and ~m remain zero, so that the model will fail to retrieve a pattern pair. The value of the overlap m(= ~m) in the solution at zero temperature with c = ~c = is shown in Figure 4. The upper, solid curve shows the stable solution corresponding to the retrieval state. The lower, dashed curve shows the unstable solution. At the critical capacity = :998, these two curves meet with each other, signaling the bifurcation at which the retrieval state disappears. The overlap at the critical capacity is m( = :998) = :93. This value is smaller than that for AAM, which is

5 α, ρ/ α. c ρ/2 Figure 5: ero-temperature capacity and loading ratio of BAM m( = :38) = :967 [5]. This means that the error-correction ability at the critical capacity is lower for BAM than for AAM. The results described so far are for BAM with c = ~c =, i.e., with the two layers having the same number of units. The relative capacity will change when we assume dierent numbers of units for the two layers of BAM. Figure 5 shows the relative capacity = p=n of BAM with ~c = =c at zero temperature, against c. The condition ~c = =c corresponds to xing the total numbers of connections to N 2. The result shows that the relative capacity decreases as the numbers of units in the two layers become dierent. On the other hand, we can take \loading ratio []" as an alternative measure of performance. The loading ratio is dened as the amount of information loaded per one connection. In the case considered here, it is given by p N(c + =c) = N 2 = c + : (3) c =2 = holds when c =, but in general =2. Figure 5 also shows the half of the loading ratio, =2, versus c. Measured by the loading ratio, making c away from causes a small improvement in the memorizing ability. Finally, we mention that the results will change if we assume dierent statistics for the memorized patterns. For example, consider the case where c = ~c = and i follows Prob[ i = ] = =2 while ^ i = i. In this case BAM functions as an autoassociative memory. Under the RS ansatz, the saddle-point equations for this case are Dz tanh ( p rz + m) Dz tanh 2 ( p rz + m) r = q [? (? q)]? (4) 2? 2 (? q) 2 which are dierent from those given in Proposition, reecting the dierence of the statistics assumed for the memorized patterns. It should be noted that these equations (4) are also dierent from the saddle-point equations (2) for AAM, although they coincide in the zero-temperature limit! +. 5 Conclusions In this study, replica analysis was performed on the equilibrium state of BAM, and the relative capacity was evaluated by numerically solving the resulting equations (9). The relative capacity of BAM with c = ~c = for zero temperature was evaluated to be :998N, which is in excellent agreement with the experimentally obtained result, :2N. The small discrepancy can be ascribed to the failure of the RS ansatz at low enough temperature, as in the case of AAM. The capacity when c and ~c deviate from was also shown. Acknowledgments T. T. would like to thank Dr. H. Yanai at Ibaraki University, Japan, for his helpful comments. References [] B. Kosko, \Bidirectional associative memories," IEEE Trans. Systems, Man, Cybern, vol. 8, pp. 49{6, 988. [2] G. Weisbuch and F. Fogelman-Soulie, \Scaling laws for the attractors of Hopeld networks," J. Physique Lett., vol. 46, no. 4, pp. L-623{63, 985. [3] R. J. McEliece, E. C. Posner, E. R. Rodemich, and S. S. Venkatesh, \The capacity of the Hop- eld associative memory," IEEE Trans. Inform. Theory, vol. IT-33, no. 4, pp. 46{482, 987. [4] S. Amari and K. Maginu, "Statistical neurodynamics of associative memory," Neural Networks, vol., pp. 63{73, 988. [5] D. J. Amit, H. Gutfreund, and H. Sompolinsky, "Storing innite numbers of patterns in a spin-glass model of neural networks," Phys. Rev. Lett., vol. 55, pp. 53{533, 985. [6] A. Crisanti, D. J. Amit, and H. Gutfreund, \Saturation level of the Hopeld model for neural network," Europhys. Lett., vol. 2, no. 4, pp. 337{34, 986. [7] K. Tokita, \The replica-symmetry-breaking solution of the Hopeld model at zero temperature: critical storage capacity and frozen eld distribution," J. Phys. A: Math. Gen., vol. 27, no. 3, pp. 443{4424, 994.

6 [8] T. Stiefvater, K.-R. Muller, and R. Kuhn, \Averaging and nite-size analysis for disorder: the Hopeld model," Physica A, vol. 232, nos. {2, pp. 6{73, 996. [9] M. Okada, \A hierarchy of macrodynamical equations for associative memory," Neural Networks, vol. 8, no. 6, pp. 833{838, 995. [] H.-F. Yanai, Y. Sawada, and S. Yoshizawa, \Dynamics of an auto-associative neural network model with arbitrary connectivity and noise in the threshold," Network, vol. 2, pp. 295{34, 99. [] K. Haines and R. Hecht-Nielsen, \A BAM with increased information storage capacity," Proc. IEEE Conf. Neural Networks, San Diego, vol., pp. I-8{9, 988. [2] S. Amari, \Statistical neurodynamics of various versions of correlation associative memory," Proc. IEEE Conf. Neural Networks, San Diego, vol., pp. I-633{64, 988. [3] C. S. Leung, L. W. Chan, and J. Sum, \Attraction basin of bidirectional associative memories," Int. J. Neural Syst., vol. 7, no. 6, pp. 75{725, 996. [4] J. A. Hertz, A. S. Krogh, and R. G. Palmer, Introduction to the Theory of Neural Computation, Addison-Wesley, 99. [5] J. R. L. de Almeida and D. J. Thouless, \Stability of the Sherrington-Kirkpatrick solution of a spin glass model," J. Phys. A: Math. Gen., vol., no. 3, pp. 983{99, 978. [6] V. Dotsenko, An Introduction to the Theory of Spin Glasses and Neural Networks, World Scientic, 994.

(a) (b) (c) Time Time. Time

(a) (b) (c) Time Time. Time Baltzer Journals Stochastic Neurodynamics and the System Size Expansion Toru Ohira and Jack D. Cowan 2 Sony Computer Science Laboratory 3-4-3 Higashi-gotanda, Shinagawa, Tokyo 4, Japan E-mail: ohiracsl.sony.co.jp