1 Introduction Tasks like voice or face recognition are quite dicult to realize with conventional computer systems, even for the most powerful of them

Size: px

Start display at page:

Download "1 Introduction Tasks like voice or face recognition are quite dicult to realize with conventional computer systems, even for the most powerful of them"

Laurence Beasley
5 years ago
Views:

1 Information Storage Capacity of Incompletely Connected Associative Memories Holger Bosch Departement de Mathematiques et d'informatique Ecole Normale Superieure de Lyon Lyon, France Franz Kurfess Department of Computer Science New Jersey Institute of Technology Newark, New Jersey submitted for publication to \Neural Networks", Oct Abstract The memory capacities for auto- and hetero-associative incompletely connected memories are calculated. First, the capacity is computed for xed parameters of the system. Optimization yields a maximum capacity between 0.53 and 0.69 for hetero-association and half of it for auto-association, improving previous results. The maximum capacity requires sparse input and output patterns and grows with increasing connectivity of the memory. Furthermore, parameters can be chosen in such a way that the information content per pattern asymptotically approaches 1. 1

2 1 Introduction Tasks like voice or face recognition are quite dicult to realize with conventional computer systems, even for the most powerful of them. On the contrary, these tasks are easily performed by the human brain despite its highly inferior computation speed. This indicates that it is rather a conceptual problem of the conventional computer model and one could ask for alternatives. An important dierence to the human brain is that some of its functionality is based on an associative memory. The main advantages of this kind of memory is its noise tolerance and the nearly constant retrieval time. Therefore it has been intensively studied in the last decades (Steinbuch 1961, Kohonen 1979, Palm 1980, Willshaw et al 1993) and some interesting applications and theoretical results have been obtained. Most of the theoretical models are fully connected memories, which can be applied to investigate the actual digital implementations of articial associative memories. However, some future realizations like optical or analog associative memories may not possess fully connected layers, due to physical properties of the implementations. Hence incompletely connected models should also be considered to supply a theoretical foundation for these memories. Another reason to study incompletely connected associative memories is that the corresponding parts of the brain are not at all fully connected. For example, the neurons of some parts of the hippocampus are connected to not more than 5% of the neurons in their neighborhood (Amaral et al 1990). The obtained results from these investigations could improve the understanding of the brain, even if the considered models are far from neurobiological reality. A naturally arising question about a memory concerns its storage capacity. For fully connected memories, Palm (1980) calculated it for a simple model and found a maximum capacity of log(2) 0:69 for the heteroassociative memory and log(2)=2 0:34 for the auto-associative memory. Furthermore, he mentioned a capacity of 0.26 for the incompletely connected hetero-associative memory. In contrast to the fully connected memory there's no natural retrieval strategy for the incompletely connected case. Buckingham and Willshaw (1993) investigated several possibilities and analyzed their performances. Our calculations are based on one of their retrieval strategies, for the capacity of a memory depends on the retrieval technique used. 2

3 We will recalculate the capacity for both, the hetero- and the autoassociative memory, and improve the value of 0.26 given in (Palm 1980). We obtain for the maximum capacity C of hetero-association an increasing function between 0.53 for low connectivity and 0.69 for full connectivity, conrming Palm's result of C = log(2). Auto-association achieves maximum capacities between 0.27 and Description of the Model This section contains a description of the associative memories on which the following calculations are based. The hetero-associative case is described in detail; the auto-associative memory discussed in the second section diers only slightly from the hetero-associative memory. 2.1 Hetero-association The memory considered here is similar to the completely connected model used in (Palm 1980). The input patterns X s and the output patterns Y s are binary vectors of nite dimension, i.e. X s 2 f0; 1g M and Y s 2 f0; 1g N. A total number of R associations between input and output patterns should be stored in the memory, i.e. s = 1::R. The neural network used for the memory task consists of one input and one output layer of M respectively N neurons, binary weights w ij 2 f0; 1g for the existing synapses between the two layers and threshold functions i for the output units which are set individually during the retrieval phase. The two layers are incompletely connected and therefore the net can be represented as a partially lled matrix W 2 f0; 1; bg N M in which the blank b stands for a missing matrix element. The memory is trained with a one{step Hebbian learning rule and the values for the existing weights w ij are calculated by: w ij = _ s=1::r (Y s i X s j ) To retrieve the response for an input pattern X rst the vector Y 0 = W X is calculated where the sums in the matrix product are over the existing 3

4 weights w ij. To obtain the nal response Y the threshold functions are determined and applied to Y 0 : MX Y i = i ( w ij X j ) j=1 The calculation of the memory capacity requires some further specications of the model. The number K of \ones" in the output pattern Y s is xed for all patterns in order to simplify the correction of the obtained response. To obtain a consistent model the number L of \ones" in the input pattern Y s is also xed. This restriction can be replaced by a probability for the occurrence of a "one" in an input unit, which gives roughly the same results. Furthermore the net contains a total of ZMN uniformly distributed connections between the input and the output layer where Z, 0 < Z 1, is the connectivity of the network. The threshold functions are set by using the activities of the input pattern X (see Buckingham and Willshaw 1993). The individual activity A i for an output unit Y i is dened as A i = X j X j where the sum is over the existing synapses w ij between input neuron i and output neuron j. Note that the activities A i can range from 0 to L, depending on the distribution of the existing connections. The threshold for the output unit Y i is set to i = A i which means that an output neuron res if its dendritic sum equals the input activity A i. Hence, if a learned input pattern X s is applied to the net, the answer contains all the K \ones" of its answer Y s but may contain O additional wrong \ones". In this point the net has the same behavior as Palm's model. The described retrieval strategy can be eciently implemented by switching to the complement matrix W of W, i.e. w ij = 1? w ij : An output unit returns \one" if its dendritic sum equals its input activity, i.e. if the existing connections between the high input units and the output unit are high. Switching to the complement matrix, this is equivalent to corresponding low connections, i.e. if its dendritic sum equals zero. Y i = 1? 4 _ j w ij X j

5 Therefore no additional calculations are required to retrieve the output vector Y. 2.2 Auto-association The same model as for the hetero-association is used, except for the following natural modications: The dimensions of the in- and output vectors are the same, i.e. N = M and both vectors contain the same number of \ones", i.e. K = L. Note that the obtained matrix is symmetrical and therefore only roughly half of it is needed, which explains the lower capacity for auto-association. Having specied the associative memories as targets of our investigations, we can now proceed to calculate the memory capacities. 3 Capacity of the Hetero-associative Memory First, the capacity is calculated for a given memory. In a next step, this capacity is optimized with respect to the connectivity Z. The obtained values are then numerically veried. 3.1 Theoretical Results for Fixed Parameters The following calculations are partly inspired by the calculations for the completely connected model from (Palm 1980). First, the expectation value of the information stored in the memory is determined. The denition of information is based on the corresponding expression of Shannon's Information Theory. The total information I is the sum of the individual informations stored per output pattern I s : I = s=1 I s is the dierence between the information contained in the output pattern Y s and the information needed to determine this pattern given the memory's RX I s 5

6 response to X s : I s = ld N K!? ld O + K K Therefore we obtain for the expectation value of the total information E(I) = R E " ld K?1 X = R E i=0 K?1 X = R E i=0 N K ld ld!? ld N? i K + O? i N? i K + O? i! O + K K Using the general inequation 1 E(ldA) lde(a) a lower bound can be derived: K?1 X N? i E(I) R ld (1) i=0 K + E(O)? i The expectation number of the wrong \ones" E(O) in the response needs to be determined next. E(O) depends highly on the distributions of \ones" in the connection matrix. Unfortunately the occurrences of \ones" in the same row or column are not independent. But for sparse 2 input patterns and a large number of stored patterns the dependence is very weak. Since the maximum capacity is reached for this choice of parameters, the independence of the occurrences is assumed for this calculation. For a more detailed discussion see (Palm 1980). The expectation number E(O) of wrong \ones" is given by!!# E(O) = (N? K)Pr ( (W X) i = A i jy i = 0) (W X) i = A i holds if and only if all existing connections between the output and the high input units have high weights. Since independence for the weights is assumed we obtain: E(O) (N? K)(1? Z Pr(w ij = 0)) L = (N? K) 1? Z 1 LK? MN R?1! L 1 which derives from the algebraic geometric inequation 2 Here, sparse means that the input vector contains only a few \ones". (2) 6

7 which completes the calculation of the stored information. The expectation value for the memory capacity E(C) is then given by E(C) = E(I) ZMN for the memory contains a total of ZMN storage units. 3.2 Maximum Capacity In order to maximize the capacity, some approximations are required. As (A? i)=(b? i) A=B for A B, and N K + E(O), we obtain by (1) N E(I) RKld K + E(O) For sparse output patterns, i.e. i K << N, this gives a tight approximation. The substitution of E(O) by (2) yields: E(I) RK ld RK ld RK ld = RKL ld N K + (N? K) 1? Z 1? LK MN N?R LK K + (N? K) 1? Ze MN N?R LK N 1? Ze MN 1?R LK 1? Ze MN L R?1 L L (3) The approximation in (3) overestimates the memory capacity, but the real value will be close to the estimation if K << E(O). This requires sparse input patterns and a certain loss of information, i.e. there are many more wrong than genuine \ones" in the answer. But if furthermore E(O) << N, even a high percentage of the information can be stored (see Section 5). Now, writing R = p(z) MN for the number of stored patterns, the expectation value of the total capacity LK is: 1 E(C) p(z)ld 1? Ze?p(Z) (4) 7

8 p(z) Z Figure 1: Optimal values for p(z) where R = p(z) MN LK stored patterns. is the number of Numerical optimization of E(C) in (4) yields values for p(z) between 0.7 and 1.0 (Figure 1). With these values we obtain a theoretical maximum capacity between 0.53 and 0.69 which depends only on the connectivity Z (Figure 2). This is much higher than the previously estimated 0.26 (Palm 1980) and brings the capacities of incompletely connected networks close to those of completely connected memories. 3.3 Numerical Verication To verify the obtained result the capacity has been optimized for M = N = 100 (Figure 3) and M = N = 1000 (Figure 4), testing a wide range of parameters. Both gures contain the theoretical capacity from Figure 2, the eective capacity from the optimization and the capacity for K = L = 2 and R = p(z) MN LK which is called capacity for estimated parameters. For M = N = 100, the theoretical and the real capacities are indistinguishable whereas the capacity for estimated parameters is not yet optimal. For M = N = 1000 all three capacities are indistinguishable, conrming the theoretical result. For the verication, the formulas (1) and (2) were used even if they don't give the exact capacity, since the exact formula requires very time-intensive 8

9 C Z Figure 2: Maximum capacity of the incompletely connected memory. The capacity C increases with the connectivity Z. C Z Figure 3: For 100 neurons the theoretical and the real capacity are indistinguishable (upper curve) whereas the capacity for estimated parameters is not yet optimal (lower curve). 9

10 C Z Figure 4: For 1000 neurons the theoretical, the real and the capacity for estimated parameters are indistinguishable. calculations. The obtained results were conrmed by the exact formula for the optimal parameters, showing in these cases only insignicantly dierent capacities. In this chapter the capacity for hetero-associative memories has been calculated and then optimized for xed connectivity. These theoretical values have been conrmed by numerical evaluations of the exact formula. 4 Capacity of the Auto-associative Memory The corresponding calculations are performed for the auto-associative memory. The capacity is optimized in the same manner as for hetero-association and is therefore only briey outlined. Since the result is quite similar to hetero-association no numerical verications were performed. The amount of information stored in the auto-associative memory is de- ned in the same way as for the hetero-associative memory. Clearly the correction information per pattern must be dened dierently, since the input pattern equals the output pattern. To retrieve a pattern the high units are determined successively, one at each step, starting with an \empty" vector. A step consists of the application of the actual pattern to the memory 10

11 and the determination of a further genuine \one" among the obtained high output units. The sum of the information of these determinations equals the correction information. Hence the information stored in the memory is given by 3 E(I) R K?1 X i=0 ld N? i K + E(O i )? i where E(O i ) is the expectation number of spurious \ones" at the i th step. Note that the applied input pattern at the i th step contains exactly i \ones". Similar to the hetero-association case E(O i ) (N? K)(1? Z Pr(w ij = 0)) i = (N? K) and comparable approximations yield and nally, with R = p(z) N 2 K?1 X K 2! 1 R?1 i 1? Z 1? N 2 A 0 1i A E(I) R 1 K2?R i=0 1? Ze N 0 2 = )ld K 2 we obtain 1? Ze K2?R N 2 E(C) 1 2 p(z)ld 1 1? Ze?p(Z) Considering that only half of the symmetrical matrix is necessary the result is the same as in the case of hetero-association and the same optimization yields values between 0.26 and 0.34 for the maximum capacity. 5 Information Content per Pattern Besides the maximum capacity of an associative memory the information stored per pattern is also of interest. The corresponding value will be calculated and then optimized. Again, numerical verications conrm the theoretical results. 3 see (Palm 1980) for the derivation of the formula 1 A 11

12 c(z) Z Figure 5: c(z) log N \ones" in the input pattern are required to store almost the entire information of the pattern. The information contained in a pattern is I p = ld N K! KldN in the case of sparse output patterns. The condition RKldN = E(I) guarantees that roughly all information of the patterns is stored since RKldN is the total information of the patterns. For the optimal information E(I) = 1 RKLld this provides the following condition for L: 1?Ze?p(Z) L log N log 1 1?Ze?p(Z) = c(z) log N (5) where c(z) is the function shown in Figure 5. Low connectivity requires a larger number of input \ones" than high connectivity, for a certain level of activity must be present to obtain a high percentage of information storage. Based on the above value of L the average activity A = ZL is a decreasing function of connectivity, resulting in values between 2:7 log N and 1:4 log N. Remember the necessary condition K << E(O) in section 3.2 to obtain 12

13 a high capacity. It can be modied to: K << (N? K)(1? Ze?p(Z) ) L N(1? Ze?p(Z) ) L Applying the logarithm yields log K < log N + L log(1? Ze?p(Z) ) which gives another condition for L: L < log N? log K log 1 = c(z)(log N? log K) (6) 1?Ze?p(Z) Thus, comparing the conditions (5) and (6), high capacity and high percentage of stored information are achieved if log K << log N and L = c(z) log N << N. In particular for small connectivity this is reached only for very large N. It is striking that, neglecting log K in (6), the same function, i.e. c(z) log N, is derived by two totally dierent approaches. Figure 6 shows the maximum capacity and the corresponding fraction of stored information for a very large memory. The number of \ones " is chosen between c log N and c(log N? log K) which results in high capacities and high information contents. For L = c(z)(log N? log K), maximum capacity is given whereas L = c(z) log N leads to an approximately complete information storage. The dependency of the capacity on the number of \ones" in the input pattern is illustrated in Figure 7 for connectivity Z = 0:5. The number of patterns has been optimized to obtain maximum capacity. The increasing function represents the information content per pattern. Figure 8 shows the capacity and the information content for an increasing number of patterns R and connectivity Z = 0:5. The capacity increases rst with the number of pattern before it begins to decrease when too many \ones" in the matrix are stored. The information content is a steadily decreasing function as expected. The previous results show that high information content per pattern is possible and can be realized together with high capacity for appropriate choices of parameters. 13

14 Z Figure 6: Maximum capacity (upper curve) and information content for N = M = 10 20, K = log N, L = (Z)(log N? 1 MN 2 log K and R = p(z). LK L Figure 7: Maximum capacity (increasing curve) and information content as function of L. The xed parameters are N = M = 1000, K = 10 and Z = 0:5 whereas the number of stored patterns is optimized. The capacity decreases with L whereas the information content increases with the number of \ones". 14

15 ,000 20,000 R Figure 8: Capacity and information content (decreasing function) for increasing numbers of patterns R. The xed parameters are N = M = 1000, Z = 0:5, K = 10 log N and L = 30 c(0:5) log N. 6 Discussion The memory capacities of incompletely connected associative memories have been investigated. The maximum capacities range from 0.53 to 0.69, depending on the connectivity of the network, and can be obtained for sparse input and output patterns and convenient numbers of stored associations. Additionally, more restricted choices of parameters guarantee a high information storage per pattern which asymptotically approaches 1. The memory loses its stored information if too many weights are set to \one" in the matrix. Sparse input patterns achieve therefore better storage capacities, since the information content does not increase with the number of \ones" in the input patterns, whereas the number of \ones" in the matrix is increased. On the other hand, a large number of \ones" in the input pattern leads to high information content. Memories with low connectivity require more \ones" in the input patterns in order to achieve a high information content, since a minimum activity is needed. The advantage of our model is that only spurious \ones" {and no \zeros"{ occur in the answer and that the number of output \ones" K is xed. There- 15

16 fore no information is required to determine the number of wrong \ones" O. These specications result in a high capacity, but are a restriction for the biological validity of the model. Furthermore, our retrieval strategy implies that an output unit is set to \high" if it has no connection to any of the high input units, i.e. if the activity is zero. This does not seem biological plausible and shows the limit of our model. Further work should investigate the memory capacity for other retrieval strategies, modied learning rules or dierent architectures of associative incompletely connected memories. The present work can be seen as basis for such investigations. 16

17 7 List of Symbols Used A i C c(z) I I s K L M N O p(z) R W X Y Z i individual activity capacity of the memory constant for optimal number of input \ones" stored information stored information per pattern number of \ones" in the output pattern number of \ones" in the input pattern length of the input pattern length of the output pattern number of wrong \ones" constant for optimal number of patterns number of stored patterns weight matrix input pattern output pattern connectivity of the network threshold function 17

18 References [1] Amaral, D. G., Ishizuka, N., & Claiborne, B. (1990). Neurons, numbers and the hippocampal network. progress in Brain Research, 83, [2] Buckingham, J., & Willshaw, D. (1983). On setting unit thresholds in an incompletely connected associative net. Network, 4, [3] Hogan, J., & Diederich, J. (1995). Random Neural Networks of Biologically Plausible Connectivity. Technical Report, Queensland University of Technology, Australia. [4] Kohonen, T. (1979). Content-Addressable memories. New York: Springer Verlag. [5] Palm, G. (1980). On associative memory. Biological Cybernetics, 36, [6] Palm, G., Kurfess, F., Schwenker, F., & Strey, A. (1993). Neural associative memories. Technical Report, Universitat Ulm, Germany. [7] Steinbuch, K. (1961). Die Lernmatrix. Kybernetik, 1, 36. [8] Vogel, D., & Boos W. (1993). Minimally connective, auto-associative, neural networks. Connection Science, Vol. 6, No. 4,

Information storage capacity of incompletely connected associative memories

Information storage capacity of incompletely connected associative memories Holger Bosch a, Franz J. Kurfess b, * a Department of Computer Science, University of Geneva, Geneva, Switzerland b Department