Alpha Beta bidirectional associative memories: theory and applications

Transcription

1 Neural Processing Letters (7) 6: 4 DOI.7/s Alpha Beta bidirectional associative memories: theory and applications María Elena Acevedo-Mosqueda Cornelio Yáñez-Márquez Itzamá López-Yáñez Received: October 6 / Accepted: April 7 / Published online: June 7 Springer Science+Business Media B.V. 7 Abstract In this work a new Bidirectional Associative Memory model, surpassing every other past and current model, is presented. This new model is based on Alpha Beta associative memories, from whom it inherits its name. The main and most important characteristic of Alpha Beta bidirectional associative memories is that they exhibit perfect recall of all patterns in the fundamental set, without requiring the fulfillment of any condition. The capacity they show is min(n,m),beingn and m the input and output patterns dimensions, respectively. Design and functioning of this model are mathematically founded, thus demonstrating that pattern recall is always perfect, with no regard to the trained pattern characteristics, such as linear independency, orthogonality, or Hamming distance. Two applications illustrating the optimal functioning of the model are shown: a translator and a fingerprint identifier. Keywords Bidirectional associative memories Alpha Beta associative memories Perfect recall Fingerprint identifier Introduction The area of Associative Memories, as a relevant part of Computer Science, has achieved ample importance and dynamism in the activities developed by numerous research groups around the globe, specifically among those working in topics related to theory and applications of pattern recognition and classification. M. E. Acevedo-Mosqueda (B) C. Yáñez-Márquez I. López-Yáñez Laboratorio de Inteligencia Artificial, Centro de Investigación en Computación, Instituto Politécnico Nacional, Av. Juan de Dios Bátiz s/n, México, DF 7738, México eacevedo@ipn.mx C. Yáñez-Márquez cyanez@cic.ipn.mx I. López-Yáñez ilopezb5@sagitario.cic.ipn.mx 3

2 M. E. Acevedo-Mosqueda et al. The fundamental purpose of an associative memory is to correctly recall complete patterns from input patterns, which may be altered with additive, subtractive, or mixed noise []. In the design of an associative memory there are two phases: before the pattern recalling phase comes the learning phase, during which the associative memory is built. In order to do so, associations are made; these associations are pairs of patterns, one input pattern and the other an output pattern. If for every association the input pattern is equal to the output pattern, the resulting memory is autoassociative; otherwise, the memory is heteroassociative. The latter means that an autoassociative memory can be considered as a particular case of an heteroassociative memory []. Through out time, Associative Memories have been developed in parallel to Neural Networks, from the conception of the first model of artificial neuron [3], to neural networks models based on modern concepts, such as Morphological Mathematics [4], going through such important works as those by the pioneers in perceptron-like neural networks models [5 7]. In 98 John J. Hopfield presents to the world his associative memory, model inspired by physics concepts which also has the particularity of an iterative algorithm [8]. This work incited a renewal of researchers interest on topics regarding both associative memories as well as neural networks, which had been dormant for some years. In this sense, neural networks had a recess of 3 years, parting from the publication in 969 of the book Perceptrons [9] by Minsky and Papert. In it, the authors demonstrated that the perceptron had severe limitations. On the side of associative memories, it was Karl Steinbuch who, in 96, developed an heteroassociative memory able to work as a binary pattern classifier. He called this first model of associative memory Lernmatrix []. Eight years later, Scottish researchers Willshaw et al. [] presented the Correlograph, elemental optical device capable of working as an associative memory. There is also another important antecedent to Hopfiled memory: two classical model of associative memory were independently presented in 97 by Anderson [] and Kohonen [3]. Due to their importance and the similitude of the involved concepts, both models are generically named Linear Associator. The work presented by Hopfield has enormous relevance since his model of neural network demonstrates that the interaction of simple processing elements, similar to neurons, give place to the rise of collective computational properties, such as the stability of memories. However, the Hopfield model of associative memory has two disadvantages: first, it is notable the small pattern recalling capacity, being only.5n, where n is the dimension of the stored patterns. Second, the Hopfield memory is only autoassociative; that is, it is incapable of associating patterns which are different. With the intention of solving the second disadvantage of the Hopfield model, in 988 Kosko [4] created a model of heteroassociative memory parting from the Hopfield memory: the Bidirectional Associative Memory (BAM), which is based on an iterative algorithm, the same as the Hopfield memory. The main characteristic of Kosko BAM is the heteroassociativity, made possible on the basis of the Hopfield model. Kosko proposed a solution based on the same matrix representing the Hopfield model, with which he was capable of realizing the learning phase in both directions: to obtain an output pattern from an input pattern and vice versa, to obtain an input pattern from an output pattern. From this comes the name of bidirectional. Even though the Kosko model was successful in obtaining a heteroassociative memory, the other aforementioned Hopfield memory disadvantage was not solved by the BAM: the Kosko bidirectional associative memory has a very low pattern learning and recovering capacity, dependant on the minimum of the dimensions of the input and output patterns. 3

3 Alpha Beta bidirectional associative memories 3 The same year of publication of the BAM, 988, the first tries of scientific research to improve the BAM pattern learning and recovering capacity appeared [5]. Since then, diverse research groups from around the globe have tried, through the years, to minimize this disadvantage of the BAM. In order to do so, many scientific concepts and mathematical techniques have been used, from multiple training to Householder coding [6] and genetic algorithms [7], and in some cases linear programming. However, the achievements have been modest and scarce. Actually, the vast majority of proposals of new bidirectional associative memory models presented in scientific journals and high level symposiums, do not even guarantee the fundamental set complete recovery. In other words, these models are not capable of recovering all of the learned patterns, and they fail in one or more [4 43]. In the present work, we present the theoretical foundation on which is based the design and functioning of the Alpha Beta bidirectional associative memories, which exhibit perfect recall of all patterns in the fundamental set. Alpha Beta associative memories In this section, basic concepts about associative memories are presented. Also, and since Alpha Beta associative memories are the basis for Alpha Beta bidirectional associative memories, the theoretical foundation of Alpha Beta associative memories is presented, as described in [44].. Basic concepts The fundamental purpose of an associative memory is to correctly recall complete patterns from input patterns, which may be altered with additive, substractive, or mixed noise. The concepts used in this section are presented in [,,44]. An Associative Memory can be formulated as an input output system, idea that is schematized as follows: x M y In this diagram, input and output pattern are represented by column vectors, denoted by x and y, respectively. Every input pattern makes up an association with its corresponding output pattern, similarly to an ordered pair. For instance, the patterns x and y in the former diagram make up the association (x, y). Input and output patterns will be denoted by bold letters, x and y, adding natural numbers as superscript for symbolic discrimination. For instance, to an input pattern x corresponds the output pattern y, and together they form the association (x, y ). In the same manner, for a specific positive integer number k, the corresponding association will be (x k, y k ). Associative memory M is represented by a matrix generated from a finite set of associations, known beforehand: this is the fundamental set of associations, orsimply fundamental set. The fundamental set is represented as follows: {(x µ, y µ ) µ =,,...,p} where p is a positive, integer number representing the cardinality of the fundamental set. 3

4 4 M. E. Acevedo-Mosqueda et al. The patterns that make up the associations of the fundamental set are called fundamental patterns. The nature of the fundamental set gives us an important criterion by which it is possible to classify associative memories: AmemoryisAutoassociative if it holds that x µ = y µ µ {,,...,p}, then one of the requisites is that n = m. AmemoryisHeteroassociative is that where µ {,,...,p} for which x µ = y µ. Notice that there can be heteroassociative memories with n = m. In problems where associative memories intervene, two important phases are considered: the learning phase, where the associative memory is generated from the p associations of the fundamental set, and the recalling phase, where the associative memory operates on an input pattern, as shown in the diagram presented at the beginning of this section. In order to specify the patterns components, the notation of two sets, which we will arbitrarily call A and B, is required. The components of the column vectors representing patterns, both input and output, will be elements from the set A, while the components of matrix M will be elements from the set B. There are no prerequisites or limitations with respect to the choice of the two sets, and therefore they do not need to be different or have special characteristics. This makes the number of possibilities to choose A and B infinite. By convention, every column vector representing an input pattern has n components whose values belong to the set A, and every column vector representing an output pattern has m components whose values belong to the set A. Inotherwords: x µ A n yy µ A m µ {,,...,p} The jth component of a column vector is indicated by the same letter of the vector, not in bold, putting a j as subscript ( j {,,..., n} or j {,,..., m}, accordingly). The jth component of a column vector x µ is represented by x µ j With the described basic concepts and the former notation, it is possible to express the two phases of an associative memory:. Learning phase (Generation of the associative memory). Find the adequate operators and a way to generate a matrix M that will store the p associations of the fundamental set {(x, y ), (x, y ),...,(x p, y p )},wherex µ A n and y µ A m µ {,,...,p}. If µ {,,...,p} such that x µ =y µ, the memory will be heteroassociative; if m = n and x µ = y µ µ {,,...,p}, the memory will be autoassociative.. Recalling phase (Operation of the associative memory). Find the adequate operators and sufficient conditions to obtain the output fundamental pattern y µ, when the memory M is operated with the input fundamental pattern x µ. The latter must hold for every element of the fundamental set and for both modes: autoassociative and heteroassociative. It is said that an associative memory M exhibits perfect recall if, during the recalling phase, when presented with a pattern x ω as input (ω {,,...,p}), M answers with the corresponding output fundamental pattern y ω. A bidirectional associative memory is also an input output system, only that the process is bidirectional. The forward direction is described in the same manner as a common associative memory: when presented with an input x, the system delivers an output y. The backward direction takes place presenting to the system an input y in order to receive and output x. 3

5 Alpha Beta bidirectional associative memories 5. Alpha Beta Associative Memories In this section the definitions of the α and β operations, and the matrix operations that make use of the original operations, are presented. Emphasis is put in the Alpha Beta associative memory learning and recalling phases, both for the max and min kinds, since they are fundamental to the Alpha Beta BAM design. Also, we present two Theorems that set the theoretical basis for the Alpha Beta Bidirectional Associative Memories. The theoretical foundation Theorems for the Alpha Beta autoassociative memories are numbered according to the original numeration appearing in [44]. The Alpha Beta associative memories are of two kinds and are able to operate in two different modes. The operator α is useful at the learning phase while the operator β is the basis for the pattern recall phase. The heart of the mathematical tools used in the Alpha Beta model, are two binary operators designed specifically for these memories. These operators are defined as follows: First, we define the sets A ={, } and B ={,, }, then the operators α and β are defined in tabular form: α : A A B β : B A A x y α(x, y) x y β(x, y) The sets A and B, theα and β operators, along with the usual (minimum) y (maximum) operators, form the algebraic system ( A, B, α, β,, ) which is the mathematical basis for the Alpha Beta associative memories. Definition of four matrix operation is required. Of these operations, only four particular cases will be used: [ αmax Operation : P mxr α Q rxn = ] fij α, m n where f ij α = r k= α(p ik, q kj ) [ βmax Operation : P mxr β Q rxn = ] f β ij, m n where f β ij = r k= β(p ik, q kj ) αmin Operation : P mxr α Q rxn = [ ] hij α, where m n hα ij = r α(p ik, q kj ) k= βmin Operation : P mxr β Q rxn = [ ] h β ij, where m n hβ ij = r β(p ik, q kj ) k= Below are shown some simplifications obtained when these four operations are applied to vectors: 3

6 6 M. E. Acevedo-Mosqueda et al. Let x A n and y A m ;theny α x t is a matrix with dimensions m n, and it holds that: k= α(y, x ) k= α(y, x )... k= α(y, x n ) k= α(y, x ) k= α(y, x )... k= α(y, x n ) y α x t = y α x t = k= α(y m, x ) k= α(y m, x )... k= α(y m, x n ) The symbol represents both operations α and α when a column vector of dimension m is operated with a row vector of dimension n: y α x t = y x t = y α x t Let x A n and P be a matrix of dimension m n: Operation P m n β x yields as a result a column vector of dimension m, whose ith component has the form: ( P m n β x ) i = n j= β ( p ij, x j ). Operation P m n β x yields as a result a column vector of dimension m, whose ith component has the form: ( P m n β x ) i = n j= β ( p ij, x j )..3 Alpha Beta Autoassociative Memories Below are shown some characteristics of Alpha Beta autoassociative memories:. The fundamental set takes the form {(x µ, x µ ) µ =,,...,p}.. Both input and output fundamental patterns are of the same dimension, denoted by n. 3. The memory is a square matrix, for both kinds, V (max) and (min). If x µ A n then: V = [ ] v ij n n and = [ λ ij ]n n. Next, the learning and recalling phases of the Alpha Beta autoassociative memories max are presented. m n Learning Phase Step. Foreveryµ =,,,p, from the pair (x µ, x µ ) is built the matrix: [ x µ (x µ ) t] n n Step. The binary operator max is applied to the matrices obtained in step : V = p µ= [ x µ (x µ ) t] The ijth entry is given by the following expression: v ij = p ( ) α x µ µ= i, x µ j and since α: A A B, wehavethatv ij B, i {,,...,n}. j {,,...,n}. 3

7 Alpha Beta bidirectional associative memories 7 Recalling Phase The Alpha Beta autoassociative memories of kind recalling phase has two possible cases. In the first case, the input pattern is a fundamental pattern. That is, the input is a pattern x ω, with ω {,,...,p}. In the second case, the input pattern is NOT a fundamental pattern, but a distorted version of at least one of the fundamental patterns. This means that if the input pattern is x, there must exist at least one index value ω {,,...,p} corresponding to the fundamental pattern in respect to which x is an altered version by one of the three types of noise: additive, substractive or mixed. Case Fundamental pattern. A pattern x ω, with ω {,,...,p}, is presented to the autoassociative Alpha Beta memory of type and we do operation β :V β x ω. The result of the former operation will be a column vector of dimension n. ( V β x ω) {[ ] } i = n β(v ij, x ω j ) = n p β j= j= µ= α(xµ i, x µ j ), x ω j Case Altered pattern. A binary pattern x (an altered pattern of some fundamental pattern x ω ), which is a column pattern of dimension n, is presented to the autoassociative Alpha Beta memory of type and the operation β is done: V β x. As in Case, the result of the former operation is a column vector of dimension n, whose ith component is expressed in the following manner: ( V β x ) {[ ] } i = n β(v ij, x j ) = n p β j= j= µ= α(xµ i, x µ j ), x j Theorem 4.3 Let {(x µ, x µ ) µ =,,...,p} be the fundamental set of an autoassociative Alpha Beta memory of type represented by V, and let x A n be a pattern altered with additive noise with respect to some fundamental pattern x ω, with ω {,,...,p}. If x is presented to V as input, and also for every i {,...,n} it holds that j = j {,...,n}, which is dependant on ω and i such that v ij α(x ω, x j ), then recall V β x is perfect; that is to say that V β x = x ω. Theorem 4.3 shows us that autoassociative Alpha Beta memories of type are immune to a certain amount of additive noise. Next, the learning and recalling phases of the Alpha Beta Autoassociative Memories min are presented. Learning Phase Step. Foreveryµ =,,,p, from the pair (x µ, x µ ) is built the matrix: [ x µ (x µ ) t] n n Step. The binary operator min is applied to the matrices obtained in step : = p µ= [ x µ (x µ ) t] The ijth entry is given by the following expression: λ ij = p µ= α ( x µ i, x µ j and since α: A A B, wehavethatλ ij B, i {,,...,n}. j {,,...,n}. ). 3

8 8 M. E. Acevedo-Mosqueda et al. Recalling Phase The Alpha Beta autoassociative memories of kind recalling phase has two possible cases. In the first case, the input pattern is a fundamental pattern. That is, the input is a patternx ω, with ω {,,...,p}. In the second case, the input pattern is NOT a fundamental pattern, but a distorted version of at least one of the fundamental patterns. This means that if the input pattern is x, there must exist at least one index value ω {,,...,p} corresponding to the fundamental pattern in respect to which x is an altered version by one of the three types of noise: additive, substractive or mixed. Case Fundamental pattern. A pattern x ω, with ω {,,...,p}, is presented to the autoassociative Alpha Beta memory of type and we do operation β : β x ω. The result of the former operation will be a column vector of dimension n. ( β x ω) {[ ] } i = n β(λ ij, x ω j ) = n p β j= j= µ= α(xµ i, x µ j ), x ω j Case Altered pattern. A binary pattern x (an altered pattern of some fundamental pattern x ω ), which is a column pattern of dimension n, is presented to the autoassociative Alpha Beta memory of type and the operation β is done: β x. As in Case, the result of the former operation is a column vector of dimension n, whose ith component is expressed in the following manner: ( β x ) {[ ] } i = n β(λ ij, x j ) = n p β j= j= µ= α(xµ i, x µ j ), x j Theorem 4.33 Let {(x µ, x µ ) µ =,,...,p} be the fundamental set of an autoassociative Alpha Beta memory of type represented by, and let x A n be a pattern altered with substractive noise with respect to some fundamental pattern x ω, with ω {,,...,p}. If x is presented to memory as input, and also for every i {,...,n} it holds that j = j {,...,n}, which is dependant on ω and i, such that λ ij α(x ω, x j ), then recall β x is perfect; that is to say that β x = x ω. Theorem 4.33 confirms that the autoassociative Alpha Beta memory of type are immune to a certain amount of substractive noise. 3 Alpha Beta Bidirectional Associative Memory Before going into detail over the processing of an Alpha Beta BAM, we will define the following. In this work we will assume that Alpha Beta associative memories have a fundamental set denoted by {(x µ, y µ ) µ =,,...,p}x µ A n and y µ A m, with A ={, }, n Z +, p Z +, m Z + and < p min( n, m ). Also, it holds that all input patterns are different; M that is x µ = x ξ if and only if µ = ξ. If µ {,,...p} it holds that x µ = y µ, the Alpha Beta memory will be autoassociative; if on the contrary, the former affirmation is negative, that is µ {,,...,p} for which it holds that x µ = y µ, then the Alpha Beta memory will be heteroassociative. Definition (One-hot) Let the set A be A ={, } and p Z +, p >, k Z +,such that k p. Thekth one-hot vector of p bits is defined as vector h k A p for which it holds that the kth component is h k k = and the ret of the components are hk j =, j = k, j p. 3

9 Alpha Beta bidirectional associative memories 9 x Stage Stage y Stage 4 Stage 3 Fig. Graphical schematics of the Alpha Beta bidirectional associative memory x Stage k h v = ( h ( () () k ) ) p Stage Modified Linear Associator y Fig. Schematics of the process done in the direction from x to y. Here are shown only Stage and Stage. Notice that vkk =,vik = i = k, i p, k p Definition (Zero-hot) Let the set A be A ={, } and p Z +, p >, k Z +,such that k p. Thekth zero-hot vector of p bits is defined as vector h k A p for which it holds that the kth component is h k k = and the ret of the components are hk j =, j = k, j p. Definition 3 (Expansion vectorial transform) Let the set A be A ={, } and n Z +, ym Z +. Given two arbitrary vectors x A n and e A m, the expansion vectorial transform of order m, τ e : A n A n+m,isdefinedasτ e (x, e) = X A n+m, a vector whose components are: X i = x i for i n and X i = e i for n + i n + m. Definition 4 (Contraction vectorial transform) Let the set A be A ={, } and n Z +, ym Z + such that m < n. Given one arbitrary vector X A n+m, the contraction vectorial transformoforderm, τ c : A n+m A m,isdefinedasτ c (X, m) = c A m, a vector whose components are: c i = X i+n for i < m. In both directions, the model is made up by two stages, as shown in Fig.. For simplicity, first will be described the process necessary in one direction, in order to later present the complementary direction which will give bidirectionality to the model (see Fig. ). ThefunctionofStageistoofferay k as output (k =,...,p) givenax k as input. Now we assume that as input to Stage we have one element of a set of p orthonormal vectors. Recall that the Linear Associator has perfect recall when it works with orthonormal vectors. In this work we use a variation of the Linear Associator in order to obtain y k, parting from a one-hot vector v k in its kth coordinate. For the construction of the modified Linear Associator, its learning phase is skipped and a matrix M representing the memory is built. Each column in this matrix corresponds to each output pattern y µ.inthisway,whenmatrixm is operated with a one-hot vector v k,the corresponding y k will always be recalled. The task of Stage is: given a x k or a noisy version of it ( x k ),theone-hot vector v k must be obtained without ambiguity and with no condition. 3

10 M. E. Acevedo-Mosqueda et al. The process in the contrary direction, which is presenting pattern y k (k =,...,p) as input to the Alpha Beta BAM and obtaining its corresponding x k, is very similar to the one described above. The task of Stage 3 is to obtain a one-hot vector v k given a y k.stage4isa modified Linear Associator built in similar fashion to the one in Stage. 3. Theoretical foundation of Stages and 3 Below are presented five Theorems and nine Lemmas with their respective proofs, as well as an illustrative example of each one. This mathematical foundation is the basis for the steps required by the complete algorithm, which is presented in Sect. 3.. These Theorems and Lemmas numbering corresponds to the numeration used in [45]. By convention, the symbol will be used to indicate the end of a proof. Theorem 4. Let {(x µ, x µ ) µ =,,...,p} be the fundamental set of an autoassociative Alpha Beta memory of type max represented by V, and let x A n be a pattern altered with additive noise with respect to some fundamental pattern x ω with ω {,,...,p}. Lets assume that during the recalling phase, x is presented to memory V as input, and lets consider an index k {,...,n}. The kth component recalled ( V β x ) k is precisely xω k if and only if it holds that r {,...,n}, dependant on ω and k, such that ν kr α(x ω k, x r ). Proof ) By hypothesis we assume that ( V β x ) k = xω k. By contradiction, now suppose false that r {,...,n} such that ν kr α(xk ω, x r ). The former is equivalent to stating that r {,...,n}ν kr >α(xk ω, x r ),whichisthesametosayingthat r {,...,n}β (ν kr, x r ) > β [ α(xk ω, x ] r ), x r = x ω k. When we take minimums at both sides of the inequality with respect to index r, wehave: n r= β (ν kr, x r ) > n r= x ω k = xω k and this means that ( V β x ) k = n β (ν kr, x r ) > x ω r= k, which contradicts the hypothesis. ) Since the conditions of Theorem 4.3 hold for every i {,...,n}, wehavethat V β x = x ω ; that is, it holds that ( V β x ) = x ω i i, i {,...,n}. When we fix indexes i ( and j such that i = k y j = r (which depends on ω and k) we obtain the desired result: V β x ) k = xω k. Example 3. Let p = 4andn = 4. The fundamental set for an associative memory contains tour pairs of patterns {(x µ, x µ ) µ =,, 3, 4}. Each vector x µ is a column vector with values in the set A 4 and the values for each vector components are the following: x =, x =, x3 =, x4 = The matrix of the autoassociative Alpha Beta memory of type max for this fundamental set is: V = 3

11 Alpha Beta bidirectional associative memories Now, lets suppose we present to matrix V a noisy version x of vector x ω with ω =, with additive noise, whose components are: x = then the recalled value will be : V β x = We can see that vector x was recalled perfectly. However, our interest lies in the recall of components and whether the condition of theorem 4. holds. The first component recalled by the autoassociative Alpha Beta memory of type max is equal to zero, which is precisely the value held by the first component of the second pattern. This is, ( V β x ) = = x. Now lets see whether the condition of r {,...,n}, such that ν kr α(xk ω, x r ), holds. For our example k = andω =. For r =, ν = andα(x, x ) = α(, ) =, that is ν >α(x, x ), does not hold. For r =, ν = andα(x, x ) = α(, ) =, that is ν >α(x, x ), does hold. For r = 3, ν 3 = andα(x, x 3) = α(, ) =, that is ν 3 >α(x, x 3), does not hold. For r = 4, ν 4 = andα(x, x 4) = α(, ) =, that is ν 4 >α(x, x 4), does not hold. Therefore, exists r = such that ν α(x, x ). Lemma 4. Let {(X k, X k ) k =,...,p} be the fundamental set of an autoassociative Alpha Beta memory of type max represented by V, with X k = τ e (x k, h k ) for k =,...,p, and let F = τ e (x k, u) A n+p be a version of a specific pattern X k, altered with additive noise, being u A p the vector defined as u = p i= hi. If during the recalling phase F is presented to memory V, then component Xn+k k will be recalled in a perfect manner; that is ( V β F ) n+k = X n+k k =. Proof This proof will be done for two mutually exclusive cases. Case Pattern F has one component with value. This means that j {,...,n + p} such that F j = ; also, due to the way vector X k is built, it is clear that Xn+k k =. Then α(xn+k k, F j ) = α(, ) = and since the maximum allowed value for a component of memory V is we have ν (n+k) j α(xn+k k, F j ). According to Theorem 4., Xn+k k is perfectly recalled. Example 3. Taking the fundamental set of example 3., patterns X k for k =,, 3, 4are built, using the expansion vectorial transform from Definition 3: X =, X =, X 3 =, X 4 = 3

12 M. E. Acevedo-Mosqueda et al. The matrix of the autoassociative Alpha Beta memory of type max is: V = Now, having k = 3 we will use vector X 3 and obtain its noisy version F, with additive noise: X 3 =, the noisy vector F, with additive noise, is: F = When F is presented to memory V, the recalled vector is: = V β F = 3 X Seventh component of both vectors Recalling that for this example n = 4andk = 3, then ( V β F ) 4+3 = X =. Therefore, the seventh component of third vector is perfectly recalled. Case Pattern F does not contain a component with value. That is F j = j {,..., n+ p}. This means that it is not possible to guarantee the existence of a value j {,...,n + p} such that ν (n+k) j α(x k n+k, F j ), and therefore Theorem 4. cannot be applied. However, we will show the impossibility of ( V β F ) n+k =. The recalling phase of the autoassociative Alpha Beta memory of type max V, when having vector F as input, takes the following form for the n + kth recalled component: ( V β F ) n+k = n j= β(ν (n+k) j, F j ) = n j= β {[ p µ= α(x µ n+k, X µ j ) ], F j } DuetothewayvectorX k is built, besides X k n+k =, it is important to notice that X µ n+k =, µ = k, and from here we can establish the following: p µ= α(x µ n+k, X µ j ) = α(x k n+k, X k j ) = α(, X k j ) 3

13 Alpha Beta bidirectional associative memories 3 is different from zero regardless of the value of X k j. According to F j = j {,...,n+ p}, we can conclude the impossibility of ( V β F ) n+k = n β(α(, X k j ), ) j= being zero. That is ( V β F ) n+k = = X k n+k. Example 3.3 Using the matrix V, obtained in Example 3., and with k =, we get vector F, which is a noisy version by additive noise of X, whose component values are: X =, the noisy vector F, with additive noise, is: F = When F is presented to memory V, the recalled vector is: V β F = Fifth component of both vectors X = For this example n = 4yk =, then ( V β F ) 4+ = X 4+ =. Therefore, the fifth component of the first pattern is perfectly recalled. Theorem 4. Let {(X k, X k ) k =,...,p} be the fundamental set of an autoassociative Alpha Beta memory of type max represented by V, with X k = τ e (x k, h k ) for k =,...,p, and let F = τ e (x k, u) A n+p be a pattern altered with additive noise with respect to some specific pattern X k,beingu A p the vector defined as u = p i= hi. Lets assume that during the recalling phase, F is presented to memory V as input, and the pattern R = V β F A n+p is obtained. If when taking vector R as argument, the contraction vectorial transform r = τ c (R, n) A p is done, the resulting vector r has two mutually exclusive possibilities: k {,...,p} such that r = h k,orr is not a one-hot vector. Proof From the definition of contraction vectorial transform we have that r i = R i+n = (V β F) i+n for i p, and in particular, by making i = k we have r k = R k+n = (V β F) k+n.however,bylemma4. ( V β F ) n+k = X n+k k, and since Xk = τ e (x k, h k ),the value Xn+k k is equal to the value of component hk k =. That is, r k =. When considering that r k =, vector r has two mutually exclusive possibilities: it can be that r j = j = k in which case r = h k ; or happens that j {,...,p}, j = k for which r j =, in which case it is not possible that r is a one-hot vector, given Definition. 3

14 4 M. E. Acevedo-Mosqueda et al. Example 3.4 Using the matrix V obtained in Example 3., and with k =, the component values of X and its respective noisy pattern F, by additive noise, are: X = and F = When F is presented to memory V, the recalled vector R is: R = V β F = When this vector is taken as argument and the contraction vectorial transform is done we obtain vector r, r = and according to Definition, we can see that r is the second one-hot vector of 4 bits. Now lets follow the same process for k = 4. Then, X 4 and its respective noisy pattern F, with additive noise, are: X 4 = and F = When F is presented to memory V we obtain vector R: R = V β F = 3

15 Alpha Beta bidirectional associative memories 5 When this vector is taken as argument and the contraction vectorial transform is done we obtain vector r, r = and according to Definition, r is not a one-hot vector. Theorem 4.3 Let {(x µ, x µ ) µ =,,...,p} be the fundamental set of an autoassociative Alpha Beta memory of type min represented by, and let x A n be a pattern altered with substractive noise with respect to some fundamental pattern x ω with ω {,,...,p}.lets assume that during the recalling phase, x ω is presented to memory as input, and consider an index k {,...,n}. The kth recalled component ( β x ) k is precisely xω k if and only if it holds that r {,...,n}, dependant on ω and k, such that λ kr α ( xk ω, x ) r. Proof ) By hypothesis it is assumed that ( β x ) k = xω k. By contradiction, now lets suppose it is false that r {,...,n} such that λ kr α ( xk ω, x ) r. That is to say that r {,...,n}λ kr <α ( xk ω, x ) r, which is in turn equivalent to r {,...,n}β (λkr, x r ) < β [ α ( xk ω, x ) ] r, xr = x ω k. When taking the maximums at both sides of the inequality, with respect to index r, wehave n r= β (λ kr, x r ) < n r= x ω k = xω k and this means that ( β x ) k = n r= β (λ kr, x r ) < xk ω, affirmation which contradicts the hypothesis. ) When conditions for Theorem 4.33 [9] are met for every i {,...,n}, wehave β x = x ω. That is, it holds that ( f β x ) = x ω i i i {,...,n}. When indexes i and ( j are fixed such that i = k and j = r, depending on ω and k, we obtain the desired result β x ) k = xω k. Example 3.5 The fundamental set components values from example 3. are shown below, with p = 4andn = 4, x =, x =, x3 =, x4 = The matrix of the autoassociative Alpha Beta memory of type min for this fundamental set is: = Now lets assume we present matrix with a noisy version x of vector x ω with ω = 4, altered by substractive noise, whose component values are: x = then the recalled pattern will be β x = 3

16 6 M. E. Acevedo-Mosqueda et al. As can be seen, vector x 4 was perfectly recalled. However, we are interested in corroborating component recall, and also that the condition for Theorem 4.5 is met. The second component recalled by the autoassociative Alpha Beta memory min is equal to one which is exactly the second component value of the fourth pattern. That is, ( β x ) = = x4.now let us see whether the condition of r {,...,n}, such that λ kr α(xk ω, x r ), holds. For our example, k = andω = 4. For r =, λ = andα(x 4, x ) = α(, ) =, this is λ α(x 4, x ), does hold. For r =, λ = andα(x 4, x ) = α(, ) =, that is λ <α(x 4, x ), does not hold. For r = 3, λ 3 = andα(x 4, x 3) = α(, ) =, that is λ 3 <α(x 4, x 3), does not hold. For r = 4, λ 4 = andα(x 4, x 4) = α(, ) =, that is λ 4 <α(x 4, x 4), does not hold. Therefore, exists r = suchthatα(x 4, x ). Lemma 4. Let {( X k, X k) k =,...,p } be the fundamental set of an autoassociative Alpha Beta memory of type min represented by, with X k = τ e (x k, h k ) for k =,...,p, and let G = τ e (x k, w) A n+p be a pattern altered with substractive noise with respect to some specific pattern X k,beingw A p a vector whose components have values w i = u i, and u A p the vector defined as u = p i= hi. If during the recalling phase, G is presented to memory, then component X n+k k is recalled in a perfect manner. That is, ( β G ) n+k = X n+k k =. Proof This proof will be done for two mutually exclusive cases. Case Pattern G has one component with value. This means that j {,...,n + p} such that G j =. Also, due to the way vector X k is built, it is clear that X n+k k =. Because of this, α( X n+k k, G j ) = α(, ) = and, since the minimum allowed value for a component of memory is, we have λ (n+k) j α( X n+k k, G j ). According to Theorem 4.3, X n+k k is perfectly recalled. Example 3.6 Taking the fundamental set of Example 3., the patterns X k for k =,, 3, 4 are built, by using the expansion vectorial transform of Definition 3. 3 X =, X =, X 3 =, X 4 =

17 Alpha Beta bidirectional associative memories 7 The matrix of the autoassociative Alpha Beta memory of type min is: = Now, taking k = 3 we use vector X 3 and we obtain its noisy version, with substractive noise: X 3 =, the noisy vector G, with substractive noise, is: G = When presenting G to matrix, the recalled pattern is: = β G = 3 X Seventh component of both vectors Λ Lets recall that, for this example, n = 4andk = 3; then ( β G ) 4+3 = X =. Therefore, the seventh component of the third vector is perfectly recalled. Case Pattern G has no component with value ; that is, G j = j {,...,n + p}.this means that it is not possible to guarantee the existence of a value j {,...,n + p} such that λ (n+k) j α( X k n+k, G j ), and therefore Theorem 4.3 cannot be applied. However, lets show the impossibility of ( β G) n+k =. Recalling phase of the autoassociative Alpha Beta memory of type min with vector G as input, takes the following form for the n + kth recalled component: ( β G ) n+k = n j= β(λ (n+k) j, G j ) = n j= β {[ p µ= α( X µ n+k, X µ j ) ], G j } Duetothewayvector X k is built, besides that X k n+k =, it is important to notice that X µ n+k =, µ = k, and from here we can state that p µ= α( X µ n+k, X µ j ) = α( X k n+k, X k j ) = α(, X k j ) 3

18 8 M. E. Acevedo-Mosqueda et al. is different from regardless of the value of X k j. Taking into account that G j = j {,...,n + p}, we can conclude that it is impossible for ( β G ) n+k = n ( ( ) ) β α, X k j, j= to be equal to. That is, ( β G ) n+k = = X k n+k. Example 3.7 By using matrix obtained in Example 3.6 and with k =, we obtain vector G, which is the noisy version, with substractive noise, of X, whose component values are: X =, the noisy vector G, with substractive noise, is: G = When presenting G to matrix, the recalled pattern is: Λ β G = Sixth component of both vectors X = For this example, n = 4andk =, then ( β G ) 4+ = X 4+ =. Therefore, the sixth component of the second pattern is perfectly recalled. Theorem 4.4 Let {( X k, X k) k =,...,p } be the fundamental set of an autoassociative Alpha Beta memory of type min represented by, with X k = τ e (x k, h k ) for k =,...,p, and let G = τ e (x k, w) A n+p be a pattern altered with substractive noise with respect to some specific pattern X k,beingw A p a vector whose components have values w i = u i, and u A p the vector defined as u = p i= hi. Lets assume that during the recalling phase, G is presented to memory as input, and the pattern S = β G A n+p is obtained as output. If when taking vectors as argument, the contraction vectorial transform s = τ c (S, n) A p is done, the resulting vector S has two mutually exclusive possibilities: k {,..., p} such that s = h k,ors is not a one-hot vector. Proof From the definition of contraction vectorial transform we have that s i = S i+n = ( β G) i+n for i p, and in particular, by making i = k we have s k = S k+n = ( β G) k+n.however,bylemma4. ( β G ) n+k = X n+k k, and since X k = τ e (x k, h k ), the value X n+k k is equal to the value of component h k k =. That is, s k =. When considering that s k =, vector s has two mutually exclusive possibilities: it can be that s j = j = k in which case s = h k ; or happens that j {,...,p}, j = k for which s j =, in which case it is not possible that s is a zero-hot vector, given Definition. 3

19 Alpha Beta bidirectional associative memories 9 Example 3.8 By taking matrix obtained in Example 3.6 and with k =, the component values of X and its respective noisy pattern G, with substractive noise, are: X = y G = When presenting G to matrix, the recalled pattern is: S = β G = When this vector is taken as argument and the contraction vectorial transform is done we obtain vector s, s = and according to Definition, we can see that s is the first zero-hot vector of 4 bits. Now let us do the same process for k = 4. Then, X 4 and its respective noisy pattern G, with substractive noise, are: X 4 = and G = When presenting G to matrix, the recalled pattern S is: S = β G = 3

20 M. E. Acevedo-Mosqueda et al. When taking this vector as argument and the contraction vectorial transform is done we obtain vector s, S = According to Definition, s is not a zero-hot vector. Lemma 4.3 Let {( X k, X k) k =,...,p } be the fundamental set of an autoassociative Alpha Beta memory of type max represented by V, with X k = τ e (x k, h k ) A n+p k {,...,p}. If t is an index such that n + t n + pthenν ij = j {,...,n + p}. Proof In order to establish that ν ij = j {,...,n + p}, given the definition of α, itis enough to find, for each t {n +,...,n + p}, an index µ for which X t µ = in the expression that produces the tjth component of memory V,whichisν tj = p µ=,α(x t µ, X µ j ).Due to the way each vector X µ = τ e (x µ, h µ ) for µ =,...,p is built, and given the domain of index t {n +,...,n + p}, for each t exists s {,...,p} such that t = n + s. Thisis why two useful values to determine the result are µ = s and t = n + s, because Xn+s s =. Then, ν tj = p µ=,α(x t µ, X µ j ) = α(x n+s s, X s j ) = α(, X s j ), value which is different from. That is, ν ij = j {,...,n + p}. Example 3.9 Let us take vectors X k for k =,, 3, 4 from Example 3.: X =, X =, X 3 =, X 4 = The matrix of the autoassociative Alpha Beta memory of type max is: V = For t {5, 6, 7, 8} and j {,,...,8} ν ij = Lemma 4.4 Let {( X k, X k) k =,...,p } be the fundamental set of an autoassociative Alpha Beta memory of type max represented by V, with X k = τ e (x k, h k ) for k =,...,p, and let F = τ e (x k, u) A n+p be an altered version, by additive noise, of a specific pattern X k,beingu A p the vector defined as u = p i= hi. Let us assume that during the recalling phase, F is presented to memory as input. Given a fixed index t {n +,...,n + p} such that t = n + k, it holds that(v β F) t = if and only if the following logic proposition is true: j {,...,n + p}(f j = ν tj = ). 3

21 Alpha Beta bidirectional associative memories Proof DuetothewayvectorsX k = τ e (x k, h k ) and F = τ e (x k, u) are built, we have that F t = is the component with additive noise with respect to component X k t =. ) There are two possible cases: Case Pattern F does not contain components with value. That is, F j = j {,...,n + p}. This means that the antecedent of proposition F j = ν tj = is false, and therefore, regardless of the truth value of consequence ν tj =, the expression j {,...,n+p}(f j = ν tj = ) is true. Case Pattern F contains at least one component with value. That is, r {,...,p} such that F r =. By hypothesis (V β F) t =, which means that the condition for perfect recall of X k t = is not met. In other words, according to Theorem 4. expression [ j {,...,n + p} such that ν tj α(x k t, F j )] is true, which is equivalent to j {,...,n + p} it holds that ν tj α(x k t, F j ) In particular, for j = r, and taking into account that X k t =, this inequality ends up like this: ν tr >α(x k t, F r ) = α(, ) =. That is, ν tr =, and therefore the expression j {,...,n + p}(f j = ν tj = ) is true. Example 3. Let us use vector X 3, with k = 3, from Example 3.9, and build its noisy vector F, with additive noise X 3 =, F = When presenting F to matrix V, presented in the aforementioned example, we obtain: = V β F = F = V (V β F) 6 = t = 6 t = 6 3

22 M. E. Acevedo-Mosqueda et al. ) Assuming the following expression is true j {,...,n + p}(f j = ν tj = ), there are two possible cases: Case Pattern F does not contain components with value. That is, F j = j {,...,n + p}. When considering that ( V β F ) = n+p t j= β ( ) ν tj, F j, according to the definition of β, it is enough to show that j {,...,n + p}ν tj =, which is guaranteed by Lemma 4.3. Then, it has been proven that ( V β F ) t = n+p j= β ( ) n+p ν tj, F j = j= β ( ν tj, ) =. Case Pattern F contains at least one component with value. That is, r {,...,p} such that F r =. By hypothesis we have that j {,...,n + p}(f j = ν tj = ) and, in particular, for j = r and ν tr =, which means that ( V β F ) t = n+p j= β ( ) ν tj, F j = β (ν tr, F r ) = β (, ) =. Corollary 4. Let {( X k, X k) k =,...,p } be the fundamental set of an autoassociative Alpha Beta memory of type max represented by V, with X k = τ e (x k, h k ) for k =,...,p, and let F = τ e (x k, u) A n+p be an altered version, by additive noise, of a specific pattern X k,beingu A p the vector defined as u = p i= hi. Let us assume that during the recalling phase, F is presented to memory as input. Given a fixed index t {n +,...,n + p} such that t = n + k, it holds that(v β F) t = if and only if the following logic proposition is true: j {,...,n + p}(f j = AND sν tj = ). Proof In general, given two logical propositions P and Q, the proposition (P if and only if Q) is equivalent to proposition ( P if and only if Q). If P is identified with equality (V β F) t = andq with expression j {,...,n + p}(f j = ν tj = ), by Lemma 4.4 the following proposition is true: { [(V β F) t = ] if and only if [ j {,...,n+ p}(f j = ν tj = )]}. This expression transforms in the following equivalent propositions: {(V β F) t = if and only if j {,...,n + p} such that (F j = ν tj = )} {(V β F) t = if and only if j {,...,n + p} such that [ (F j = ) OR ν tj = ]} {(V β F) t = if and only if j {,...,n+p} such that [ [ (F j =)] AND (ν tj =)]} {(V β F) t = if and only if j {,...,n + p} such that [(F j = ) AND ν tj = ]} Example 3. Taking X 3 and F from Example 3., when presenting F to V we have: 3 V β F = F = (V β F) 5 =, t = 5 (V β F) 8 =, t = 8 = V t = 5

23 Alpha Beta bidirectional associative memories 3 Lemma 4.5 Let {( X k, X k) k =,...,p } be the fundamental set of an autoassociative Alpha Beta memory of type min represented by, with X k = τ e (x k, h k ) A n+p k {,...,p}. If t is an index such that n + t n + pthenλ tj = j {,...,n + p}. Proof In order to establish that λ tj = j {,...,n + p}, given the definition of α, itis enough to find, for each t {n +,...,n + p}, an index µ for which X t µ = in the expression leading to obtaining the tjth component of memory,whichisλ tj = p µ= α( X t µ, X µ j ). In fact, die to the way each vector X k = τ e (x k, h k ) for µ =,...,p is built, and given the domain of index t {n +,...,n + p}, for each t exists s {,...,p} such that t = n + s; therefore two values useful to determine the result are µ = s and t = n + s, because X n+s s =, then λ tj = p µ= α( X t µ, X µ j ) = α( X n+s s, X s j ) = α(, X µ j ), value different from. That is, λ tj = j {,...,n + p}. Example 3. Let us take vectors X k for k =,, 3, 4 from Example 3.6 X =, X =, X 3 =, X 4 = The matrix of the autoassociative Alpha Beta memory of type min is: = For t {5, 6, 7, 8} and j {,,...,8} λ ij = Lemma 4.6 Let {( X k, X k) k =,...,p } be the fundamental set of an autoassociative Alpha Beta memory of type min represented by, with X k = τ e (x k, h k ) for k =,...,p, and let G = τ e (x k, w) A n+p be an altered version, by substractive noise, of a specific pattern X k,beingw A p a vector whose components have values w i = u i, and u A p the vector defined as u = p i= hi. Let us assume that during the recalling phase, G is presented to memory as input. Given a fixed index t {n +,...,n + p} such that t = n + k, it holds that ( β G) t =, if and only if the following logical proposition is true j {,...,n + p}(g j = λ tj = ). Proof Duetothewayvectors X k = τ e (x k, barh k ) and G = τ e (x k, w) are built, we have that G t = is the component with substractive noise with respect to component X k t =. ) There are two possible cases: Case Pattern G does not contain components with value. That is, G j = j {,..., n + p}. This means that the antecedent of logical proposition G j = λ tj = is 3

24 4 M. E. Acevedo-Mosqueda et al. false and therefore, regardless of the truth value of consequent λ tj =, the expression j {,...,n + p}(g j = λ tj = ) is true. Case Pattern G contains at least one component with value. That is, r {,...,n + p} such that G r =. By hypothesis ( β G) t =, which means that the perfect recall condition of X k t = is not met. In other words, according to Theorem 4.3 expression [ j {,...,n + p} such that λ tj α( X k t, G j )] is true, which in turn is equivalent to j {,...,n + p} it holds that λ tj <α(x k t, G j ) In particular, for j = r and considering that X k t =, this inequality yields: λ tr <α( X k t, G r )= α(, ) =. That is, λ tr =, and therefore the expressión j {,...,n + p}(g j = λ tj = ) is true. Example 3.3 Let us use vector X 4, with k = 4, from Example 3., and build its noisy vector G, with substractive noise. X 4 =, G = When presenting G to, shown n the mentioned example, we have = Λ β G = G = Λ t = 5 (Λ β G) 5 = t = 5 ) Assuming the following expression to be true, j {,...,n+ p}(g j = λ tj = ), there are two possible cases: Case Pattern G does not contain components with value. That is, G j = j {,..., n + p}. When considering that ( β G ) t = n+p j= β ( λ tj, G j ), according to the β definition, 3

25 Alpha Beta bidirectional associative memories 5 it is enough to show that j {,...,n + p}λ tj =, which is guaranteed by Lemma 4.5. Then, it is proven that ( β G ) t = n+p j= β ( ) λ tj, G j. Case Pattern G contains at least one component with value. That is, r {,...,n + p} such that G r =. By hypothesis we have that j {,...,n + p}(g j = λ tj = ) and, in particular, for j = r and λ tr =, which means that ( β G ) t = n+p j= β ( ) λ tj, G j = β (λ tr, G r ) = β (, ) =. Corollary 4. Let {( X k, X k) k =,...,p } be the fundamental set of an autoassociative Alpha Beta memory of type min represented by, with X k = τ e (x k, h k ) for k =,...,p, and let G = τ e (x k, w) A n+p be an altered version, by substractive noise, of a specific pattern X k,beingw A p a vector whose components have values w i = u i, and u the vector defined as u = p i= hi. Let us assume that during the recalling phase, G is presented to memory as input. Given a fixed index t {n +,...,n + p} such that t = n + k, it holds that ( β G) t = if and only if the following logic proposition is true: j {,...,n + p}(g j = AND λ tj = ). Proof In general, given two logical propositions P and Q, the proposition (P if and only if Q) is equivalent to proposition ( P if and only if Q). IfP is identified with equality ( β G) t = andq with expression j {,...,n + p}(g j = λ tj = ), by Lemma 4.6 the following proposition is true: { [( β G) t = ] if and only if [ j {,...,n + p}(g j = λ tj = )]}. This expression transforms into the following equivalent propositions: {( β G) t = if and only if j {,...,n + p} such that (G j = λ tj = )]} {( β G) t = if and only if j {,...,n + p} such that [ (G j = ) OR λ tj = ]} {( β G) t = if and only if j {,...,n+ p} such that [ [ (G j =)] AND (λ tj =)]} {( β G) t = if and only if j {,...,n + p} such that [G j = AND λ tj = ]} Example 3.4 Taking X 4 and G from Example 3.3, when presenting G to Λ β G = (Λ β G) 7 = t = 7 (Λ β G) 6 = t = 6 G = = Λ t = 6 3

26 6 M. E. Acevedo-Mosqueda et al. Lemma 4.7 Let {( X k, X k) k =,...,p } be the fundamental set of an autoassociative Alpha Beta memory of type max represented by V, with X k = τ e (x k, h k ) for k =,...,p, and let {( X k, X k) k =,...,p } be the fundamental set of an autoassociative Alpha Beta memory of type min represented by, with X k = τ e (x k, h k ), k {,...,p}. Then, for each i {n +,...,n + p} such that i = n + r, with r i {,...,p}, it holds that: ν ij = α(, X r i j ) and λ ij = α(, X r i j ) j {,...,n + p}. Proof DuetothewayvectorsX k = τ e (x k, h k ) and X k = τ e (x k, h k ) are built, we have that X r i i = and X r i i =, besides X µ i = and X µ i = µ = r i such that µ {,...,p}. Because of this, and using the definition of α, α(x r i i, X r i j ) = α(, Xr i j ) and α(x µ i, X µ j ) = α(, X µ j ), which implies that, regardless of the values of Xr i j and X µ j, it holds that α(x r i i, X r i j ) α(x µ i, X µ j ), from whence ν ij = p α(x µ µ= i, X µ j ) = α(xr i i, X r i j ) = α(, Xr i j ) We also have α( X r i i, X r i j ) = α(, X r i j ) and α( X µ i, X µ j ) = α(, X µ j ), which implies that, regardless of the values of X r i j and X µ j, it holds that α( X r i i, X r i j ) α( X µ i, X µ j ), from whence λ ij = p α( X µ µ= i, X µ j ) = α( X r i i, X r i j ) = α(, X r i j ) µ {,...,p}, j {,...,n + p}. Example 3.5 Let us take vectors X k for k =,, 3, 4 from Example 3. X =, X =, X 3 =, X 4 =, V = In this example i {5, 6, 7, 8} and j {,,...,8}. Since we are using the autoassociative Alpha Beta memory of type max, during the learning phase the maximum value must be taken. Using the definition of α, the maximum allowed value is, which can be reached through α(, ) =. With a value the maximum that can be obtained is α(, ) =. Therefore, the component yielding a maximum value will be that whose value is. The only component, out of the four vectors for i = 5, having a value of, is the one in X.Thatis, X5 = will determine a maximum value, then ν 5 j = α(x5, Xr i j ) = α(, Xr i j ). 3 For i = 6, X6 =, then ν 6 j = α(x6, Xr i j ) = α(, Xr i j ). For i = 7, X7 3 =, then ν 7 j = α(x7 3, Xr i j ) = α(, Xr i j ). For i = 8, X8 4 =, then ν 8 j = α(x8 4, Xr i j ) = α(, Xr i j ).

27 Alpha Beta bidirectional associative memories 7 Now let us take vectors X k for k =,, 3, 4 from Example 3.6 X =, X =, X 3 =, X 4 =, = In the case of the autoassociative Alpha Beta memory of type min, during the learning phase the minimum value must be taken. The minimum allowed value is and appears when α(, ) =. Therefore, the component yielding a minimum value will be that whose value is. The only component, out of the four vectors for i = 5, having a value of, is the one in X.Thatis,X5 = will determine a minimum value, then λ 5 j = α( X 5, Xr i j ) = α(, X r i j ). For i = 6, X 6 =, then λ 6 j = α( X 6, Xr i j ) = α(, X r i j ). For i = 7, X 7 3 =, then λ 7 j = α( X 7 3, X r i j ) = α(, X r i j ). For i = 8, X 8 4 =, then λ 8 j = α( X 8 4, X r i j ) = α(, X r i j ). Corollary 4.3 Let {( X k, X k) k =,...,p } be the fundamental set of an autoassociative Alpha Beta memory of type max represented by V, with X k = τ e (x k, h k ) k {,...,p}, and let {( X k, X k) k =,...,p } be the fundamental set of an autoassociative Alpha Beta memory of type min represented by, with X k = τ e (x k, h k ), k {,...,p}. Then, ν ij = λ ij +, i {n +,...,n + p}, i = n +r i, with r i {,...,p} and j {,...,n}. Proof Let i {n +,...,n + p} and j {,...,n} be two indexes arbitrarily selected. By Lemma 4.7, the expressions used to calculate the ijth components of memories V y take the following values: ν ij = α(, X r i j )yλ ij = α(, X r i j ) Considering that for j {,...,n}x r i j = X r i j, there are two possible cases: Case X r i j = = X r i j. We have the following values: ν ij = α(, ) = andλ ij = α(, ) =, therefore ν ij = λ ij +. Case X r i j = = X r i j. We have the following values: ν ij = α(, ) = yλ ij = α(, ) =, therefore ν ij = λ ij +. Since both indexes i and j were arbitrarily chosen inside their respective domains, the result ν ij = λ ij + is valid i {n +,...,n + p} and j {,...,n}. Example 3.6 For this example we shall use memories V y, presented in Example 3.5. V =, = 3

28 8 M. E. Acevedo-Mosqueda et al. With n = 4andp = 4, i {5, 6, 7, 8} and j {,, 3, 4}. ν 5 = λ 5 + = + = ν 6 = λ 6 + = + = ν 7 = λ 7 + = + = ν 8 = λ 8 + = + = ν 5 = λ 5 + = + = ν 6 = λ 6 + = + = ν 7 = λ 7 + = + = ν 8 = λ 8 + = + = ν 53 = λ 53 + = + = ν 63 = λ 63 + = + = ν 73 = λ 73 + = + = ν 83 = λ 83 + = + = ν 54 = λ 54 + = + = ν 64 = λ 64 + = + = ν 74 = λ 74 + = + = ν 84 = λ 84 + = + = Lemma 4.8 Let {( X k, X k) k =,...,p } be the fundamental set of an autoassociative Alpha Beta memory of type max represented by V, with X k = τ e (x k, h k ) k {,...,p}, and let {( X k, X k) k =,...,p } be the fundamental set of an autoassociative Alpha Beta memory of type min represented by, with X k = τ e (x k, h k ), k {,...,p}. Also, if we define vector u A p as u = p i= hi, and take a fixed index r {,...,p}, let us consider two noisy versions of pattern X r A n+p : vector F = τ e (x r, u) A n+p which is an additive noise altered version of pattern X r, and vector G = τ e (x r, w) A n+p, which is a substractive noise altered version of pattern X r,beingw A p a vector whose components take the values w i = u i i {,...,p}. If during the recalling phase, G is presented as input to memory and F is presented as input to memory V, and if also it holds that ( β G) t = for an index t {n +,...,n + p}, fixed such that t = n + r, then (V β F) t =. Proof Due to the way vectors X r, F y G are built, we have that F t = is the component in the vector with additive noise corresponding to component Xt r,andg t = is the component in the vector with substractive noise corresponding to component X t r. Also, since t = n + r, we can see that Xt r =, that is Xt r =, and X t r =. There are two possible cases: Case Pattern F does not contain any component with value. That is, F j = j {,...,n + p}. By Lemma 4.3 ν tj = j {,...,n + p},thenβ(ν tj, F j ) j {,...,n + p}, which means that ( V β F ) = n+p t j= β(ν tj, F j ) =. In other words, expression (V β F) t = is false. The only possibility for the theorem to hold, is for expression ( β G) t = to be false too. That is, we need to show that ( β G) t =. According to Corollary 4., the latter is true if for every t {n +,...,n + p} with t = n + r, exists j {,...,n + p} such that (G j = AND λ tj = ). Now,t = n + r indicates that s {,...,p}, s = r such that t = n+s, and by Lemma 4.7 α( X t s, X s j ) α( X t µ, X µ j ) µ {,...,p}, j {,...,n + p}, from where we have λ tj = p j= α( X t µ, X µ j ) = α( X t s, X s j ), and by noting the equality X t s = X n+s s =, it holds that: λ tj = α(, X s j ) j {,...,n + p} On the other side, i {,...,n} the following equalities hold: X i r = xi r = and X i s = xi s and also, taking into account that x r = x s, it is clear that h {,...,p} such that xh s = xr h ; meaning xh s = = X h s and therefore 3 λ th = α(, ) =

29 Alpha Beta bidirectional associative memories 9 Finally, since i {,...,n} it holds that G i = X i r = xi r =, in particular G h =. Then we have proven that for every t {n +,...,n + p} with t = n +r,exists j {,...,n + p} such that (G j = AND λ tj = ), and by Corollary 4. it holds that ( β G) t =, thus making expression ( β G) t = isfalse. Case Pattern F contains, besides the components with value of, at least one component with value. That is, h {,...,n + p} such that F h =. Due to the way vectors G and F are built i {,...,n}g i = F i and, also, necessarily h n and thus F h = G h =. By hypothesis t {n +,...,n + p} fixed such that t = n + r and ( β G) t =, and by Lemma 4.6 j {,...,n + p}(g j = λ tj = ). Given the way vector G is built we have that j {n +,...,n + p}g j =, thus making the former expression like this: j {,...,n}(g j = λ tj = ).LetJ be a set, proper subset of {,...,n}, defined like this: J ={j {,...,n} G j = }. The fact that J is a proper subset of {,...,n} is guaranteed by the existence of G h =. Now, t = n + r indicates that s {,...,p}, s = r such that t = n + s, and by Lemma 4.7 ν tj = α(, X s j ) and λ tj = α(, X s j ) j {,...,n + p}, from where we have that j J, X s j =, because if this was not the case, λ tj =. This means that for each j J X s j = = G j which in turn means that patterns X r and X s coincide with value in all components with index j J. Let us now consider the complement of set J, which is defined as J c ={j {,...,n} G j = }. The existence of at least one value j J c for which G j = y X s j = is guaranteed by the known fact that x r = x s.letus see, if X s j = j J c then j {,...,n} it holds that X s j = G j, which would mean that x r = x s.since j J c for which G j = and X s j =, this means that j J c for which F j = andx s j =. Now, β(ν tj, F j ) = β(α(, X s j ), ) = β(α(, ), ) = β(, ) =, and finally ( V β F ) t = n+p j= β(ν tj, F j ) = β(ν tj, F j ) = Example 3.7 From Example 3. let us takex and build the noisy vector F, with additive noise; from Example 3.6 take X and build the noisy vector G, with substractive noise, and use the memories V and shown in Example 3.5. X =, F =, X =, G = V =, = 3

30 3 M. E. Acevedo-Mosqueda et al. Now we present F to V and G to in order to obtain: = V β F, Λ β G = With n = 4andp = 4, t {5, 6, 7, 8} and for this example r =, thus t = 4 + = 6. Then, for t = 5( β G) 5 =,(V β F) 5 =. for t = 7( β G) 7 =,(V β F) 7 =. for t = 8( β G) 8 =,(V β F) 8 =. Lemma 4.9 Let {( X k, X k) k =,...,p } be the fundamental set of an autoassociative Alpha Beta memory of type max represented by V, with X k = τ e (x k, h k ) k {,...,p}, and let {( X k, X k) k =,...,p } be the fundamental set of an autoassociative Alpha Beta memory of type min represented by, with X k = τ e (x k, h k ), k {,...,p}. Also, if we define vector u A p as u = p i= hi, and take a fixed index r {,...,p}, let us consider two noisy versions of pattern X r A n+p : vector F = τ e (x r, u) A n+p which is an additive noise altered version of pattern X r, and vector G = τ e (x r, w) A n+p, which is a substractive noise altered version of pattern X r,beingw A p a vector whose components take the values w i = u i i {,...,p}. If during the recalling phase, G is presented as input to memory and F is presented as input to memory V, and if also it holds that (V β F) t = for an index t {n +,...,n + p}, fixed such that t = n + r, then ( β G) t =. Proof Due to the way vectors X r, F y G are built, we have that F t = is the component in the vector with additive noise corresponding to component Xt r,andg t = is the component in the vector with substractive noise corresponding to component X t r. Also, since t = n + r, we can see that Xt r =, that is Xt r =, and X t r =. There are two possible cases: Case Pattern G does not contain any component with value. That is, G j = j {,...,n + p}. By Lemma 4.5 λ tj = j {,...,n + p}, thus β(λ tj, G j ) = j {,...,n + p}, which means that ( β G ) t = n+p j= β(λ tj, G j ) =. In other words, expression ( β G) t = is false. The only possibility for the theorem to hold, is for expression (V β F) t = to be false too. That is, we need to show that (V β F) t =. According to Corollary 4., the latter is true if for every t {n +,...,n + p} with t = n + r, exists j {,...,n + p} such that (F j = AND ν tj = ). Now, t = n + r indicates that s {,...,p}, s = r such that t = n + s, and by Lemma 4.6 α(xt s, X s j ) α(x t µ, X µ j ) µ {,...,p}, j {,...,n + p}, from where we have ν tj = p µ= α(x t µ, X µ j ) = α(x t s, X s j ), 3

31 Alpha Beta bidirectional associative memories 3 and by noting the equality Xt s = Xn+s s =, it holds that: ν tj = α(, X s j ) j {,...,n + p} () On the other side, i {,...,n} the following equalities hold: Xi r = xi r = and Xi s = xi s and also, taking into account that x r = x s, it is clear that h {,...,p} such that xh s = xr h ; meaning xs h = = X h s and therefore ν th = α(, Xh s ) = α(, ) = Finally, since i {,...,n} it holds that F i = Xi r = xi r =, in particular F h =. Then we have proven that for every t {n +,...,n + p} with t = n + r, exists j {,...,n + p} such that (F j = AND ν tj = ), and by Corollary 4. it holds that (V β F) t =, thus making expression (V β F) t = isfalse. Case Pattern G contains, besides the components with value of, at least one component with value. That is, h {,...,n + p} such that G h =. Due to the way vectors G and F are built i {,...,n}g i = F i and, also, necessarily h n and thus F h = G h =. By hypothesis t {n +,...,n + p} fixed such that t = n + r and (V β F) t =, and by Lemma 4.4 j {,...,n + p}(f j = ν tj = ). Given the way vector F is built we have that j {n +,...,n + p}g j =, thus making the former expression like this: j {,...,n + p}(f j = ν tj = ).LetJ be a set, proper subset of {,...,n}, defined like this: J ={j {,...,n} F j = }. The fact that J is a proper subset of {,...,n} is guaranteed by the existence of G h =. Now, t = n +r indicates that s {,...,p}, s = r such that t = n + s, and by Lemma 4.7 ν tj = α(, X s j ) and λ tj = α(, X s j ) j {,...,n + p}, from where we have that j J, X s j =, because if this was not the case, ν tj =. This means that for each j JX s j = = F j which in turn means that patterns X r and X s coincide with value in all components with index j J. Let us now consider the complement of set J, which is defined as J c ={j {,...,n} F j = }. The existence of at least one value j J c for which F j = yx s j = is guaranteed by the known fact that x r = x s.let us see, if X s j = j J c then j {,...,n} it holds that X s j =F j, which would mean that x r = x s.since j J c for which F j =andx s j =, this means that j J c for which G j =and X s j =. Now, β(λ tj, G j )=β(α(, X s j ), )=β(α(, ), )=β(, )=, and finally ( β G ) t = n+p j= β(λ tj, G j ) = β(λ tj, G j ) = Example 3.8 From Example 3., let us takex and build the noisy vector F, with additive noise; from Example 3.6 let us takex and build the noisy vector G, with additive noise; and let us use the memories V y shown in Example 3.5. X =, F =, X =, G =, 3

32 3 M. E. Acevedo-Mosqueda et al. V =, = Now we present F to V and G to in order to obtain: = V β F, G = With n = 4andp = 4, t {5, 6, 7, 8}, and for this example r =, thus t = 4 + = 5. Then, for t = 6( β G) 6 =,(V β F) 6 =. for t = 7( β G) 7 =,(V β F) 7 =. for t = 8( β G) 8 =,(V β F) 8 =. Theorem 4.5 (Main Theorem) Let {( X k, X k) k =,...,p } be the fundamental set of an autoassociative Alpha Beta memory of type max represented by V, with X k = τ e (x k, h k ) k {,...,p}, and let {( X k, X k) k =,...,p } be the fundamental set of an autoassociative Alpha Beta memory of type min represented by, with X k = τ e (x k, h k ), k {,...,p}. Also, if we define vector u A p as u = p i= hi, and take a fixed index r {,...,p}, let us consider two noisy versions of pattern X r A n+p : vector F = τ e (x r, u) A n+p, which is an additive noise altered version of pattern X r, and vector G = τ e (x r, w) A n+p, which is a substractive noise altered version of pattern X r,beingw A p a vector whose components take the values w i = u i i {,...,p}. Now, let us assume that during the recalling phase, G is presented as input to memory and F is presented as input to memory V, and patterns S= β G A n+p and R = V β F A n+p are obtained. If when taking vector R as argument the contraction vectorial transform r = τ c (R, n) A p is done, and when taking vector S as argument the contraction vectorial transform s = τ c (S, n) A p is done, then H = (r AND s) will be the kth one-hot vector of p bits, where s is the negated of s. Proof From the definition of contraction vectorial transform we have that r i = R i+n = (V β F) i+n and s i = S i+n = ( β G) i+n for i p, and in particular, by making i = k 3 Λ β

33 Alpha Beta bidirectional associative memories 33 we have r k = R k+n = (V β F) k+n and s k = S k+n = ( β G) k+n. By Lemmas 4. and 4. we have ( V β F ) n+k = X n+k k = and( β G ) n+k = X n+k k = ; and thus: H k = r k AND s k = AND = AND =. Now, by Lemma 4.8 we know that if ( β G) t = tal que t = i + n is a fixed index with t = n + k,then(v β F) t = ; thus: H i = r i AND s i = (V β F) t AND ( β G) t = AND = AND = On the other side, by Lemma 4.9 it is known that if (V β F) q = for a fixed index q = i + n such that q = n + k,then( β G) q =. According to the latter: H i = r i AND s i = (V β F) q AND ( β G) q = AND = AND = Then H i = fori = k yhi = fori = k. Therefore, and according to Definition, H will be the kth one-hot vector of p bits. 3. Theoretical Foundation of Stages and 4 In this section is presented the theoretical foundation which serves as the basis for the design and operation of Stages and 4, whose main element is an original variation of the Linear Associator. Let {(x µ, y µ ) µ =,,...,p} with A ={, }, x µ A n and y µ A m be the fundamental set of the Linear Associator. The Learning Phase consists of two stages: For each of the p associations (x µ, y µ ) find matrix y µ (x µ ) t of dimensions m x n. The p matrices are added together to obtain the memory M = p y µ (x µ) t [ = mij ]m n µ= in such way that the ijth component of memory M is expressed as: m ij = p y µ i x µ j µ= The Recalling Phase consists of presenting an input pattern x ω to the memory, where ω {,,...,p}, and do operation p M x ω = y µ (x µ) t x ω µ= The following form of expression allow us to investigate the conditions that must be met in order for the proposed recalling method to give perfect outputs as results: [ (x M x ω = y ω ω ) ] t x ω + µ =ω [ (x y µ µ ) ] t x ω For the latter expression to give pattern y ω as result, it is necessary that two equalities hold: [ (x ω ) t x ω] = [ (x µ ) t x ω] = as long as µ = ω. 3

34 34 M. E. Acevedo-Mosqueda et al. This means that, in order to have perfect recall, vectors x µ must be orthonormals. If that happens, then, for µ =,,...,p, wehave: y y y (x ) t y (n) =.. (x, ) y x,...,x (n) n = ym ym (n) Therefore, y y y (x ) t y (n) =.. (x, ) y x,...,x (n) n = ym ym (n) y p y p (x p ) t y p =... M = y p m y p (n) (x p, x p,...,x n p ) y p (n) = y p m(n) y p y y3 y p n y µ (x µ) t y y y3 y p = ym y m y3 m y m p µ= Taking advantage of the characteristic shown by the Linear Asssociator when the input patterns are orthonormals, and given that, by Definition, one-hot vectors v k with k =,...,p are orthonormals, we can obviate the learning phase by avoiding the vectorial operations done by the Linear Associator, and simply put the vectors in order, to form the Linear Associator. Stages and 4 correspond to two modified Linear Associators, built with vectors y and x, respectively, of the fundamental set. 3.3 Algorithm In this section we describe, step by step, the processes required by the Alpha Beta BAM, in the Learning Phase as well as in the Recalling Phase (by convention only) in the direction x y, the algorithm for Stages and. The following algorithm describes the steps needed by the Alpha Beta bidirectional associative memory in order to realize the learning and recalling phases in the direction x y. Learning phase. For each index k {,...,p} do expansion: X k = τ e (x k, h k ). Create an Alpha Beta autoassociative memory of type max V with the fundamental set {(X k, X k) } k =,...,p 3. For each index k {,...,p} do expansion: X k = τ e (x k, h k ) 3

35 Alpha Beta bidirectional associative memories Create an Alpha Beta autoassociative memory of type min with the fundamental set {( X k, X k) k =,...,p} 5. Create a matrix consisting of a modified Linear Associator with patterns y k y y y p y LAy = y y p..... yn y n y n p Recalling phase. Present, as input to Stage, a vector of the fundamental set x µ A n for some index µ {,...,p}. Build vector u A p in the following manner: u = 3. Do expansion: F = τ e (x µ, u) A n+p 4. Operate the Alpha Beta autoassociative memory max V with F, in order to obtain a vector R of dimension n + p p i= h i R = V β F A n+p 5. Do contraction r = τ c (R, n) A p 6. If ( k {,...,p} such that h k = r) it is assured that k = µ (based on Theorem 4.), and the result is h µ. Thus, operation LAy r is done, resulting in the corresponding y µ. STOP. Else { 7. Build vector w A p in such way that w i = u i, i {,...,p} 8. Do expansion: G = τ e (x µ, w) A n+p 9. Operate the Alpha Beta autoassociative memory min with G, in order to obtain a vector S of dimension n + p S = β G A n+p. Do contraction s = τ c (S µ, n) A p. If ( k {,...,p} such that h k = s) it is assured that k = µ (based on Theorem 4.4), and the result is h µ. Thus, operation LAy s is done, resulting in the corresponding y µ. STOP. Else {. Do operation t = r s, where is the symbol of the logical AND. The result of this operation is h µ (based on Theorem 4.5). Operation LAy t is done, in order to obtain the corresponding y µ.stop.}} 4Results In this section, two applications that make use of the algorithm described in the former section are presented, illustrating the optimal functioning which the Alpha Beta bidirectional associative memories exhibit. 3

36 36 M. E. Acevedo-Mosqueda et al. Fig. 3 (a) The corresponding translation of departure is partida. (b) With just a part of the Word departure the correct result is still given. (c) The translator works in an optimal manner even if the input word is misspelled The first application is a Spanish English/English Spanish translator. As second application, a fingerprint verifier is presented. For both applications implementation, the programming language Visual C++ 6. was used. The programs were executed on a Sony VAIO laptop computer with a Pentium 4 processor at.8 GHz. 4. Spanish English/English Spanish translator This program is able to translate words in English to Spanish and vice versa. For the Learning Phase two text files were used. These files contain words each, in English and Spanish, respectively. With both files the Alpha Beta bidirectional associative memory is created. The learning process lasts, approximately, 6. The program was executed on a Sony VAIO laptop computer with a Pentium 4 processor at.8 GHz. After the Alpha Beta BAM is built, during the Recalling Phase, a word is written as input, either in English or Spanish, and the translation mode is chosen. The word in the corresponding language appears immediately. An example can be seen in Fig. 3a. The word to be translated is departure and its corresponding translation in Spanish is partida. The translator presents other advantages as well. For instance, let us suppose that only a part of the word departure is written as input, say departu. The program will give as output the word partida (see Fig. 3b). Now, let us assume that instead of writing the last e, a w is written as a typo. The result can be seen in Fig. 3c. In this example we can see that, a misspell, which at the pattern level means that a noisy pattern has been given as input, does not limit the translator performance. The advantages shown by the translator stress out the advantages presented by the Alpha Beta bidirectional associative memories. These memories are immune to certain amounts and kinds of noise, properties which have not been characterized yet. 3

37 Alpha Beta bidirectional associative memories 37 Fig. 4 The bidirectional process of the model is shown in both screens. (a)the input pattern is a fingerprint, its corresponding output pattern a number. (b) A number is chosen as input pattern and its corresponding fingerprint is perfectly recalled 4. Fingerprint verifier The first part of the identification consists in building the Alpha Beta BAM; this is, the learning phase is realized. The input patterns set is made up of 4 fingerprints, which were obtained from the Fingerprint Verification Competition (FPV) [46]. Originally, these images have dimensions of 4 3 pixels. The Advanced Batch Converter image editor was used to modify the fingerprints dimensions, such that it was possible to present 4 images on one screen. The final dimensions of the images are 8 7 pixels. The output patterns set contains 4 images, each representing a number between and 4, inclusive. The dimensions of this images are pixels. For the verifier implementation it will be assumed that x represents the input patterns set. Given that fingerprints are represented with bidimensional images, and Alpha Beta BAMs work with column vectors, it becomes necessary to convert the images to a vectorial form. Therefore, input vectors dimension is n = 8 7 = 3, 6. In a similar way, the output patterns set s represented by y, and its images are converted to vector of dimension m = =. In this case, the number of trained patterns is p = 4. The learning phase process, whose algorithm was described in the former section, lasts approximately min and 34 s. Once the Alpha Beta BAM has been built, the program allows to choose any one of the learned fingerprints, or any one number associated to the fingerprints. If the chosen pattern is a fingerprint, the recalled pattern is the number associated to that specific fingerprint (see Fig. 4a). On the other hand, if a number is selected, the result of recall is its corresponding fingerprint (see Fig. 4b). For the recalling of a fingerprint parting from a number, the same algorithm that was described in the prior section is used, only this time the pattern presented to the verifier is a member of the pattern set y. In order to test the efficacy of the verifier, each of the 4 images depicting fingerprints was picked, obtaining in each case the perfect recall of its corresponding number. In the same way, the 4 numbers were individually presented as input; the recall phase delivered in a perfect manner, each of the corresponding fingerprints. From that we can observe that the verifier exhibits perfect recall of all learned patterns. Actually, in this example only 4 pairs of patterns were used; however, the verifier can asso- 3