Neural networks: Associative memory

Transcription

1 Neural networs: Associative memory Prof. Sven Lončarić 1

2 Overview of topics l Introduction l Associative memories l Correlation matrix as an associative memory l Error correction learning l Pseudoinverse matrix as an associative memory l Discussion l Problems 2

3 Introduction l In neurobiological context, a memory represents relatively permanent neural changed caused by an interaction of an organism and environment l For memory to be useful, it must be accessible by the neural system l A memory is filled through a learning process l Memories can be divided into: l l Short term memory (contains the current state of the environment) Long term memory (contains permanently stored nowledge) 3

4 Introduction l his section deals with a distributed memory similar to the brain that uses associations l Associative memory is an important part of human memory l he main property of an associative memory is mapping of input patterns into output patterns of neural activity 4

5 Introduction l During learning an input pattern called a ey is presented to the memory that transforms it into a memorized pattern l During recall a noised or incomplete version of the original ey is presented to the memory l Regardless of the imperfect input ey, the associative memory outputs the corresponding memorized output pattern 5

6 Properties of associative memories l An associative memory is distributed l Input pattern (ey) and output (memorized pattern) are vectors l Information is stored in memory using a large number of neurons l Information contained in the ey determines the address of the pattern in the memory l Memory is noise tolerant l Possible interactions of memorized patterns posibility of errors 6

7 ypes of associative memories l Autoassociative memory: l l Input vector (ey) is associated to itsself in the memory Dimension of input and output vectors is the same l Heteroassociative memory: l l Arbitrary input vectors (eys) are associated with arbitrary memorized vectors Dimension of input and output vectors can be different l In both cases a memorized pattern can be recalled using an input vector that is incomplete or noised version of the original ey 7

8 ypes of associative memories l Linear associative memory: l Neurons wor in linear mode (linear combination) l Let a and b be input and output from associative memory then the input-output mapping is represented by b=ma, where M is the memory matrix Stimulus a memory matrix M Output b l Nonlinear associative memory l Input-output relation is described by: b=f(m,a)a, where f(.,.) is a nonlinear function 8

9 A model of associative memory l A model of linear associative memory with artificial neurons is shown below: a 1 b 1 a 2 b 2 a p b p input layer output layer 9

10 Memory mapping l Let us assume that we have a memory with one input layer and one layer with p linear neurons l Let us assume that for input a we obtain output b l Let us assume that the memory stores q inputoutput pairs l We could express the input-output relations as: b =W()a, = 1,, q where W() is a matrix of dimensions p p that depends only on a and b 10

11 Memory mapping l For q input-output pairs we obtain matrices W(1),, W(q) l Furthermore, we could form a matrix of dimensions p p that represents the sum of matrices W(): M = q = 1 W( ) l Matrix M defines a relation between the input and the output patterns 11

12 Memory mapping l A memory matrix M can also be represented using a recursive expression: M = M -1 + W(), = 1, 2,, q where M 0 = 0 l M -1 is an old matrix value for the first -1 associations l M is an updated matrix that also taes into account the -th association l As the number of stored associations q increases the influence of individual new pairs to the memory decreases 12

13 Correlation matrix l Let us assume that a memory has matrix M that represents the stored associations a, b, where = 1, 2,, q l An estimate of the matrix M (called correlation matrix) can be calculated using: Mˆ = q = 1 l he term b a is the outer product of ey a and stored pattern b and is a matrix of dimension p p b a 13

14 Correlation matrix l his estimate can be written as: a 1 a M ˆ = 1 2 q =! aq [ ] 2 b b b BA where A=[a 1 a 2... a q ] and B=[b 1 b 2... b q ] l A is the matrix of input patterns (eys) of dimension p q l B is the matrix of stored patterns of dimension p q 14

15 15 Memory recall l Let us assume that a pattern a is input into associative memory that has stored q patterns l he output of the memory will be: l Furthermore, we can rewrite this as: ( ) = = = = = q q 1 1 ˆ b a a a a b Ma b ( ) ( ) = + = q 1 b a a b a a b

16 Memory recall l Let us assume that the input vectors (eys) are normalized: a a =1 l hen it holds that: where: b = b + v v = q ( a ) a = 1 b 16

17 Interpretation of memory recall b = b + v l he first term in the above expression represents the desired memory recall for ey a l he second term represents a crosstal between ey a and other stored eys (i.e. noise) l If input patterns are statistically independent than the second term represents Gaussian noise l his noise limits the number of reliably stored patterns 17

18 Interpretation of memory recall ( a ) a = 1 l Let us assume that the input vectors a (eys) compose an orthonormal set of vectors, i.e. 1, = aa = 0, l In that case the noise v edna nuli v = l he number of linearly independent vectors of dimension p is equal to p q l his means that the memory capacity is equal to the vector dimension (in this case p) b 18

19 Discussion l If a set of input patterns is not orthogonal it is possible to use Gram-Schmidt procedure to orthonormalize the ste of linearly independent vectors l A disadvantage of this simple approach is that the memory has no way of error correction l o overcome this drawbac it is possible to use an error correction algorithm 19

20 Error correction learning l Let M(n) be the matrix learned in step n l Input vector a is presented to the memory and gives output vector M(n) a l An error vector can be defined as: e (n) = b - M(n) a where b is the output associated with input a l he error vector can be used for learning as: (correction) = (learning-rate) (error) (input) 20

21 Error correction learning l In this case we can write: ΔM(n) = η e (n) a = η [b - M(n) a ] a l Correction ΔM(n) is used to update matrix M: M(n+1) = M(n) + ΔM(n), M(0)=0 l A constant positive parameter η generates a short term memory because recent patterns will be better memorized compared to older patterns l For this reason the parameter η is sometimes decreased with time n to approach zero when memory is full 21

22 Pseudoinverse memory l Another type of linear associative memory is a memory that minimizes the error of associative memory: e = B MA ˆ where A is a ey matrix of dimension p q, and B is a matrix of desired outputs of dimension p q l Euclidean norm gives the error of the associative memory 22

23 Pseudoinverse memory l Linear algebra tells us that the error e is minimal for: + Mˆ = BA where A + is a pseudoinverse matrix of matrix A l he above equation is called the pseudoinverse learning rule and the memory is called pseudoinverse memory l he sufficient condition for the perfect association is: + A A = I l hen it holds that: M ˆ + A = BA A = BI = B 23

24 Discussion l In some applications better resistance to noise is shown by correlation memory and in some other cases pseudoinverse memory shows better results 24

25 Problems l Problem 3.1. l Modify expressions for correlation memory assuming different dimensions of input and output vector. l Problem 3.2. l Let input vectors be defined as: a 1 =[ ], a 2 =[ ], a 3 =[ ] and output vector as: b 1 =[5 1 0], b 2 =[-2 1 6], b 3 =[-2 4 3] Determine a memory matrix M and show that the memory recall is correct. 25