Lecture 23: Associative memory & Hopfield Networks

Transcription

1 Lecture 3: Assocatve memory & Hopfeld Networks

2 Feedforward/Feedback NNs Feedforward NNs The coectos betwee uts do ot form cycles. Usually produce a respose to a put quckly. Most feedforward NNs ca be traed usg a wde varety of effcet algorthms. Feedback or recurret NNs There are cycles the coectos. I some feedback NNs, each tme a put s preseted, the NN must terate for a potetally log tme before t produces a respose. Usually more dffcult to tra tha feedforward NNs.

3 Supervsed-Learg NNs Feedforward NNs Perceptro Adale, Madale Backpropagato (BP) Artmap Learg Vector Quatzato (LVQ) Probablstc Neural Network (PNN) Geeral Regresso Neural Network (GRNN) Feedback or recurret NNs Bra-State--a-Box (BSB) Fuzzy Cogtve Map (FCM) Boltzma Mache (BM) Backpropagato through tme (BPTT)

4 Usupervsed-Learg NNs Feedforward NNs Learg Matrx (LM) Sparse Dstrbuted Assocatve Memory (SDM) Fuzzy Assocatve Memory (FAM) Couterprogato (CPN) Feedback or Recurret NNs Bary Adaptve Resoace Theory (ART) Aalog Adaptve Resoace Theory (ART, ARTa) Dscrete Hopfeld (DH) Cotuous Hopfeld (CH) Dscrete Bdrectoal Assocatve Memory (BAM)

5 Neural Networks wth Temporal Behavor Icluso of feedback gves temporal characterstcs to eural etworks: recurret etworks. Two ways to add feedback: Local feedback Global feedback Recurret etworks ca become ustable or stable. Ma terest s recurret etwork s stablty: eurodyamcs. Stablty s a property of the whole system: coordato betwee parts s ecessary.

6 The Hopfeld NNs I 98, Hopfeld, a Caltech physcst, mathematcally ted together may of the deas from prevous research. A fully coected, symmetrcally weghted etwork where each ode fuctos both as put ad output ode. Used for Assocated memores Combatoral optmzato

7 Assocatve Memores A assocatve memory s a cotet-addressable structure t that t maps a set of put patters to a set of output patters. Two types of assocatve memory: autoassocatve ad heteroassocatve. Auto-assocato retreves a prevously stored patter that most closely resembles the curret patter. Hetero-assocato the retreved patter s, geeral, dfferet from the put patter ot oly cotet but possbly also type ad format.

8 Assocatve Memores Auto-assocato A memory A Hetero-assocato Tehra memory Cty

9 Lear assocatve memory The lear assocator s oe of the smplest ad frst studed assocatve memory models A feedforward type etwork where the output s produced a sgle feedforward computato The puts coected to the outputs va the coecto weght matrx W = [w j ] m x It s W that stores the N dfferet assocated patter pars {(X k, Y k ) k =,,..., N} where the puts ad outputs are ether - or +

10 Lear assocatve memory Buldg a assocatve memory s costructg W such that whe a put patter s preseted, the stored patter assocated wth the put patter s retreved ecodg W k s for a partcular assocated patter par (X k, Y k ) are computed as (w j ) k = (x ) k (y j ) k The W = a N k=n W k a s the proportoalty or ormalzg costat to prevet the syaptc values from gog too large whe there are a umber of assocated patter pars to be memorzed, usually a = /N. The coecto weght matrx costructo above smultaeously stores or remembers N dfferet assocated patter pars a dstrbuted maer.

11 Lear assocatve memory After ecodg or memorzato, the etwork ca be used for retreval decodg (X = W - Y) Gve a stmulus put patter X, decodg or recollecto s accomplshed by computg the et put to the output uts usg the prevous formula. The, y (- or +) ca be computed by thresholdg the euro j The put patter may cota errors ad ose, or may be a complete verso of some prevously ecoded patter. Nevertheless, whe preseted wth such a corrupted put patter, the etwork wll retreve the stored patter that s closest to actual put patter. So, the model s robust ad fault tolerat,.e., the presece of ose or errors results oly a mere decrease rather tha total degradato the performace of the etwork. Assocatve memores beg robust ad fault tolerat t are the byproducts of havg a umber of processg elemets performg hghly parallel ad dstrbuted computatos.

12 Lear assocatve memory Tradtoal measures of assocatve memory performace are ts memory capacty ad cotet- addressablty. Memory capacty refers to the maxmum umber of assocated patter pars that t ca be stored ad correctly retreved Cotet-addressablty s the ablty of the etwork to retreve the correct stored patter It ca be show that usg Hebb's learg rule buldg the coecto weght matrx of a assocatve memory yelds a sgfcatly low memory capacty. Due to the lmtato brought about by usg Hebb's learg rule, several modfcatos ad varats have bee proposed to maxmze the memory capacty.

13 Ituto Mapulato of attractors as a recurret eural etwork paradgm. We ca detfy attractors wth computatoal objects. I order to do so, we must exercse cotrol over the locato of the attractors the state space of the system. A learg algorthm wll mapulate the equatos goverg the dyamcal behavor so that a desred locato of attractors are set. Oe good way to do ths s to use the eergy mmzato paradgm (e.g., by Hopfeld).

14 Ituto N uts wth full coecto amog every ode (o selffeedback). Gve M put patters, each havg the same dmesoalty as the etwork, ca be memorzed attractors of the etwork. Startg wth a tal patter, the dyamc wll coverge toward the attractor of the bas of attracto where the tal patter was placed.

15 Example of usg Hopfeld NNs

16 Addtve model of a euro Low put resstace Uty curret ga Hgh output resstace

17 Addtve model of a euro The total curret flowg toward the put ode of the olear elemet s: =N w j x (t) + I j The total t curret flowg away from the put odes of the olear elemet s v j(t)/r j + C j dv j(t)/dt By applyg Krchoff's curret law, we get C j dv j (t)/dt + v j (t)/r j = =N w j x (t) + I j Gve the duced local feld v j (t), we may determe the output of euro j by usg the olear relato x j (t) = (v j () )

18 Addtve model of a euro So, gorg tereuro propagato tme delays, we may defe the dyamcs of the etwork by the followg system of coupled frst-order ODEs: C -v N j dv j (t)/dt = j (t)/r j + = w j x (t) + I j ; j =,...,N x j (t) = (v j () ) For example: (v j() ) = /(+exp(- v j()) Ths ca be coverted to: dx (t)/dt = x (t)/r + ( N w x (t)) +K dx j (t)/dt = -x j (t)/r j + ( =N w j x (t)) + K j

19 Hopfeld Model The Hopfeld etwork (model) cossts of a set of euros ad a correspodg set of ut delays, formg a multple-loop feedback system The umber of ffeedback kloops s equal lto the umber of euros. The output of each euro s fed back va a ut delay elemet, to each of the other euros the etwork..e., there s o self-feedback the etwork.

20 Hopfeld Archtecture

21 Hopfeld Model: learg Let t, t, t M deote a kow set of N- M dmesoal fudametal memores The weghts are computed usg Hebb s rule W j = (/M) q=m t q,j t q, ; j W j = 0 ; j = t q, : the -th elemet of t q W becomes a N N matrx The elemets of vector t q are equal to - or + Oce computed, the syaptc weghts are kept fxed

22 Hopfeld Model: talzato Let t probe deote a ukow N-dmesoal probe put vector preseted to the etwork The algorthm s talzed by settg x j (0) = t j,probe ; j =,...,N where x j (0) s the state of euro j at tme = 0, d t th j th l t f th t t ad t j,probe s the j-th elemet of the evctor t probe

23 Hopfeld Model: terato The states of the euros (.e., radomly ad oe at a tme) are terated asychroously (dfferece equato for dscrete-tme ad dfferetal equatos for cotuous-tme) utl covergece. For dscrete-tme t s x j ( + ) = sg [ =N W j x ()] ; j =,,...,N The covergace of the above rule s gaurateed (we wll see why!)

24 Hopfeld Model: Outputtg Let X fxed deote the fxed pot (stable state) fxed p ( ) computed at the ed of the prevous step The resultg output vector y of the etwork s set as y = X fxed Step s the storage phase, whle the last three steps costtute the retreval phase

25 The Dscrete Hopfeld NNs w w w 3 w 3 w 3 w 3 w w 3 w w w 3 w... 3 I I I 3 I v v v 3 v

26 The Dscrete Hopfeld NNs w j =w j w w w 3 w = 0 w w w w 3 w 3 w 3 w w 3 w w w 3 w... 3 I I I 3 I v v v 3 v

27 The Dscrete Hopfeld NNs w j =w j w w w 3 w = 0 w 3 w 3 w 3 w w 3 w w w 3 w... 3 I I I 3 I v v v 3 v

28 State Update Rule Asychroous mode w w w 3 w 3 w 3 w 3 w w 3 w w w 3 w... 3 Update rule H ( t) w jv j ( t) I j j I I I 3 I v v v 3 v H ( t) 0 ( ) sg ( v t H t ) H ( t) 0

29 Eergy Fucto H ( t) wjvj ( t) I j j H ( t ) 0 v ( t ) H ( t ) 0 w w w 3 w 3 w 3 w 3 w w 3 w w w 3 w... 3 I I I 3 I E wj vv j I v j v v v 3 v If E s mootocally decreasg, the system s stable. (Due to Lyapuov Theorem)

30 The Proof E wj vv j I v j Suppose that at tme t +, the k th euro s selected for update. w w w 3 w 3 w 3 w 3 w w 3 w w w 3 w Ther values are ot chaged at tme t t I I I 3 I Et () wv () tv () t Iv () t wv () tv () t Iv () t j j k k k k j k jk k v v v 3 v Et ( ) wv( t) v( t) Iv( t) wv( t) v( t) Iv( t) j j k k k k j k jk k

31 The Proof E E ( t ) E ( t ) w v ( t ) v () t I v ( t ) w v ( t ) v () t I I v ( t ) k k k k k k k k wv k ( t) Ik [ vk( t) vk( t)] H ( t)[ v ( t) v ( t)] k k k () j () j () () k k () () k k () j k jk k E t w v t v t I v t w v t v t I v t E( t) w v ( t) v ( t) I v ( t) w v ( t) v ( t) I v ( t) j j k k k k j k jk k

32 The Proof E E( t) E( t) w v ( t) v () t I v ( t) w v ( t) v () t I v ( t) k k k k k k k k wv k ( t) Ik [ vk( t) vk( t)] H ( t)[ v ( t) v ( t)] k k k v k (t) H k (t+) v k (t+) E 0 0 E < 0 < 0 0 < 0 < 0 0

33 The Neuro of Cotuous Hopfeld NNs v w I v =a(u ) v w... u v u =a (v ) w u a ( u ) C g v v v

34 The Dyamcs v v w I d u ( v u ) w C j( j u) gu I w v dt w... w ( v ) u w v ( v ) w u C a ( u ) v v u du dt C g j j du C w v j w g u I dt j j j j g u j j G du C w G I jv j u dt j j

35 The Cotuous Hopfeld NNs I I I 3 I d C w G I u jv j u dt j j w w w3 w 3 w 3 w w 3 w 3 w 3 w 3 w w u u u 3 g g 3 u w... C g C C 3... v v v 3 v C g v v v 3 v

36 The Cotuous Hopfeld NNs d C u w v G u j j I dt j j du C w dt... I I I 3 I v w w w 3 w w 3 w w 3 w u C g C g C w 3 w 3 du dt du dt u u 3 C 3 3 g 3 j j j j j w u w v v v 3 v C 3 du dt... j j w 3 j w w Cj 3 j v v j G u j G u g G u 3 G u v v v 3 v w j j j v j 3 I I I I 3

37 Lyapuov Eergy Fucto j j 0 I I I 3 I w w w 3 w 3 w 3 w w 3 3 w 3 w 3 w w 3 w w... u u u 3 u C g C g C 3 g 3... v v v 3 v C g v v v 3 v

38 Lyapuov Eergy Fucto u=a (v) v =a(u ) I I I 3 I uw w w 3 w 3 v 0 a ( x) dx w 3 w w 3 w 3 w v v w 3 a ( v ) u w u u u 3 u g g 3 u w... C g C v v v 3 v C 3... C g v v v 3 v

39 Stablty of Cotuous Hopfeld NNs de dt du C v u I Dyamcs w j j G dt j j v wj v j Iv a () v dv 0 j j E v G de dv dv w jvj Gu I dv dt j dt j du dv C dt dt u a ( v ) u a v ( ) da v dv ( ) C du da ( v ) dv dv dt dt dv dt

40 Stablty of Cotuous Hopfeld NNs Dyamcs E v G du C v u I wj j G dt j j v wj v j Iv a () 0 v dv j j 0 de da ( v) dv C dt dv dt de dt 0 > 0 da () v dv 0 u=a (v) v

41 Stablty of Cotuous Hopfeld NNs... C w jv I I I 3 dt j I w w w 3 w w 3 w u C g C g w 3 C w 3 w 3 C3 du dt du dt u u 3 C 3 g 3 3 j w j j w ju j... du w dt v v v 3 v j j v v v 3 v C j w 3 w C j v v w3 j j v j j j G u G u g G u 3 G u 3 I I I I 3

42 Bass of Attracto

43 Bass of Attracto

44 Local/Global Mma Eergy Ladscape

45 Hopfeld Model (Summary) The Hopfeld etwork may be operated a cotuous mode or dscrete mode The cotuous mode of operato s based o a addtve model The dscrete mode of operato s based o the McCulloch-Ptts model We may establsh the relatoshp betwee the stable states of the cotuous ad dscrete Hopfeld models wth the followg assumptos: The output of a euro has the asymptotc values x j = whe v j ad x j = - whe v j - The mdpot of the actvato fucto of a euro les at the org, that s (0) = 0 Correspodgly, we may set the bas I j equal to zero for all j.

46 Hopfeld Model If the ga of the actvato fucto s very large (fty), we ca show that the eergy fucto of the formulatg the eergy fucto E of the dscrete (ad cotuous) Hopfeld models ca be rewrtte as The models tred to mmze the above eergy fucto

47 Hopfeld Model: Storage The learg s smlar to Hebba learg: w j = 0 f = j. (Learg s oe-shot.) I matrx form the above becomes: The resultg weght matrx W s symmetrc: W = W T

48 Hopfeld Model: Actvato (Retreval) Italze the etwork wth a probe patter probe. Update output of each euro (pckg them by radom) as utl x reaches a fxed pot. The fxed pot s the retreved output.

49 Storage Capacty of Hopfeld Network Gve a probe (ukow put) equal to the p ( p ) q stored patter, the actvato of the j th euro ca be decomposed to the sgal term ad the ose term (The ut hypercube s N-dmesoal ad we have M fudametal memory):

50 Storage Capacty of Hopfeld Network The sgal to ose rato s obtaed as b = N/M It has bee show that memory recall of the Hopfeld etwork deterorates wth creasg load parameter b, ad breaks dow at the crtcal value b c = 0.4 Thus, M < 0.5N order that t Hopfeld retreve correctly

51 Cohe-Grossberg Theorem Cohe & Grossberg defed a geeral prcple for assessg the stablty of a certa class of eural etworks descrbed by the followg equatos (Hopfled model s a specal case): du j (t)/dt = a j (u j )[b j (u j ) - =N c j (u ) ] ; j =,...,N They proved that the eergy fucto E descrbed below ca be cosdered as a Lyapuov fucto for the above system E = (/) =N j=n c j (u ) j (u j ) - j=n uj 0 b j (k) j (k)dk j (k) = (d/dk) j (k)

52 Assocatve Memores Also amed cotet-addressable memory. Autoassocatve Memory (Hopfeld Memory) Heteroassocatve Memory (Bdrecto Assocatve Memory) Stored Patters ( x, y ) ( x, y ) p p ( x, y ) x y R R m x x y Autoassocatve y Heteroassocatve

53 Hopfeld Memory 0 Neuros Fully coected 4,400 weghts Stored Patters Memory Assocato

54 Hopfeld Memory 0 Neuros Fully coected 4,400 weghts Stored Patters Memory Assocato

55 The Storage Algorthm Suppose that the set of stored patters s of dmeso. W w w w w w w x k ( x k, x k,, x k ) T k,, p. k x {, },,. p p k k x x k ( k ) T j j W x x pi wj k k 0 j w w w

56 Aalyss Suppose that x x. p k k T Wx x ( x ) pix k x px ( p) x p p k k k k T x xj j W x ( x ) pi w j k k 0 j

57 Example ( ) p k k T p W x x I p k k j k j x x j w ( ) T x ( ) k p W x x I 0 k w j j (,,,) (,,,) T x x ( ) T x x ( ) T x x ) ( ) ( T T W ) ( ) ( T T x x x x 0 0 0

58 Example ) T x (,,,) 3 4 x (,,,) T W j j w wj xx j j E( x) w xx j Ix T x Wx ( xx xx) 3 4

59 Example ) T x (,,,) 3 4 x (,,,) T E=4 E( x) ( x x x x ) 3 4 Stable E=0 E=4

60 Example ) T x (,,,) 3 4 x (,,,) T E=4 E( x) ( x x x x ) 3 4 Stable E=00 E=44

61 Bdrecto Memory y y y y( t ) a Wx( t). Y Layer.. y( t) Fo orward Pass W=[w j ] m... X Layer Backwa ard Pass s T x () t ( t ) a ( t) x W y x x x m

62 Bdrecto Memory y ( t ) a Wx ( t ) Stored Patters y y y. Y Layer.. W y ( t ) For rward Pass W=[w j ] m Backw ward Pas ss ( x, y ) ( x, y ) p p ( x, y ) x () t T ( t ) a ( t)... X Layer x x x m x W y

63 The Storage Algorthm Stored Patters ( x, y ) ( x, y ) p p ( x, y ) x k ( x, x,, x ) T x {,} y k m ( y, y,, y ) p k k k y W y ( x ) w j p k y T k k x j y {,} T

64 Aalyss Suppose x k s oe of the stored vector: p k k k T k y a Wx a y ( x ) x k p k k k T k a m ( x ) x a my k k k ' p k k W y ( x ) k k y y Eergy Fucto: 0 a m y k y y E( xy, ) xw y y Wx y Wx T T T T T

65 Applcatos of Hopfeld Memory Patter restorato Patter completo Patter geeralzato Patter assocato

66 Readg S Hayk, Neural Networks: A Comprehesve y, p Foudato, 007 (Chapter 4).