Chapter3 Pattern Association & Associative Memory

Transcription

1 Cher3 Per Aoco & Aocve Memor Aocg er hch re mlr, corr, cloe roxm l, cloe ucceo emorl Aocve recll evoe oced er recll er b r of evoe/recll h comlee/ o er To e of oco. For o er d heero-oco! : relg o dffere er uo-oco : relg r of er h oher r

2 Archecure of NN ocve memor gle ler h/ou u ler o ler for bdrecol oc. Lerg lgorhm for AM Hebb lerg rule d vro grde dece Al orge cc ho m er c be remembered correcl memor covergece AM model for hum memor

3 Trg Algorhm for Smle AM Neor rucure: gle ler oe ouu ler of o-ler u d oe u ler mlr o he mle eor for clfco Ch. _ x _m _ x _m _m _m Gol of lerg: o ob e of egh _ from e of rg er r {:} uch h he led o he u ler, comued he ouu ler for ll rg r : : f T for ll

4 Smlr o hebb lerg for clfco Ch. Algorhm: bolr or br er If For ech rg mle :: cree f boh d re ON br or hve he me g bolr ll. P W { P Hebb rule The, Ied of obg W b erve ude, c be comued from he rg e b clculg he ouer roduc of d. fer ude } for ll P rg er

5 Ouer roduc. Le d be ro vecor. The for rculr rg r : Ad I volve 3 eed loo,, order of rrelev o P /* for ever rg r */ o /* for ever ro W */ o m /* for ever eleme ro */ [ ] m m m m m m T W ,... P T P W : +

6 Doe h mehod rovde good oco? Recll h rg mle fer he egh re lered or comued Al o oe ler, hoe er o he oher, e.g. M o l ucceed ech egh co ome formo from ll mle W f + + T T T P T P T W cro-l erm rcl erm

7 Prcl erm gve he oco beee d. Cro-l reree correlo beee : d oher rg r. Whe cro-l lrge, ll recll omehg oher h. T If ll re orhogol o ech oher, he, o mle oher h : corbue o he reul. There re mo orhogol vecor -dmeol ce. Cro-l cree he P cree. Ho m rbrr rg r c be ored AM? C be more h llog ome o-orhogol er hle eeg cro-l erm mll? Sorge cc more ler

8 Del Rule Smlr o h ued Adle The orgl del rule for egh ude: Exeded del rule For ouu u h dffereble cvo fuco Derved follog grde dece roch. x α _ ' f x α I J J J J IJ J J J IJ J J J IJ J J J J J IJ J J IJ x f E x f x f f E ' ' ' m E

9 me he ude rule for ouu ode BP lerg. Wor ell f S re lerl deede eve f o orhogol.

10 Exmle of heero-ocve memor Br er r : h 4 d. Tol eghed u o ouu u: _ x Acvo fuco: hrehold f _ > f _ Wegh re comued b Hebb rule um of ouer roduc of ll rg r W Trg mle: P T,, 3, 4,

11 T 3 3 T T 4 4 T W Comug he egh

12 recll: x x mlr o S d S x,,,, cl,, cl, o uffcel mlr o cl del-rule ould gve me or mlr reul.

13 Exmle of uo-ocve memor Sme heero-ocve e, exce. Ued o recll er b o or comlee vero. er comleo/er recover A gle er,,, - ored egh comued b Hebb rule ouer roduc W rg. o mg fo more o W W W W o recogzed

14 Dgol eleme ll dome he comuo he mulle er re ored P. Whe P lrge, W cloe o de mrx. Th cue ouu u, hch m o be oed er. The er correco oer lo. Relce dgol eleme b zero. W W ' W ' 3 W ' W ' rog

15 Sorge Cc # of er h c be correcl ored & reclled b eor. More er c be ored f he re o mlr o ech oher e.g., orhogol o-orhogol orhogol W W o ored correcl I W correcl reclled hree er c be All

16 Addg oe more orhogol er he egh mrx become: W The memor comleel deroed! Theorem: b eor ble o ore u o - muull orhogol M.O. bolr vecor of - dmeo, bu o uch vecor. Iforml rgume: Suoe m orhogol vecor re ored h he follog egh mrx: f zero dgol m ohere Hebb rule... m

17 m m W : comoe h he,...,,...,,..., Le r o recll oe of hem,... ce re M.O. d ce T

18 [ ] m m m Therefore, m W Whe m <, c correcl recll elf he m, ouu vecor, recll fl I ler lgebrc erm, egevecor of W, hoe correodg egevlue -m. he m, W h egevlue zero, he ol egevecor, hch rvl egevecor.

19 Ho m muull orhogol bolr vecor h gve dmeo? c be re m, here m odd eger. The mxmll: M.O. vecor Follo u queo: Wh ould be he cc of AM f ored er re o muull orhogol rdom Abl of er recover d comleo. Ho fr off er c be from ored er h ll ble o recll correc/ored er Suoe x ored er, x cloe o x, d x fxw eve cloer o x h x. Wh hould e do? Feed bc x, d hoe ero of feedbc ll led o x

20 Exmle: Ierve Auoocve Neor x,,, W I geerl: ug curre ouu u of he ex ero x l recll u xi fxi-w, I,, ul xn xk here K < N A comlee recll u : x',,, x' W,,, x" x" W 3,,, 3,,, x Ouu u re hrehold u

21 Dmc Sem: e vecor xi If N-, xn ble e fxed o fxnw fxn-w xn If xk oe of he ored er, he xk clled geue memor Ohere, xk urou memor cued b crol/erferece beee geue memore Ech fxed o geue or urou memor rcor h dffere rco b If! N-, lm-crcle, The eor ll ree xk, xk+,..xnxk he ero coue. Iero ll eveull o becue he ol umber of dc e fe 3^ f hrehold u re ued. If gmod u re ued, he em m coue evolve forever cho.

22 Dcree Hofeld Model A gle ler eor ech ode boh u d ouu u More h AM Oher lco e.g., comborl omzo Dffere form: dcree & couou Mor corbuo of Joh Hofeld o NN Treg eor dmc em Iroduce he oo of eerg fuco & rcor o NN reerch

23 Dcree Hofeld Model DHM AM Archecure: gle ler u erve boh u d ouu ode re hrehold u br or bolr egh: full coeced, mmerc, d zero dgol re exerl x u, hch m be re or erme

24 Wegh: To ore er,,, P bolr: me Hebb rule h zero dgol br: coverg o bolr he corucg W.

25 Recll Ue u vecor o recll ored vecor boo cll he lco of DHM Ech me, rdoml elec u for ude Recll Procedure.Al recll u vecor x o he eor: : x,,....whle covergece fl do..rdoml elec u.. Comue _ x +.3. Deerme cvo of Y f _ > θ f _ θ f _ < θ.4. Perodcll e for covergece.

26 Noe:. Ech u hould hve equl robbl o be eleced e.. Theorecll, o guree covergece of he recll roce, ol oe u lloed o ude cvo me durg he comuo. Hoever, he em m coverge fer f ll u re lloed o ude her cvo he me me. 3. Covergece e: curre ex 4. uull e o zero. θ x 5. e. _ x + ool.

27 Exmle: Sore oe er: br er,,, bolr couerr - gve he mew W Recll u x,,,, fr o b re rog Y eleced Y 4 eleced _ x + + _ 4 x Y,,, Y,,, + Y3 eleced _ 3 x3 + 3 Y,,, 3 3 Y eleced + _ x + Y,,, + The ored er correcl reclled

28 Covergece Al of DHM To queo:.wll Hofeld AM coverge o h gve recll u?.wll Hofeld AM coverge o he ored er h cloe o he recll u? Hofeld rovde er o he fr queo B roducg eerg fuco o h model, No fcor er o he ecod queo o fr. Eerg fuco: Noo hermo-dmc hcl em. The em h edec o move ord loer eerg e. Alo o Luov fuco. Afer Luov heorem for he bl of em of dfferel equo.

29 I geerl, he eerg fuco E, here he e of he em e me, mu f o codo. E bouded from belo E The eerg fuco defed for DHM c. E mooocll ocreg. E + E + E couou vero : E& E.5 x + θ Sho E + A +, Y eleced for ude + + Noe : + ol oe u c ude me E + E x + + θ.5 x + θ +

30 erm hch re dffere he o r re hoe volvg E, + [, + x x, θ θ ] + ce : f & + + _ < θ E + < f & + + _ > θ E + < ohere, + + E _ + + Sho E bouded from belo, ce, x, θ, re ll bouded, E bouded.

31 Comme:.Wh coverge. Ech me, E eher uchged or decree mou. E bouded from belo. There lm E m decree. Afer fe umber of e, E ll o decree o mer h u eleced for ude. eher or _ + θ.the e he em coverge ble e. Wll reur o h e fer ome mll erurbo. I clled rcor h dffere rco b 3.Error fuco of BP lerg oher exmle of eerg/luov fuco. Becue I bouded from belo E> I mooocll o-creg W ude log grde dece of E

32 P: mxmum umber of rdom er of dmeo c be ored DHM of ode Hofeld obervo: Theorecl l: Cc Al of DHM P P.5,.5 P P, log log P/ decree becue lrger led o more erferece beee ored er. Some or o modf HM o cree cc o cloe o, W red o comued b Hebb rule.

33 M O Wor: Oe oble reo for he mll cc of HM h doe o hve hdde ode. Tr feed forrd eor h hdde ler b BP o eblh er uo-ocve. Recll: feedbc he ouu o u ler, mg dmc em. Sho ll coverge, d ored er become geue memore. I c ore m more er eem O^ I er comlee/recover cbl decree he cree # of urou rcor eem o cree exoell ouu hdde u ouu hdde u Auo-oco Heero-oco ouu hdde u

34 Archecure: Bdrecol AMBAM To ler of o-ler u: X-ler, Y-ler U: dcree hrehold, coug gmod c be eher br or bolr.

35 Wegh: T W Hebb/ouer roduc Smmerc: Cover br er o bolr he corucg W Recll: P m Bdrecol, eher b o recll Y or b Y Recurre: f _,... f _ here x + _ here x _ X o recll X Ude c be eher chroou HM or chroou chge ll Y u oe me, he ll X u he ex me + m x m f x _ +,... f x _ +

36 Al dcree ce Eerg fuco: lo Luov fuco L.5 XWY m The roof mlr o DHM Hold for boh chroou d chroou ude hold for DHM ol h chroou ude, due o lerl coeco. Sorge cc: x T + YW T X ο mx, m T XWY T