1 n. w = How much information can the network store? 4. RECURRENT NETWORKS. One stored pattern, x (1) : Since the elements of the vector x (1) should

Transcription

1 4. RECURRENT NETWORKS Ho much formao ca he eor sore? Ths chaper preses some ypes of recurre eural eors: Frs, eors h (srog egh cosras are reaed, eemplfed by Hopfeld eors, folloed by a se of recurre eors ha arse f feedbac coecos are cluded mullayer percepro eors. 4. Hopfeld eors Oe sored paer, ( : Sce he elemes of he vecor ( should ( sasfy h(, he eghs ca be chose, e.g., as ( ( (4. because hs guaraees ha sg( a sce. A commo ( The recurre eor roduced by Hopfeld 98 acs as a choce s assocave memory; afer sorg a umber of daa paers, he eor ca upo preseao of a e paer decde hch oe of he sored paers shos he bes correspodece h. The sored vecors hus ac as aracors h her correspodg bass of araco. ( ( (4.3 Cosder a recurre eor h odes ha oupu eher + or -. Le he odes have he acvao fuco + f a 0 h( a h f < 0 a (4. hch ca be compared h he Hebba rule (.5! Several sored paers, (, (,..., (m : A possbly s o add he corbuos of he vecors o be sored ad hope for he bes,.e., ha he eor ll be able o sore all paers: m p ( p (

2 Ho much ca be sored? ( p ( r δ pr (4.7 Qualave aalyss: We sh ha ( p h( a p (4.5 ( p, For a daa vecor r e hus ge a ( r ( r ( r + p p r p ( r ( r ( r + c (4.6 here c s a cross-al erm. If c < hs erm does o domae, so he desred resul s obaed. If oe aemps o sore oo may vecors, c > ad he eor fals o ac as a assocave memory for he daa vecors. here δ s he Kroecer dela. For hs case, he cross-al erm of (4.6 vashes, so oe should be able o sore m m ma orhogoal daa vecors. Hoever, f Eq. (4.4 s appled h m all us ge oly self loops, hch meas ha he raed eor alays eacly reproduces s pus! Therefore m <, eve for orhogoal vecors. For praccal cases here he pus are o orhogoal, or cao be orhogoalzed, a heorecal mamum of he umber of sored paers s m ma (4.8 A rule of humb ofe appled s ha he umber of paers possble o sore a Hopfeld eor s 5 % of he umber of odes. Hopfeld roduced a eergy fuco I dervg resuls for he Hopfeld eor, s ofe assumed ha he daa vecors hold radom varables, ad he he bas erms ca be omed (as doe above. The problem ges much more complcaed f he daa E, p (4.9 vecors are correlaed. The oher ereme, ur,.e., orhogoal vecors gves 64 65

3 ha as sho o decrease f he eor follos he dyamc rule (4.. Daa vecors ha have bee sore ca herefore be recosruced by mmzg Eq. (4.9, sce he vecors ac as aracors he sysem. (4.0 oe may re E 443 C cosa (4. Sce he cosa C s umpora a he mmzao, oe may se 0 (4..e., om he self loops. Eergy ladscape ad bass of araco If he egh mar s symmercal,.e., If he vecors o be sored are o radom vecors, may be movaed o clude a hreshold erm (bas, 0, for each ode. Ths gves he eergy fuco E 0 (

4 Ho shall he sae of he eor be updaed o deerme he oupu for a e pu vecor? Asychroously: a. Selec a ode,, by radom ad updae s eghs by Eq. (4.. b. Le every ode updae self (depede of oher by Eq. (4. h a probably, p. Sychroously: Le all odes updae smulaeously by Eq. (4. o he bass of old formao,.e., (+ h(a (. The sychroous updae s, hoever, lely o gve rse o cyclc saes ad should o be used! The e fgure (Hay 994 shos rag daa cossg of seve umbers ad a perod (0,,, 3, 4, 6, 9,. fed o a Hopfeld eor as bary pels (blac/he ad he resuls ha evolve afer preseg a dsored 6. Trag daa (upper ad erpreao of a dsored

5 Travelg Salesma Problem (TSP Combaoral problems are usually dffcul o solve because of a large umber of alerave soluos. Hoever, for may such problems s o ecessary o fd he bes soluo; a suffcely good soluo could do! A classcal eample s he Travelg Salesma Problem: A salesma has o vs ces arbrary order, bu he should mmze he ravelg dsace. The problem s rcy! There are roues, bu sce he order does o maer (.e., A B C s aalogous o B C A ad C A B a problem h 3, ad he dreco ca be reversed (C B A, e ge!/( dsc roues.! / > Combaoral eploso maes a ehausve search for large mpossble; f oe cy s added, he umber of roues o be suded ll crease by a facor of ( +! ( +! ( +!!( + Ho ca he problem be formulaed o be solved by a Hopfeld eor? Use a eor h odes, he ros of hch deoe he ces (A,B,C,... ad he colums he umber of order of he correspodg cy o he roue (see fgure belo. For 4 ces, e hus eed a eor h 6 odes, here he odal oupus are bary (0/ ad mplemeed by sgmod acvao fucos h hgh ga. The follog codos ca be saed: Every cy should appear oce; every ro has oly oe. Every umber of order s uque; every colum has oly oe The dsace should be mmzed. These codos ca be formulaed mahemacally (Hopfeld ad Ta 985 as pealy erms ha ogeher ae a form correspodg o he eergy fuco of Eq. (4.3 yeldg he egh mar W. A graphcal erpreao of he aco of he eghs s provded he belo fgure. Ros ad colums have eral hbo, ad oher us are coeced by eghs proporoal o he dsace, d, beee he ces queso. 70 7

6 4. Recurre eors hou egh resrcos 4.. Bacgroud Ths seco preses some feaures of fully or parally recurre eural eors, ad maly eor cofguraos ha arse f layered feedforard eors are augmeed by feedbac coecos. Schemac of he coeco eghs a Hopfeld TSP eor A dscree formulao of he problem cao be used, sce he mehod ges suc local mma; herefore, a couous couerpar of he model s used. A radom roue, a ypcal soluo, ad he (hus far bes o soluo o a 30-cy problem are gve he fgure belo. A fac ha has araced very lle aeo s ha he oupu sgals of feedforard eors represe he aracg fed pos of dyamc sysems. Cosder a sysem descrbed by a se of frs order dffereal equaos (Peda 988 τ + σ ( a + I (4.4 h he fed pos 0 + σ ( a + I or σ ( a + I (4.5 Three roues for a 30-cy TSP Equao (4.5 s realzed by a feedforard eor here s he oupu of ode, a s s pu from oher odes 7 73

7 a (4.6 ac as auoomous sysems; hey ca deec bu o produce me sequeces. σ s he acvao fuco of he ode ad I s a pu from he evrome (cf. Hopfeld s equaos. If he egh mar W { } s loer ragular (.e., > eq. (4.6 he fed pos of he sysem ca be calculaed by passg he pus hrough he eor from he pu o he oupu layer. If he egh mar s o loer ragular,.e., he mar has o-zero elemes o or above he dagoal, oe has o resor o eraos o deerme he fed pos. Such sysems are descrbed by recurre eors. I he specal case h a symmerc egh mar h zeroes o he dagoal, ad 0,.e., he codos for Hopfeld eors ad Bolzma maches (Hopfeld 98, Acley e al. 985, he fed pos are sable. More eresgly, recurre eors ca epress he dyamcs of sysems ha are descrbed by paral dffereal equaos. Thus, s possble o acle problems here dyamc memory s requred o solve he as. Noe ha feedforard eors h apped delay les cao If eq. (4.4 s acled by Euler s mehod ( + ( τ τ ( + σ ( + ( + σ τ τ ( a ( + I ( ( a ( + I ( (4.7 ad he me sep s chose o equal he me cosa,.e., τ, he equao ca be re as ( a ( I ( ( + σ (4.8 + Ths epresso descrbes he oupus of he odes a me dscree recurre eor. The equao shos ha such eors hardly ca be used o solve dffereal equaos properly, bu by sudyg he eors s possble o ga eresg sghs o he behavor of dyamc sysems. 4.. Neor archecures I addo o fully recurre eors, several parally recurre eor cofguraos have bee proposed, maly o acle problems here emporal aspecs are ceral (e.g., voce recogo

8 Jorda (986 roduced a eor h oupus ha are fed bac o fcous pu odes (sae us ha are equpped h self-loops for epoeal eghg of old saes. True pus are roduced hrough pla us, he ma aco of hch s o fre dffere me sequeces. odes ca ac as depede hdde odes ad smulaeously provde he eor h dyamc memory (Nerrad e al , Elma (989, 990 proposed a smlar cofgurao here he oupus of he hdde us are coeced o coe us Sorea e al. (987 preseed a eor here he pu us have selfloops, hch mae possble o represe he hsory of he pus. Laer hs chaper some resuls ll be preseed by a class of parally recurre me-dscree eors (Bulsar ad Saé 99a,b ha arses s a feedforard eor s eeded by me-delayed feedbac coecos from he oupu or hdde layer o fcous pus possble fcous oupu odes a rag of he al saes of he fcous pu odes. The secod em above maes sae esmao possble sce he behavor of hese odes s o eplcly deermed by he rag daa; merely, he 3, Eample of a parally recurre eor Every recurre eor has a correspodg feedforard eor, hch ca be obaed by urollg (ufoldg he eor me (Rumelhar e al Epaso of he above eor yelds he feedforard eor o he e page. The fgure serves o llusrae ha he fcous oupu odes maes possble o carry ou olear rasformaos of prevous pus eve hough he recurre eor lacs a hdde layer. Sce he al saes of he fcous pus are esmaed by he rag algorhm, s possble o errup he ufoldg a a gve me

9 E ( e e T e (4.9 here s he fed po (ha s obaed as me s used, e ge E l e l (4.0 Epaso of he eor of he prevous fgure Trag algorhms Trag of fed pos I s possble o sho (Almeda 987, Peda 987 ha he bacpropagao equaos ca be derved for recurre eors h aracg fed pos. If he quadrac obecve fuco If he egh chages, l, are small oe may appromae o be seady sae. Ulzg eqs. (4.5 ad (4.6 for he las dervave eq. (4.0, l a a σ '( a l σ '( a δ σ '( a l here δ s he Kroecer dela. The equao ca also be re + l + l l l (

10 l or compacly h σ '( a σ '( a δ l l (4. L σ '( a δ l l (4.3 L δ σ '( a (4.4 If he equaos for all are colleced a sysem of equaos, e ge L u l T σ' ( a l (4.5 here he mar L {L }, σ (a s a colum vecor h he elemes σ (a ad u s he u vecor he h coordae dreco. Ths gves l L from hch follos E l e u σ' ( a T T L T l u σ' ( a l (4.6 (4.7 Eq. (4.7 ca, erms of he bac-propagao equaos, be re as ηδ h l l '( T T δ e L u σ a l (4.8 These equaos, hoever, requre a verso of L. A soluo o hs problem s o roduce e varables z obeyg L z e (4.9 he soluo of hch s he seady-sae of he aulary sysem z Lz + e z + + e ( σ '( a z (4.30 These are he equaos for a aalogous error propagao eor, here he eghs,, have bee replaced by σ, ad h he error erm, e, as pu sead of I (cf. eq. (4.4. I he search he fed pos of eq. (4.4 ca, e.g., be deermed frs, e he error erms,, he z from eq. (4.30 ad, fally, he eghs are updaed by eq. ( Trag of raecores Trag mehods for feedforard eors ca be drecly appled o recurre eors afer ufoldg he eors o equvale feedforard oes. Hoever, a drabac of hs approach s he memory requremes gro dramacally h he legh of he me sequece suded. A umber of more geeral mehods for rag of recurre eors have bee preseed; a summary s gve by, e.g., Herz e al. (

11 Pearlmuer (989 derved a rag algorhm for recurre eors couous me (eq. (4.4, ha ca ra boh eghs ad me cosas. The mehod s based o a smlar dea as he approach preseed above for deermg he fed pos,.e., o he roduco ad soluo of a se of aulary varables ( he form of dffereal equaos. Wllams ad Zpser (989 proposed a mehod for real me rag of recurre eors dscree me. Ths ll be preseed e: hle he oupu aes place durg he e me sep, y σ ( a ( p+ (4.33 here O. Le T (p deoe he se of oupus a me p for hch arges are avalable. The error ca o be re as e ( p 0 y f T oherse. (4.34 Cosder a eor h N us ad M eeral pus (ecludg he bas. Le he oupu vecor be deoed by y ad he pu vecor by o dffereae beee pu odes ad rue eor odes (ha carry ou rasformao of he sgals. I ha follos, me ll be deoed by p (superscrped ad parehess. Collecg y (p ad (p a vecor, z (p, ad referrg o he se of dces here z s a oupu u as O ad he se here z s a pu u as I, e ge z y f O f I The oal pu o ode a me p s gve by a l l ( I O z l (4.3 (4.3 Recurre eor for real-me learg The saaeous error summed over all oupus s hus E O ( e hle he oal error summed over me s (

12 E p E (4.36 A search (, e.g., he egave grade dreco a every me o gves a egh correco, egh chage ( p, ha ca be summed over all p o yeld he oal p (4.37 s ( p+ σ '( a ls + δ z l O (0 ha s egraed from s 0. A every me sa p he errors calculaed from eq. (4.34 ad a e value of are sered o (4.39 e are s from eq. (4.39, hch Eercse: Derve a epresso for bacpropagao equaos of subseco... accordace h he η O e s (4.40 I s cusomary o assume ha he odes al saes be depede of he eghs,.e., y (0 0 (4.38 The above equaos hold for O, I ad (I O. The sysem s, as a rule, o appled o mmze he error over all he me seps epereced, eq. (4.36, bu, sead, he search s carred ou based o he saaeous error, eq. (4.35. A dyamc sysem of aulary varables, { s }, s creaed for he purpose o yeld he egh correcos. Noe ha he dervave of he sgmod eq. (4.33 s smply gve by ( p+ ( p+ σ '( a y ( y (4.4 A complcao of he algorhm s ha he raecory ha s produced by he eor o oly depeds o he dyamcs of he eor, bu also o he chages he eghs ha ere carred ou hle he raecory as produced. I order o o mae hs fluece he valdy of he algorhm, he value of he learg rae, η, s geerally chose o be small

13 4..4 Some llusrao eamples or I ha follos some resuls by he recurre eors descrbed o p. 76 appled o es problems ll be gve. I s relavely smple o sho ha cos( α s( α ( p ( p s( α + cos( α ( p ( p (4.44 eors h lear acvao fucos ca descrbe boh VAR ad VARMA models (Saé 996b. Sce hs s raher obvous, ll o be deal h hs coe, bu sead some olear problems ll be dscussed. Pearlmuer (989, 990 raed a fully recurre eor h o oupu, four hdde ad o pu us o follo a crcular pah. If K+ rag observaos, (,, are geeraed h a cosa agle creme, α, e.g., as r cos( p α p 0,..., K r s( p α (4.4 s easly sho ha a fully recurre eor h o lear odes ca solve he problem eacly, sce ( r cos( α ( r cosα ( r s( α ( r sα p p p p + α ( cos α r sα + α ( cos α + r cosα p p s α s α (4.43 Fully recurre -ode eor If he equaos for a recurre eor h o oupus ad o rue pus are re, e ge 3 4 h cos( α 4 s( α ( p 3 ( p ( p 4 ( p 4 (4.45 (

14 The belo fgure shos he performace (doed le of he eor mmcg he arge profle (sold le. Due o he oleary of he model, a small error remas. Ths as ca be eacly solved by he eor o p. 87 h sgmods as acvao fucos. The belo fgure llusraes ho he eor oupus approach he perodc aracor he sared from o dffere al saes. Appromao of a crcular phase porra by a recurre -ode eor Jorda (986, 989 has sho ha a eor ca be augh o reproduce a seres of pos ha correspod o he coordaes of he corers of a square. If he corer coordaes are produced clocse, he as s a subproblem of he crcle case dscussed above, h α π/, yeldg 3 4 ( p 4 ( p 3 (4.47 Sequece produced by a recurre -ode eor ha has bee raed o h he corers of a square. Pearlmuer (989, 990 has also llusraed ha a eor ca be raed o follo a egh-le raecory. The e fgure shos ho a four-ode recurre eor (of he d ouled o p. 76 solves he problem. I ca be sho ha hdde us are requred he eor sce he desred raecory crosses self

15 Egh-le raecory (sold les ad resuls by four-ode recurre eural eors (doed les. The above raecores ca also be lear for cases here ose s prese (Bulsar ad Saé 995. The eors here fd he uderlyg perodc aracors,.e., hey fler ou rreleva formao from he me sequeces. 90