Stable Fixed Point Assignment Problems in Neural Networks

Transcription

1 Poceedigs of the 17th Iteatioal Syposiu o Matheatical heoy of Netwoks ad Systes, Kyoto, Japa, July 24-28, 26 up1.1 Stable Fixed Poit Assiget Pobles i Neual Netwoks Hioshi Iaba ad Yoshiitsu Shoji Abstact he poble of assigig a pescibed set of vectos to locally asyptotically stable fixed poits of a syste aises i ipleetig associative eoy usig a eual etwok. his pape discusses this poble fo eual etwoks of the discete state space type i the faewok of systes ad cotol theoy. Although the so-called othogoal pojectio ethod is easoably poweful ad widely used to costuct such a etwok fo associative eoy, thee is yet aothe ipotat poble to be ivestigated. hat is the poble of how to avoid fictitious fixed poits ceated aoud desied fixed poits o how to elage ad /o adjust the doais of attactio of desied fixed poits. Fistly a geealized othogoal pojectio ethod is studied, ad secoday itoducig a state feedback stuctue i to a eual etwok it is show that it is possible to desig a cotol law such that without chagig the aleady assiged fixed poits each fixed poit achieves a axiu covegece agi to ipove the capability as associative eoy. Fially, to illustate the esults, ueical exaples fo associative eoy ae woked out. Keywods eual etwok, stability, stability agi, associative eoy I I. INRODUCION N the dyaical systes theoy, the stability aalysis of fixed poits is a vey ipotat poble. O the othe had, i the eual etwok theoy, the poble of assigig abitaily give vectos to fixed poits of a eual etwok ay becoe a ai coce. I fact, i ipleetig associative eoy o patte ecogitio usig a dyaical eual etwok, the desied tue ifoatio is eoized as a locally asyptotically stable fixed poit of the etwok. he, the eoized ifoatio ca be ecalled by oly givig a icoplete cotet o a potio of the eoized ifoatio, which is take as a iitial state of the etwok, so that its state coveges asyptotically to the fixed poit that cotais the desied ifoatio. his type of eoy is quite diffeet fo the eoy used i odiay digital coputes i which ifoatio is stoed i a eoy device with a uique addess fo each eoy uit ad the cotet of ifoatio is ecalled by eely specifyig the addess. hee have bee studied the two types of eual etwoks, i.e., the discete state space with discete tie ad Hioshi Iaba is with Depatet of Ifoatio Scieces, okyo Deki Uivesity, Hatoyaa-achi, Hiki-gu, Saitaa , Japa (eail: iaba@cck.dedai.ac.jp). Yoshiitsu Shoji, Ikegai sushiki Copoatio, 4 Kozuka, Fujisawa, Kaagawa , Japa (eail: syouji-y@shoa.ikegai.co.jp) the cotiuous state space with discete o cotiuous tie. I ipleetig associative eoy, the fixed poit assiget poble has attacted a geat deal of attetio, howeve thee is aothe ipotat poble to be studied, that is, the poble of how to elage ad/o adjust the covegece agi of each assiged stable fixed poit i ode to ipove the capability of associative eoy. Howeve, this poble has ot bee thooughly studied. I additio to this, thee is still aothe challegig but exteely difficult poble, that is, the poble of how to hadle ifoatio coupted by stuctual defoatio. Fo the discete type, a vaiety of ethods fo assigig give vectos to locally asyptotically stable fixed poits has bee studied ad futhe the poble of elagig the covegece agi of each fixed poit has bee exaied, see e.g., [1]-[9]. I paticula, the papes [8]-[9] poposed ad studied ethods fo ipovig o axiizig the covegece agis of assiged stable fixed poits, ad this poble has bee faily udestood. Fo the cotiuous type, thee have also appeaed a ube of ivestigatios, see, e.g., [1]-[13] ad the efeeces thee. Howeve, thee ae still a ube of essetial pobles usolved, icludig eve the fixed-poit assiget poble ad the othe pobles etioed above. Fo istace, whe dealig with two-diesioal iages coupted by shape defoatio, the poble becoes exteely difficult [13]. his pape deals with eual etwoks of the discete state space type, oe specifically, those descibed by the McCulloch-Pitts odel [1]. Fo this type a vaiety of ethods fo assigig a set of pescibed vectos as asyptotically stable fixed poits have bee studied [2]-[9]. Aog the, the so-called othogoal pojectio ethod is easoably poweful ad widely used. Howeve, fo this ethod thee is yet aothe ipotat poble to be ivestigated. It is the poble of how to avoid fictitious fixed poits ceated aoud desied fixed poits o how to elage ad /o adjust the doais of attactio of desied fixed poits because the capability of ecallig ifoatio as associative eoy is depedet o the doais of attactio. I paticula, it has bee poited out [8] [9] that, although ay give vectos ca be assiged to stable fixed poits of a give eual etwok by eas of the othogoal pojectio ethod, soe fictitious fixed poits ay be also ceated i viciities of the assiged stable fixed poits. heefoe, this ay cause a fatal poble that ot oly the ecallig pocess leads to expected ifoatio but also the covegece agi becoes uexpectedly sall. his pape fist poposes ad studies a geealized o- 91

2 thogoal pojectio ethod, ad the discusses a ethod fo axiizig the covegece agis of the desied fixed poits by itoducig a state feedback ito the eual etwoks. Fially, to illustate the theoetical esults obtaied, soe ueical exaples of siple eual etwoks ipleetig associative eoy ae peseted. II. PROCEDURE FOR PAPER SUBMISSION Fist, basic defiitios ad otatios used i the sequel ae itoduced. Let B : = { 1,1} ad coside a geeal dyaical syste ove B descibed i the followig fo: xk ( + 1) = f( xk ( )), x() = x. (1) Deote the solutio of (1) by x( kx ; ) o siply x( k ). he, a vecto ξ B is said to be a fixed poit o a equilibiu solutio of (1) if f ( ξ ) = ξ o equivaletly xkξ ( ; ) = ξ fo all k. he doai of attactio of a fixed poit ξ B is defied as D( ξ ): = { x k such that x( k; x ) = ξ}. (2) Next let the Haig distace betwee ξ ad ζ i B be deoted by dh ( ζ, ξ ), ad the δ -ball ceteed at ξ B by N ( ξ ), i.e., δ Nδ ( ξ): = { ζ d H ( ζ, ξ) δ}, δ. he, a fixed poit ξ B is said to be locally asyptotically stable (o siply stable) if D( ξ ) N 1( ξ ). Fially, fo a fixed poitξ B, defie ( ξ ): = ax{ δ N ( ξ) D ( ξ)}, (3) which will be used as a easue of covegece agi of the fixed poit. Next, coside a eual etwok of the McCulloch-Pitts odel [1] defied ove the discete state space B as x( k+ 1) = Sg{ Wx( k) h}, : (4) x() = x whee xk ( ) B is the state, W R the coectio atix, h R the theshold vecto ad Sg( ) desigates the vecto-valued sig fuctio, i.e., whee δ Sg( ξ) : = [sgξ sg ξ ] ξ = [ ξ ξ ] R ad 1 1 1, η > sg η : = 1, η. Futhe, deote the solutio of by x ( kx ; ) o siply. he it is clea that all the popeties of solutio x ( kx ; ), o equivaletly, of dyaical eual etwok ae deteied by the paaete set ( W, h ), ad hece ay desig poble of such a etwok ca be descibed as a poble of choosig a appopiate paaete set ( W, h ). Fially let < ad P : = { ξ,, ξ } B be a set of distict vectos, called a set of pototype vectos o siply a pototype set, which epesets the set of ifoatio to be stoed i a eual etwok. Futhe, itoduce a Lyapuov fuctio fo odel (4) by 1 Ex ( ): = xwx+ xh. (5) 2 he we cite the followig theoes [2], [5], [6], which povide a ethod fo assigig a give set of vectos to its asyptotically stable fixed poits of a eual etwok. HEOREM 1. Coside a dyaical eual etwok of the odel (4). If the coectio atix W R is oegative defiite ove the set { 1,,1}, the fo ay iitial state x () = x (i) x ( k + 1; x) x ( k; x) Ex ( ( k+ 1; x)) < Ex ( ( kx ; )) (ii) the tajectoy x( kx ; ) coveges to a asyptotically stable fixed poit of the odel with fiite steps. HEOREM 2 (he Othogoal Pojectio Method, OPM). Coside a dyaical eual etwok of the odel (4) with the paaete set ( W, h ) give by W : =ΞΞ R, Ξ= : [ ξ ξ ] B (6) h: = [ h1 h], hk 1 whee Ξ B deotes the Mooe-Peose geealized ivese. he the followig stateets hold: (i) Each ξ P is a fixed poit of. (1) (2) ( ) (ii) E( ξ ) = E( ξ ) = = E( ξ ) E( x), x (iii) has o liit cycles. he ae the Othogoal Pojectio Method (OPM) coes fo the fact that the atix W : =ΞΞ epesets the othogoal pojectio opeato fo R oto the subspace spaed by the colu vectos i Ξ (hece it is oegative defiite). 92

3 III. HE GENERALIZED ORHOGONAL PROJECION MEHOD his sectio poposes ad studies a geealized othogoal Pojectio ethod. o begi with, the followig fact is cited. HEOREM 3. Let X R be a atix. he the Mooe-Peose ivese X R satisfies 1 X = li X ( λi + XX ) λ + (7) 1 = li X ( λi + X X). λ + Now the ext theoe holds, but its poof is oitted hee. HEOREM 4. Let P : = { ξ,, ξ } B be a set of give pototype vectos with < ad defie Ξ= : [ ξ ξ ].Futhe, choose a ite- ge d satisfyig 1 d < ( )/2 ad a positive fuctio q:{,, d} (, ). Moeove fo each i = 1,,, defie pi( x): = q( dh( x, ξ )) ad D i : = Nd( ξ ). Fially itoduce a oegative fuctio E : R [, ) by i= 1 x Di EW ( ): = p ( x ) Wx ξ i that ii- he, thee exists a uique atixw R izes EW ( ) ad is give as whee 1 β W = αγξ I +Ξ Ξ Ξ γ 2. (8) d 2s α = qs () C s= s d 4( s s) β = qs () s C = s ( 1) d 4( s s) γ = qs ()1. s C = s ( 1) Now, otice that fo ay a > Sg( Wx h) = Sg{ a( Wx h)} = Sg( awx ah). heefoe, HEOREM 4 ay iply that it is eaigful to set 1 Wε =Ξ εi +Ξ Ξ Ξ, ε > (1) h: = [ h1 h], hk 1 as a paaete set ( W, h) of the odel (4) because it follows fo HEOREM 3 that (9) W ε ε + W : = li =ΞΞ. (11) Based o the above aguets, (1) will be called the Geealized Othogoal Pojectio Method (GOPM) copaig with (6) i HEOREM 2. I ipleetig the odel (4), choose a coectio atixw ε with sufficietly sall ε > so that Sg( Wεξ h) = Sg( ΞΞ ξ h) = ξ, i = 1,,, while soe of fictitious fixed poits ca disappea.. IV. MAXIMIZAION OF CONVERGENCE MARGIN Fist, we coside the odel (4) with a abitay paaete set ( W, h ), ad itoduce to this syste a iput vecto uk ( ) R ad a state feedback with a special fo to defie the McCulloch-Pitts Model with feedback (F) as follows: xk ( + 1) = Sg{ Wxk ( ) h+ uk ( )}, x() = x F: uk ( ) = FSg{ Vxk ( ) g} + θ (12) o equivaletly i the closed loop fo xk ( + 1) = Sg[ Wxk ( ) h+ FSg{ Vxk ( ) g} +θ ] F: x() = x (13) whee F R with, V R, g R ad θ R. ( FV,,, gθ, ) fos a feedback paaete set to be chose so as to ipove the covegece agi without chagig its pe-assiged fixed poits. Let the solutio of F be deoted by xf ( kx ; ) o siply xf ( k ) ad coside a pototype vecto set P : = { ξ,, ξ } B with <. Now, fo each i = 1,,, deote by d i the iiu distace fo ξ P to the othes ( j) ξ fo j i, i.e., ( j) di : = i{ dh( ξ, ξ ) j = 1,, ad j i}. (14) he, the followig theoe holds [8]. HEOREM 5. Let P : = { ξ,, ξ } B be a pototype set of distict vectos, ad defie Ξ= : [ ξ ξ ]. Coside the odel (4) with a abitay paaete set ( W, h) ad the F odel (12) o (13) with the feedback paaete set give by 93

4 : = F : = aξ, V : =Ξ g : = [ g g ], g : = d θ : = a = Ξ[1 1] whee, deotig W ( w ij ) to be ay costat satisfyig 1 i i ( j) ξ a j= 1 = ad ( ) (15) h = h i, a > is chose { j= 1 ij i} a 1 > ax 2 w + h. (116 1 i he, lettig x B be a iitial state ad k be a itege, the tajectoies x ( k) of odel ad x ( k) of F odel satisfy the followig popeties: (i) M ( ) If x ( k j ) N ( ξ ), the F F j= 1 1)/2 x ( k+ 1) = x ( k + 1). F 1)/2 (ii) If x ( k) N ( ξ ) fo soe k ad soe ξ, the x ( k + 1) = ξ. F (iii) Evey ξ P is a fixed poit fo F odel, that is, fo all i ξ = Sg{ Wξ h+ u} u = FSg{ Vξ g} + θ. Futhe, the followig coollay ca be easily veified usig HEOREM 5. COROLLARY 1. Let all the otatios be the sae as those i the pevious HEOREM 5. he, the followig stateets ae satisfied: (i) Fo evey ξ P, D( ξ ) = { ξ k such that x ( k; ξ) N ( ) ad 1)/2 ξ N j= 1 ( d j 1)/2 ( j) x ( l; ξ) ( ξ ), l < k}. (ii) Fo evey ξ P, N( d 1)/2 ( ξ ) D ( ξ ). i (iii) If di 3, the ξ is a locally asyptotically stable fixed poit. (iv) If the odel has o liit cycles, the the F odel has also o liit cycles. Fially, the followig theoe ca be easily veified usig HEOREM 5 ad COROLLARY 1. HEOREM 6. Let all the otatios be the sae as those i the pevious HEOREM 5 except a abitay paaete set ( W, h) is eplaced with the paaete set costucted the Othogoal Pojectio Method as i HEOREM 2. he, if di 3 fo all i = 1,, (i) each assiged fixed poit ξ is locally asyptotically stable () (ii) thee is o othe fixed poit i ( 1)/2 ( i N d ξ ) i ad its covegece agi is give as ( ξ ) = ( d i 1) 2, that is, each fixed poit ξ has the axiu covegece agi. Figue 1 explais the esult of heoe 6 togethe with HEOREM 5, hat is, o the outside of the covegece agi 1)/2 1 the two tajectoies xf ( k) ad x ( kx ; ) ae exactly the sae util they each the covegece agi ad the at the ext oet ξ * xf ( k ) iediately oves to the fixed poit but x ( k) ay tavel oe ad evetually each a fictitious fixed poitξ ceated by odel. Covegece agi = ( d 1)/2 i i ξ = x ( k + 1) M (A assiged fixed poit fo F ad odels) Iitial state x () * ξ = x ( k + 1) (A Fictitious fixed poit ceated by odel) V. NUMERICAL EXALES x ( k 1) = x ( k 1) F Soe ueical exaples wee pefoed fo the geealized othogoal pojectio ethod obtaied i Sectio III, ad the esults showed that soe fictitious fixed poits ae eoved ad the capability fo associative eoy is defiitely ipoved. Howeve, the ipoveet sees uch less tha the ethod poposed i Sectio IV. Due to the shotage of space, all the ueical esults ae oitted, ad oly those esults obtaied usig the ethod developed i Sectio IV ae peseted hee. EXALE 1. Fist, we coside a associative eoy fo eoizig Eglish alphabets A, B,, Z ad blak. x ( k) = x ( k) F Figue 1. he ajectoies of ad F odels 94

5 Each chaacte is divided ito 1 ( = 1 1 ) pixels ad each pixel is epeseted by 1 o -1 accodig to black o ( A) ( B) ( Z) ( ) white, foig the set P = { ξ, ξ,, ξ, ξ } of 27 pototype vectos i B 1. he paaete set ( W, h) of the odel is costucted by the Othogoal Pojectio Method descibed i HEOREM 2 with h =, ad the feedback paaete set ( FV,,, gθ, ) of the F odel is deteied accodig to HEOREM 5. Fist, it is checked that, fo these pototype vectos, all the iiu distaces ae coputed ad tu out to satisfy da, db,, dz, d 4. heefoe, it follows fo HEOREM 6 that all the pototype vectos ae assiged to locally asyptotically stable fixed poits of the F odel. Figue 2 depicts the tajectoies of both ad F odels statig fo the sae iitial state 1 x = x() obtaied fo ξ ( A) by addig 15 % oises. he esult shows that the F odel tajectoy xf ( k ) ( A) coveges to the expected pototype vectoξ, but supisig eough the odel tajectoy x ( kx ; ) * coveges to a fictitious fixed poitξ, which is ceated by the Othogoal Pojectio Method ad diffes oly at ( A) * oe pixel, that is, dh ( ξ, ξ ) = 1. hat is to say that the Othogoal Pojectio Method esues to assig all the pototype vectos to fixed poits but siultaeously ay poduce a fictitious fixed poit just ext to a pototype vecto as deostated i this exaple. EXALE 2. Next, we coside a associative eoy fo eoizig Eglish wods. A wod to be eoized is coposed of at ost five (5) alphabets ad the followig 3 wods ae eoized: CA APPLE MOON MOUSE PEACH EARH IGER LEMON SUN WOLF MELON VENUS WHALE GRAPE MARS JAPAN SHIP ROSE CHINA RAIN LILY INDIA PLANE PANSY SPAIN BIKE ULIP IALY BOA OLIVE I this tie, each chaacte is divided ito ( = 576) pixels, so that each chaacte is epeseted by a vecto i 576 B. heefoe each wod cosistig at ost 5 chaactes 288 is epeseted by a vecto i B ad the 3 pototype (CA) (OLIVE) vectos ae deoted as ξ,, ξ i B 288. Fo the associative eoy fo these wods, we also costuct two types of eual etwoks, usig odel odel F odel odel F odel x ( k) x ( k ) M Iitial state 1 x = B Coveget to A fixed poit ξ ( ) Coveget to a fictitious * ( A) fixed poitξ ξ Figue 2. ajectoies of ad F odels Figue 3. Associative eoies fo Eglish wods usig odel ad F odel 95

6 ad F odel, espectively. Each type etwok is coposed of five (F) odels, each of which is costucted as i EXALE 1 to eoize ξ, ξ,, ( A) ( B) ( Z ) ( ) ξ, ξ i B 576. he these five etwoks ae coected each othe, though a ewly itoduced hidde etwok to each idividual etwok, i such a way that each wod is assiged to a asyptotically stable fixed poit i the 288 space B of the total etwok but the pe-assiged fixed poits i each idividual etwok fo eoizig ( A) ( B) ( Z) ( ) ξ, ξ,, ξ, ξ ae uchaged. A detailed desciptio fo this costuctio is give i [7]. Figue 3 depicts the copute siulatio esults of the two associative eoies fo Eglish wods usig odel ad F odel. As see fo the figue, the iitial state is a vey osy ifoatio which is geeated fo the pototype vecto ξ (APPLE) of APPLE by addig 45 % oises. It is see that i the associative eoy usig the odel a fictitious fixed poit (i.e., a eaigless wod) is ceated ea the wod APPLE ad the coect ifoatio caot be ecalled, while i the associative eoy usig the F odel the coect ifoatio APPLE is ecalled. VI. CONCLUDING REMARKS his pape dealt with the poble of assigig a give set of poits to the locally asyptotically stable fixed poits i a eual etwok. I paticula, a geealized othogoal pojectio ethod was poposed. Futhe fo the viewpoit of systes ad cotol theoy a ethod fo axiizig the covegece agi of each assiged fixed poit was studied, which is vital fo ipleetig a associative eoy by a eual etwok to ipove the capability of associative eoy. I fact itoducig a state feedback stuctue ito a eual etwok it was show that it is possible to desig a state feedback law such that the covegece agi of each fixed poit i its closed loop syste is axiized without chagig all the pe-assiged asyptotically stable fixed poits. Fially, to show the effectiveess of the esult obtaied, soe ueical exaples fo associative eoy wee peseted. Moe sophisticated eual etwoks wee cosideed by itoducig liit cycles to eoize ifoatio [6]. Futhe, ecetly eual etwoks of cotiuous state spaces with cotiuous tie have bee cosideed i the faewok of the systes ad cotol theoy [13]. Fo these etwoks, the sae poble studied i this pape should be ivestigated. he authos thak the studets, I. Saito,. Sakuai ad K. Nakajia, fo pefoig ueical siulatios. REFERENCES [1] W. S. McCulloch ad W. Pitts, A logical calculus of the idea iaet i evous activity, Bulleti of Matheatical Biophysics, vol. 5, pp , [2] L. Pesoaz, I. Guyou ad G. Deyfu, Collective coputatioal popeties of eual etwoks: New leaig echais, Physical Reviews A, vol. 34, pp , [3]. Kohoe, Self-Ogaizatio ad Associative Meoy, Spige-Velag, 1988 (2 d Editio). [4] Yves Kap ad Mati Hasle, Recusive Neual Netwoks fo Associative Meoy, Wiley, 199. [5] A. Michel ad J. A. Faell, Associative eoies via atificial eual etwoks, IEEE Cotol Magazie, pp. 6-17, Apil 199. [6] K. Ishii ad H. Iaba, Associative eoies usig dyaical eual etwoks with liit cycles, Poceedigs of the ISCIE Iteatioal Syposiu o Stochastic Systes heoy ad Its Applicatios, pp , Novebe 1993, Osaka, Japa. [7] H. Iaba ad H. Iaba, Multi-odule eual etwoks ad thei applicatios to associative eoies, Poceedigs of IEEE Iteatioal Cofeece o Neual Netwoks ad Sigal pocessig, vol. 1, pp , 1995, Najig, Chia. [8] Y. Shoji ad H. Iaba, A odule eual etwok ad its basic behavios, Poceedigs of IEEE Iteatioal Cofeece o Neual Netwoks, vol. 2, pp , Jue 1997, Housto, exas, USA. [9] H. Iaba ad Y. Shoji, A geealized othogoal pojectio ethod fo desigig dyaical eual etwoks, Abstacts of the 3 th IS- CIE Syposiu o Stochastic Systes heoy ad Its Applicatios, pp. 29-3, Novebe 1998, Kyoto, Japa. [1] A. Fuchs ad H. Hake, Patte ecogitio ad associative eoy as dyaical pocesses i a syegetic syste, I, aslatioal ivaiace, selective attetio, ad decopositio of scees, Biological Cybeetics, vol. 6, pp , [11] A. Fuchs ad H. Hake, Patte ecogitio ad associative eoy as dyaical pocesses i a syegetic syste, II, decopositio of coplex scees, siultaeous with espect to taslatio, otatio, ad scalig with A, Biological Cybeetics, vol. 6, pp , [12] H. Hake, Syegetic Coputes ad Cogitio, Spige-Velag, Beli, 24. [13] H. Iaba ad. Ooki, Locally Asyptotically Stable Fixed Poit Assiget Pobles i Neual Netwoks ad Applicatio to Associative Meoy, to appea i Poceedigs of 26 IEEE Wold Cogess o Coputatioal Itelligece, July 26, Vacouve, Caada. ACKNOWLEDGMEN his wok was suppoted i pat by the Japaese Miisty of Educatio, Sciece, Spots ad Cultue ude both the Gat-Aid of Geeal Scietific Reseach C ad the 21st Cetuy Cete of Excellece (COE) Poga. 96