Advances in Theory of Neural Network and Its Application

Transcription

1 Joural of Behavoral a Bra Scece Publshe Ole May 06 ScRes Avaces Theory of Neural Network a Its Applcato Baha Mashoo Greg Mllbak 50 a Playa Street 304 Sa Fracsco USA At Praxs Techcal Group Ic Naao Caaa Receve March 06; accepte May 06; publshe 4 May 06 Copyrght 06 by authors a Scetfc Research Publshg Ic Ths work s lcese uer the Creatve Coos Attrbuto Iteratoal cese (CC BY Abstract I ths artcle we trouce a large class of optzato probles that ca be approxate by eural etworks Furtherore for soe large category of optzato probles the acto of the correspog eural etwork wll be reuce to lear or quaratc prograg therefore the global optu coul be obtae eately Keywors Neural Network Optzato Hopfel Neural Network ear Prograg Cohe a Grossberg Neural Network Itroucto May probles the ustry volve optzato of certa coplcate fucto of several varables Furtherore there are usually set of costras to be satsfe The coplexty of the fucto a the gve costras ake t alost possble to use eterstc ethos to solve the gve optzato proble Most ofte we have to approxate the solutos The approxatg ethos are usually very verse a partcular for each case Recet avaces theory of eural etwork are provg us wth copletely ew approach Ths approach s ore coprehesve a ca be apple to we rage of probles at the sae te I the prelary secto we are gog to trouce the eural etwork ethos that are base o the works of D Hopfel Cohe a Grossberg Oe ca see these results at (secto-4 [] a (secto-4 [] We are gog to use the geeralze verso of the above ethos to f the optu pots for soe gve probles The results ths artcle are base o our coo work wth Greg Mllbak of praxs group May of our proucts use eural etwork of soe sort Our expereces show that by choosg approprate tal ata a weghts we are able to approxate the stablty pots very fast a effcetly I secto- a secto-3 we trouce the How to cte ths paper: Mashoo B a Mllbak G (06 Avaces Theory of Neural Network a Its Applcato Joural of Behavoral a Bra Scece

2 B Mashoo G Mllbak exteso of Cohe a Grossberg theore to larger class of yac systes For the goo referece to lear prograg see [3] wrtte by S Gass The appearace of ew geerato of super coputers wll gve eural etwork uch ore vtal role the ustry ache tellget a robotcs O the Structure a Applcatot of Neural Networks Neural etworks are base o assocatve eory We gve a cotet to eural etwork a we get a aress or etfcato back Most of the classc eural etworks have put oes a output oes I other wors every eural etworks s assocate wth two tegers a Where the puts are vectors R a outputs are vectors R eural etworks ca also cosst of eterstc process lke lear prograg They ca cosst of coplcate cobato of other eural etworks There are two k of eural etworks Neural etworks wth learg abltes a eural etworks wthout learg abltes The splest eural etworks wth learg abltes are perceptros A gve perceptro wth put vectors R a output vectors R s assocate wth treshhol vector ( θ θ θ a atrx ( w The atrx W s calle atrx of syaptcal values It plays a portat role as we wll see The relato betwee output vector O ( o o o a put vector vector S ( s s s s gve by o ( k g w s θ k k wth g a logstc fucto usually gve as g( x tah ( β x wth > β > 0 Ths eural etwork s trae usg eough uber of correspog patters utl syaptcal values stablze The the perceptro s able to etfy the ukow patters ter of the patters that have bee use to tra the eural etwork For ore etals about ths subect see for exaple (Secto-5 [] The eural etwork calle back propagato s a extee verso of sple perceptro It has slar structure as sple perceptro But t has oe or ore layers of euros calle he layers It has very powerful ablty to recogze ukow patters a has ore learg capactes The oly proble wth ths eural etwork s that the syaptcal values o ot always coverge There are ore avace versos of back propagato eural etwork calle recurret eural etwork a teporal eural etwork They have ore verse archtect a ca perfor te seres gaes forecastg a travellg salesa proble For ore forato o ths topc see (secto-6 [] Neural etworks wthout learg echas are ofte use for optzatos The results of DHopfel Cohe a Grossberg see (secto-4 [] a (secto-4 [] o specal category of yacal systes prove us wth eural etworks that ca solve optzato probles The put a out put to ths eural etworks are vectors R for soe teger The put vector wll be chose raoly The acto of eural etwork o soe vector X R cosst of uctve applcatos of soe fucto f : R R whch prove us wth fte sequece X X X where X f ( X f ( X A output (f exst wll be the lt of of the above sequece of vectors These eural etworks are resulte fro gtzg the correspog fferetal equato a as t s has bee prove that the ltg pot of the above sequece of vector coce wth the ltg pot of the traectory passg by X Recet avaces theory of eural etworks prove us wth robots a coprehesve approach that ca be apple to we rage of probles At ths e we ca cate soe of the a ffereces betwee eural etwork a covetoal algorth The back propagato eural etworks gve the put wll prove us the out put o te But the covetoal algorth has to o the sae ob over a over aga O the other ha realty the algorths rvg the eural etworks are qute assy a are ever bug free Ths eas that the syste ca crash oce gve a ew ata Hece the covetoal ethos wll usually prouce ore precse outputs because they repeat the sae process o the ew ata Aother efect of the eural etworks s the fact that they are base o graet esce etho but ths etho s slow at the te a ofte coverge to the wrog vector Recetly other etho calle Kala flter (see (secto-59 [] whch s ore relable a faster bee suggeste to replace the graet esce etho 3 O the Nature of Dyac Systes Iuce fro ergy Fuctos I orer to solve optzato probles usg eural etwork achery we frst costruct a correspog eergy fucto such that the optu of wll coce wth the optu pot for the optzato proble Next the eergy fucto that s usually postve wll uce the yac syste The traectores of wll coverge hyperbolcally to local optus of our optzato proble Fally we costruct the eural etwork NN whch s the gtze verso of where epeg o tal pots t wll coverge to soe local optu As we cate secto- certa category of yac syste whch s 0

3 B Mashoo G Mllbak calle Hopfel a ts geeralzato whch s calle Cohe a Grossberg yac syste wll uce a syste of eural etworks that are able to solve soe well kow NP probles More recetly the ore avace yac systes base o geeralzato of the above yac systes bee use [4] to to solve or prove ay terestg probles clug four color theore I the followg sequece of leas a theores we are gog to show that f the yac syste satsfes certa coutg coto the t ca be uce fro a eergy fucto whch s ot usually postve a all ts traectores coverge hyperbolcally to a correspog attractor pots Furtherore the attractor correspog to the global optu s locate o the o trval traectory Note that the eergy fucto that s uce fro optzato scearo s always postve a the correspog yac syste G s a coutg yac syste Suppose we are gve yac syste as the followg x t Q x x x x x R Also the above equato ca be expresse as ( ( ( x t Q x Defto We say that the above syste satsfes the coutg coto f for each two ces k k we have Q x Q x Ths s very slar to the propertes of coutg squares the VJoes ex theory [5] The avatage of coutg syste as we wll show later s that each traectory Xx 0 ( t passg through a tal pot x0 R wll coverge to the crtcal pot x a x s asyptotcally stable I partcular ote that f the yac syste s uce fro a eergy fucto the the uce eural etwork N s robot a stable I the sese that begg fro oe pot x0 R the eural etwork wll asyptotcally wll coverge to a crtcal pot Ths property plus soe other techques ake t possble to f the optu value of The followg lea wll lea us to the above coclusos ea Suppose the yac syste has a coutg property The there exsts a fucto ( x actg o R such that for every teger we have ( x x t Q x Furtherore for every traectory X x 0 0 t a t 0 oly o the correspog crtcal pot t 0 Q x Proof et us pck up a teger Next efe Q( xx The for ay teger we x Q x x Q x x Q x t x o whch ( have ( ( fally we have ( ( t x x t x t 0 A the equalty hols e t 0 oly o the crtcal pot x o whch t 0 QD But the proble s that the uce ea s ot always a postve fucto I the case that s a postve fucto we have the followg lea ea 3 Followg the otato as the above suppose s a coutg syste a that 0 s aalytcal fucto Next let x R be the crtcal pot for the syste whch s o o trval traectory wth ( x 0 The x s asyptotcally stable Proof Followg the efto of apoov fucto a usg ea the fact that ( x 0 ples that regarg to the traectory passg through x t s asyptotcally stable QD There are soe cases that we ca choose to be a postve fucto as we wll show the followg lea ea 4 Keepg the sae otatos as the above suppose that there exsts a uber α R such that α ( x for all x R The there exsts a postve eergy fucto F for the yac syste Proof et us efe F actg o R by F( x ( x α The F s a postve fucto Furtherore x x t Q a over the traectores t 0 t s equal to zero oly over the attractg pots Ths wll coplete the proof of the lea QD

4 B Mashoo G Mllbak ea 5 et be a coutg yac syste et 0 be the uce eergy fucto The f x R s a pot o a o trval traectory o whch acheves a optu α the the traectory passg through x wll coverge asyptotcally to x Proof Set F( x ( x α the usg ea 4 s a eergy fucto for a sce F ( x 0 we are oe QD Suppose s a coutg yac syste et 0 be a uce eergy fucto The a goal of the correspog eural etwork N s to reach a pot x R at whch wll get ts optu value I ea 5 we prove that ay o trval traectory passg through x wll coverges to x asyptotcally I the followg we show that geeral the above property hols for ay attractg pot of coutg yac systes ea 6 Suppose s a coutg yac syste a x a attractve pot The the traectory passg through x wll coverge asyptotcally to x Proof et δ ( x Next efe the fucto F actg o R by F ( ( ( x x δ I orer to coplete the proof we have to show that wth regar to the traectory X( tx passg through x F ( x s a apuov fucto For ths t s eough to show that F t < 0 But ( ( ( δ ( ( δ Fally the fact that ( X( tx δ F t x t X tx t > 0 ples that F t < 0 a ths coplete the proof QD Suppose s a coutg yac syste a x s a pot o soe o trval traectory at whch reaches ts fu We wat to f a cotos that guaratees the exstece of a o trval traectory passg through x Defto 6 Keepg the sae otato as the above for a coutg yac syste we call 0 caocal f for each crtcal pot x wth ( x 0 we have l x x ( x Q < Before proceeg to the ext theore let us set the followg otatos et C ( δ x R x δ a ( Oδ x R x δ Furtherore for x R let x δ Cδ be the frst pot at whch the traectory X( tx wll tersect C δ ea 7 Followg the above otatos suppose wthout loss of geeralty x 0 a that there s o traectory passg through a pot x x a covergg to x The there exsts ρ > 0 a a sequeces ( 0 a ( x O such that for each N X( tx wll tersect C ρ frst te at the pot x ρ Cρ Proof Otherwse for each ρ > 0 there exsts a postve uber > ρ 0 such that for ay postve uber < ρ a a pot x O the traectory X ( tx wll le O ρ Ths ples the exstece of the sequece ( y covergg to x such that for each N ( y x 0 N hece assug that s aalytc ths ples that ( x 0 whch s a cotracto QD Theore 8 Keepg the sae otato as the above suppose we have a coutg syste wth the uce eergy fucto beg caocal Suppose there exsts a pot x R wth ( 0 x The there exsts a o trval traectory X ( t covergg to x Proof Suppose there s o o trval traectory passg through x Next for ay > 0 let us choose x R wth x x < Furtherore let X( tx be the traectory through the pot x Now let ρ a be as cate at ea 7 Next for N X tx passg through et coser ( be the frst pot at whch X( tx tersect C ρ We have ( X( tx tσ Q ( X( tx G( X( tx Now coser the traectory X( tx the te at whch the traectory X( tx arrves at x ρ Ths ples t t ρ ( ( ( ( 0 ( ( 0 ( ( ρ ρ t x ρ t ρ X t x X x x x G X t x t x et us eote by

5 B Mashoo G Mllbak Thus applyg ea value theore ples ( ( ( ( ρ τ ρ wth ( 0 t ρ x x G X x t τ ( To coplete the proof of the Theore 8 we ee the followg lea ea 9 Keepg the sae otatos as the above the there exsts a ω > 0 such that for N large eough there exsts a postve uber α > 0 wth the property that for every ( large eough 0 < t ω < α Proof Otherwse there exsts a sequece of ubers ( ω k k such that ωk 0 as k where for each k N we have t as Thus coserg the quato ( a the fact That for the ω k ces X tx < ω t 0 we ca wrte large eough we have ( ( t ω t ( ( ( ( ω X t ω x G X x ε τ ( 0 t ω τ Ths usg the fact that s caocal wll lea to cotracto QD To coplete the proof of theore 8 ote that for every pot x the pots locate o the traectory ( Φ x t of x a t The fucto Φ ( x t acts o the 0 α whch s a copact set Hece there exsts a uber egth such that X t ca be expresse as a cotuous fucto ( set [ ] ω C ω ( ( t t ω egth X t x < egth for all N Next for each k N t 0 we ve ( X tx to k part k each of legth equal to ω et us efe the followg pots p k x p p p k k k be the k pots correspog to the above parttos Furtherore coser the followg coutable set of pots Q p 0 k k > k > Furtherore for each trple ( k let us efe a pot p k to ( k be a lt pot of the set ( pk > Fally coser the followg set ( k S p k > k k The the set S the closure of S R s a o trval traectory passg through x But ths s a cotracto to our assupto QD ea 0 Keepg the sae otatos as the above suppose for a gve coutg syste the uce eergy fucto s postve fucto The s caocal Proof If s ot caocal the there exst a creasg sequeces of postve ubers ( α as a ( 0 as such that for each N there exsts a pot u R at the stace of less tha fro u wth ( u Q ( k k u α Next we ca assue that there exsts a le U( t R coectg sequece of pots u to the ltg U u Q u u Q u u pot u such that ( But ( ( thus usg Hoptal lea χ l ( u Q l ( ( u t t t ( Q t et us set g( u Q The by the above lt ( u Q lt g( u g ( u lt g( u g ( u 0 0 the usg Hoptal lea we have χ l t g ( u ( g( u + + suppose usg ucto that χ l ( 0 0 t ( g u t ( g( u t χ lt ( g( u t ( g( u t Now f for each N lt ( g( u t 0 get that ( lt ( g( u t 0 but lt ( g( u t 0 χ lt ( u Q 0 Next f cotue ths process the we get by Hoptal lea the usg the fact that ( ( g t whch s a cotracto Hece there exsts a teger N such that g u t s aalytc fucto we Therefore the above arguets ply that a ths s a cotracto to the assupto QD 3

6 B Mashoo G Mllbak As a result of the ea 0 we get that f 0 the syste s always caocal hece we have the the followg corollary Corollary Keepg the sae otatos as the above The results of Theore 8 hols as log as 0 At ths pot we have to eto that o trval traectores wll supply us wth uch ore chace of httg the global optu oce we perfor a rao search to locate t For soe yac systes whch s expresse the usual for x t Q( x the coutg coto oes ot hol For exaple coser the Hopfel eural etwork a ts correspog yac syste u t u+ w g( u where g( x tah ( x both se of the 'th equato the above by g( u u g ( u followg as ( u t g ( u g ( u u w g ( u It s clear that the above syste oes ot posses coutg propertes et us ultply to get the yac syste gve the + therefore we get et us eote u g( u ( ( ( ( ( ( g u t g u u + w g u ( a Q u t the the yac syste ca be expresse as u t Q It s clear that f U ( t s a traectory of the syste a prove ( 0 the U ( g u t s a traectory of syste too Now the coutg property hols for the rght se of quato ( hece usg the sae techques as before we ca costruct the eergy fucto ( u for the syste ( such that ( u u Q furtherore we have ( ( ( ( ( u t t g u u t Ths ples that t < 0 except o the attractve pots Now by the results of the ea 3 ths ples U tu the coverget asyptotcally to the correspog attractve pot u that for ay traectory ( 0 As we etoe before ore geeralze verso of Hopfel yac syste whch s calle Cohe a Grossberg yac syste s gve as the followg where the set of coeffcets c ( a ( u 0 g ( u 0 ( ( ( ( u t a u b u c g u (3 wll satsfy c c Furtherore kewse the syste ( syste (3 s ot a coutg syste But f we ultply both se of the th equato syste (3 by g ( u the we get a yac syste where ts rght se s coutg Hece each of the traectores coverge asyptotcally to correspog attractg pot 4 Reucto of Certa Optzato Probles to ear or Quaratc Prograg I solvg optzato probles usg eural etwork we frst for a eergy fucto ( u correspog to the optzato proble Next the above eergy wll uces the yac syste that ts traectores coverge to local optu solutos for the optzato proble u actg o R the uce yac syste s gve the followg Gve the eergy fucto ( u t u Q As we showe s a coutg syste As a exaple coser the travellg salesa proble As t has bee expresse secto 4 page 77 of [] the eergy fucto s expresse as the followg k k ( k ( k k γ k + ( k ( k 05 u u + u + 05 u + u where we are represet the pots to be vste by travellg salesa as ( p a the stace of 4

7 B Mashoo G Mllbak pot p to the pot p by Sce solvg optzato probles usg eural etwork we are tereste varables that are of 0 or ature therefore let us efe ew set of varables by ( k vk ( tah ( λ0 uk v k + Thus the yac syste correspog to ca be wrtte as the followg l ( γ l ( + γ γ γ u v t u + u + u + u k k l l l lk lk l lk As we prove the pot at whch reaches ts optu s a ltg pot u of a traectory belogg to the yac syste as gve the above Next as we ha l vk t uk λ0 tah ( λ0 uk yk t l yl l + yl + + yl l + yl γ ( l ( But uk t ( λ0 ( tah ( λ0 vk vk t hece we get the followg yac syste l u k t l γ ( ul l + ulk + γ + ulk l + ulk γ γ ( l Thus as we showe the syste s a coutg syste a ts crtcal pots coces wth the crtcal pots of the syste Now coser the followg set of varables y y u q where we choose the set q to satsfy ( + γ γ q q q q s l l l l l + ls + ls l + ls Hece we wll have the followg set of equatos l ( l l ls + γ ls ls l γ l q + q + q + q s Replacg u k by y k equatos of syste ples l ( l ( + yk t l yl l + yls + yls l + yls γ But y k 0 as t whch ples that uk qk as t hece ths ples that the above k equatos together wth the optalty of the expresso F uk ( u k k + + u k s a syste of quaratc lear prograg that wll gve us the optu value uch faster that usual eural etwork I fact ths etho ca be apple to ay types of optzato probles whch guarates fast coverget to esre crtcal pot et us coser the Four color Theore The slar scearo to Four color Theore s to coser the have two perpecular axs X a Y a sets of pots P ( x o X axs a the set of pots P ( y o the Y axs wth x ( 0 a y ( 0 et us set the followg four pots O ( 0 0 A ( 0 B ( 0 C ( We wat to coect the pots of the sets P to P such that p the coectg le wll tersect oly out se the square OACB Now suppose we take a pot ( 0 a q ( 0 p ( a q ( 0 q wll gve us the 0 Next coectg the pots p a le Y Coectg the pots p a q wll gve us the le Y We wat to choose the les Y a Y such that the pot z whch s the tersecto of Y a Y wll stay outse the square OABC Suppose z z z The the above coto s equvalet to the fact that ether z or z 0 But ( ( ( the ether ( the ether ( z Now the above coto s equvalet to the followg set of equaltes Case- If Case- If or or syetrcally or Next let us take the set of followg varables ( w 0 k a ( 0 k ( k Furtherore coser the followg equaltes w wth ( a 5

8 B Mashoo G Mllbak ( w 0 k 0 k for all ( ( w 0 k 0 k for all ( Furtherore let us set the followg eergy fucto to be optze w0 k + w 0 k + λ ( w0 ( k 0 k k k k w + k cotos of Case- a Case- over the ces Now the above syste s equvalet to f the optu soluto for colorg So as before assug W ( tah ( λ0v + we have w0 k t( v0 k a w t v also we ust eet the ( k 0 k 0 t v w c λ λ wth c k Ths ples ( 0 k ( 0 k Next let us efe w0 y + q k k k O the other ha t s easy to show that ( ( 0 k t v k λ w 0 k λ Next let us efe w0 k zk + pk Fally f the followg equaltes hols q k c λ p k k λ wth where the ces satsfy the cotos of Case- a Case- the as t tes to fty we have w0 k qk a w0 k pk At ths pot usg the above arguets t s eough to f Q a P satsfyg the above equaltes a wll optze the followg expresso ( a k q + p k k k k k Therefore the above equatos together wth optzato expresso wll for a syste of lear prograg that wll coverge to the optal soluto at o te 5 Cocluso I ths artcle we trouce the ethos of approxatg the soluto to optzato probles usg eural etworks achery I partcular we prove that for certa large category of optzato probles the applcato of eural etwork ethos guarates that the above probles wll be reuce to lear or quaratc prograg Ths wll gve us very portat cocluso because the soluto of the optzato probles these categores ca be reache eately Refereces [] Hetz J Krough A a Paler R (99 Itroucto to the Theory of Neural Coputato Aso Wesley Copay Bosto [] Hayk S (999 Neural Networks: A Coprehesve Fouato to Pretce Hall Ic Upper Sale Rver [3] Gass S (958 ear Prograg McGraw Hll New York [4] Yoshyasu T (99 Neural a Parallel Processg The Kluwer Iteratoal Seres geerg a Coputer Scece: SeS064 [5] Joes V a Suer VS (997 Itroucto to Subfactors Cabrge Uversty Press Cabrge 6