Alternative Dynamic Network Structures for Non-linear System Modelling

Transcription

1 Alteratve Dyamc Networ Structures for No-lear System Modellg K. P. Dmopoulos ad C. Kambhampat CITY Lberal Studes, Thessalo, Greece, Afflated Isttuto of the Uversty of Sheffeld, U.K., The Uversty of Hull, Hull, Abstract Hopfeld Neural Networs have bee used as uversal detfers of o-lear systems, because of ther heret dyamc propertes. However the desg decso of the umber of euros the Hopfeld etwor s ot easy to mae, order for the etwor model to have the ecessary complexty, extra euros are requred. Ths poses a problem sce the role of the states that these euros represet s ot clear. Addg a hdde layer the Hopfeld etwor model creases the complexty of the model wthout posg the extra states problem. Alteratvely breag the problem dow by havg dfferet tercoected Hopfeld etwors modelg each state, also crease the complexty of the problem. A comparso betwee the three approaches (tradtoal Hopfeld, Hopfeld wth a hdde layer, ad multple tercoected Hopfeld etwors) dcates equvalece betwee the three structures, but wth the alteratve cases havg creased coectvty the feedbac matrx, ad lmted coectvty the weght matrces.. Itroducto Hopfeld Neural Networs (HNNs) have bee very popular lterature as uversal o-lear models. The dyamc propertes of the HNNs maes them deal for capturg the complete o-lear dyamcs of ay uow o-lear process, as log as accurate estmates of the process states exst. However, very ofte whe HNNs represet the dyamcs of a process, they ofte have more teral states tha the process they are represetg. The extra states are hard to talse, as they do ot represet ay of the process real states. Nevertheless, the extra complexty offered by the extra euros s sometmes ecessary for the HNN to approxmate the process correctly. I ths paper we propose ways to crease the complexty of the eural etwor model, wth out creasg the exteral states, thus allevatg the problem.. HNNs as o-lear systems Hopfeld Neural Networs are eural etwors that exhbt dyamc propertes due to teral/exteral feedbac. [-5]. These propertes allow for the teral dyamcs of uow processs to be detfed cotrast to the usual put-output detfcato that Feed Forward Neural Networs are lmted to. Hopfeld Networs have of the form: = Bx+ Wσ ( x) +Γu y = x or a more precsely: = β x + wσ ( x ) + γ u = y = x () () where x R s a vector wth the states of the etwor, x ɺ R s the frst dervatve of x, B R s the feedbac matrx (usually a dagoal matrx), W R s the weght matrx, Γ R s the put matrx, ad σ ( x) s a sgmodal fucto le tah( x ). Wth out loss of geeralty, from ow o t wll be assumed that h( x) = x. The equatos that characterse a Hopfeld etwor are those of a o-lear cotrol affe system: = f ( x) + g( x) u y= h( x) (3)

2 Therefore, Hopfeld etwors are a class of olear cotrol affe systems, ad thus all aalyss of such systems ca be also appled to them... Relatve Order of HNN Relatve order s a varat property of physcal systems. It defes the umber of tmes the put has to be tegrated order to affect the output. Accordg to Isdor [6]: The system (3) s sad to have relatve order r at a pot x 0 f L L h( x ) = 0 for all x a eghbourhood of x 0 g f ad all <r- r 0 L L h( x ) 0. g f Where L f h(x) s the Le dervatve of h(x) the drecto of f(x) defed as: h Lf h( x) = f ( x) x Sce the Hopfeld etwor s a cotrol affe system of the form (3), the cocept of the relatve order, s also be applcable t. By applyg the relatve order defto to the HNN, Dmopoulos ad Kambhampat have proved that the structure of the HNN ca determe ts relatve order, ad vce versa [4], ad that the relatve order of the HNN wll be upper lmted by the umber of states the eural etwor has. Specfcally: Relatve Order Proposto: For the Hopfeld etwor (); If γ = 0, =, ( ), ad γ 0, the relatve order of the system s r [, ], f the followg codtos are satsfed: w, = 0 w,3, w,( ), w, = 0 w( r 3).( r ), w( r 3),( ), w( r ), = 0 w( r ), r, w( r ),( ), w( r ), 0 If γ 0, the the relatve order r s. If γ = 0, =,, m, γ 0, = ( m+ ),, the relatve order of the system s r, wth r [, m+ ], f the followg codtos are satsfed: w,( m+ ),, w, = 0 w,3,, w,( ), w,( m+ ),, w, = 0 w,4,, w,( ), w3,( m+ ),, w3, = 0 w( r 3),( r ),, w( r 3),( ), w( r ),( m+ ),, w( r ), = 0 w( r ), r,, w( r ),( ), w( r ),( m+ ) 0 If γ r 0, the the relatve order r wll be. If γ = 0, [, m) ( m, ], γ m 0, the the relatve order of the system s r, where r [, m) ( m, ]. Proofs of the above propostos ca be foud [4]... Zero Dyamcs of HNN Zero dyamcs are aother varat property of physcal systems. I may staces they play a role exactly smlar to that of the zeros of the trasfer fucto a lear system [6]. Zero dyamcs are the dfferetal equatos that descrbe the teral behavour of the o-lear system whe the system s output s forced to be zero. Let us assume a Hopfeld etwor of the form of () ad further assume that ths etwor has a relatve order r< ad has the approprate structure as defed the relatve order proposto. The because of the structure of the weght matrces, the Hopfeld wll loo le: + = β x + wσ ( x ) =,, r = = β x + wσ ( x ) + γ u = r+,, = y= x Sce the output of the etwor s the frst state, ad sce each of the frst r states s a fucto of oly the frst r states, the the frst r states wll be zero, ad the zero dyamcs of the Hopfeld wll be gve by: γ = β x + w wr σ ( x ) = r+, = r+ γ r

3 I order for the verse of the model to be stable, these zero dyamcs have to be stable. Stablty of these ca be the same way as stablty of a Hopfeld ca be determed; by learsg ad examg the Egevalues. The learsed zero dyamcs are gve by: wth B=0 ad A gve by: zɺ = Az+ Bu h β A= h, β h h h β r+, r+ r+ r+,,( r+ ), γ h = w wr σ ( x ), = r+, γ r Whe recurret etwors represet the dyamcs of a system, they ofte have more teral states tha the system they are represetg. Suppose that ths s the case wth a etwor wth states that s modellg a o-lear process of m states where r<m< (r beg the relatve order of both process ad model). The the o-lear process wll have m-r zero dyamcs, whle the etwor model wll have -r zero dyamcs. Therefore the extra states of the etwor model represet extra zero dyamcs. Thus t s possble for the etwor to have more zero dyamcs tha the process t models. Nevertheless, trag a state etwor to detfy a m state process, where m<, has the same effect as wth trag the same etwor to detfy a state process usg lmted formato (say the frst m states) about the o-lear process. Sce we have o formato about the last -m states, t s dffcult ot oly to decde the role they play but also how to talse them. 3. HNN wth a hdde layer Extra euros are eeded order to crease the complexty of the Hopfeld etwor, wthout creasg the umber of states. Ths ca be acheved by addg a hdde layer sde the Hopfeld etwor. I ths case, the etwor s composed out of three layers. The put ad the output layer are composed of euros,.e. the umber of states the etwor s tryg to model. The hdde layer s composed of m euros. It s these euros that wll provde the extra complexty to the model. Ths archtecture was proposed [7] ad s descrbed by the followg equatos: = Ax+ Cσ ( Dx+ E) + Fu y = x (4) where A R s the dagoal feedbac matrx, much m the same wth the feedbac matrx (), C R m m ad D R are the weght matrces, E R s a bas matrx, F R s the put weght matrx ad x R s the state vector. I a more detaled form (4) becomes: m = a x + cσ ( d x + e ) + fu = = (5) y= x 3.. Equvalece betwee Smple HNN ad HNN wth a hdde layer Comparg the descrpto of the hdde layered etwor () wth the descrpto of the Hopfeld etwor of (5) t s clear that the proposed structure has may attrbutes of the orgal structure. To beg wth, both structures utlse exteral feedbac. Each state s drectly coected to tself, separately from the coecto wth the state vector. Secodly the method of coecto of the put to the etwor s detcal both structures. Fally, both structures use the weghted sgmod of a fucto of tme (x(t) the case of the Hopfeld, Dx(t)+E the case of the hdde layer structure). Therefore oe ca expect that the olear propertes vestgated the prevous chapter wll also hold for ths structure. Cosder the proposed structure for a momet. Gve that = Ax+ Cσ ( Dx+ E) + Fu from (4) let us defe a ew auxlary varable m ξ = Dx+ E, ξ R. The the dervatve of ths ew varable s gve by ɺ ξ = Dx ɺ, ad substtutg (4) to ths we obta: ɺ ξ = DAx+ DCσ ( Dx+ E) + DFu (6) Now let us defe a ew state vector [, ] T + z= x ξ R m. By combg (4) ad (6) we ca see that: x A 0 0 C F z ɺ ɺ= z σ ( z) ɺ u ξ = + + DA 0 0 DC DF (7)

4 But ths s smlar to the form of the Hopfeld but wth the matrces B, W ad Γ gve by: A 0 0 C F B=, W, DA 0 = Γ= 0 DC DF It could be oted that the B matrx s o loger a dagoal matrx. Although the frst states are fedbac to themselves, the last m states are fed wth oly the frst states. Ths hts to a specal sgfcace of the frst states. Uder closer specto of (7) t ca be see that the frst states are the revealed states or the approxmatos to the states that the etwor s traed for. Aother mportat observato s that the weght matrx W s sparse. The revealed states are ot drectly coected here. Istead there s a drect coecto through the auxlary states ad the feedbac matrx. To summarse, the proposed hdde layered archtecture of put euros ad m hdde oes s equvalet to a Hopfeld structure of N = + m euros but wth the feedbac matrx ot beg dagoal, ad the weght matrx beg sparse. 4. Multple Itercoected HNNs The proposed structure s composed of may smple Hopfeld Neural Networs, each modellg a aspect of the problem, thus breag the ma problem to smaller oes. Ths structure was spred from multple or staced etwors [8-]. These types of etwors have bee successfully employed varous applcatos ragg from mage recogto to patter predcto. I some of those cases the etwors where traed to solve the same problem ad the fal decso was tae by meas le votg. I our case, smaller etwors are employed for dfferet parts of the problem, the soluto to whch rses from the combato of the smaller etwors. As a example cosder a system wth states, smlarly to (3): = f ( x) + g ( x) u = f ( x) + g ( x) u = f ( x) + g ( x) u y = x (8) We ca cosder ths as put-output problems stead of oe put-state problem wth tracg parameters. Specfcally we ca tra smaller etwors each modellg oe state of (8) but wth puts (the regular put u ad the other - states) stead of ust u. Therefore the resultg structure for the th etwor modellg the th process state wll loo le: m zɺ, = β, z, + w,(, ) σ ( z, ) = + γ,, z, + γ,, + u = m zɺ, = β, z, + w,(, ) σ ( z, ) = (9) + γ,, z, + γ,, u + = m zɺ, m = β,,,(, ) (, ) m z m + w m σ z = + γ, m, z,+ γ, m, + u = where m s the umber of euros for ths etwor ad the state of the process s modelled by the frst state of the etwor. I the case where the weghts have three dexes, the frst oe detfes the etwor, the secod the state, ad the last the coecto they belog to. Therefore the weght w 3,6,4 s a weght belogg to the thrd etwor coectg the sxth state to the fourth state. Fgure : Multple Itercoected etwor structure modelg a o-lear process

5 4.. Equvalece betwee Smple HNN ad multple tercoected HNNs As t ca be see from (9), the proposed structure s composed out of smaller Hopfeld etwors. As far as each etwor s cocered, the process that t s tryg to model has multple puts ad a sgle output. Therefore from a collectve pot of vew the collecto of the etwors are modellg a collecto of processes each wth may puts but oe output, as t ca be see Fgure. Ths has the advatage that t s possble to tra each etwor depedetly of the rest. Sce each etwor structure has to model oly oe varable, t s much easer to tra (Fgure ). Solvg ths for x ( t) we get x ( t) = x ( t) + ( t) (0) Ths expresses that at ay tme the state approxmato of the etwor model equals the state of the process plus a small value. I (0), ( t ) ca be approxmated by ose wth zero average (Fgure ). Now, let us cosder the structure Fgure. Each of the dvdual etwors wll be descrbed by (9). Defe a state vector ξ as the composto of all the states of the etwors: [,, ] ξ = ξ ξ ξ T T = z, z, m,, z, z, m,, z, z, m () ad let the matrces Γ, C, W, be: γ,, + γ,, 0 0 w,, w,, m γ,, + Γ, =, C =, W = γ, m, 0 0 w, m, w, m, m γ, m, + The by combg etwors of the form (9) ad substtutg for the states wth () we get the descrpto: Fgure : Trag of a multple tercoected etwor structure. Oce all the etwors are traed, t s a smple matter of puttg together the buldg blocs to create the complex represetato requred. A secod advatage of ths structure s ose mmuty. Gve that ose was cluded the states that were fed to the etwors as puts durg trag, the overall structure wll be mmue to dsturbaces. Sce each etwor output wll be a approxmato to the real state of the process, t ca be therefore cosdered to be the state of the process wth some ose added. Ths ca be see more clearly f we cosder the followg. Let at tme t, x ( t ) be the th state of the process, ad xˆ ( t ) be the th state of the etwor. Let ( t ) be the fucto whch defes the dfferece betwee x ( t) ad x ( t ). Assumg that the etwor s traed, the at ay tme ths dfferece must be a eghbourhood of zero: x ( t) x ( t) = ( t), ( t) ( ε, ε ) t 0 0 B Γ Γ W 0 0 C Γ B Γ 0 W 0 C ɺ ξ = ξ + σ ( ξ ) + u Γ Γ B 0 0 W C () Ths descrpto s a Hopfeld etwor. Nevertheless there are subtle dffereces betwee ths represetato ad (). The feedbac matrx retas ts dagoal, but ow there are addtoal feedbac coectos betwee the states. The Γ matrces have zero elemets everywhere except the frst colum. Ths hts to a specal sgfcace for these states. As a matter of fact these are the frst states of each etwor ad therefore accordg to our desg of the multple structure, the approxmatg states to the o-lear states. Thus each state the ew structure has feedbac coectos to tself (because of the B matrces) ad to the states that approxmate the states of the process (because of the Γ matrces). Furthermore, the weght matrx s a sparse matrx. The dstrbuted system approach of the multple etwor structure ca be clearly see here: the weght matrx provdes coectos betwee eghbourhoods of

6 states. I each case, these sates correspod to the states of a etwor the multple etwor structure aloe. The coectos betwee these eghbourhoods of states are left to weghts the feedbac matrx. Fally the put matrx s exactly the same format as the orgal Hopfeld cofgurato of (). The smlartes betwee the two proposed structures should be oted. I both cases, the feedbac matrces have creased coectvty, allowg formato to pass drectly from oe state to the other. O the other had, the coectvty the weght matrces has bee lmted ad fewer states are requred to pass through the sgmod fucto. A frst glace to ths dcates that t s ot mportat. Nevertheless, most eural etwor applcatos are mplemeted software rather tha hardware because of practcal mplcatos. Calculatg the sgmod of a state ad the multplyg t wth a weght s less effcet tha ust multplyg the two umbers together, ot ust because of the extra calculato eed, but because calculatg the sgmod (a o-lear fucto) of a umber s also computatoally expesve. 5. Alteratve Neural Networs as olear systems The etwor structures preseted ths secto are equvalet to the Hopfeld etwor, wth the feedbac matrx o loger a dagoal matrx, ad the weght matrx beg sparse. The equvalet equatos (7) ad () of the alteratve structures stll descrbe cotrol affe o-lear systems ad thus the cocepts dscussed the prevous part stll apply to them. As a mater of fact, wth lttle effort, that aalyss ca be easly exteded to cover the alteratve structures. 5.. Relatve order of alteratve structures Cosder the propostos about the relatve order of the Hopfeld etwor. There the relatve order s defed from the structures of the weght ad the put matrces. I the case of smple Hopfeld etwors the feedbac matrx s ot tae to accout, because t coects a state wth tself. I order to exted the propostos for the cases of the alteratve etwor structures, the effect of the feedbac matrx has to be tae to accout. Here the effect of the weght matrx relato wth that of the feedbac matrx has to be cosdered. The feedbac matrx s assocated wth the states drectly, whle the weght matrx s assocated wth the threshold fucto, whch s a sgmod. So geerally, both matrces are multpled wth a fucto of the states, ad therefore whatever ssues arse for the structure of oe, wll also arse for the structure of the other. Cosder a fully coected etwor: = β x β x + w σ ( x ) + + w σ ( x ) = β x β x + w σ ( x ) + + w σ ( x ) = β x β x + w σ ( x ) + + w σ ( x ) + γ u y= x whch ca be re-arraged as: = [ β x + w σ ( x )] + + [ β x + w σ ( x )] = [ β x + w σ ( x )] + + [ β x + w σ ( x )] = [ β x + w σ ( x )] + + [ β x + w σ ( x )] + γ u y= x Cosderg the above structure, t ca be see that the etwor s charactersed by fuctos of the form β x + wσ ( x ). Ths also dcates to a smlar behavour of the structures of the feedbac ad weght matrces as far as the relatve order s cocered. I order for the relatve order to belog the rego of [,] the combato of the feedbac matrx ad the weght matrx must gve rse to a structure smlar to that of the structure of the weght matrx the Relatve Order Proposto. 5.. Zero dyamcs of the alteratve structures Prevously we demostrated that the extra states of a Hopfeld Neural Networ could descrbe extra zero dyamcs. It has also bee demostrated that the equvalet structure of the hdde layered Hopfeld, the frst states approxmate the process states, ad the last m states (where m s the umber of euros the hdde layer) are extra states. Therefore these states descrbe extra zero dyamcs. Smlarly, the equvalet form of the multple tercoected etwors, oly the frst state of each etwor approxmates a state of the o-lear process. Sce the remag states each etwor also appear the equvalet structure, they must also descrbe extra zero dyamcs. Geerally, from the pot of vew of a Hopfeld etwor wth more states tha the process t s modellg, we ca say that the Hopfeld etwor s tryg to model a process wth a equal umber of

7 states but wth usg lmted owledge of the process states to tra. I the case of the HNN wth a hdde layer, the extra states are teral, ad do ot appear the output of the etwor. Therefore o matter the umber of hdde euros, the etwor wll always try to model a process wth the same umber of observable states. Smlarly, the multple tercoected Hopfeld structure, each sub-etwor s modellg a state of the process ad t s that state that s used to coect t to the other sub-etwors. The other states of each sub-etwor oly appear that etwor ad ot the output of the resultg structure. Therefore, the resultg structure wll always model a process wth the correct umber of states. V N N N N L = + + [ ( )] = + + V = + N+ N L 5.3. Whch etwor? We have argued that a Hopfeld etwor ca be used to model the dyamcs of a o-lear process. Whe extra complexty the etwor model s eeded, there exst three alteratves. The frst s to use a Hopfeld etwor but wth extra euros. Ths gves rse to the problem of detfyg the role of the extra euros whe these do ot represet extra zerodyamcs. If the umber of euros of the etwor s N, the the umber of varables (V S ) that eed trag (ad therefore the dmesoalty of the trag problem) s: N varables from the feedbac matrx, N N varables from the weght matrx, ad N varables from the put matrx, a total of: VS = N+ N N+ N = N + N A secod soluto s to add a hdde layer to the Hopfeld model, thus creasg ts complexty. Ths structure has bee see to be equvalet to the frst, wth the umber of put / output euros ad the hdde oes equal to the total umber of euros the frst case..e. a hdde layer Hopfeld wth put / output euros ad m hdde (where s also the umber of states of the process), s equvalet form to a smple Hopfeld wth N = + m euros. But because the equvalet structure s sparse, the umber of varables that eed trag (V L ) ca be see from (4) as varables from the A matrx, m varables from the C matrx, m from the D matrx, m from the E matrx, ad from the F matrx. A total of V = + m+ m + m+ = + m+ ( m) L But sce m= N the total umber of varables terms of umber of modelled states ad equvalet euros s Fgure 3: The three cases wth varyg umber of euros ther equvalet forms, whe the process has states Fally, the thrd opto s to use may smpler tercoected Hopfeld etwors, each modellg oe state of the process. Assumg that each sub-etwor has m euros, the the overall structure s equvalet to the frst case, wth the total umber of euros the equvalet form N = m where s aga the umber of states of the process (ad cosecutvely umber of sub-etwors the structure). The the umber of varables that eed trag (V M ) are the umber of subetwors tmes the umber of varables each subetwor: ( ) VM = m+ m N N N VM = + m= V M N = N+ These three equatos have serous mplcatos o the choce of etwor model. I the frst case the umber of varables ca be see to be depedet of the umber of process states, but t s proportoal to the square of the umber of euros the etwor. Therefore as the umber of extra euros creases, the umber of varables that eed trag creases proportoally to the square of that value. I the secod case, the relato betwee the umber of varables that eed trag ad the total umber of euros all layers s of a frst degree, but s also proportoal to the

8 square of the umber of process states. Fally, the case of the multple tercoected etwors, the umber of varables that eed trag s proportoal to the square of total umber of euros all the etwors, but s versely proportoal to the umber of states of the process. The mplcatos of these ca be see more clearly Fgure 3 ad Fgure 4. ad multple eural etwors of dfferet cofguratos were traed as models for ths process. 6.. The o-lear process The process selected for the smulatos s a wellow, well-used o-lear system [3, 4, -8]. The Sgle L Mapulator (SLM) s essetally a pedulum, ad the cotrol problem s to cotrol at ay pot tme, ts posto ad velocty. Gve a weghtless rod of legth l, a dmesoless mass m s placed at oe ed ad the other s pvoted to a fxed pot space such way as to allow the rod to move oly oe drecto. The frcto coeffcet at the pvot pot s ν. Ths system s llustrated Fgure 5. Gve a put torque u at the pvot, the at a agle θ, usg Newto s Laws we ca get: T = I ɺɺ θ u mgl sθ vɺ θ = mlɺɺ θ ɺɺ v θ = g sθ ɺ θ+ u ml ml Fgure 4: The three cases wth varyg umber of states the process, whe ther equvalet forms have 36 euros. Where the umber of states the process that wll be modelled are ot may, but creased complexty s eeded, the most effcet structure terms of umber of varables that eed trag s the Hopfeld wth the hdde layer. The the effect of a small the equato wll be mmal, ad at the same tme t s possble to crease the complexty of the model by addg euros the hdde layer sce ther effect s lear. O the other had, where creased complexty s eeded modelg a process wth may states, the multple tercoected etwor structure would be most effcet, sce the umber of varables that eed trag wll be versely proportoal to the umber of states. At ths pot we should ote that whle mmsg the umber of varables that eed trag s desrable, there should be a lmt the mmsato. As someoe would expect there s a trade off betwee umber of varables ad complexty of the model. 6. Smulatos To test the modellg abltes of the proposed structures, a well-ow o-lear process s selected, Fgure 5: The Sgle L Mapulator Trasferrg the above equato to state space by settg x equal to the posto ad x equal to the velocty, ad by settg m=g, l=m ad v=6g m /s we get: = x = 9.8s x 3x + 0.5u (4) y = x Ths system has a relatve order of. Oce the system s learsed aroud the equlbrum pot ( xo, u o ) = (0,0), we see that the system s stable, ad that t has egevalues at.5± Specfyg the trag algorthm I order to compare the proposed eural etwor structures wth smple HNN, fve sets of trag

9 expermets were performed usg a smple Geetc Algorthm (GA). Each set composed of te trals for the same structure, wth each tral startg from a dfferet radom umber (seed for the radom umber geerator). The same te seeds were used for all sets. I all tests drect ecodg was used to desg the gee sequeces that ecode the eural etwors: The sequece was composed out of real values, each uquely correspodg to a weght the matrces. I all cases, the ftess was defed based o the Normalsed Mea Square Error (NMSE) [4] of the states as defed by: N P (max m ) p m NMSE = 00 ( x x ) N = P = Where N s the umber of states of the system, P s the umber of trag patters, max s the maxmum output value of output, m s the mmum value output of output, x p s the system output ad x m s the model output. Furthermore, trag was stopped whe ether the NMSE of the best etwor dropped bellow 5 0-6, or whe the algorthm fshed the 300 th geerato, whchever crtera was reached frst. Ths was because from prelmary tests t was observed that the NMSE dd ot mprove sgfcatly after 300 geeratos. I all cases a populato of 50 etwors was used. A mutato rate of 40% was used all GA smulatos. Although ths percetage seems large for a mutato rate for a GA, t s ot the percetage for each gee the gee sequece, but t s the possblty that oe gee the gee sequece wll mutate. Ths esures that at most oly oe gee a geotype wll mutate. Ths s wated because may cases the geomes are sort legth, ad therefore f two gees mutate, a large percetage of the geome chages. The pseudo-code descrbg the trag algorthm follows:. Create ad talse a ew populato of 50 etwors.. For every etwor the populato, test t ad calculate the NMSE. 3. Short the populato from smallest NMSE to largest. 4. If the stoppg crtera are reached END SIMULATION. Else go to Do crossover ad mutato operatos usg the 0 etwors wth the lowest NMSE, creatg 8 ew etwors substtutg the 8 etwors wth the hghest NMSE. Go to. The stoppg crtera are descrbed above Trag the models The frst set of expermets volved trag smple HNNs wth fve euros, as models of the SLM. Ths way the etwor model has more states tha the process. Ths wll form the bass of comparso wth the proposed structures. The results of ths set are dsplayed table T, the appedx. Fgure 6, shows the phase portrat of the best etwor ths set. I order to test the Hopfeld etwor wth the hdde layer, a etwor wth two output euros (oe for each process state) ad three odes the hdde layer was used. As we have see ths s equvalet to the fve-euro smple HNN. For ths set of smulatos a GA was used to tra the etwor smlarly to the prevous case. For reasos of comparso, the same radom umber seeds where used to produce te etwors. The results for ths set are preseted table T, the appedx. Fgure 7, shows the phase portrat of the best etwor ths set. Fgure 6: The phase portrat of the best 5-euro etwor compared wth the process, for dfferet tal codtos. I the fal set of smulatos the multple etwors archtecture was tested. Sce the o-lear process has two states, a double Hopfeld etwor was traed wth each etwor modellg a state. Trag the etwors was doe wth a GA, wth each sub-etwor traed ot separately, but parallel wth each other. Smlarly wth all prevous sets, the trag procedure was repeated te tmes wth te dfferet tal seeds for the radom umber geerator. The results for ths set are preseted the appedx, table T3. Fgure 8, shows the phase portrat of the best etwor ths set.

10 6.4. Dscusso of results I Fgure 6, Fgure 7 ad Fgure 8, the phase portrats of the best etwors of each set, are llustrated for dfferet tal codtos. I order to llustrate the ablty of the etwors to extrapolate, the tal codtos used the phase portrats are outsde the rego used for trag the etwors. Ths rego s llustrated the attached detal, each fgure. We ca clearly see how the etwors are qute capable of mtatg the phase portrats of the process the rego they were traed, whle outsde that rego they do ot perform as well. I partcular, the fve-euro etwor orgally seems to perform very well sde the trag rego, but o a closer loo, oe ca clearly see that the performace s ot as good as those the ext two cases. I the case of the Hopfeld etwor wth hdde layer, the performace s equally good both regos (Fgure 7). Usg etwors of ths type does ot compromse the ablty to detfy the o-lear dyamcs betwee the two regos. Fgure 7: The phase portrat of the best Hopfeld wth a hdde layer compared wth the process, for dfferet tal codtos. Furthermore, Fgure 8 the phase portrat of the best double etwor structure s llustrated (case T-). Comparg ths wth the prevous phase portrats the advatages of ths structure become evdet. Ths etwor outperforms all the prevous cases, sde ad outsde the trag regos. Comparg the results obtaed from trag a HNN wth a hdde layer ad that of ts equvalet fve-euro etwor, t ca be see that o average the Hopfeld wth the hdde layer performs worst tha the equvalet fve-euro case. Uder closer examato though, t becomes evdet that more etwors of the frst case have test errors the order of 0-4 tha the secod. Fgure 8: The phase portrat of the best double Hopfeld compared wth the process, for dfferet tal codtos. A secod alteratve s to use multple tercoected etwors, each modellg a state of the process. Earler t was show that ths structure s also equvalet to a larger Hopfeld etwor. I order to vestgate the approxmatg propertes of ths structure a umber of double etwors were traed to approxmate the process. I each case, the each subetwor was composed of two euros, ad was traed to model a state of the process. Ths structure s roughly equvalet to the fve-euro case, ad the Hopfeld wth the hdde layer. Comparg ths case wth ts equvalets, oe ca see that o average the double structure outperforms both prevous cases, for both tra ad test put sets. By specto of the tables T, T ad T3 located at the appedx, we ca see that the smple HNN wth 5 euros has the largest average trag error, followed by the HNN wth the hdde layer. The best average trag error (lowest value) s acheved by the double HNNs. Uder closer specto, we ca see that the trag error of all double HNNs s located the rage of 0-5, whle ths s also true about most HNN wth a hdde layer. I cotrast, there s oly case (case 4 of table T) of a smple HNNs where the trag error s so low. Ths s a clear dcato that the alteratve etwors structures are easer to tra. I order to esure that the etwors are ot overtraed (perform very well oly for data used durg trag, ad ot very well for other data), a test error for all etwors s calculated, usg data usee durg trag. It s usual to assume that f a Neural Networ (NN) has a small trag error but a large test error,

11 the the NN has over-traed sce t o loger performs well wth other data from what t was traed wth. By spectg table T3 we ca clearly see that the double HNN structure aga outperforms the other two structures, havg a average test error the rage of 0-4, whle the average trag error s the rage of 0-5. Ths s true for all double HNNs, dcatg that oe of the double HNNs have over-traed. The average test error of the HNN wth the hdde layer s above that of the smple HNN, ad after closer specto of tables T we ca see that there are fve cases where the test error s much larger tha the trag error (cases, 5, 7, 9 ad 0). Therefore these fve cases the etwors have over-traed. Fally, table T, there s oly oe case whe the test error much larger tha the test error (case 4). It s very terestg to observe case 9 table T. I ths case, the test error s very hgh respect to all other cases. However ths s also true about the trag error. Therefore ths case the etwor has ot overtraed; rather t ot traed very well. The test errors dcate that there multple HNN structures are also harder to over-tra, tha smple HNNs ad HNNs wth a hdde layer. However, the opposte ca be clamed about HNNs wth a hdde layer. 7. Coclusos It has bee show that ay extra states of a etwor traed as a model of a o-lear process, are coected to the o-lear zero dyamcs, ad are therefore essetal for a good approxmato. Ths has the effect of mag the trag more dffcult by creasg the dmesos of the search space, ad therefore the umber of teratos for the trag algorthm ecessary to lower the trag error to a satsfactory pot. Oe way of percevg the effect of trag a m state etwor to detfy a state process, where m>, s smlar to trag the same etwor to detfy a m state process usg lmted formato (say the frst states) about the o-lear process. Sce we have o formato about the last m- states, t s dffcult ot oly to decde what the roles of the extra states are, but also how to talse them. Two alteratves have bee proposed. The frst s to use a Hopfeld etwor wth as may states as the process, but to crease ts complexty wth the use of a hdde layer. It was show that, a structure le ths s equvalet to a smple HNN wth a total umber of euros equal to the umber of euros both put ad hdde layers of the proposed structure. However, oly some of the euros are drectly coected. The advatage of ths etwor type s that whle t creases the complexty of the etwor, the problem assocated wth choosg tal codtos s o loger preset. The secod alteratve vestgated, was to use multple tercoected etwors, each modellg a sgle state of the process, thus breag dow the problem. Such a structure wth etwors ad m euros was foud to be equvalet to a sgle Hopfeld eural etwor wth tmes m euros but wth the feedbac matrx cludg extra feedbac betwee the states, ad the weght matrx beg sparse. I a comparso of the umber of varables eeded to be traed equvalet forms each case, t was foud that all cases as the umber of equvalet euros crease the umber of varables crease wth a hgher rate the case of the smple HNN, wth a slower rate the Hopfeld wth the hdde layer, ad wth much slower rate the case of the tercoected Hopfeld etwors. Expermetal results dcated that both proposed structures (multple tercoected Hopfeld etwors ad HNN wth hdde layer) have better approxmato capabltes tha the Smple Hopfeld etwor of equvalet umber of euros, ad better extrapolato capabltes. I addto multple HNN structures are easer to tra ad harder to over-tra, whle HNNs wth hdde layer are easy to tra, but there s a hgh probablty that the etwor wll over-tra. 8. Refereces [] J. J. Hopfeld, "Neural Networs ad Physcal Systems Wth Emerget Collectve Computatoal Abltes," Proceedgs of the Natoal Academy of Sceces of the Uted States of Amerca-Bologcal Sceces, vol. 79, pp , 98. [] K. P. Dmopoulos ad C. Kambhampat, "Apror Iformato Networ Desg," Dealg wth Complexty: A Neural Networ Approach., M. Kary, K. Warwc, ad V. Kurova., Eds., 998. [3] K. P. Dmopoulos ad C. Kambhampat, "Multple Itercoected Hopfeld Networs for Itellget Global Learsg Cotrol," preseted at IJCNN, Washgto D.C. USA, 999. [4] K. P. Dmopoulos, C. Kambhampat, ad R. J. Craddoc, "Effcet recurret eural etwor trag corporatg a pror owledge," Mathematcs ad Computers Smulato, vol. 5, pp. 37-6, 000. [5] N. J. Dmopoulos, "A study of the asymptotc-behavor of eural etwors," IEEE Trasactos O Crcuts ad Systems, vol. 36, pp , 989. [6] A. Isdor, Nolear Cotrol Systems. A Itroducto, d ed: Sprger-Verlag, 989. [7] C. Kambhampat, A. Delgado, J. D. Maso, ad K. Warwc, "Stable Recedg Horzo Cotrol Based o

12 Reccuret Networs," preseted at IEE Cotrol Theory Applcatos, 997. [8] S. W. Lee ad S. Y. Km, "Itegrated segmetato ad recogto of hadwrtte umerals wth cascade eural etwor," IEEE Trasactos O Systems Ma ad Cyberetcs Part C- Applcatos ad Revews, vol. 9, pp , 999. [9] J. Ls, "The sythess of the raed eural etwors applyg geetc algorthm wth the dyamc probablty of mutato," From Natural to Artfcal Neural Computato, vol. 930, pp , 995. [0] D. H. Wolpert, "Staced Geeralzato," Neural Networs, vol. 5, pp. 4-59, 99. [] D. V. Srdhar, R. C. Seagrave, ad E. B. Bartlett, "Process modelg usg staced eural etwors," Ache Joural, vol. 4, pp , 996. [] A. Delgado ad C. Kambhampat, "Dfferetal Geometrc Cotrol of No-lear System wth Recurret Neural Networs," Departmet of Cyberetcs, Uversty of Readg, Readg [3] A. Delgado, C. Kambhampat, ad K. Warwc, "Approxmato of o-lear systems usg dyamc recurret eural etwors," preseted at Iteratoal coferece o detfcatos egeerg systems, Uv. of Wales, Swasea, U.K., 996. [4] A. Delgado ad C. Kambhampat, "Global learsg cotrol usg recurret etwors," preseted at Europea Robotcs ad Itellget systems Coferece (EURICON), Malaga, Spa, 994. [5] A. Delgado, "Iput/output learsato of cotrol affe systems usg eural etwors," Departmet of Cyberetcs. Readg: Uversty of Readg, 996. [6] A. Delgado, C. Kambhampat, ad K. Warwc, "Iput/output learzato usg dyamc recurret-eural etwors," Mathematcs ad Computers Smulato, vol. 4, pp , 996. [7] C. Kambhampat, R. J. Craddoc, M. Tham, ad K. Warwc, "Iteral model cotrol of olear systems through the verso of recurret eural etwors," World Cogress o Computatoal Itellgece, Alasa., 998. [8] A. Delgado, C. Kambhampat, ad K. Warwc, "Dyamc recurret eural etwors for system detfcato ad cotrol," IEE Proc. Cotrol Theory Appl., vol. 4, pp , Appedx T. Smple HNN wth 5 euros TEST Seed tra error test error Epochs E-04.65E E E E E E E E E E E E E E E E-03.65E E E Aver 7.46E E T. HNN wth 3 Hdde euros TEST Seed tra error test error Epochs E E E E E E E E E-05.90E E E E E E E E-05.56E E E Aver.6E-04.06E T3. Double HNN wth euros each etwor TEST Seed tra error test error Epochs E-05.E E-05.0E E-05.40E E-05.90E E-05.08E E-05.30E E-05.94E E-05.59E E-05.46E E-05.98E Aver.55E-05.79E