Pareto Genetic Design of GMDH-type Neural Networks for Nonlinear Systems

Transcription

1 Pareto Genetc Desgn of GMDH-tye Neural Networs for Nonlnear Systems N. Narman-zaeh an A. Jamal Deartment of Mechancal Engneerng, Unversty of Gulan, PO Box 3756, Rasht, Iran Abstract. In ths aer, Genetc Algorthms (GAs) are eloye for mult-objectve Pareto otmal esgn of Grou Metho of Data Hanlng (GMDH)-tye neural networs that have been use for moellng of a nonlnear system. In ths way, GAs wth a secfc encong scheme s frstly resente to evolutonary esgn of the generalze GMDH-tye neural networs n whch the connectvty confguratons n such networs are not lmte to ajacent layers. Mult-objectve GAs wth a new versty reservng mechansm are seconly use for Pareto otmzaton of such GMDH-tye neural networs. The mortant conflctng objectves of GMDH-tye neural networs that are consere n ths wor are, namely, Tranng Error (TE), Precton Error (PE) an Number of Neurons (N) of such neural networs. It s shown that the obtane non-omnate Pareto onts are nclusve of those whch can be foun usng Aae s Informaton Crteron (AIC) for both tranng an recton errors.moreover, an mortant trae-off can be scovere by such Pareto otmum aroach to the esgn of GMDH-tye neural networs whch hels a esgner to select a networ comromsngly. Keywors Mult-objectve otmzaton, Genetc algorthms, GMDH, Pareto. 1 Introucton Grou Metho of Data Hanlng (GMDH) algorthm s a self-organzng aroach by whch graually comlcate moels are generate base on the evaluaton of ther erformances on a set of multnut-sngle-outut ata ars (=1,,, M). The GMDH was frst eveloe by Ivahneno [1] as a multvarate analyss metho for comlex systems moellng an entfcaton. In ths way, GMDH was use to crcumvent the ffculty of nowng a ror nowlege of mathematcal moel of the rocess beng consere. Therefore, GMDH can be use to moel comlex systems wthout havng secfc nowlege of the systems. The man ea of GMDH s to bul an analytcal functon n a feeforwar networ base on a quaratc noe transfer functon [] whose coeffcents are obtane usng regresson technque [3]. In recent years, however, the use of such self-organzng networs leas to successful alcaton of the GMDH-tye algorthm n a broa range of areas n engneerng, scence, an economcs [3-5]. The nherent comlexty n the esgn of feeforwar neural networs n terms of unerstanng the most arorate toology an coeffcents has a great mact on ther erformance. There have been extensve efforts n recent years to eloy oulaton-base stochastc search algorthms such as evolutonary methos to esgn artfcal neural networs snce such evolutonary algorthms are artcularly useful for ealng wth comlex roblems havng large search saces wth many local otma [3]. A very comrehensve revew of usng evolutonary algorthms n the esgn of artfcal neural networs can be foun n [6]. Recently, genetc algorthms have been use n a feeforwar GMDH-tye neural networ for each neuron searchng ts otmal set of connecton wth the receng layer [5]. In ths reference, authors have roose a hybr genetc algorthm for a smlfe GMDHtye neural networ n whch the connecton of neurons are restrcte to ajacent layers. All these methos evse revously have been base on sngle objectve otmzaton rocess n whch ether tranng error or recton error selecte to be mnmze wth no control of other objectves. In orer 96

2 to conser the comlexty of such networs, a Mnmum Descrton Length (MDL) aroach has been use n [ba] to nvolve a traeoff between ftness of tranng error an the number of arameters n the networ (comlexty). Recently, Aae s Informaton Crteron (AIC) for recton error of the networ has been use n [7] to traeoff such objectve functons n the esgn of a revse GMDHtye neural networ. However, n orer to obtan more robust moels of such comlex rocess, t s requre to conser all the non-commensurable conflctng objectves, namely, tranng error (TE), recton error (PE) an number of neurons (N) (reresentng the comlexty of the moels) be mnmze smultaneously n the sense of mult-objectve Pareto otmzaton rocess. The obtane Pareto front shown n fferent lane of those objectves woul vsualze the exste traeoffs whch, therefore, hel the esgner to comromse an choose the arorate networ. In Mult-objectve otmzaton roblems (MOPs), there are several objectve or cost functons (a vector of objectves) to be otmze (mnmze or maxmze) smultaneously. These objectves often conflct wth each other so that mrovng one of them wll eterorate another. Therefore, there s no sngle otmal soluton as the best wth resect to all the objectve functons. Instea, there s a set of otmal solutons, nown as Pareto otmal solutons or Pareto front [8-9] for mult-objectve otmzaton roblems. A very goo an comrehensve survey of these methos has been resente n [9]. In aton to ts oularty an effectveness, NSGA-II [8] has been mofe n [10] to enhance ts versty reservng mechansm whch wll be use n ths wor. In ths aer, EAs wth a new encong scheme are use to evolutonary esgn the generalze structure GMDH-tye (GS-GMDH) neural networs n whch the connectvty confguraton n such networs s not lmte to ajacent layers for moellng an recton of a nonlnear system. In ths way, mult-objectve EAs (non omnate sortng genetc algorthm, NSGA-II) wth a new versty reservng mechansm are ale for Pareto otmzaton of such GS-GMDH-tye neural networs. The mortant conflctng objectves of the GS-GMDH neural networs that are consere n ths wor are, namely, tranng error (TE), recton error (PE) an number of neurons (N). The total numbers of ranomly generate ata are 100 from whch 50 are ranomly use for evaluatons of TE whlst the remanng 50 ata are use for evaluaton of PE. All these 3 conflctng objectves are consere n a 3-objectve otmzaton rocess whch consequently leas to a comlete Pareto set of solutons of GMDH-tye neural networs moels. It s shown that AIC formulaton ether for tranng error or for recton error wll conce to only two non-omnate otmum onts obtane from such Pareto aroach of GMDH-tye neural networs. The results of ths wor emonstrate that some useful an nformatve esgn concets regarng the otmal choces of moels can be unvele by the combnaton of GMDH-tye neural networs an mult-objectve EAs. Moelng Usng GMDH Neural Networs By means of GMDH algorthm a moel can be reresente as set of neurons n whch fferent ars of them n each layer are connecte through a quaratc olynomal an thus rouce new neurons n the next layer. Such reresentaton can be use n moellng to ma nuts to oututs. The formal efnton of the entfcaton roblem s to fn a functon fˆ so that can be aroxmately use nstea of actual one, f n orer to rect outut ŷ for a gven nut vector X = ( x, x, x,..., x ) as 1 3 n close as ossble to ts actual outut y. Therefore, gven M observaton of mult-nut-sngle-outut ata ars so that y = f(x, x, x,..., x ) (=1, M), (1) 1 3 n t s now ossble to tran a GMDH-tye neural networ to rect the outut values ŷ for any gven nut vector X = ( x, x, x,..., x ), that s 1 3 n yˆ = f(x ˆ, x, x,..., x ) (=1, M). () 1 3 n The roblem s now to etermne a GMDH-tye neural networ so that the square of fference between the actual outut an the recte one s mnmze, that s M = 1 ˆ [ f (x, x, x,..., x ) y ] mn. (3) 1 3 n 97

3 General connecton between nuts an outut varables can be exresse by a comlcate screte form of the Volterra functonal seres n the form of n n n n n n y = a 0 + a x + a j x x j + a j x x j x , (4) o = = j = = j = = where s nown as the Kolmogorov-Gabor olynomal []. Ths full form of mathematcal escrton can be reresente by a system of artal quaratc olynomals consstng of only two varables (neurons) n the form of yˆ = G(x, x j ) = a 0 + a 1 x + a x j + a 3 x x j + a 4 x + a5x j. (5) In ths way, such artal quaratc escrton s recursvely use n a networ of connecte neurons to bul the general mathematcal relaton of nuts an outut varables gven n equaton (4). The coeffcents a n equaton (5) are calculate usng regresson technques [1-3] so that the fference between actual outut, y, an the calculate one, ŷ for each ar of ( x, x j ) as nut varables s mnmze. Inee, t can be seen that a tree of olynomals s constructe usng the quaratc form gven n equaton (5) whose coeffcents are obtane n a least-squares sense. In ths way, the coeffcents of each quaratc functon G are obtane to otmally ft the outut n the whole set of nut-outut ata ar, that s M ( y y ) o E = = 1 mn. (6) M In the basc form of the GMDH algorthm, all the ossbltes of two neenent varables out of total n nut varables are taen n orer to construct the regresson olynomal n the form of equaton (5) that best fts the eenent observatons ( y, =1,,, M) n a least-squares sense. Consequently, n n( n 1) = neurons wll be bult u n the frst hen layer of the fee forwar networ from the observatons { ( y,x, x ); (=1,,, M)} for fferent, q { 1,,..., n}. In other wors, t s now q ossble to construct M ata trles { ( y,x,xq ) ; (=1, M)} from observaton usng such, q { 1,,..., n} n the form q y1 x xq y. x M xmq y M Usng the quaratc sub-exresson n the form of equaton (5) for each row of M ata trles, the followng matrx equaton can be realy obtane as A a = Y, (7) where a s the vector of unnown coeffcents of the quaratc olynomal n equaton (5) a = {a 0,a 1,...,a 5 }, (8) an T Y = { y1, y, y3,..., y M }, (9) s the vector of outut s value from observaton. It can be realy seen that 1 x 1 q q q A = 1 x xq x xq x xq. (10) 1 xm xmq xm xmq xm xmq The least-squares technque from multle-regresson analyss leas to the soluton of the normal equatons n the form of T 1 T a = ( A A) A Y, (11) 98

4 whch etermnes the vector of the best coeffcents of the quaratc equaton (5) for the whole set of M ata trles. It shoul be note that ths roceure s reeate for each neuron of the next hen layer accorng to the connectvty toology of the networ. However, such a soluton rectly from normal equatons s rather suscetble to roun off errors an, more mortantly, to the sngularty of these equatons..1. Alcaton of SVD to the esgn of GMDH-Tye neural networs The SVD of a matrx, A R M 6 s a factorsaton of the matrx nto the rouct of three matrces, column-orthogonal matrxu R M 6, agonal matrx W R 6 6 wth non-negatve elements (sngular values), an orthogonal matrx V R 6 6 such that A = U W V T. (1) The roblem of otmal selecton of vector of the coeffcents n equatons (7) (11) s frstly reuce to fnng the mofe nverson of agonal matrx W [11] n whch the recrocals of zero or near zero sngulars (accorng to a threshol) are set to zero. Then, such otmal a s calculate usng the followng relaton a = V [ ag (1/ w )] U T Y. (13) j Such arametrc entfcaton roblem s art of the general roblem of moellng when structure entfcaton s consere together wth the arametrc entfcaton roblem smultaneously. In ths wor, an encong scheme eveloe by authors [10] s use n an evolutonary aroach for smultaneous etermnaton of structure an arametrc entfcaton of GMDH neural networs.. Alcaton of GA n the toology esgn of GMDH-Tye neural networs GAs as stochastc methos are commonly use n the tranng of neural networs n terms of assocate weghts or coeffcents an have successfully erforme better than tratonal graent-base technques [5]. The lterature shows that a we range of evolutonary esgn aroaches ether for archtectures or for connecton weghts searately, n aton to efforts for them smultaneously [6]. In the most GMDH-tye neural networ, neurons n each layer are only connecte to neurons n ts ajacent layer as t was the case n Methos I an II revously reorte n [5]. Tang ths avantage, t s ossble to resent a smle encong scheme for the genotye of each nvual n the oulaton [1]. The encong scheme n generalze GMDH (GS-GMDH) neural networs must emonstrate the ablty of reresentng fferent length an sze of such neural networs an s now resente n summary. a b c ab bc a abbc abbcaa Fg. 1. A GS-GMDH networ structure of a chromosome In fgure 1, neuron a n the frst hen layer s connecte to the outut layer by rectly gong through the secon hen layer. Therefore, t s now very easy to notce that the name of outut neuron (networ s outut) nclues a twce as abbcaa. In other wors, a vrtual neuron name aa has been constructe n the secon hen layer an use wth abbc n the same layer to mae the outut neuron abbcaa as shown n the fgure 1. It shoul be note that such reetton occurs whenever a neuron asses some ajacent hen layers an connects to another neuron n the next n, or 3 r,or 4 th,or followng hen layer. In ths encong scheme, the number of reetton of that ñ neuron eens on the number of asse hen layers, ñ, an s calculate as. It s easy to realze that a chromosome such as abab bcbc, unle chromosome abab acbc for examle, s not a val one n GS-GMDH networs an has to be smly re-wrtten as abbc. 99

5 Parents a b b c a b a b c Offsrngs a b b c a b c a b Fg.. Crossover oeraton for two nvuals n GS-GMDH networs a ab a ab b c a bc abbc abbcaa b c bc abbc Fg. 3. Crossover oeraton on two GS-GMDH networs.3 Genetc Oerators for GS-GMDH Networ Reroucton The genetc oerators of crossover an mutaton can now be mlemente to rouce two offsrngs from two arents. The natural roulette wheel selecton metho s use for choosng two arents roucng two offsrngs. The crossover oerator for two selecte nvuals s smly accomlshe by exchangng the tals of two chromosomes from a ranomly chosen ont as shown n fgure. It shoul be note, however, such a ont coul only be chosen ranomly from the 1 n + 1 set {,,..., l } where n l s the number of hen layers of the chromosome wth the smaller length. It s very event from fgures an 3 that the crossover oeraton can certanly exchange the bulng blocs nformaton of such GS-GMDH neural networs. In aton, such crossover oeraton can also rouce fferent length of chromosomes whch n turn leas to fferent sze of GS-GMDH networ structures. Smlarly, the mutaton oeraton can contrbute effectvely to the versty of the oulaton. Ths oeraton s smly accomlshe by changng one or more symbolc gts as genes n a chromosome to another ossble symbol, for examle, abbcaa to abbcca. It shoul be note that such evolutonary oeratons are accetable only f a val chromosome s rouce. Otherwse, these oeratons are smly reeate untl a val chromosome s constructe. 3 Mult-objectve otmzaton Mult-objectve otmzaton whch s also calle multcrtera otmzaton or vector otmzaton has been efne as fnng a vector of ecson varables satsfyng constrants to gve otmal values to all objectve functons [8-10]. In general, t can be mathematcally efne as: fn the vector X = [ x,x,...,x n ] T 1 to otmze [ f (X), f (X),..., f (X)] T a b c ab ab F(X) = 1, (14) subject to m nequalty constrants g (X) 0, = 1 to m, (15) an equalty constrants h j (X) = 0, j = 1 to, (16) a b c ab a ab abaa 100

6 where n X R s the vector of ecson or esgn varables, an F(X) R s the vector of objectve functons. Wthout loss of generalty, t s assume that all objectve functons are to be mnmze. Such mult-objectve mnmzaton base on the Pareto aroach can be conucte usng some efntons: Defnton of Pareto omnance A vector U = [ u, u,..., u ] R omnates to vector V [ v, v,..., v ] R 1 ) f an only f { 1,,..., }, u j whch s smaller than = 1 (enote by U V u v j { 1,,..., } : u j < v j. It means that there s at least one v j whlst the rest u s are ether smaller or equal to corresonng v s. Defnton of Pareto otmalty n A ont X Ω ( Ω s a feasble regon n R satsfyng equatons (15) an (16)) s sa to be Pareto otmal (mnmal) wth resect to all X Ω f an only f F(X ) F(X). Alternatvely, t can be realy restate as { 1,,..., }, X Ω {X } f (X ) f (X) j { 1,,..., } : f j (X ) < f j (X). It means that the soluton X s sa to be Pareto otmal (mnmal) f no other soluton can be foun to omnate X usng the efnton of Pareto omnance. Defnton of Pareto front For a gven MOP, the Pareto front ƄŦ s a set of vectors of objectve functons whch are obtane usng the vectors of ecson varables n the Pareto set, Ƅ that s, ƄŦ = { F(X) = (f 1 (X), f (X),..., f (X)): X.{ Ƅ Therefore, the Pareto front ƄŦ s a set of the vectors. Ƅ of objectve functons mae from Defnton of Pareto Set For a gven MOP, a Pareto set Ƅ s a set n the ecson varable sace consstng of all the Pareto otmal vectors, Ƅ = { X Ω X Ω :F(X ) F(X) }. In other wors, there s no other X n Ω that omnates any X Ƅ n terms of ther objectove functons. 4 Moellng of a Nonlnear System usng Mult-objectve GMDH neural networs The nut outut ata use n such moellng nvolve 100 ata ars ranomely generate from a nonlnear system wth three nuts x 1, x, x 3, an a sngle outut y gven by y = (1 + x + x + x ), 1 x 1, x, x 3 5. ( 17) 1 3 There are 50 attern numbers whch have been ranomly selecte from those ata ars to tran such GMDH tye neural networs. However, a testng set whch conssts of 50 unforeseen nutoutut ata samles urng the tranng rocess, s merely use for testng to show the recton ablty of such evolve GMDH-tye neural networ moels urng the tranng rocess. The GMDH-tye neural networs are now use for such nut-outut ata to fn the olynomal moel of y n such nonlnear system rocess wth resect to ther nut arameters. In orer to esgn GMDH-tye neural networ escrbe n revous secton from a mult-objectve otmum ont of vew, a oulaton of 60 nvuals wth a crossover robablty of 0.95 an mutaton robablty of 0.1 has been use n 50 generaton that no further mrovement has been acheve for such oulaton sze. A mult-objectve otmzaton of GMDH-tye neural networs nclung all three objectves can offer more choces for a esgner. Fgure 4 ects the non-omnate onts of 3- objectve otmzaton rocess n the lane of (TE-PE). It shoul be note that there s a sngle set of non-omnate onts as a result of 3-objectve Pareto otmzaton of TE, PE an N that are shown n 101

7 that lane. Therefore, there are some onts n the lane that may omnate others n the case of 3- objectve otmzaton. However, these onts are all non-omnate when conserng all three objectves smultaneously. By careful nvestgaton of the results of 3-objectve otmzaton n that lane, the Pareto front of the corresonng -objectve otmzaton (TE-PE) can now be observe. In ths fgure, onts A an B stan for the best (TE) an the best (PE), resectvely. The corresonng values of errors, number of neurons, an the structure of these extreme otmum esgn onts are gven n table Pareto front AIC wth best tranng error AIC wth best recton error 0.1 e r o r n to 0.08 c r e P 0.06 A 0.04 C 0.0 B Tranng error Fg. 4. Precton error varaton wth tranng error n 3-objectve otmzaton. Clearly, there s an mortant otmal esgn fact between these two objectve functons whch has been scovere by the Pareto otmum esgn of GMDH-tye neural networs. Such mortant esgn fact coul not have been foun wthout the mult-objectve Pareto otmzaton of those GMDH-tye neural networs. From fgure 4 onts C s the ont whch emonstrates such mortant otmal esgn fact. Pont C n the Pareto front of otmum esgn of TE an PE, exhbts small ncrease n the value of TE n comarson wth that of ont A whlst ts PE shows sgnfcant mrovement (about 150 tmes better recton error). Therefore, ont C coul be a trae-off otmum choce when conserng the mnmum values of both PE an TE smultaneously. The structure an networ confguraton corresonng to ont C s shown n fgure 5. Fg. 6. The networ s structure of ont C n whch a, b, c an stan for x 1, x, x 3 resectvely. In orer to comare these results, AIC [13][7] has been use both for tranng an testng ata n two fferent sngle objectve otmzaton rocesses. AIC s efne by AIC = n log e (E) + (N+1) + C (18) where E, the mean square of error, s comute usng equaton 6, N s the number of neurons, n s number of tranng/testng error, an C s a constant. Tab.1. Objectve functons an structure of networs of fferent onts shown on fgure 4. Networ s chromosome No. of Tranng Precton Neurons error error Pont A bbbbbbbbaabcabab Pont B bbbbabacbbabaaaa Pont C bbbbbbbbaabcabac

8 Therefore, two otmum onts have been foun usng AIC an are shown n fgure 4. Clearly, these two onts conce wth the onts A an B corresonngly. It s then event that the Pareto otmum esgn of GMDH-tye neural networs resente n ths aer are nclusve of those obtane by AIC an also resents more effectve way of choosng trae-off otmum moels wth resect to conflctng objectve functons. It shoul be note that ont C coul be acheve by a roer weghtng coeffcents (whch s not now a ror) of recton an tranng errors usng AIC n only convex rogrammng roblems. 5 Concluson Genetc algorthms have been successfully use for mult-objectve Pareto base otmzaton of generalze GMDH-tye neural networs use for moellng an recton of a nonlnear system. Such mult-objectve otmzaton le to the scoverng of useful otmal esgn rncles n the sace of objectve functons. In ths wor, the mortant conflctng objectve functons of GMDH-tye neural networs have been selecte as Tranng Error (TE), Precton Error (PE) an Number of Neurons (N) of such neural networs. In aton to scoverng the trae-off otmum onts, t has been shown that the Pareto front obtane by the aroach of ths aer nvolves those that can be foun by Aae s Informaton Crteron whch thus exhbts the effectveness of the Pareto otmum esgn of GMDH-tye neural networs resente n ths aer. References [1] Ivahneno, A. G.,: Polynomal Theory of Comlex Systems. IEEE Trans. Syst. Man & Cybern, SMC-1, , [] Farlow, S. J.,: Self-organzng Metho n Moelng: GMDH tye algorthm. Marcel Deer Inc., [3] Iba, H., egars, H. an Sato, T.,: A numercal Aroach to Genetc Programmng for System Ientfcaton. Evolutonary Comutaton 3(4):417-45, [4] Narman-Zaeh, N., Darvzeh, A., Felez, M. E. an Gharababae, H.,: Polynomal moellng of exlosve comacton rocess of metallc owers usng GMDH-tye neural networs an sngular value ecomoston. Moellng an Smulaton n Materals Scence an Engneerng, Vol. 10, no. 6, (18), 00. [5] Narman-Zaeh, N., Darvzeh, A. an Ahma-Zaeh, G. R.,: Hybr Genetc Desgn of GMDH- Tye Neural Networs Usng Sngular Value Decomoston for Moellng an Precton of the Exlosve Cuttng Process. Proceengs of the I MECH E Part B Journal of Engneerng Manufacture, Volume: 17, Page: , 003. [6] Yao, X.,: Evolvng Artfcal Neural Networs. Proceengs of IEEE 87(9): , [7] Kono, T. an Ueno, J.,: Revse GMDH-tye Neural Networ Algorthm wth a Feebac Loo Ientfyng Sgmo Functon Neural Networ. Int. J. of Innovatve Comutng, Informaton an Control, Vol. No.5, 006. [8] Deb, K., Agrawal, S., Prata, A. an Meyarvan, T.,: A fast an eltst mult-objectve genetc algorthm: NSGA-II. IEEE Trans. On Evolutonary Comutaton 6():18-197, 00. [9] Coello Coello, C. A.,: A comrehensve survey of evolutonary base multobjectve otmzaton technques. Knowlege an Informaton Systems: An Int. Journal, (3), , [10] Narman-zaeh, N., Atashar, K., Jamal, A., Plech, A. an Yao, X.,: Inverse Moellng of Mult-objectve Thermoynamcally Otmze Turbo Engnes usng GMDH-tye Neural Networs an Evolutonary Algorthms. Engneerng Otmzaton, Vol. 37, No. 5, , 005. [11] Press, W. H., Teuolsy, S. A., Vetterlng, W. T. an Flannery B. P.,: Numercal Reces n FORTRAN: The Art of Scentfc Comutng, n Eton, Cambrge Unversty Press, 199. [1] Narman-zaeh, N., Darvzeh, A., Jamal, A. an Moen,: A. Evolutonary Desgn of Generalze Polynomal Neural Networs for Moellng an Precton of Exlosve Formng Process. Journal of Materal Processng an Technology, Vol , , Elsever, 005. [13] Aae, H.,: A new loo at the statstcal moel entfcaton, IEEE Trans. Automatc Control, vol.ac-19, no.6, ,