2003 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media,

Transcription

1 003 IEEE. Personal use of ths materal s permtted. Permsson from IEEE must be obtaned for all other uses, n any current or future meda, ncludng reprntng/republshng ths materal for advertsng or promotonal purposes, creatng new collectve works, for resale or redstrbuton to servers or lsts, or reuse of any copyrghted component of ths work n other works.

2 Tunng of the Structure and Parameters of Neural Networks usng an Improved Genetc Algorthm H.K. Lam, S.H. Lng, F.H.F. Leung and P.K.S. Tam Centre for Multmeda Sgnal Processng Department of Electronc and Informaton Engneerng The Hong Kong Polytechnc Unversty, Hung Hom, Kowloon, Hong Kong. Abstract Ths paper presents the tunng of the structure and parameters of a neural network usng an mproved genetc algorthm (GA). It wll also be shown that the mproved GA performs better than the standard GA based on some benchmark test functons. A neural network wth swtches ntroduced to ts lnks s proposed. By dong ths, the proposed neural network can learn both the nput-output relatonshps of an applcaton and the network structure. The number of hdden nodes should be chosen manually startng from a small number. The number of hdden nodes should be ncreased f the learnng performance n terms of ftness value s not acceptable. Usng the mproved GA, the structure and the parameters of the neural network can be tuned. Applcaton examples on sunspot forecastng and assocatve memory are gven to show the merts of the mproved GA and the proposed neural network. I. INTRODUCTION GA s a drected random search technque [] that s wdely appled n optmzaton problems [-, 5]. Ths s especally useful for complex optmzaton problems where the number of parameters s large and the analytcal solutons are dffcult to obtan. GA can help to fnd out the optmal soluton globally over a doman [-, 5]. It has been appled n dfferent areas such as fuzzy control [9-, 5], path plannng [], greenhouse clmate control [3], modelng and classfcaton [4] etc. A lot of research efforts have been spent to mprove the performance of GA. Dfferent selecton schemes and genetc operators have been proposed. Selecton schemes such as rank-based selecton, eltst strateges, steady-state electon and tournament selecton have been reported [3].

3 There are two knds of genetc operators, namely crossover and mutaton. Apart from random mutaton and crossover, other crossover and mutaton mechansms have been proposed. For crossover mechansms, two-pont crossover, multpont crossover, arthmetc crossover and heurstc crossover have been reported [, 3-33]. For mutaton mechansms, boundary mutaton, unform mutaton and non-unform mutaton can be found [, 3-33]. Neural network was proved to be a unversal approxmator [6]. A 3-layer feed-forward neural network can approxmate any nonlnear contnuous functon to an arbtrary accuracy. Neural networks are wdely appled n areas such as predcton [7], system modelng and control [6]. Owng to ts partcular structure, a neural network s very good n learnng [] usng some learnng algorthms such as GA [] and back propagaton algorthm []. In general, the learnng steps of a neural network are as follows. Frst, a network structure s defned wth fxed numbers of nputs, hdden nodes and outputs. Second, an algorthm s chosen to realze the learnng process. It can be seen that a fxed structure may not provde the optmal performance wthn a gven tranng perod. A small network may not provde good performance owng to ts lmted nformaton processng power. A large network, on the other hand, may have some of ts connectons redundant [8-9]. Moreover, the mplementaton cost for a large network s hgh. To obtan the network structure automatcally, constructve and destructve algorthms can be used [8]. The constructve algorthm starts wth a small network. Hdden layers, nodes and connectons are added to expand the network dynamcally [9-4]. The destructve algorthm starts wth a large network. Hdden layers, nodes and connectons are then deleted to contract the network dynamcally [5-6]. The desgn of a network structure can be formulated nto a search problem. Genetc algorthms [7-8] were employed to obtan the soluton. Pattern-classfcaton approaches [9] can also be found to desgn the network structure. Some other methods have been proposed to learn both the network structure and connecton weghts. The evoluton cycle of these methods can be summarzed by three steps as follows. ) Evaluate each ndvdual,.e., chromosome, accordng to a defned ftness functon. ) Select ndvdual for reproducton and genetc operaton. 3) Apply dfferent knds of genetc operatons to the chromosomes to obtan the next

4 generaton. An ANNA ELEONORA algorthm was proposed [36]. New genetc operator and encodng procedures (bnary) whch allows the algorthm to obtan an opportune length of the codng strng were ntroduced. Each gene conssts of two parts, connectvty bts and the connecton weght bts. The connectvty bts are to ndcate the absence or present of a lnk. The connecton weght bts are related to the value of the weght of a lnk. A GNARL algorthm was also proposed n [37]. At frst, the populaton of the chromosomes representng the network structure wll be generated randomly. The number of hdden nodes and connecton lnks for each network s randomly chosen wthn some defned ranges. Three steps were proposed to generate an offsprng: copyng the parents, determnng the mutatons to be performed and mutatng the copy. The severty of mutatons measurng the performance of the network wll be used to anneal the structural and parametrc smlarty between parent and offsprng. The networks wth hgh smlarty wll be mutated severely, on the contrary, the networks wth low smlarty wll be mutated slghtly. Mutaton of the copy s separated nto two classes, parametrc mutatons whch alter the connecton weghts and structural mutatons alter the number of hdden nodes and the presence of lnks n the network. An evolutonary system named EPNet can also be found for evolvng the neural networks. Rank-based selecton and fve mutatons were employed to modfy the network structure and connecton weghts. The fve mutatons are hybrd tranng, node deleton, connecton deleton, connecton addton and node addton. Hybrd tranng, based on a modfed back-propagaton wth adaptve learnng rate and smulated annealng, s the mutaton used to modfy the connecton weghts. The other four mutatons are used to grow and prune hdden node nodes and connectons of the network. Some other algorthms can also be found to evolve the network structure and connecton weghts smultaneously. In ths paper, a three-layer neural network wth swtches ntroduced n some lnks s proposed to facltate the tunng of the network structure. As a result, for a gven fully connected feedforward neural network, t may no longer be a fully connected network after learnng. Ths mples that the cost of mplementng the proposed neural network, n terms of hardware mplementaton and processng tme, can be reduced. The network structure and parameters wll be tuned smultaneously usng a 3

5 proposed mproved GA. As applcaton examples, the proposed neural network wth lnk swtches tuned by the mproved GA s used to estmate the number of sunspots [7-8] and realze an assocatve memory. The results wll be compared wth those obtaned by tradtonal feed-forward networks [] traned by () the standard GA wth arthmetc crossover and non-unform mutaton [-, 5] and () the back-propagaton wth momentum and adaptve learnng rate [30]. Ths paper s organzed as follows. In sesson II, the mproved genetc algorthm s presented. In sesson III, t wll be shown that the mproved GA performs more effcently than the standard GA [-, 5] based on some benchmark test functons [3-4, 6, 7]. In sesson IV, the neural network wth lnk swtches, and tunng of ts structure and parameters usng the mproved GA, wll be presented. Applcaton examples wll be presented n sesson V. A concluson wll be drawn n sesson VI. II. IMPROVED GENETIC ALGORITHM Genetc algorthms (GAs) are powerful searchng algorthms. The standard GA process [-, 5] s shown n Fg.. Frst, a populaton of chromosomes s created. Second, the chromosomes are evaluated by a defned ftness functon. Thrd, some of the chromosomes are selected for performng genetc operatons. Forth, genetc operatons of crossover and mutaton are performed. The produced offsprng replace ther parents n the ntal populaton. In ths reproducton process, only the selected parents n the thrd step wll be replaced by ther correspondng offsprng. Ths GA process repeats untl a user-defned crteron s reached. In ths paper, the standard GA s modfed and new genetc operators are ntroduced to mprove ts performance. The mproved GA process s shown n Fg.. Its detals wll be gven as follows. A. Intal Populaton The ntal populaton s a potental soluton set P. The frst set of populaton s usually generated randomly. 4

6 pop_ sze P p, p,, p () no _ vars p p p p p, =,,, pop_sze; =,,, no_vars () para mn max p para (3) where pop_sze denotes the populaton sze; no_vars denotes the number of varables to be tuned; p, =,,, pop_sze; =,,, no_vars, are the parameters to be tuned; para mn and para max are the mnmum and maxmum values of the parameter p for all. It can be seen from () to (3) that the potental soluton set P contans some canddate solutons p (chromosomes). The chromosome p contans some varables p (genes). B. Evaluaton Each chromosome n the populaton wll be evaluated by a defned ftness functon. The better chromosomes wll return hgher values n ths process. The ftness functon to evaluate a chromosome n the populaton can be wrtten as, ftness f p ) (4) ( The form of the ftness functon depends on the applcaton. C. Selecton Two chromosomes n the populaton wll be selected to undergo genetc operatons for reproducton by the method of spnnng the roulette wheel []. It s beleved that hgh potental parents wll produce better offsprng (survval of the best ones). The chromosome havng a hgher ftness value should therefore have a hgher chance to be selected. The selecton can be done by assgnng a probablty q to the chromosome p : 5

7 q f ( p ) pop_ sze f ( p ), =,,, pop_sze (5) The cumulatve probablty qˆ for the chromosome p s defned as, qˆ q, =,,, pop_sze (6) The selecton process starts by randomly generatng a nonzero floatng-pont number, 0 Then, the chromosome p s chosen f qˆ d qˆ, =,,, pop_sze, and q ˆ0 0 d.. It can be observed from ths selecton process that a chromosome havng a larger f( p ) wll have a hgher chance to be selected. Consequently, the best chromosomes wll get more offsprng, the average wll stay and the worst wll de off. In the selecton process, only two chromosomes wll be selected to undergo the genetc operatons. D. Genetc Operatons The genetc operatons are to generate some new chromosomes (offsprng) from ther parents after the selecton process. They nclude the crossover and the mutaton operatons.. Crossover The crossover operaton s manly for exchangng nformaton from the two parents, chromosomes p and p, obtaned n the selecton process. The two parents wll produce one offsprng. The detals of the crossover operaton are as follows. Frst, four chromosomes wll be generated accordng to the followng mechansms, os os os p os os _ vars (7) p c os no os os os p ( w) max p, p w (8) c no_ vars max os os os p ( w) mn p, p w (9) 3 c no_ vars mn 6

8 os ( p max p mn )( w) ( p p ) os os osno_ vars (0) w 4 c no_ vars max p para para para () max max max no_ vars mn p para para para () mn mn mn where w 0 denotes the weght to be determned by users, max p p, denotes the vector wth each element obtaned by takng the maxmum among the correspondng element of p and p. For nstance, max 3, Smlarly, mn p mnmum value. For nstance, mn 3, 3 p, gves a vector by takng the. Among os c to 4 os c, the one wth the largest ftness value s used as the offsprng of the crossover operaton. The offsprng s defned as, os os os os os os no _ vars c (3) os denotes the ndex whch gves a maxmum value of f os, =,,,3,4. The offsprng generated by the crossover operaton wll undergo the mutaton operaton. If the crossover operaton can provde a good offsprng, a hgher ftness value can be reached n less teraton. In general, two-pont crossover, multpont crossover, arthmetc crossover or heurstc crossover can be employed to realze the crossover operaton [, 3-33]. The offsprng generated by these methods may not be better than that form our approach. As seen from (7) to (0), the offsprng spreads over the doman: (7) and (0) wll move the offsprng near centre regon of the concerned doman (as w n (0) approaches, 4 os c approaches p p boundary (as w n (8) and (9) approaches, c ), and (8) and (9) wll move the offsprng near the doman os c and 3 os c approaches p max and p mn respectvely).. Mutaton 7

9 The offsprng (3) wll then undergo the mutaton operaton. The mutaton operaton s to change the genes of the chromosomes. Consequently, the features of the chromosomes nherted from ther parents can be changed. Three new offsprng wll be generated by the mutaton operaton: nos os os os b nos b nos b nos no_ var s no_ var s where b, =,,, no_vars, can only take the value of 0 or, no_ var s, =,, 3 (4) nos, =,,, no_vars, are randomly generated numbers such that para mn max os nos para. The frst new offsprng ( = ) s obtaned accordng to (4) wth that only one b ( beng randomly generated wthn the range) s allowed to be and all the others are zeros. The second new offsprng s obtaned accordng to (4) wth that some randomly chosen b are set to be and others are zero. The thrd new offsprng s obtaned accordng to (4) wth all b =. These three new offsprng wll then be evaluated usng the ftness functon of (4). A real number wll be generated randomly and compared wth a user-defned number p 0 a. If the real number s smaller than p a, the one wth the largest ftness value f l among the three new offsprng wll replace the chromosome wth the smallest ftness f s n the populaton. If the real number s larger than p a, the frst offsprng wll replace the chromosome wth the smallest ftness value f s n the populaton f f l f s ; the second and the thrd offsprng wll do the same. p a s effectvely the probablty of acceptng a bad offsprng n order to reduce the chance of convergng to a local optmum. Hence, the possblty of reachng the global optmum s kept. In general, varous methods lke boundary mutaton, unform mutaton or non-unform mutaton [, 3-33] can be employed to realze the mutaton operaton. Boundary mutaton s to change the value of a randomly selected gene to ts upper or lower bound. Unform mutaton s to change the value of a randomly selected gene to a value between ts upper and lower bounds. Non-unform mutaton s capable of fne-tunng the parameters. The value of a randomly selected gene wll be ncreased or decreased by a weghted random number. The weght s usually a monotonc decreasng functon of the number of teratons. In our approach, we have three offsprng generated n the mutaton 8

10 process. From (4), the frst mutaton s n fact a unform mutaton. The second mutaton allows some randomly selected genes to change smultaneously. The thrd mutaton changes all genes smultaneously. The second and the thrd mutatons allow multple genes to be changed. Hence, the doman to be searched s larger as compared wth a doman characterzed by changng a sngle gene. As the ntal values are generated randomly, the genes wll have a larger space for mprovng the ftness value when the ftness value s small. On the contrary, when the ftness values are large and nearly steady, changng the value of a sngle gene (the frst mutaton) may be enough as some genes may have reached the optmal values. After the operaton of selecton, crossover, and mutaton, a new populaton s generated. Ths new populaton wll repeat the same process. Such an teratve process can be termnated when the result reaches a defned condton, e.g., the change of the ftness values between the current and the prevous teraton s less than 0.00, or a defned number of teratons s reached. III. BENCHMARK TEST FUNCTIONS Some benchmark test functons [3-4, 6, 7] are used to examne the applcablty and effcency of the mproved GA. Sx test functons, f (x), =,, 3, 4, 5, 6 wll be used, where x x x x n. n s an nteger denotng the dmenson of the vector x. The sx test functons are defned as follows, n ) x f (x, 5. x 5. (5) where n = 3 and the mnmum pont s at f (0, 0, 0) = 0 x x x n ( ) 00 f x,.048 x. 048 (6) where n = and the mnmum pont s at f (0, 0) = 0. f n 3 ) 6n floor ( x ) (x, 5. x 5. (7) 9

11 where n = 5 and the mnmum pont s at f 3 ([5., 5],, [5., 5]) = 0. The floor functon, floor(), s to round down the argument to an nteger. f n 4 4 ) x Gauss(0,) (x,.8 x. 8 (8) where n = 3 and the mnmum pont s at f 4 (0, 0, 0) = 0. Gauss(0, ) s a functon to generate unformly a floatng-pont number between 0 and nclusvely. 5 f5 ( x ), x (9) k 6 x a where a a , 3 k = 500 and the maxmum pont s at f 5 (3, 3). n x 0cos x f 6 ( x ) 0, 5. x 5. (0) where n = 3 and the mnmum pont s at f 6 (0, 0, 0) = 0. It should be noted that the mnmum values of all functons n the defned doman are zero except for f 5( x). The ftness functons for f to f 4 and f 6 are defned as, ftness, =,, 3, 4, 6. () ( x) f and the ftness functon for f 5 s defned as, ftness f 5 ( x) () The proposed GA goes through these 6 test functons. The results are compared wth those obtaned by the standard GA wth arthmetc crossover and non-unform mutaton [, 3-33]. For each test functon, the smulaton takes 500 teratons and the populaton sze s 0 for the proposed and the 0

12 standard GAs. When the standard GA s used, the probablty of crossover s set at 0.8 for all functons, and the probablty of mutaton for functons f to f 6 are 0.8, 0.8, 0.7, 0.8, 0.8, 0.35 respectvely. The shape parameters b of the standard GA [] for non-unform mutaton, whch s selected by tral and error through experments for good performance, are set at b = 5 for f, f and f 5, b = 0. for f 3, b = for f 4 and f 6. For the proposed GA, the values of w are set to be 0.5, 0.99, 0., 0.5, 0.0 and 0.0 for the sx test functons respectvely. The probablty of acceptance p a s set at 0. for all functons. These values are selected by tral and error through experments for good performance. The ntal values of x n the populaton for a test functon are set to be the same for both the proposed and the standard GAs. For tests to 6, the ntal values are, ,, , 0 0 and respectvely. The results of the average ftness values over 00 tmes of smulatons based on the proposed and standard GAs are shown n Fg. 3 and tabulated n Table I. Generally, t can be seen that the performance of the proposed GA s better than that of the standard GA. IV. NEURAL NETWORK WITH LINK SWITCHES AND TUNING USING THE IMPROVED GA In ths secton, a neural network wth lnk swtches s presented. By ntroducng a swtch to a lnk, the parameters and the structure of the neural network can be tuned usng the mproved GA. A. Neural Network wth Lnk Swtches Neural networks [5] for tunng usually have a fxed structure. The number of connectons has to be large enough to ft a gven applcaton. Ths may cause the neural network structure to be unnecessarly complex and ncrease the mplementaton cost. In ths secton, a multple-nput-multple-output three-layer neural network s proposed as shown n Fg. 4. The man dfferent pont s that a unt step functon s ntroduced to each lnk. Such a unt step functon s defned as,

13 0 f 0 ( ), (3) f 0 Ths s equvalent to addng a swtch to each lnk of the neural network. Referrng to Fg. 4, the nput-output relatonshp of the proposed multple-nput multple-output three-layer neural network s as follows, y ( s ) v z ( t) ( s ) b ( s ) b nh nn k ( t) ( s k) wklogsg k k (t) z, k =,,, n out (4), =,,, n n, are the nputs whch are functons of a varable t; n n denotes the number of nputs; n h denotes the number of the hdden nodes; w k, =,,, h n ; k =,,, n out, denote the weght of the lnk between the -th hdden node and the k-th output; v denotes the weght of the lnk between the -th nput and the -th hdden node; s denotes the parameter of the lnk swtch from the -th nput to the -th hdden node; s k denotes the parameter of the lnk swtch from the -th hdden node to the k-th output; n out denotes the number of outputs of the proposed neural network; denote the bases for the hdden nodes and output nodes respectvely; s and b and b k s k denote the parameters of the lnk swtches of the bases to the hdden and output layers respectvely; logsg() denotes the logarthmc sgmod functon: logsg ( ), (5) e (t) y k the weghts, k =,,, n out, s the k-th output of the proposed neural network. By ntroducng the swtches, w k and v, and the swtch states can be tuned. It can be seen that the weghts of the lnks govern the nput-output relatonshp of the neural network whle the swtches of the lnks govern the structure of the neural network. B. Tunng of the Parameters and Structure

14 The proposed neural network can be employed to learn the nput-output relatonshp of an applcaton usng the mproved GA. The nput-output relatonshp s descrbed by, d d y ( t) g z ( t), t =,,, n d (6) d d d d d d d d where z t) z ( t) z ( t) z ( ) and t) y ( t) y ( t) y ( ) ( n t n y are the gven ( n t nputs and the desred outputs of an unknown nonlnear functon g () respectvely. n d denotes the number of nput-output data pars. The ftness functon s defned as, out ftness err (7) err d n t out k n y d k ( t) y n d k ( t) (8) The obectve s to maxmze the ftness value of (7) usng the mproved GA by settng the chromosome to be s w s v s b s b for all,, k. It can be seen from (7) and k k (8) that a larger ftness value mples a smaller error value. k k V. APPLICATION EXAMPLES Two applcaton examples wll be gven n ths secton to llustrate the merts of the proposed neural networks tuned by the mproved GA. A. Forecastng of the Sunspot Number An applcaton example on forecastng of the sunspot number [7-8, 7] wll be gven n ths secton. The sunspot numbers from 700 to 980 are shown n Fg. 5. The cycles generated are non-lnear, non-statonary, and non-gaussan whch are dffcult to model and predct. We use the proposed 3-layer neural network (3-nput-sngle-output) wth lnk swtches for the sunspot number d ( forecastng. The nputs, z, of the proposed neural network are defned as z t) y ( t ), d z ( t) y ( t ) d 3( and z t) y ( t 3) where t denotes the year and ( ) s the sunspot numbers at 3 y d t

15 the year t. The sunspot numbers of the frst 80 years (.e. 705 t 884 ) are used to tran the proposed neural network. Referrng to (4), the proposed neural network used for the sunspot forecastng s governed by, n h ( s) vz( t) ( s ) b ( s 3 y ( t) ( s ) wlogsg ) b (9) The value of n h s changed from 3 to 7 to test the learnng performance. The ftness functon s defned as follows, ftness err (30) 884 y ( t) y( t) err (3) 80 t705 d The mproved GA s employed to tune the parameters and structure of the neural network of (9). The obectve s to maxmze the ftness functon of (30). The best ftness value s and the worst one s 0. The populaton sze used for the mproved GA s 0; w = 0.9 and p a = 0. for all values of n h. The lower and the upper bounds of the lnk weghts are defned as 3 v, wk, b, b h n 3 n h and, s, s, s, s, =,,, 3; =,,, n h, k = [6]. The chromosomes used for the mproved GA are s w k s v s b sk b. The ntal values of all the lnk weghts between the nput and hdden layers are and those between the hdden and output layers are respectvely. The ntal values of the swtches are all 0.5. For comparson purpose, a fully connected 3-layer feed-forward neural network (3-nput--output) [] s traned by () the standard GA wth arthmetc crossover and non-unform mutaton [-, 5], and () back-propagaton wth momentum and adaptve learnng rate [30]. On the other hand, the proposed neural network s also traned wth the standard GA for comparson. For standard GA, the populaton sze s 0, the probablty of crossover s 0.8 and the probablty of mutaton s 0.. The shape parameters b of the standard GA wth arthmetc crossover and non-unform 4

16 mutaton, whch s selected by tral and error through experments for good performance, s set to be. For the back-propagaton wth momentum and adaptve learnng rate, the learnng rate s 0., the rato to ncrease learnng rate s.05, the rato to decrease the learnng rate s 0.7, the maxmum valdaton falures s 5, the maxmum performance ncrease s.04, and the momentum constant s 0.9. The ntal values of the lnk weghts are the same as those n the proposed neural network. For all approaches, the learnng processes are carred out by a personal computer wth a P4.4GHz CPU. The number of teratons for all approaches s 000. The tuned neural networks are used to forecast the sunspot number durng the years Fg. 6 shows the smulaton results of the forecastng usng the proposed neural network traned wth the mproved GA (dashed lnes) and the actual sunspot numbers (sold lnes). The number of hdden nodes n h s changed from 4 to 8. The smulaton results for the comparsons are tabulated n Table II and Table III. From Table II, t s observed that the proposed neural network traned wth the mproved GA provdes better results than those of the proposed neural network wth standard GA, the tradtonal feed-forward neural network traned wth standard GA and back-propagaton wth momentum and adaptve learnng rate n terms of accuracy (ftness values) and number of lnks. The tranng error (governed by (3)) and the forecastng error (governed by 980 t885 d y ( t) y ( t) 96 ) are tabulated n Table III. It can be observed from Table III that our approach performs better than the tradtonal approaches. Refer to Table III, the best result s obtaned when the number of hdden node s 6. The number of connected lnk s 8 after learnng (the number of lnks of a fully connected network s 3, ncludng the bas lnks). It s about 4.9% reducton of the number of lnks after learnng. The tranng error and the forecastng error n term of mean absolute error (MAE) are.5730 and respectvely. B. Assocatve Memory Another applcaton example on tunng an assocatve memory wll be gven n ths secton. In ths example, the assocatve memory, whch maps ts nput vector nto tself, has 0 nputs and 0 5

17 outputs. Thus, the desred output vector s ts nput vector. Referrng to (4), the proposed neural network used for the sunspot forecastng s reused, y ( s ) v z ( t) ( s ) b ( s ) b nh nn k ( t) ( s k) wklogsg k k, =,,, 0, k =,,, 0 (3) 50 sets of nput vector (each nput vector has the property that z ( t) ) wll be employed to tran the proposed neural network. The value of n h s changed from 4 to 8 to test the learnng performance. The ftness functon s defned as follows, ftness err (33) zk ( t) yk ( t) err 00 (34) 0 t k 00 The mproved GA s employed to tune the parameters and structure of the neural network of (3). The obectve s to maxmze the ftness functon of (33). A larger value of the ftness functon ndcates a smaller value of err of (34). The best ftness value s and the worst one s 0. The populaton sze used for the mproved GA s 0; w = 0.8 and p a = 0. for all values of n h. The lower and the upper bounds of the lnk weghts are defned as 3 v, w k, b, b k n h 3 n h and, s, s, s, s, =,,, 3; k k =,,, n h, k = 0 [6]. The chromosomes used for the mproved GA are s k w k s v s b s k b k. The ntal values of the lnk weghts are all zero. For comparson purpose, the proposed neural network traned by the standard GA (wth arthmetc crossover and non-unform mutaton [-, 5]), a fully connected 3-layer feed-forward neural networks (0-nput-0-output) [] traned by the standard GA and back-propagaton (wth momentum and adaptve learnng rate [30]) are used agan. For the standard GA, the populaton sze s 0, the probablty of crossover s 0.8 and the probablty of mutaton s The shape parameters b of the standard GA wth arthmetc crossover and non-unform mutaton, whch s selected by tral and error 6

18 through experments for good performance, s set to be 3. For the back-propagaton wth momentum and adaptve learnng rate, the learnng rate s 0., the rato to ncrease the learnng rate s.05, the rato to decrease the learnng rate s 0.7, the maxmum valdaton falures s 5, the maxmum performance ncrease s.04, and the momentum constant s 0.9. The ntal values of the lnks weghts are the same as those of the proposed approach. The number of teratons for all approaches s 500. The smulaton results are tabulated n Table IV. It can be seen from Table IV that the ftness values for dfferent n h by the standard GA (wth arthmetc crossover and non-unform mutaton) and the back-propagaton (wth momentum and adaptve learnng rate) are smlar to those by our approach, whch offer smaller networks. VI. CONCLUSION An mproved GA has been proposed n ths paper. By usng the benchmark test functons, t has been shown that the mproved GA performs more effcently than the standard GA. Besdes, by ntroducng a swtch to each lnk, a neural network that facltates the tunng of ts structure has been proposed. Usng the mproved GA, the proposed neural network s able to learn both the nput-output relatonshp of an applcaton and the network structure. As a result, a gven fully connected neural network can be reduced to a partly connected network after learnng. Ths mples a lower cost of mplementaton of the neural network. Applcaton examples on forecastng the sunspot numbers and tunng of an assocatve memory usng the proposed neural network traned wth the mproved GA have been gven. The smulaton results have been compared wth those obtaned by a tradtonal feed-forward network traned by () the standard GA wth arthmetc crossover and non-unform mutaton, and () the back-propagaton wth momentum and adaptve learnng rate. ACKNOWLEDGEMENT The work descrbed n ths paper was substantally supported by a Research Grant of Centre for 7

19 Multmeda Sgnal Processng, The Hong Kong Polytechnc Unversty (proect number A43). REFERENCES [] J.H. Holland, Adaptaton n natural and artfcal systems, Unversty of Mchgan Press, Ann Arbor, MI, 975. [] D.T. Pham and D. Karaboga, Intellgent optmzaton technques, genetc algorthms, tabu search, smulated annealng and neural networks, Sprnger, 000. [3] Y. Hanak, T. Hashyama and S. Okuma, Accelerated evolutonary computaton usng ftness estmaton, IEEE SMC '99 Conference Proceedngs of IEEE Internatonal Conference on Systems, Man, and Cybernetcs, vol., 999, pp [4] K.A. De Jong, An analyss of the behavor of a class of genetc adaptve systems, Ph.D. Thess, Unversty of Mchgan, Ann Arbor, MI., 975. [5] Z. Mchalewcz, Genetc Algorthm + Data Structures = Evoluton Programs, second, extended edton, Sprnger-Verlag, 994. [6] G.X. Yao and Y. Lu "Evolutonary Programmng made Faster," IEEE Trans., on Evolutonary Computaton, vol. 3, no., July 999, pp.8-0 [7] M. L, K. Mechrotra, C. Mohan and S. Ranka, Sunspot Numbers Forecastng Usng Neural Network, 5th IEEE Internatonal Symposum on Intellgent Control, 990. Proceedngs, pp.54-58, 990. [8] T.J. Cholewo, J.M. Zurada, Sequental network constructon for tme seres predcton, Internatonal Conference on Neural Networks, vol.4, pp , 997. [9] B.D. Lu, C.Y. Chen and J.Y. Tsao, Desgn of adaptve fuzzy logc controller based on lngustc-hedge concepts and genetc algorthms, IEEE Trans. Systems, Man and Cybernetcs, Part B, vol. 3 no., Feb. 00, pp [0] Y.S. Zhou and L.Y. La Optmal desgn for fuzzy controllers by genetc algorthms, IEEE Trans., Industry Applcatons, vol. 36, no., Jan.-Feb. 000, pp

20 [] C.F. Juang, J.Y. Ln and C.T. Ln, Genetc renforcement learnng through symbotc evoluton for fuzzy controller desgn, IEEE Trans., Systems, Man and Cybernetcs, Part B, vol. 30, no., Aprl, 000, pp [] H. Judette and H. Youlal, Fuzzy dynamc path plannng usng genetc algorthms, Electroncs Letters, vol. 36, no. 4, Feb. 000, pp [3] R. Caponetto, L. Fortuna, G. Nunnar, L. Occhpnt and M. G. Xbla, Soft computng for greenhouse clmate control, IEEE Trans., Fuzzy Systems, vol. 8, no. 6, Dec. 000, pp [4] M. Setnes and H. Roubos, GA-fuzzy modelng and classfcaton: complexty and performance, IEEE. Trans, Fuzzy Systems, vol. 8, no. 5, Oct. 000, pp [5] K. Belarb, and F. Ttel Genetc algorthm for the desgn of a class of fuzzy controllers: an alternatve approach, IEEE Trans., Fuzzy Systems, vol. 8, no. 4, Aug. 000, pp [6] M. Brown and C. Harrs, Neuralfuzzy adaptve modelng and control, Prentce Hall, 994. [7] S. Amn and J.L. Fernandez-Vllacanas, Dynamc Local Search, Second Internatonal Conference On Genetc Algorthms n Engneerng Systems: Innovatons and Applcatons, pp9-3, 997. [8] Xn Yao, Evolvng Artfcal Networks, Proceedngs of the IEEE, vol. 87, no. 7, pp , 999. [9] Xn Yao and Young Lu, A new Evolutonary System for Evolvng Artfcal Neural Networks, IEEE Tran. On Neural Networks, vol. 8, no. 3, pp , 997. [0] Faa-Jeng Ln, Chh-Hong Ln, Po-Hung Shen, Self-constructng fuzzy neural network speed controller for permanent-magnet synchronous motor drve, IEEE Trans., Fuzzy Systems, vol. 9, no. 5, pp , Oct. 00. [] Y. Hrose, K. Yamashta and S. Hya, Back-propagaton algorthm whch vares the number of hdden unts, Neural Networks, vol. 4, no., pp. 6-66, 99. [] A. Roy, L.S. Km and S. Mukhopadhyay, A polynomal tme algorthm for the constructon and 9

21 tranng of a class of multplayer perceptons, Neural Networks, vol. 6, no. 4, pp , 993. [3] Ncholas. K. Treadold and Tamas D. Gedeon, Explorng constructve Cascade Networks, IEEE Trans. Neural Networks, vol. 0, no. 6, pp , Nov, 999. [4] Chn-Ch Teng and Benamn W. Wah, Automated learnng for reducng the confguraton of a feedforward neural network, IEEE Trans. Neural Networks, vol. 7, no. 5, pp , Sep, 996. [5] Y.Q. Chen, D.W. Thomás and M.S. Nxon, Generatng-shrnkng algorthm for learnng arbtrary classfcaton, Neural Networks, vol. 7 no. 9, pp , 994. [6] M.C. Moze and P. Smolensky, Usng relevance to reduce network sze automatcally, Connect. Sc., vol., no., pp. 3-6, 989. [7] H.K. Lam, S.H. Lng, F.H.F. Leung and P.K.S. Tam, Tunng of the Structure and Parameters of Neural Network usng an Improved Genetc Algorthm, Proceedngs of the 7th Annual Conference of the IEEE Industral Electroncs Socety, IECON 00, Denver, Nov, 00, pp [8] G.P. Mller, P.M. Todd and S.U. Hegde, Desgnng neural networks usng genetc algorthms, Proc. 3 rd Int. Conf. Genetc Algorthms and Ther Applcatons, J. D. Schaffer, Ed. San Mateo, CA: Morgan Kaufmann, 989, pp [9] N. Weymaere and J. Martens, On the ntalzaton and optmzaton of multplayer perceptrons, IEEE Trans. Neural Networks, vol. 5, pp , Sept [30] Haykn, Smon S., Neural networks: A comprehensve foundaton, nd Ed., Upper Saddle Rver, N. J.: Prentce Hall, 999. [3] Xufeng Wang and M., Elbuluk, Neural network control of nducton machnes usng genetc algorthm tranng, Industry Applcatons Conference, 996. Thrty-Frst IAS Annual Meetng, IAS '96., Conference Record of the 996 IEEE, vol. 3, 996, pp [3] Lawrence Davs, Handbook of genetc algorthms, Van Nostrand Renhold, New York, 99. [33] M. Srnvas, L.M. Patnak Genetc algorthms: a survey, IEEE Computer, vol. 7, ssue 6, pp. 0

22 7-6, June 994. [34] J.D. Schaffer, D. Whtley, and L.J. Eshelman Combnatons of genetc algorthms and neural networks: a survey of the state of the art, Proceedngs of Internatonal Workshop on Combnatons of Genetc Algorthms and Neural Networks, COGANN-9, 99, pp -37. [35] S. Bornholdt and D. Graudenz General asymmetrc neural networks and structure desgn by genetc algorthms: a learnng rule for temporal patterns, Conference proceedngs of Internatonal Conference on Systems, Man and Cybernetcs, 993, vol., 993, pp [36] V. Manezzo, Genetc evoluton of the topology and weght dstrbuton of neural networks, IEEE Trans. Neural Networks, vol.5, ssue, pp , Jan [37] P.J. Angelne, G.M. Saunders and J.B. Pollack, An evolutonary algorthm that constructs recurrent neural networks, IEEE Trans. Neural Networks, vol.5, ssue, pp , Jan. 994

23 Procedure of the standard GA begn 0 // : teraton generaton ntalze P() //P(): populaton for teraton t evaluate f(p()) // f(p()):ftness functon whle (not termnaton condton) do begn + select parents p and p from P(-) perform genetc operatons (crossover and mutaton) reproduce a new P() evaluate f(p()) end end end Fg.. Procedure of standard GA. Procedure of the mproved GA begn 0 // : teraton ntalze P() //P(): populaton for teraton t evaluate f(p()) // f(p()):ftness functon whle (not termnaton condton) do begn + select parents p and p from P(-) perform crossover operaton accordng to equatons (7) to (3) perform mutaton operaton accordng to equaton (4) to three offsprng nos, nos and nos3 // reproduce a new P() f random number < pa // pa: probablty of acceptance The one among nos, nos and nos3 wth the largest ftness value replaces the chromosome wth the smallest ftness value n the populaton else f f(nos) > smallest ftness value n the P(-) nos replaces the chromosome wth the smallest ftness value end f f(nos) > smallest ftness value n the updated P(-) nos replaces the chromosome wth the smallest ftness value end f f(nos3) > smallest ftness value n the updated P(-) nos3 replaces the chromosome wth the smallest ftness value end end Fg.. Procedure of the mproved GA.

24 Ftness Value Ftness Value No. of Iteratons (a). The averaged ftness value of the test functon f ( x) obtaned by the mproved (sold lne) and standard (dotted lne) GAs No. of Iteratons (b). The averaged ftness value of the test functon f ( x) obtaned by the mproved (sold lne) and standard (dotted lne) GAs. 3

25 Ftness Value Ftness Value No. of Iteratons (c). The averaged ftness value of the test functon f 3( x) obtaned by the mproved (sold lne) and standard (dotted lne) GAs No. of Iteratons (d). The averaged ftness value of the test functon f 4 ( x) obtaned by the mproved (sold lne) and standard (dotted lne) GAs. 4

26 Ftness Value Ftness Value No. of Iteratons (e). The averaged ftness value of the test functon f 5 ( x) obtaned by the mproved (sold lne) and standard (dotted lne) GAs No. of Iteratons (f). The averaged ftness value of the test functon f 6 ( x) obtaned by the mproved (sold lne) and standard (dotted lne) GAs. Fg. 3. Smulaton results of the mproved and standard GAs. 5

27 s Logsg w s z v s w n out y v n h Logsg n out s v n n z n n s n n n h w n h b s n h Logsg b s n out y n out Logsg swtch Fg. 4. Proposed 3-layer neural network Fg. 5. Sunspot numbers from year 700 to

28 Sunspot numbers Sunspot numbers Year (a) Number of hdden nodes ( n h ) = Year (b). Number of hdden nodes ( n h ) = 5. 7

29 Sunspot numbers Sunspot numbers Year (c). Number of hdden nodes ( n h ) = Year (d). Number of hdden nodes ( n h ) = 7. 8

30 Sunspot numbers Year (e). Number of hdden nodes ( n h ) = 8. Fg. 6. Smulaton results of a 96-year predcton usng the proposed neural network wth the proposed GA (dashed lne) and actual sunspot numbers (sold lne) for the years Test functon Proposed GA Standard GA f ( ) x f ( ) x f ( ) x f ( ) x f ( ) x f 6 ( x) Table I. Smulaton results of the proposed GA and the standard GA based on the benchmark test functons. 9

31 Our Approach Standard GA wth proposed neural network n h Ftness Values Number of Lnks Ftness Values Number of Lnks (a) Standard GA wth tradtonal neural network Back-Propagaton wth Momentum and Adaptve Learnng Rate n h Ftness Values Number of Lnks Ftness Values Number of Lnks (b) Table II. Smulaton results for the applcaton example of forecastng the sunspot number after 000 teratons of learnng. Our Approach Standard GA wth proposed neural network n Tranng error Forecastng error Tranng error Forecastng error h (a) Standard GA wth tradtonal neural network Back-Propagaton wth Momentum and Adaptve Learnng Rate n Tranng error Forecastng error Tranng error Forecastng error h (b) Table III. Tranng error and forecastng error n mean absolute error (MAE) for the applcaton example on forecastng the sunspot number. 30

32 Our Approach Standard GA wth proposed neural network n h Ftness Values Number of Ftness Values Number of Lnks Lnks (a) Standard GA wth tradtonal neural network Back-Propagaton wth Momentum and Adaptve Learnng Rate n h Ftness Values Number of Ftness Values Number of Lnks Lnks (b) Table IV. Smulaton results for the applcaton example of assocatve memory after 500 teratons of learnng. 3