HIERARCHICAL RANK DENSITY GENETIC ALGORITHM FOR RADIAL-BASIS FUNCTION NEURAL NETWORK DESIGN

Transcription

1 HIERARCHICAL RANK DEITY GENETIC ALGORITHM FOR RADIAL-BASIS FUNCTION NEURAL NETWORK DESIGN Gary G. Yen Hamng Lu Intellgent Systems and Control Laboratory School of Electrcal and Computer Engneerng Oklahoma State Unversty Stllwater, OK Abstract In ths paper, we propose a genetc algorthm based desgn procedure for a radal-bass functon neural network. A Herarchcal Rank Densty Genetc Algorthm (HRDGA) s used to evolve both the neural network s topology and parameters. In addton, the rank-densty based ftness assgnment technque s used to optmze the performance and topology of the evolved neural network to deal wth the conflcton between the tranng performance and network complexty. Instead of producng a sngle optmal network, HRDGA provdes a set of near-optmal neural networks to the desgners or the decson makers so that they can have more flexblty for the fnal decson-makng based on ther preferences. In terms of searchng for a near-complete set of canddate networks wth hgh performances, the networks desgned by the proposed algorthm prove to be compettve, or even superor, to three selected tradtonal radal-bass functon networks for predctng Mackey-Glass chaotc tme seres. I. INTRODUCTION Snce the orgnal emergence of Artfcal Neural Networks n 1940 s, there has been an extensve qualtatve and quanttatve analyss on dfferent classes of neural networks possessng varous archtectures and tranng algorthms. Wthout a proven gudelne, the desgn of an optmal neural network for a gven problem s often regarded as an ad hoc process. Gven a suffcent of neurons, more than one neural network structure (.e., wth dfferent weghtng coeffcents and s of neurons) can be traned to solve a gven problem wthn an error bound f gven enough tranng tme. The decson of whch network s the best s often decded by whch network wll better meet the user s needs for a gven problem. It s known that the performance of neural networks s senstve to the of hdden neurons. Too few neurons can result n underfttng problems (poor approxmaton), whle too many neurons may contrbute to overfttng problems. Obvously, achevng a better network performance and smplfyng the network topology are two conflctng objectves. Ths has promoted research on how to dentfy an optmal and effcent neural network structure. AIC (Akake Informaton Crteron) [1] and PMDL (Predctve Mnmum Descrpton Length) [] are two well-adopted approaches. However, AIC can be nconsstent and has a tendency to overft a model, whle PMDL only succeeded n relatvely smple neural network structures and seemed very dffcult to extend to a complex NN structure optmzaton problem. Moreover, all of these approaches tend to produce a sngle neural network by each run, whch does not offer the desgners wth alternatve choces. Over the past decade, evolutonary algorthms have been successfully appled to the desgn of network topologes and the choce of learnng parameters [3]. They reported some encouragng results that are comparable wth conventonal neural network desgn approaches. However, the multobjectve trade-off characterstc of the neural network desgn has not been well studed and appled n the real world applcatons. In ths paper, we propose a Herarchcal Rank Densty Genetc Algorthm (HRDGA) for neural network desgn n order to evolve a set of near-optmal neural networks. Wthout loss of generalty, we wll restrct our dscussons to the radal bass functon neural network. The remander of ths paper s organzed as follows. Secton II dscusses the neural network desgn dlemma and the dffculty of fndng a sngle optmal neural network. Secton III revews a Herarchcal Genetc Algorthm based Neural Network (HGA-NN) desgn approach and apples herarchcal genotype representaton to a Radal-Bass Functon (RBF) neural network desgn. Secton IV ntroduces the proposed rank-densty ftness assgnment technque for multobjectve genetc algorthms and descrbes HRDGA parameters and desgn procedure. Secton V presents a feasble applcaton on the Mackey-Glass chaotc tme seres predcton usng HRDGA evolved neural networks. A tme seres wth a specfc chaotc character s traned and the performance s compared wth those of the k-nearest neghbors, generalzed regresson and orthogonal least square tranng algorthms. Fnally, Secton VI provdes some concludng remarks along wth pertnent observatons. II. NEURAL NETWORK DESIGN DILEMMA To generate a neural network that possesses the practcal applcablty, several essental condtons need to be consdered. 1) A tranng algorthm that can search for the optmal parameters (.e., weghts and bases) for the specfed network structure and tranng task. ) A rule or algorthm that can determne the network complexty and ensure t to be suffcent for solvng the gven tranng problem. 3) A metrc or measure to evaluate the relablty and generalzaton of the produced neural network. The desgn of an optmal neural network nvolves all of these three problems. As gven n [4], the ultmate goal of the

2 constructon of a neural network wth the nput-output relaton y = f s the mnmzaton of the expectaton ( x, ω) of a cost functon g ( X, Y) as where ( T f E[ g ( f ( X, Y)] = g ( f ( x, y) f ( x, y dxdy (1) T T x, y ) f x,y ( x, y) denotes the jont pdf that depends on the nput vector x and the target output vector y. Gven a network structure, a famly of nput-output relatons F, parameterzed by ω, consstng of all = { f ( x, ω)} network functons that may be formed wth dfferent choces of the weghts can be assgned. The structure ' s sad to be domnated by " f F. In order to choose the ' F" optmal neural network, we need to fnd the determnaton of the network functon (x) (.e., the determnaton of the respectve weghts ω ) that gves the mnmal cost value wthn the famly F f f ( x) = f ( x, ω ) arg mn E[ g ( f ( X, Y)], () = ω and the determnaton of the network structure that realzes the mnmal cost value wthn a set of structures {} = arg mn E[ g ( f ( X), Y)]. (3) F T Obvously, the solutons of ths task need not result nto a unque network. In [5], f several structures 1, meet,l the crteron as shown n Equaton (3), the one wth the mnmal of hdden neurons s defned as an optmal. However, as a neural network can only tune the weghts by the gven tranng data sets, and these data sets are always fnte, there wll be a trade-off between NN learnng capablty and the varaton of the hdden neuron s. A network wth nsuffcent neurons mght not be able to approxmate well enough the functonal relatonshp between nput and target output. On the other hand, f the of neurons s excessve, the realzed network functon wll depend greatly on the resultng realzaton of the gven lmted tranng set. Ths trade-off characterstc mples that a sngle optmal neural network s very dffcult to fnd as extractng f from by usng a fnte tranng data set ( x) F s a dffcult task, f not mpossble [5]. Therefore, nstead of tryng to obtan a sngle optmal neural network, fndng a set of near-optmal networks wth dfferent network structures seems more feasble. Each ndvdual n ths neural network set may provde dfferent tranng and testng performances for dfferent tranng and testng data sets. Moreover, the dea of provdng a set of canddate networks to the decson makers can offer more flexbltes n selectng an approprate network judged by ther own preferences. For ths reason, genetc algorthms and multobjectve optmzaton technques can be ntroduced n neural network desgn problems to evolve network topology along wth parameters and present a set of alternatve canddates network. L III. RBF NEURAL NETWORK DESIGN In the lterature of usng genetc algorthms to assst neural networks desgn, several approaches have been proposed for evolvng NN structure together wth weghts and bases [3,6-7]. Among all these methods, we ncorporate a herarchcal genotype representaton nto an RBF neural network desgn. Herarchcal Genetc Algorthm (HGA) was frst proposed by Ke, et. al., [8] for fuzzy controller desgn usng two layer genes to evolve membershp. Based on ths dea, Yen and Lu [7] desgned an HGA Neural Network (HGA- NN). In the HGA-NN, a three-level HGA s used to evolve a Mult-layer Perceptron neural network. Each canddate chromosome corresponds to a neural network and the frst, second and thrd level genes represent the network layer, the neuron n each layer and the parameter values of each neuron, respectvely. By usng ths herarchcal genotype codng, the problem of one phenotype mappng dfferent genotypes can be prevented [7]. In a smlar sprt, HGA s talored n ths paper to evolve an RBF (Radal-Bass Functon) neural network. A radalbasc functon can be formed as m f ( x) = ω exp( x c ) (4) = 1 where c denotes the center of the th localzed functon, ω s the weghtng coeffcent connectng the th Gaussan neuron to the output neuron, and m s the of Gaussan neurons n the hdden layer. Wthout loss of generalty, we choose the varance as unty for each Gaussan neuron. In HGA based RBF neural network desgn, genes n the genotype are classfed nto three categores: control genes, weght genes and center genes. The lengths of these three knds of genes are the same. The value of each control gene (0 or 1) determnes the actvaton status (off or on) of the correspondng weght gene and center gene. The weght genes and center genes are represented by real values. Control genes and weght genes are randomly ntalzed and the center genes are randomly selected from gven tranng data samples. Fgure 1 shows the genotype and phenotype of HGA based RBF neural network. Control genes Weght genes Center genes Fgure 1 Genotype and Phenotype of HGA based RBF neural network IV. MULTIOBJECTIVE GENETIC ALGORITHMS As dscussed n Secton II, neural network desgn problems have a multobjectve trade-off characterstc n

3 terms of optmzng network topology and performance. Therefore, multobjectve genetc algorthm can be mplemented n NN desgn procedure. A. Multobjectve Genetc Algorthms (MOGAs) Snce the 1980 s, several Multobjectve Genetc Algorthms (MOGAs) have been proposed and appled n Multobjectve Optmzaton Problems (MOPs) [9]. These algorthms share the same purpose searchng for a unformly dstrbuted, near-optmal and near-complete famly of non-domnated ndvduals, a so-called Pareto front [9], whch descrbes the trade-off among contradcted objectves as shown n Fgure. For example, consderng the NN desgn dlemma ntroduced n Secton II, a neural network desgn problem can be regarded as a class of MOPs as mnmzng network structure and mprovng network performance, whch are two conflctng objectves. Therefore, searchng for a near-complete set of non-domnated and nearoptmal canddate networks as the desgn solutons (.e., Pareto front) s our goal. Second, to mantan the dversty of the obtaned Pareto front, we adopt an adaptve cell densty evaluaton scheme as shown n Fgure 3. The cell wdth n each objectve dmenson can be formed as d where K max f ( x) mn f ( x) x X x X =, 1,..., n K d s the wdth of the cell n =, (6) the th dmenson, denotes the of cells desgnated for the th dmens on (.e., n Fgure 4, K 1 and K 8 ), and X denotes the decson vector space. As the maxmum and mnmum ftness values n objectve space wll change wth dfferent generatons, the cell sze w ll vary from generat on to generaton to mantan the resoluton of the densty calculaton. The densty value of an ndvdual s defned as the of the ndvduals located n the same cell. f 1 = = f 1 Fgure 3 Densty map and densty grd Fgure Graphcal llustraton of the Pareto optmalty B. Rank-densty based ftness assgnment MOGA As MOGAs are desgnated to fnd a near-optmal and near complete set of Pareto solutons, the ftness assgnment scheme s qute dfferent from generc GAs, whch are desgned to search for a sngle optmal soluton. In ths paper, based on the herarchcal phenotype formulaton, we propose a new rank-densty based ftness assgnment scheme n a multobjectve genetc algorthm to assst neural network desgn. Three essental steps were appled n ths technque. Frst, an Automatc Accumulated Rankng Strategy (AARS) s appled to calculate the Pareto rank value, whch represents the domnated relatonshp among ndvduals. In AARS, assume at generaton t, ndvdual y s domnated by (t ) p ndvduals,, y 1 ( t ) p y L, whose rank values are already kn own as r( y ), L, r(y ), t). Its rank value can be computed by r( y, t) = 1+ ( t ) p j = 1, t 1 ( t p r( y j, t). (5) Therefore, by AARS, ndvdual s densty nformaton s ncluded n ts rank value, thus the populaton dversty wll be kept by penalzng those domnated ndvduals. Thrd, because rank and densty values represent ftness and populato n dversty, respectvely, the new rank-densty ftness formulaton can convert any multobjectve optmzaton problem nto a b-objectve optmzaton problem. Here, populaton rank and densty values are desgnated as the two ftness values for GA to mnmze. Before ftness evaluaton, the entre populaton s dvded nto two subpopulatons wth equal szes; each subpopulaton s flled wth ndvduals that are randomly chosen from the current populaton accordng to rank and densty value, respectvely. Afterwards, the entre populaton s shuffled, and crossover and mutaton are then performed. Meanwhle, snce we take the mnmzaton of the populaton densty value as one of the objectves, t s expected that the entre populaton wll move toward an opposte drecton to the Pareto front when the populaton densty value s beng mnmzed. Although movng away from the true Pareto front can reduce populaton densty value, obvously, these ndvduals are harmful to the populaton to converge to the Pareto front. To prevent harmful offsprng survvng and affectng the evolutonary drecton and speed, a forbdden regon concept s proposed n the replacement scheme for the densty subpopulaton, thereby preventng the backward effect. The forbdden regon ncludes all the cells domnated by the selected parent. The offsprng located n the forbdden regon wll not survve n the next generaton, and thus the selected parent wll not be replaced. As shown n Fgure 4, assumng our goal s to mnmze objectves f 1 and f, and a resultng offsprng of the selected parent p s located n the

4 forbdden regon, thus ths offsprng wll be elmnated even f t reduces the populaton densty, because ths knd of offsprng has the tendency to push the entre populaton away from the desred evolutonary drecton. rank and densty ftness values of each ndvdual wll be evaluated. After crossover, the offsprng replaces the low ftness parents and a new generaton s formed. Matng s then teratvely processed. 4) Stoppng crtera When the desred of generatons s met, the evolutonary process stops. V. TIME SERIES PREDICTION The cell where the selected parent p s located Vald range where parent p s offsprng can be located Forbdden regon where parent p s offsprng cannot be located Fgure 4 Illustraton of the vald range and the forbdden regon Fnally, The smple eltsm scheme [9] s also appled for bookkeepng the Pareto ndvduals obtaned n each generaton. These ndvduals are compared to acheve the fnal Pareto front after the evoluton process has stopped. C. HRDGA for NN desgn To assst RBF network desgn, the proposed Herarchcal Ran k Densty based Genetc Algorthm (HRDGA) s appled to carry out the ftness evaluaton and matng selecton schemes. The HRDGA operators are desgned as follows. 1) Chromosome representaton In HRDGA, each ndvdual (chromosome) represents a canddate neural network. The control genes are bnary bts (0 or 1). For the weght and center genes, real values are adopted as the gene representaton to reduce the length of the chromosome. The populaton sze s fxed and chosen ad hoc by the dffculty of the problem to be solved. ) Crossover and mutaton We used one-pont crossover n the control gene segments and two-pont crossover n the other two gene segments. The crossover ponts were randomly selected, and the crossover rates were chosen to be 0.8, 0.7 and 0.7 for the control, weght and center genes, respectvely. One-pont mutaton was appled n each segment. In the control gene segment, common bnary value mutaton was adopted. In the weght and center gene segments, real value mutaton was performed by addng a Gaussan (0,1), whch denotes a Gaussan functon wth zero mean and unt varance. The mutaton rates were set to be 0.1, 0.05 and 0.05 for the control, weght and center genes, respectvely. 3) Ftness evaluatons and matng selecton Snce we are tryng to use HRDGA to optmze the neural network topology along wth ts performance, we need to convert them nto the rank-densty doman. Therefore, the orgnal ftness network performance and of neurons of each ndvdual n a generaton s evaluated and ranked, and the densty value s calculated. Then the new Snce the proposed HRDGA was desgned to evolve the neural network topology together wth ts best performance, t proves useful n solvng complex problems such as tme seres predcton or pattern classfcaton. For a feasblty check, we use the HRDGA asssted NN desgn to predct a Mackey-Glass chaotc tme seres: d( x( t)) a x( t τ) = b x( t) (7) c d( t) (1 + x ( t τ)) where τ = 150, a = 0., b = 0. 1 and c = 10. The network s set to predct x( t + 6) based on x( t), x( t 6), x( t 1) and x( t 18 ). In the proposed HRDGA, 150 ntal center genes are selected, 150 control genes a nd 150 weght genes are ntally generated as well. Populaton sze was set to be 400. For comparson, we appled three other center selecton methods KNN (K-Nearest Neghbor) [10], GRNN (Generalzed Regresson Neural Network) [11] and OLS (Orthogonal Least Square Error) [1] on the same tme seres predcton problem. For KNN and GRNN types of networks, 70 networks are generated wth the neuron s ncreasng from 11 to 80 wth the step sze equals to one. Each of these networks wll be traned by KNN and GRNN methods, and the stop crtera s the same wth the one we appled n HRDGA. For the OLS method, the selecton of the tolerance parameter ρ determnes the trade-off between the performance and complexty of the network. A smaller ρ value wll produce a neural network wth more neuron s, whereas a larger ρ generally results n a network wth less of neurons. Therefore, by usng dfferent ρ values, we generated a group of neural networks wth varous tranng performances and s of hdden neurons. For the gven Mackey-Glass tme seres predcton problem, w e selected 40 dfferent ρ values, whch are from 0.01 to 0.4 wth the step sze of Optmal k value n KNN s determned accordng to reference [10]. The stop crtera for KNN, GRNN and OLS algorthms s ether the epochs exceeds 5,000, or the tranng Sum Square Error () between two sequental generatons s smaller than For HRDGA, the stoppng generaton s set to be 5,000. We used the frst 50 seconds of the data as the tranng data set, and then the data from , , and seconds were used as the testng data sets (labeled #1, # and #3) to be predcted by four dfferent approaches. Each

5 approach runs 30 tmes wth dfferent parameter ntalzatons to obtan the average results. Fgure 5(a)-(d) shows the average s of tranng data set and three testng data sets by the resultng neural networks wth dfferent of hdden neurons. Fgure 6(a)-(d) shows the correspondng approxmated Pareto fronts (.e., nondomnated sets) by the selected four approaches. Table 1 shows the best tranng and testng performances and ther correspondng s of hdden neurons over 30 runs. From Fgures 5 6, we can see, comparng to KNN and GRNN, HRDGA and OLS algorthms have much smaller tranng and testng errors for the same network structures. KNN traned networks produce the worst performances, because the RBF centers of the KNN algorthm are randomly selected, whch make KNN to acheve a local optmum soluton. Snce GA always seeks global optmum, and the orthogonal result s near optmal, the performances of OLS are comparable to HRDGA. (a) Tranng data set (a) Tranng data set (b) Testng data set #1 (b) Testng data set #1 (c) Testng data set # (c) Testng data set # (d) Testng data set #3 Fgure 5 Tranng and testng performances for the resultng neural networks wth dfferent of hdden neurons (d) Testng data set #3 Fgure 6 The correspondng Pareto fronts (non-domnated sets)

6 Table1 - Structure and performance comparson among four algorthms for Tranng set for Testng set #1 for Testng set # for Testng set #3 KNN GRNN OLS HRDGA Moreover, from Fgure 5, we can see that for all tranng algorthms, when the network complexty ncreases, the tranng error decreases. However, ths phenomenon s only partally mantaned for the relatonshp between the testng performances and the network complexty. Before the of hdden neurons reaches a certan threshold, the testng error decreases as the network complexty ncreases. After that, the testng error has the tendency to fluctuate even when the of hdden neurons ncreases. Ths occurrence can be consdered as that the resultng networks are overftted. The network wth the best testng performance before overfttng occurs s called the optmal network and judged as the fnal sngle soluton by conventonal NN desgn algorthms [4]. However, from Fgures 5 6 and Table 1, t s very dffcult to fnd a sngle optmal network that can offer the best performances for all the testng data sets, snce these data sets possess dfferent trats. Therefore, nstead of searchng for a sngle optmal neural network, HRDGA can be a more reasonable and applcable opton snce t results n a near-complete set of near-optmal networks.. Fgure 7 Relatonshp between ρ values and network complexty From the smulaton results, although KNN and GRNN provde worse tranng and testng results comparng to the other two approaches, they have the advantage that the desgner can control the network complexty by ncreasng or decreasng the neuron s at wll. On the other hand, although the OLS algorthm always provdes near-optmal network solutons wth good performances, the desgners cannot manage the network structure drectly. The trade-off characterstc between network performance and complexty totally depends on the value of tolerance parameter ρ. Same ρ value means completely dfferent trade-off features for dfferent NN desgn problems. In addton, as shown n Fgure 7, the relatonshp between ρ value and network topology s a nonlnear, many-to-one mappng, whch may cause a redundant computaton effo rt n order to generate a near-complete neural network soluton set. Compared wth the other three tranng approaches, HRDGA does not have problems n desgnng trade-off parameters, because t treats each objectve equally and ndependently, and ts populaton dversty preservng technques help to buld a near-unformly dstrbuted non-domnated soluton set. VI. CONCLUSION From the results presented above, HRDGA shows potental n estmatng neural network topology and weghtng parameters for complex problems when a heurstc estmaton of the neural network structure s not readly avalable. For the gven Mackey Glass chaotc tme seres predcton, HRDGA shows compettve, or even superor performances comparng wth the other three selected tranng algorthms n terms of search ng for a set of nondomnated, near-complete neural network solutons wth hgh-qualty tranng and testng performances. Whle we consdered radal-bass functon neural networks, the proposed herarchcal genetc algorthm may be easly extended to the desgns of other ne ural networks (.e., feedforward, feedback, or self-organzed). REFERENCES [1] N. Murata, S. Yoshzawa and S. A mar, Network nformaton crteron determnng the of hdden unts for an artfcal neural network model, IEEE Trans. Neural Networks, vol. 5, pp , [] X. M. Gao, S. J. Ovaska and Z. O. Hartmo, Speech sgnal restoraton usng an optmal neural network structur e, n Proc. IEEE Int. Conf. Neural Networks, pp , [3] X. Yao, Evolvng artfcal neural network, Internatonal Journal of Neural Systems, vol. 4, pp. 03-, [4] A. Doerng, M. Galck and H. Wtte, Structure optmzaton of neural networks wth the A-Algorthm, IEEE Trans. Neural Networks, vol. 8, pp , [5] S. Geman, E. Benenstock and R. Dousat, Neural networks and the bas/varance dlemma, Neural Comput, vol., pp , [6] B. Zhang and D. Cho, Evolvng neural trees for tme seres predcton usng Bayesan evolutonary algorthms, n Proc. 1 st IEEE Symp. Combnaton of Evolutonary Computaton and Neural Networks, pp. 17-3, 000. [7] G. G. Yen and H. Lu, Herarchcal genetc algorthm based neural network desgn, n Proc. 1 st IEEE Symp. Combnaton of Evolutonary Computaton and Neural Networks, pp , 000. [8] T. Y. Ke, K. S. Tang, K. F. Man and P. C. Luk, Herarchcal genetc fuzzy controller for a solar power plant, n Proc. IEEE Int. Symp. Industral Electroncs, pp , [9] C. M. Fonseca and P. J. Flemng, An overvew of evolutonary algorthms n multobjectve optmzaton, Evol. Comput., vol. 3, pp. 1-16, [10] T. Kaylan and S. Dasgupta, A new method for ntalzng radal bass functon classfers, n Proc. IEEE In t. Conf. Systems, Man, and Cybernetcs, pp , [11] P. D. Wasserman, Advanced Method n Neural Computng, New York: Van Nostrand Renhold, 1993 [1] S. Chen, C. F. Cowan and P. M. Grant, Orthogonal least square learnng algorthm for radal bass functon networks, IEEE Trans. Neural Networks, vol., pp , 1991.