A Multiple Evolutionary Neural Network Classifier Based on Niche Genetic Algorithm *

Transcription

1 Fourth Internatonal Conference on atural Comutaton A ultle Evolutonar eural etwork Classfer Based on che Genetc Algorthm * Jang Wu, hange Tang, Taong L,, Yue Jang, Shangu Ye, Xun Ln,, Je Zuo School of Comuter Scence, School of Economc Informaton Engneerng, Schuan Unverst, Southwestern Unverst of Fnance and Economcs, Chengduhna, Chengduhna, tangchange@cs.scu.edu.cn, wu_t@swufe.edu.cn Abstract Classfcaton s mortant n data mnng. In ths aer, a ultle Evolutonar eural etwork Classfer Based on che Genetc Algorthm (C- G s resented, whch establshes classfers b a grou of three-laer feed-forward neural networks wth hgh accurac and good dverst. The neural networks are traned b nche genetc algorthm based on clusterng. The class label of the dentfng data can frst be evaluated b each neural network, and the fnal classfcaton result s obtaned accordng to the dnamc votng rule. Exermental results on 6 data sets show that C-G ncreases the redctve accurac b 5.6%, 5.5% and 8.5% resectvel comared wth BP, GA and L tranng methods and b 6.0%, 6.% and 4.0% comared wth aïve Baesan classfer4.5 and SV. can frst be evaluated b each neural network, and the fnal classfcaton result s obtaned accordng to the dnamc votng rule. Exermental results show that C-G s more effectve n mrovng the redctve accurac than other tradtonal algorthms. The rest of ths aer s organzed as follows. Secton ntroduces the framework of C-G. Secton 3 analzes the rncle of classfer ensemble. Secton 4 descrbes the desgn of C-G. Secton 5 rovdes exermental evaluatons. Fnall Secton 6 concludes ths aer.. Framework of C-G. The framework of C-G s llustrated n Fgure. Introducton Classfcaton s ver mortant n data mnng. The oular classfcaton algorthms nclude decson tree [], Baesan classfer [], suort vector machne [3], neural network [4], etc. has been wdel used n classfcaton because of ts nose toleraton and modelng ablt n comlex roblems. In order to mrove the accurac of classfcaton, we roose a ultle Evolutonar eural etwork Classfer Based on che Genetc Algorthm (C-G. The algorthm establshes subclassfers b a grou of neural networks. These neural networks are traned b nche genetc algorthm smultaneousl. The class label of the dentfng data *Ths work was suorted b the atonal Scence Foundaton of Chna under Grant o and atonal Ke Proect of Scentfc and Techncal Suortng Programs Funded b nstr of Scence & Technolog of Chna under Grant o. 006BAI05A0. Fgure. Framework of C-G 3. Prncle of classfer ensemble Extensve exerments show that the unselectve combnaton of all sub-classfers does not alwas mrove the ensemble redctve accurac. Krogh and Vedelsb [5] ut forward a formula of generalzaton /08 $ IEEE DOI 0.09/ICC

2 error n neural network ensembles. In ths secton, we analze the relatonsh between ensemble erformance and sub-classfers, and dscuss how to effectvel combne sub-classfers. Consder a two-class classfcaton roblem. Assume that dataset conssts of nstances (x,, where x s an attrbute set, s a class label, the value of s ether 0 or, =,,,, S s the ensemble of sub-classfers (c,c,, c, each weght of subclassfer s w ( =,,,, where w 0 and w +w + +w = and f (x s the outut (class label of the th sub-classfer for the nut x, so the value of f (x s ether 0 or. The sum of f (x s called the ensemble outut of sub-classfers for the nut x, denoted as f(x : f ( x = w f ( x ( If the ensemble outut f(x < 0.5, then x belongs to class 0, otherwse t belongs to class : f 0 f ( x < 0.5 x = otherwse ( ( E s the generalzaton error of ensemble classfer on the entre dataset: E = = ( f ( (3 x E s the weghted average of the generalzaton error of sub-classfers on the entre dataset: E = = w ( f ( x (4 D s the dfference between each sub-classfer and the ensemble outut on the entre dataset: D = = w ( f ( x f ( x (5 Prooston The generalzaton error of ensemble classfer satsfes: E = E D (6 The analss above shows that a good ensemble s one where the sub-classfers n the ensemble are both accurate and tend to have dverst n ther redctons. So, f we want to mrove the generalzaton ablt of ensemble classfer, we should mrove the accurac of sub-classfers and at the same tme boost ther dverst. 4. Ensemble -classfer based on nche GA The ke stes of C-G are as follows: ( Constructon. che GA s used to evolve c subclassfers satsfed wth good dverst and redefned accurac. ( Combnaton. Sub-classfers are combned wth Dnamc Votng (DV. 4.. Sub-classfer constructon based on nche GA Tradtonal GA alwas converges to a sngle best classfer and thus t s not arorate for multmodal otmzaton. As mrovement, we resent a novel nche GA algorthm. In each GA teraton, the oulaton s arttoned nto several classes b clusterng. Then the ftness of best ndvdual n each cluster s unchanged and the other ndvduals n the oulaton are enalzed b a reducton n ther ftness. Ths causes oulaton dverst ressure that allows the oulaton to mantan ndvduals at local otma Chromosome encodng. Accordng to the structure of neutral network, the chromosome encodng ncludes the weghts of lnks and the bases of nodes. The length of a chromosome s comuted as follows: l=(n +n o n h + n h + n o (7 Where n s the number of nodes of nut laer; n o s the number of nodes of outut laer; n h s the number of nodes of hdden laer. The encodng s as follows: the weghts of lnks between nut laer and hdden laer the weghts of lnks between hdden laer and outut laer the bases of hdden nodes the bases of outut nodes Ftness functon. Ftness functon s used to measure the accurac and generalzaton ablt of the ndvdual (neutral network. The ftness functon s defned as below: f = (8 n ( C T / n + (, Where 0 C, = T C T = (9, otherwse C, s the redcted classfcaton outut (class label of the th ndvdual on the th case. T s the actual class label of the th case. n s the number of cases. Obvousl, when C, = T ( =,,,n, the value of f reachs the maxmum value Genetc oerators. ( Crossover oerator Crossover oerator s as follows: for two ndvduals f = [w, w n, v,, v m ], f = [w,, w n, v,, v m ]-G decdes f the should be recombned or not accordng to crossover rate P c. 406

3 Where w s the weght,v s the bases. At the same tme, two bnar temlates are constructed. When the value equals to 0, the gene n new chromosome s from f, otherwse t s from f. Fgure llustrates the crossover oerator. f = [w,,w n, v,,v m ] f = [w,,w n, v,,v m ] t = [ 0 ] t = [ ] f ' = [w,w n,v,,v m ] f ' = [w,,w n, v,,v m ] Fgure. Examle of crossover ( Selecton oerator We use eltsm and roulette-wheel samlng strateg to select ndvdual to ass on to the next generaton. For the ndvdual whose ftness s the hghest (eltsm, t wll be assed on to the next generaton drectl. (3 utaton oerator The stes of mutaton are as below: a For each ndvdual-g decdes f t should mutate or not accordng to mutate rate P m ; b for the gene ξ to mutate, a random value r s added. Accordng to the exerence, when the value of gene should be lmted n [a, b], the value of r s created randoml n [a, b], the values of new gene s as follows: ξ = (ξ+ r/. ( Classfer dstance. In GA, each ndvdual s a classfer. In order to measure the dstance between two ndvduals, we ntroduce kaa statstc [6]. Let be the number of nstances recognzed as class b classfer C x and as class b classfer C, * be the number of nstances recognzed as b classfer C x, and * be the number of nstances recognzed as b classfer C. Defne Θ and Θ as: l l * * Θ = and Θ = ( ( Where l s the number of classes and s the total number of nstances. Θ estmates the robablt that the two classfers agree, and Θ s a correcton term for Θ, whch estmates the robablt that the two classfers agree sml b chance. The dverst between the two classfers s defned as follows: Θ Θ dv _ kaa ( C x = ( Θ Dv_kaa(C x,c vares between - and. Dv_kaa(C x,c s equal to when the two classfers agree on ever nstance. egatve values occur when agreement s less than exected b chance. In C-G, dverst s regarded as the dstance between two -classfers. We normalze dv_kaa to var from 0 to : dv _ kaa( Cx Cx = ( Classfer clusterng and nche strateg. Havng calculated the arwse dstances between all ndvduals, we roceed b arttonng the oulaton nto several classes b Subtractve Clusterng (SC [7, 8]. In our work, Subtractve Clusterng algorthm s modfed to be used. The ke stes are as followng: Ste. Consder a collecton of n classfers {C, C, n }. Each classfer s a canddate for cluster centrod. The otental of classfer C s defned as [9]: n C P = ( ex( / m (4 ( ra / Where r a s a ostve constant, C,C s the normalzed kaa dstance between C and C, m s the number of classfers wthn the radus r a of C. Hence, a classfer wll have a hgh otental value f t has man neghborng classfers. Ste. The classfer wth the hghest otental s selected as the frst cluster centrod. Let C c be the classfer selected and P c be ts otental. ext, the otental of each classfer C s revsed as follows: C c P = P Pc ex( (5 ( rb / Where the constant r b s usuall greater than r a to avod havng closel saced centrods and s set to.5r a. Ste 3. The next cluster centrod C c s selected and all of the otental calculatons are revsed agan. Ths rocess s reeated untl a stong crteron s met. In each teraton-g combnes current oulaton wth the best c ndvduals n last generaton and arttons the combned oulaton nto several classes b modfed SC. After clusterng, excet for the best ndvdual n each cluster, other ndvduals n the oulaton are enalzed b a reducton n ther ftness. The ftness of a enalzed ndvdual, s as follows: f f, = (6 d Where f s the raw ftness of ndvdual, d s the dstance between ndvdual and the centrod of the cluster that ndvdual belongs to. After the enalt, the best c ndvduals n the current generaton are relcated unchanged nto the next generaton classfer ensemble 407

4 Sub-classfers are combned wth Dnamc Votng (DV. Suose that the number of sub-classfers n ensemble s c and the outut of each sub-classfer s f (x, where =,,,c, =,,,n. The outut of the ensemble classfer s: ensemble = c w f ( x (7 The algorthm C-G s descrbed as follows: Algorthm. C-G Inut: tranng dataset, ensemble sze: c Outut: ensemble classfer Begn. S = Intal Poulaton;. m = axgeneraton; // teraton generatons 3. evaluate(s; // ftness evaluaton 4. reserve(best c -classfers; 5. reeat 6. genetc-oeraton(s; 7. S = combne(s, Best c -classfers; // combne current oulaton wth the best c ndvduals n last generaton 8. SC(S; // modfed Subtractve Clusterng 9. enalze(s; //ftness enalt 0. evaluate(s;. reserve(best c -classfers;. m = m-; 3. untl ((m = 0 (All Best c -classfers reach the redefned accurac 4. ensemble_classfer =DV(Best c -classfers; 5. return ensemble_classfer End. 5. Exerments and erformance evaluaton 5.. Exermental dataset and envronment We evaluate the erformance of C-G on 6 data sets that are from the UCI reostor [0]. The man characterstcs of the 6 data sets are shown n Table. Table. Summar of 6 UCI benchmark data sets used n the exerments Dataset Records Attrbutes Classes Hddens Credt Dabetes Glass House-votes Irs Heatts The arameters of C-G are shown n Table. Table. Parameters for the exerments Parameter Value Suboulaton sze ( 400 umber of generaton ( 000 utaton rate (P m Crossover rate (P c 0.7 All results of C-G are averaged over 4-fold cross-valdaton exerments. 5.. Comared wth BP, GA and L In ths secton, we comare C-G wth the tradtonal tranng methods of sngle neural network: BP, GA, and L n terms of classfcaton accurac. Exermental results of varous models are summarzed n Table 3. Table 3. Estmaton of average redctve accurac rate on the 6 data sets Dataset BP GA L C-G Credt Dabetes Glass House-votes Irs Heatts In terms of redctve accurac-g shows sgnfcantl sueror erformance to the tradtonal tranng methods of sngle neural network: BP, GA, and L n all data sets. Fgure 3 comares the average redctve accuraces of varous models on the 6 data sets. The result shows that comared wth BP, GA, and L-G ncreases the redctve accurac b 5.6%, 5.5%, and 8.5% resectvel. Fgure 3. Comarson of average redctve accuraces of four models 5.3. Comared wth B4.5 and SV 408

5 In ths secton, we comare C-G wth the tradtonal classfcaton algorthm: B4.5 and SV. Exermental results of varous models are summarzed n Table 4. In terms of redctve accurac-g also shows sgnfcantl sueror erformance to the tradtonal classfcaton algorthms: B4.5 and SV n all data sets. C- G ncreases the redctve accurac b 6.0%, 6.% and 4.0% comared wth B4.5 and SV resectvel. Table 4. Estmaton of average redctve accurac rate on the 6 data sets Dataset B C4.5 SV C-G Credt Dabetes Glass House-otes Irs Heatts Ensemble sze In ths secton, we use C-G to examne ensemble characterstcs and rovde data analss on how to construct an otmal ensemble sze. Fgure 4 shows the relatonsh between the average redctve accurac and the ensemble sze on the 6 data sets n C-G, where the number of classfers ranges from to 0. Form Fgure 4, we can fnd that the redctve accurac steadl mroves u to an ensemble sze of 9. The erformance mrovement s unobvous when the number of sub-classfers s more than 9. In ths aer, we roose a ultle Evolutonar eural etwork Classfer Based on che Genetc Algorthm (C-G. The neural networks are traned b nche genetc algorthm based on clusterng. The class label of the dentfng data can frst be evaluated b each neural network, and the fnal classfcaton result s obtaned accordng to the dnamc votng rule. Exermental results show that C-G s more effectve n mrovng the redctve accurac than tradtonal algorthms. 7. References [] Qunlan JR. C4.5: Programs for achne Learnng. San Francsco: organ Kaufmann Publshers,. 5, 993. [] Fredman, Geger D, Goldszmdt. Baesan network classfers. achne Learnng, 9(- 3,.3 63, 997. [3].Hearst.Trends & Controverses: Suort Vector achnes, IEEE Intellgent Sstems, 3(4,.8-8,998. [4] Arulamalam G, Bouzerdoum A. A generalzed feedforward neural network archtecture for classfcaton and regresson. eural etworks, 6(5-6, , 003. [5] Krogh A, Vedelsb J. eural etwork Ensembles, Cross Valdaton, and Actve Learnng. Advance n eural Informaton Processng Sstems 7,. 3-38, 995. [6] J. Cohen, A Coeffcent of Agreement for omnal Scales, Educatonal and Pschologcal easurement 0, , 960. [7] S.L. Chu. Fuzz odel Identfcaton Based on Cluster Estmaton. Journal of Intellgent and Fuzz Sstem (3, , 994. [8] S.L. Chu. A Cluster Estmaton ethod wth Extenson to Fuzz odel Identfcaton. In Proceedngs of the 3rd IEEE Conference on Fuzz Sstems, , 994. Fgure 4 Relatonsh between ensemble sze and average redctve accurac 6. Concluson [9] Lu Yu, Qn Zheng, Lu Jang, and Sh Zhewen, ultmodal Partcle Swarm Otmzaton for eural etwork Ensemble, Journal of Comuter Research and Develoment 4(9, , 005. [0] Blake.L., erz.j. (998 UCI reostor of machne learnng databases [htt:// unverst of Calforna, Irvne, Deartment of Informaton and Comuter Scences. 409