Ensemble of GA based Selective Neural Network Ensembles

Transcription

1 Ensemble of GA based Selectve eural etwork Ensembles Jan-Xn WU Zh-Hua ZHOU Zhao-Qan CHE atonal Laboratory for ovel Software Technology anjng Unversty anjng, 0093, P.R.Chna {zhouzh, Abstract eural network ensemble s a learnng paradgm where several neural networks are jontly used to solve a problem. In ths paper, e-gase, a two-layer neural network ensemble archtecture s proposed, n whch the base learners of the fnal ensemble are also ensembles. Expermental results show that e-gase generalzes better than a popular ensemble method. The reason why e-gase works s also dscussed. We beleve that the dfferent layers of e-gase attan good generalzaton ablty for dfferent reasons. The frst layer ensembles proft from the selected ndvdual neural networks that are moderately dvergent but generalze well, whle the second layer ensemble profts from the dvergency among the frst layer ensembles. Introducton Snce neural computng has no rgorous theoretcal framework untl now, whether a neural network based applcaton wll be successful or not s almost fully determned by the practtoner. In general, the more experenced the practtoner s, the more chances the applcaton wll have of beng success. However, users are often wth lttle knowledge on neural computng. Therefore the rewards that neural network technques may return do not always appear. In the begnnng of the 990 s, Hansen and Salamon showed that the generalzaton ablty of a neural network system can be sgnfcantly mproved through ensemblng ndvdual neural networks,.e. tranng several neural networks and combnng ther results n some way[]. Later, Sollch and Krogh defned neural network ensemble as a collecton of a (fnte) number of neural networks that are traned for the same task[]. Snce t behaves remarkably well and s vey easy to use, neural network ensemble s regarded as a promsng methodology that can beneft not only experts n neural computng but also ordnary engneers. And neural network ensemble has already been used n many real domans such as handwrtten dgt recognton[3], scentfc mage analyss[4], face recognton[5][6], OCR[7], sesmc sgnal classfcaton[8], etc. Many works have been done to nvestgat why and how neural network ensemble works. The classcal one s Krogh and Vedelsby[9] s work, n whch they derved the famous equaton E = E A. It clearly demonstrates that the generalzaton ablty of the ensemble s determned by the average generalzaton ablty and the average ambguty (dvergency) of the ndvdual neural networks that consttute the ensemble. Many ensemble methods have been proposed n the lterature. The most attractve methods manly nclude smple ensemble[], AdaBoost[0], and baggng[]. These methods combne outputs of all the base learners at hand. Usually, the base learner s a neural network or a classfcaton tree. If a base learner s of hgh generalzaton error and low ambguty, addng t nto the ensemble wll defntely deterorate the ensemble s generalzaton ablty. However, there s no gurantee that such bad base learner wll never appear. Ths means that n some crcumstances usng all the base learners at hand may not be the best choce. GASE (Genetc Algorthm based Selectve Esemble) was proposed n [], whch trans several neural networks and then employs genetc algorthm to select an optmum subset of those networks to consttute an ensemble. Experments

2 show that GASE s superor to smple ensemble, even f t tends to select only a small quantty of neural networks. In ths paper, we argue that f we employ a two-layer ensemble archtecture,.e. when the base learner tself s also an ensemble, the fnal ensemble wll have better generalzaton ablty. The archtecture we used n ths paper s a smple ensemble of GASE. The fnal ensemble s composed of several GASEs, and the fnal ensemble s output s the average of all ndvdual GASE s outputs. Ths archtecture s abberevated as e-gase. By analyzng the expermental results, we beleve that GASE and e-gase promotes the fnal ensemble s generalzaton ablty n dfferent ways. The rest of ths paper s organzed as follows. In Secton, the equaton E = E A,.e. relaton between the generalzaton ablty of the ensemble, the generalzaton ablty of ndvdual base learners and the average ambguty of the base learners, s frst explaned. Then GASE s brefly ntroduced. In Secton 3, e-gase s proposed. Some experments are aslo reported. In Secton 4, The facts revealed by experments are dscussed. The reason about how and why such a two-layer ensemble archtecture works s analysed. Fnally n Secton 5, conclusons are drawn and several ssues for future work are ndcated. GASE Suppose the learnng task s to use an ensemble that comprses base learners to approxmate a functon f: R m R n. The predctons of the base learners are combned through weghted averagng, where a weght w ( =,,, ) s assgned to the ndvdual base learner f, and w satsfes equaton () and (): 0 < w < () = w = () The output of the ensemble s computed accordng to equaton (3), where f s the output of the -th base learner. f ( x) = w f ( x ) (3) = For convenence of dscusson, here we assume that each base learner has only one output component,.e. the functon to be approxmated s f: R m R. But note that t can be easly generalzed to stuatons where each base learner has multple output components. Suppose x R m s randomly sampled accordng to a dstrbuton p(x). The expected output for x s d(x). Then the error E (x) of the -th base learner on nput x and the error E(x) of the ensemble on nput x are respectvely: E E ( x) ( f ( x) d( x) ) = (4) ( x) f ( x) d( x) = (5) Then the generalzaton error E of the -th base learner on the dstrbuton p(x) and the generalzaton error E of the ensemble on the dstrbuton p(x) are respectvely: E = d xp( x) E x (6) E = d xp( x) E x (7) The average error of the base learners on nput x s: E ( x) = w E ( x) = (8) Then the average generalzaton error of the base learners on the dstrbuton p(x) s: E = d xp( x) E x (9) Accordngly, the ambguty of the -th base learner on nput x, the ambguty of the -th base learner on the dstrbuton p(x), the average ambguty of base learners on nput x, and the average ambguty of the base learners on the dstrbuton p(x) are defned respectvely as: = ( ) A x f x f x (0) A = d xp( x) A ( x) () w A ( x) A x = () = A= d xp( x) A x (3) After a few algebrac manpulatons, Krogh and Vedelsby reached the famous formula (4) whch states that the generalzaton ablty of the ensemble

3 s determned by the average generalzaton ablty and the average ambguty of the base learners that consttutes the ensemble. E = E A (4) ow we defne the correlaton between the -th and the j-th ndvdual base learner as: Cj = dxp( x) ( f ( x) d( x) )( f j ( x) d( x) ) Then, accordng to [] and [3], we have E = = j= w w C j j (5) (6) When the base learners are combned usng the smple ensemble method,.e. w =/ for every, we have = j (7) = j= E C It s proved that when usng the smple ensemble method and when formula (8) s satsfed, then omttng the k-th base learner wll mprove the ensemble s generalzaton ablty []. ( ) Cj < ( ) Ck + ( ) Ek = j= k j k = k (8) ow a concluson s arrved that after the neural networks are traned, n some cases ensemblng an approprate subset of the neural networks s superor to ensemblng all of them. The ndvdual neural network that should be omtted satsfy equaton (8). Ths statement s also partly approved by Lu, Yao and Hguch[4]. After traned several neural networks wth negatve correlaton learnng and evolutonary computaton, they used the k-means algorthms to dvde the ndvduals nto dfferent clusters. In every cluster, the fttest ndvdual network s selected as a representatve of the cluster. They compared the ensemble formed of these representatves and the ensemble formed of all the networks. o statstcally sgnfcant dfference s observed between them n ther experments. Ths observaton mples that the ensemble does not have to use all the networks to achve good performance. GASE s proposed based on ths concluson, whch frst trans several ndvdual neural networks ndependently and then employs genetc algorthm to select an optmum subset of ndvdual networks to consttute an ensemble. The selected neural networks are combned toghther usng smple averagng. However, expermental data show that GASE s superor to usng all the avalable networks at hand []. Although genetc algorthm s used n both methods of [4] and [], they are qute dfferent. In [4] genetc algorthm s used to evolve a populaton of neural networks that are negatvely correlated, whle n [] genetc algorthm s used to select a subset of neural networks to consttute the ensemble. 3 e-gase It s well known that n order for an ensemble to work well, the ndvdual neural networks should respond as ndependent as possble to an nput. If the ndependency requrement s satsfed, the ensemble s generalzaton error wll decrease when more neural networks are added nto t. However, the margnal error reduced by every newly added neural network tends to decrease when the ensemble grows larger and larger[3][5]. The GASE method underwent a genetc algorthm based selecton process. After selecton, GASE s sze,.e. the number of neural networks survved the selecton process, s rather small. Expermental data n [] show that f neural networks are traned, averagely GASE wll select only about /4 among them to form an ensemble. The benefts brought by the genetcal selecton process s enjoyable. However, we beleve that f more neural networks are ncluded, n some cases the generalzaton error of the ensemble may be further reduced. Ths s the motvaton of e-gase, whch s a natural extenson of GASE. Gven a learnng task, we may tran several ensembles usng the GASE algorthm frst. Then, an e-gase s formed by combnng these GASEs by usng the smple ensemble method,.e. averagng the output of several GASEs on an nput to form the e-gase s output. The e-gase s a two-layer ensemble archteture. Snce e-gase s formed by averagng several GASEs and every GASE s constructed by averagng several sngle neural networks, e-gase

4 Table Expermental results on smple ensemble, GASE, and e-gase Data set smple ensemble GASE e-gase error devaton Error devaton error devaton Fredman# Boston Housng Ozone Servo Table The mean error-ambguty decomposton of generalzaton error Data set smple ensemble GASE e-gase E E A E E A E E A Fredman# Boston Housng Ozone Servo may be vewed as averagng some selected sngle neural networks. So we may defne the sze of an e-gase as the number of sngle neural networks contaned n t. In ths sense, the sze of an e-gase equals the sum of szes of all ts component GASEs. We use four regresson problems that were used n [] to compare the performance of smple ensemble, GASE and e-gase. The frst problem s Fredman# proposed by Fredman [6]. There are 5 contnuous attrbutes. The data set s generated accordng to equaton (9) where the nose tem ε satsfes normal dstrbuton (0, ) and x ( =,,, 5) satsfes unform dstrbuton U[0, ]. In our experments the sze of the tranng set and the test set are respectvely 00 and 000. t ( π x x ) + 0( x 0.5) + 0x + + ε = 0sn 3 4 5x5 (9) The second problem s Boston Housng from UCI machne learnng repostory[7]. There are contnuous attrbutes and categorcal attrbute. The data set comprses 506 examples among whch 400 examples make up the tranng set and the rest 06 examples make up the test set n our experments. The thrd problem s Ozone proposed by Breman and Fredman [8]. There are 9 contnuous attrbutes. The data set comprses 366 examples. Snce the ntenton of the experments s not to compare the ablty of dealng wth mssng values, attrbute and 36 examples that has mssng values are omtted. Therefore n our experments there are 8 contnuous attrbutes and 330 examples among whch 50 examples make up the tranng set and the rest 80 examples make up the test set. The fourth problem s Servo from UCI machne learnng repostory. There are 4 categorcal attrbutes. The data set comprses 67 examples among whch 30 examples make up the tranng set and the rest 37 examples make up the test set n our experments. ote that some researchers [9] beleve that ths problem s very dffcult because t nvolves some knd of extreme nonlnearty. For each problem we use baggng on the tranng set to generate 0 sngle-hdden-layered BP networks. The smple ensemble method s formed by averagng these networks. After performng genetcal selecton, a GASE s constructed by averagng the selected networks. For every problem we perform 0 runs and record the average mean squared error and the standard devatons of these errors on the test set. An e-gase s formed by averagng 4 GASEs. So there are totally 5 runs of e-gase. The average mean squared error and correspondng standard devaton s also recorded. Expermental results are shown n Table. Statstcal tests show that on the Fredman#, Boston Housng, and Ozone data sets, GASE s generalzaton error s sgnfcantly lower than that of the smple ensemble method, and e-gase attans stll lower generalzaton errors than GASE. On the

5 Servo data set, GASE s slghtly nferor to smple ensemble. The e-gase method s performance, however, has no sgnfcant dfference wth that of the smple ensemble method. From the aforementoned statstcs we may conclude that e-gase s superor to both GASE and smple ensemble. 4 Dscusson Untl now, we are not very clear through what mechansm e-gase works so well. Followng formula (4), the generalzaton error of an ensemble (E) can be decomposed as dfference of the mean error part E and the mean ambguty part A. The mean error-ambguty decomposton of smple ensemble, GASE and e-gase on the four data sets are tabulated n Table. The mean error part E s calculated by averagng the error of the ndvdual neural networks that consst the ensemble on the test set. Then, The mean ambguty part A s calculated from formula (4) wth the help of E. It s clear that the mean error part E of GASE s qute smaller than that of smple ensemble. The mean ambguty part A of GASE s also smaller than that of smple ensemble, the decrease n E s more sgnfcant. It means that GASE may attan better generalzaton ablty by selects base learners that are of only moderate ambguty but are well-generalzed. Ths analyss s somewhat dfferent from our prevous one n [], n whch we beleve that GASE s genetcal selecton process wll ncrease ambguty. Snce every GASE has a relatvely small generalzaton error, e-gase s hard to get smaller error by further lowerng the mean error part E. From Table we can fnd that the mean error of GASE and e-gase has no obvous dfference, whle e-gase has a hgher mean ambguty. Therefore we beleve that e-gase manly profts from the dvergence among dfferent GASEs. 5 Conclusons and future work In ths paper, we re-examned the GASE,.e. Genetc Algorthm based Selectve Esemble method and proposed e-gase, a natural extenson of GASE, whch combnes several GASEs by usng smple averagng. Through analyses of expermental results, a conjecture on how and why e-gase works s proposed. We beleve that GASE works by selectng neural networks that are of only moderate ambguty but are well-generalzed. And, the e-gase method gans from the dvergence among dfferent GASEs. However, analyses presented n ths paper are very prelmnary. More experments and theoretcal works are stll needed to clarfy the rules behnd GASE and e-gase. For theoretcal analyss, we beleve that the bas-varance decomposton may be helpful [0]. Moreover, whether other knds of ensembles, e.g. weghted averaged one, can play the roles of GASE n e-gase, s an nterestng ssue for future exploraton. Acknowledgements The atonal atural Scence Foundaton of P. R.Chna and the atural Scence Foundaton of Jangsu Provnce, P. R.Chna, supported ths research. References [] L. K. Hansen and P. Salamon, eural network ensembles, IEEE Trans. Pattern Analyss and Machne Intellgence, vol., no. 0, pp , 990. [] P. Sollch, A. Krogh, Learnng wth Ensembles: How over-fttng can be useful, In Advances n eural Informaton Processng Systems 8, pp , 996. [3] L. K. Hansen, L. Lsberg, and P. Salamon, Ensemble methods for handwrtten dgt recognton, In Proc. IEEE-SP Workshop on eural etworks for Sgnal Processng, pp , 99, IEEE Computer Socety. [4] K. J. Cherkauer, Human expert level performance on a scentfc mage analyss task by a system usng combned artfcal neural networks, In Proc. 3th AAAI Workshop on Integratng Multple Learned Models for Improvng and Scalng Machne Learnng Algorthms, pp. 5-, 996, AAAI. [5] S. Gutta and H. Wechsler, Face recognton usng

6 hybrd classfer systems, In Proc. IEEE Int. Conf. on eural etworks, pp. 07-0, 996, IEEE Computer Socety. [6] F. J. Huang, Z.-H. Zhou, H.-J. Zhang, and T. Chen, Pose nvarant face recognton, In Proc. 4th IEEE Int. Conf. on Automatc Face and Gesture Recognton, pp , Grenoble, France, 000, IEEE Computer Socety. [7] J. Mao, A case study on baggng, boostng and basc ensembles of neural networks for OCR, In Proc. IEEE Int. Jont Conf. on eural etworks, vol.3, pp , 998, IEEE Computer Socety. [8] Y. Shmshon and. Intrator, Classfcaton of sesmc sgnals by ntegratng ensembles of neural networks, IEEE Trans. Sgnal Processng, vol. 46, no. 5, pp. 94-0, 998. [9] A. Krogh and J. Vedelsby, eural network ensembles, cross valdaton, and actve learnng, In Advances n eural Informaton Processng Systems 7, pp. 3-38, 995, MIT. [0] Y. Freund and R. E. Schapre, A decson-theoretc generalzaton of on-lne learnng and an applcaton to boostng, Journal of Computer and System Scences, vol. 55, no., pp. 9-39, 997. [] L. Breman, Baggng predctors, Machne Learnng, vol. 4, no., pp. 3-40, 996. [] Z.-H. Zhou, J.-X. WU, Y. Jang, and S.-F. Chen, Genetc Algorthm based Selectve eural etwork Ensemble, to appear n Proc. IJCAI 0, Seattle, USA, 00. [3] M. P. Perrone, L.. Cooper, When networks dsagree: Ensemble method for neural networks, In Mammone R.J. eds. Artfcal erual etworks for Speech and Vson, London, Chapman-Hall, pp. 6~4, 993. [4] Y. Lu, X. Yao and T. Hguch, Evolutonary Ensembles wth egatve Correlaton Learnng, IEEE Trans. Evolutonary Computaton, vol. 4, no. 4, pp , 000. [5] R. E. Schapre, Y. Freund, P. Bartlett, and W. S. Lee, Boostng the margn: A new explanaton for the effectveness of votng methods, The Annals of Statstcs, vol. 6, no. 5, pp , 998. [6] J. Fredman, Multvarate adaptve regresson splnes (wth dscusson), Annals of Statstcs, vol. 9, no., pp. -4, 99. [7] C. Blake, E. Keogh, and C. J. Merz, UCI Repostory of machne learnng databases [ ww.cs.uc.edu/~mlearn/mlrepostory.html], Dept. of Informaton and Computer Scence, Unversty of Calforna, Irvne, Calforna, 998. [8] L. Breman and J. Fredman, Estmatng optmal transformatons n multple regresson and correlaton (wth dscusson), Journal of the Amercan Statstcal Assocaton, vol. 80, pp , 985. [9] J. R. Qunlan, C4.5: Programs for Machne Learnng, Morgan Kaufmann, San Mateo, Calforna, 993. [0] L. Breman, Bas, varance and arcng classfers, Techncal Report 460, Statstcs Department, Unversty of Calforna at Berkeley, 996. (ftp://ftp.stat.berkeley.edu/users/breman/ arcall.ps.z.)