Dynamic Ensemble Selection and Instantaneous Pruning for Regression

Transcription

1 Dynamc Ensemble Selecton and Instantaneous Prunng for Regresson Kaushala Das and Terry Wndeatt Centre for Vson Speech and Sgnal Processng Faculty of Engneerng and Physcal Scences Unversty of Surrey, Guldford, Surrey, GU2 7XH Unted Kngdom Abstract. A novel dynamc method of selectng pruned ensembles of predctors for regresson problems s presented. The proposed method, known henceforth as DESIP, enhances the predcton accuracy and generalzaton ablty of prunng methods. Prunng heurstcs attempt to combne accurate yet complementary members, therefore DESIP enhances the performance by modfyng the pruned aggregaton through dstrbutng the ensemble member selecton over the entre dataset. Four statc ensemble prunng approaches used n regresson are compared to hghlght the performance mprovement yelded by the dynamc method. Expermental comparson s made usng Multple Layer Perceptron predctors on benchmark datasets. 1 Introducton In the context of ensemble methods, t s recognzed that the combned outputs of several regressors generally gve mproved accuracy compared to a sngle predctor [1]. It has also been shown that ensemble members that are complementary can be selected to further mprove the performance [1]. The selecton, also called prunng, has the potental advantage of both reduced ensemble sze as well as mproved accuracy. However the selecton of classfers, rather than regressors, has prevously receved more attenton and gven rse to many dfferent approaches to prunng [2]. Some of these methods have been adapted to the regresson problem [3]. The dynamc prunng methods n [4, 5] are classfcaton orented and rely on functons that determne the ensemble selecton based on nformaton from the tranng set. The proposed novel dynamc method for regresson smlarly uses the tranng set to determne the ensemble selecton. By dynamc, we mean that the subset of predctors s chosen dfferently dependng on the test sample and ts relatonshp to the tranng set. 2 Related Research The man objectve of usng ensemble methods n regresson problems s to harness the complementarty of ndvdual ensemble member predctons [1]. In Negatve Correlaton Learnng, dversty of the predctors s ntroduced by smultaneously tranng a collecton of predctors usng a cost functon that ncludes a correlaton penalty term [3]; thereby collectvely enhancng the performance of the entre ensemble. By weghtng the outputs of the ensemble members before aggregatng, an 643

2 optmal set of weghts s obtaned n [9] by mnmzng a functon that estmates the generalzaton error of the ensemble; ths optmzaton beng acheved usng genetc algorthms. Wth ths approach, predctors wth weghts below a certan level are removed from the ensemble. In [3] genetc algorthms have been utlzed to extract sub-ensembles from larger ensembles. In Stacked Generalzaton a meta-learner s traned wth the outputs of each predctor to produce the fnal output [1]. Emprcal evdence shows that ths approach tends to over-ft, but wth regularzaton technques for prunng ensembles over-fttng s elmnated. Ensemble prunng by Sem-defnte Programmng has been used to fnd a sub-optmal ensemble n [3]. A dynamc ensemble selecton approach n whch many ensembles that perform well on an optmzaton set or a valdaton set are searched from a pool of over-produced ensembles and from ths the best ensemble s selected usng a selecton functon for computng the fnal output for the test sample [4]. Smlarly a dynamc multstage organzatonal method based on contextual nformaton of the tranng data s used to select the best ensemble for classfcaton n [5]. Recursve Feature Elmnaton has been used n [7] as a method of prunng ensembles. Here the weghts of a traned combner are evaluated to determne the least performng predctor that s removed from the ensemble. 2.1 Reduced Error Prunng Reduced Error Prunng wthout back fttng method (RE) [2], modfed for regresson problems, s used to establsh the order of regressors n the ensemble that produces a mnmum n the ensemble tranng error. Startng wth the regressor that produces the lowest tranng error, the remanng regressors are subsequently ncorporated one at a tme nto the ensemble to acheve a mnmum ensemble error. The sub ensemble S u s constructed by ncorporatng to S u-1 the regressor that mnmzes u = 1 1 su argk mnu Cs = 1 + C where k ϵ (1,...,M)\{S 1, S 2,,S u-1 } and {S 1, S 2,,S u-1 } label regressors that have been ncorporated n the pruned ensemble at teraton u-1. For statc ensemble selecton C s calculated over the entre tranng set and expressed as C = N n= 1 k (1) f ( x ) y (2) where = 1,2,,M, f (x) s the output of the th regressor and(x n,y n ) s the tranng data where n = (1,2,,N). Therefore the nformaton requred for the optmzaton of the tranng error s contaned n the vector C. 3 Method In contrast to statc ensemble selecton, Dynamc Ensemble Selecton wth Instantaneous Prunng (DESIP) provdes an ensemble talored to the specfc test nstance based on the nformaton of the tranng set. The method descrbed here s for a regresson problem where the regressors are ordered for every ndvdual tranng nstance based on the method of RE. Therefore each ensemble selecton for every n n 644

3 tranng nstance contans the same regressors as consttuent members but aggregated n a dfferent order. However, potentally ths dynamc method can be mplemented wth any prunng technque. The mplementaton of DESIP conssts of two stages. Frst the base regressors M are traned on bootstrap samples of the tranng dataset and the regressor order s found for every nstance n the tranng set. As shown n the pseudo-code n fgure 1, ths s acheved by buldng a seres of nested ensembles, per tranng nstance, n whch the ensemble of sze u contans the ensemble of sze u-1. Takng a sngle nstance of the tranng set, the method starts wth an empty ensemble S, n step 2, and bulds the ensemble order, n steps 6 to 15, by evaluatng the tranng error of each regressor n M. The regressor that ncreases the ensemble tranng error least s teratvely added to S. Ths s acheved by mnmzng z n step 9. Therefore each regressor n M takes a unque poston n S as S grows. Ths order s archved n a two dmensonal matrx A wth regressor order n rows and tranng nstance n columns. Tranng data D = (x n, y n ), where n = (1,2,..,N) and f m s a regressor, where m = (1,2,..,M). The Archve Matrx A = (a n ) where a n s a column vector wth max ndex of m. S s also a vector wth max ndex of m. 1. For n = 1.N 2. S empty vector 3. For m = 1 M 4. Evaluate Cm = f m ( xn ) yn 5. End for 6. For u = 1 M 7. mn + 8. For k n (1,...,M)\{S 1, S 2,,S u } u 9. Evaluate = 1 1 z u C S + C k = If z < mn 11. S u k 12. mn z 13. End f 14. End for 15. End for 16. a n S 17. End for Fg 1: Pseudo-code mplementng the archve matrx wth ordered ensemble per tranng nstance. In the second stage, the regressor order that s assocated wth the tranng nstance closest to the test nstance s retreved from matrx A. Here the closeness s determned by calculatng the L1 Norm of the dstance measure between the test nstance and the tranng set. Ths s performed n steps 1 to 6 n fgure 2. All nput features of the tranng set are consdered to dentfy the closest tranng nstance, 645

4 usng the K-Nearest Neghbors method [6], where K = 1. The resultng vector g n, where n s the ndex of the tranng nstance, s searched for the mnmum value and s dentfed as the closest tranng nstance to be retreved from A. The selected ensemble has the order of regressors determned by the tranng nstance. Test and tran nstance x f,test, x f,n,tran where f = (1,2,..,F) features. From fgure 1 Archve Matrx A = (a n ) where a n s the column vector contanng the order of regressors, n = 1,2,..,N. e f s a vector wth max ndex of F and g n s a vector wth max ndex of N. 1. For n = 1.N 2. For f = 1.F 3. Evaluate e f = x f,n,tran - x f,test 4. End for 5. F g n = e f f = 1 6. End for 7. Search for the mnmum values n g n and note n 8. a n s the ensemble selecton for the test nstance. Fg 2: Pseudo-code mplementng the dentfcaton of the closest tranng nstance to the test nstance. In the mplementaton of DESIP wth RE, equaton (2) s modfed so that C s calculated for every tranng nstance. The modfed equaton s shown n equaton (3) C = f ( xn ) yn (3) Consequently S u n equaton (1) s also calculated for every tranng nstance. For the comparson of DESIP wth statc methods, four statc prunng methods were mplemented wth DESIP. They are Ordered Aggregaton (OA) as descrbed n [3] for regresson, Recursve Feature Elmnaton (RFE) n [7], ensemble optmzaton usng Genetc Algorthm (GA) [3] and Reduced Error Prunng wthout back fttng (RE), descrbed n Secton 2.1. The datasets lsted n table 3 have been used for the above comparson. 4 Results MLP archtecture usng the Levenberg Marquardt learnng algorthm wth 5 nodes n the hdden layer, as descrbed n [3] has been selected n ths experment. The tranng/test data splt s 70/30 percent, and 32 base regressors are traned wth bootstrap samples. The Mean Squared Error (MSE) s used as the performance ndcator for both tranng and test sets, and averaged over 100 teratons. Tables 1 and 2 show MSE performance of the four statc methods wth and wthout DESIP. In tables 1 and 2, grayed results ndcate the mnmum MSE over the eght methods. It s observed that the majorty of the lowest MSE values have been acheved by DESIP. Fgure 3 shows the comparson of the tranng and the test error 646

5 plots of statc methods and DESIP (wth RE mplemented) for the SERVO dataset. It s observed that pruned ensembles wth DESIP are more accurate wth fewer members than statc methods. Mean Squared Error Fg 3: Servo / Test MSE OA RFE DESIP Subensemble Sze Mean Squared Error Comparson of the MSE plots of the tranng set and the test set for OA, RFE and DESIP usng RE Servo / Tran MSE OA RFE DESIP Subensemble Sze Dataset Multpler OA RFE GA RE Servo ± ± ± ±0.24 Boston Housng ± ± ± ±0.64 Forest Fres ± ± ± ±0.07 Wsconsn ± ± ± ±0.12 Concrete Slump ± ± ± ±0.36 Auto ± ± ± ±0.93 Auto Prce ± ± ± ±0.63 Body Fat ± ± ± ±2.12 Bolts ± ± ± ±4.69 Polluton ± ± ± ±0.21 Sensory ± ± ± ±0.26 Table 1: Statc Ensemble Prunng Methods: Averaged MSE wth Standard Devaton for the 100 teratons. Dataset Multpler OA RFE GA RE Servo ± ± ± ±0.21 Boston Housng ± ± ± ±0.78 Forest Fres ± ± ± ±0.09 Wsconsn ± ± ± ±0.28 Concrete Slump ± ± ± ±0.56 Auto ± ± ± ±0.83 Auto Prce ± ± ± ±0.63 Body Fat ± ± ± ±1.57 Bolts ± ± ± ±2.43 Polluton ± ± ± ±0.26 Sensory ± ± ± ±0.16 Table 2: DESIP wth Statc Methods Adopted: Averaged MSE wth Standard Devaton for the 100 teratons. 647

6 5 Concluson Dataset Instances Attrbutes Source Servo UCI-Repostory Boston Housng UCI-Repostory Forest Fres UCI-Repostory Wsconsn UCI-Repostory Concrete Slump UCI-Repostory Auto WEKA Auto Prce WEKA Body Fat WEKA Bolts 40 8 WEKA Polluton WEKA Sensory WEKA Table 3: Benchmark datasets used Dynamc ensemble prunng utlzes a dstrbuted approach to ensemble selecton and s an actve area of research for both classfcaton and regresson problems. In ths paper, a novel method of dynamc prunng of regresson ensembles s proposed. Expermental results show that test error has been reduced by modfyng the prunng based on the closest tranng nstance. On a few datasets the proposed method has not mproved performance, and wll be nvestgated further along wth dfferent dstance measures, varyng K for K-NN and relevant feature selecton. Bas/Varance and tme complexty analyss should also help to understand the performance relatve to other statc and dynamc prunng methods wth smlar complexty. References [1] Tsoumakas G., Partalas I., Vlahavas I., An Ensemble Prunng Prmer. Supervsed and Unsupervsed Ensemble Methods and ther Applcatons. Studes n Computatonal Intellgence Volume 245, Sprnger 2009, pp [2] Martínez-Muñoz G., Hernández-Lobato D., Suárez A., An Analyss of Ensemble Prunng Technques Based on Ordered Aggregaton. IEEE Transactons on Pattern Analyss and Machne Intellgence. 31(2), pp [3] Hernández-Lobato D., Martínez-Muñoz G., Suárez A, Emprcal Analyss and Evaluaton of Approxmate Technques for Prunng Regresson Baggng Ensembles. Neurocomputng 74, 2011, pp [4] Dos Santos E.M., Sabourn R., Maupn P., A Dynamc Overproduce-and-choose Strategy for the selecton of Classfer Ensembles. Pattern Recognton 41, 2008, pp [5] Cavaln P.R., Sabourn R., Suen C.Y., Dynamc Selecton of Ensembles of Classfer Usng Contextual Informaton, Multple Classfer Systems Volume 5997, LNCS, Sprnger, 2010, pp [6] Dubey H., Pud V., CLUEKR: Clusterng Based Effcent K-NN Regresson, Advances n Knowledge Dscovery and Data Mnng, LNCS, Sprnger, Volume 7818, 2013, pp [7] Wndeatt T., Das K., Feature Rankng Ensembles for Facal Acton Unt Classfcaton. IAPR Thrd Internatonal Workshop on Artfcal Neural Networks n Pattern Recognton, [8] Cavaln P.R., Sabourn R., Suen C.Y., Dynamc Selecton Approaches for Multple Classfer Systems, Neural Computng Applcatons Volume 22 (3-4) LNCS, Sprnger, 2013, pp [9] Zhau Z.-H., Wu J., Tang W., Ensemblng Neural Networks: many could be better than all, Artfcal Intellgence, Volume 137, 2002, pp