New Boosting Methods of Gaussian Processes for Regression

Transcription

1 New Boosing Mehods of Gaussian Processes for Regression Yangqiu Song Sae Key Laboraory of Inelligen echnology and Sysems Deparmen of Auomaion singhua Universiy Beijing 00084, P.R.China Changshui Zhang Sae Key Laboraory of Inelligen echnology and Sysems Deparmen of Auomaion singhua Universiy Beijing 00084, P.R.China Absrac Feed forward neural neworks are popular ools for nonlinear regression and classificaion problems. Gaussian ProcessGP can be viewed as an RBF neural nework which have infinie number of hidden neurons. On regression problems, hey can predic boh he mean value and he variance of he given sample. Boosing is one of he mos imporan recen developmens in machine learning. Classificaion problems have dominaed research on boosing o dae. On he oher hand, he applicaion of boosing of regression has received less invesigaion. In his paper, we develop wo boosing mehods of GPs for regression according o he characerisic of hem. We compare he performance of our ensembles wih oher boosing algorihms and find ha our mehods are more sable and essenially have less over-fiing problems han he oher mehods. I. INRODUCION Neural neworks are popular ools for nonlinear regression and classificaion problems. Mackay0 poins ha, feed forward neural neworks can be viewed as defining a prior probabiliy disribuion over non-linear funcions, and he neural nework s learning process can be inerpreed in erms of he poserior probabiliy disribuion over he unknown funcion. Neal shows ha he properies of an RBF neural nework wih one hidden layer converges o a Gaussian process as he number of hidden neurons ends o infiniy when he prior of he parameers is Gaussian. hus, we can ignore he parameers of he neural neworks and work direcly wih he inpu and he oupu of he daa. We find ha Gaussian process has a good characerisic ha is no possessed by many oher non-parameric regressors. I can no only regress he mean value of he given poin, bu also can esimae he variance. In bayesian inference framework, i can use he MCMC echnique or MAP esimaion o solve he hyperparameers problem05, compared wih oher kernel mehods such as he Suppor Vecor MachineSVM4 which needs users give he value of he hyper-parameers. However, is raining ime is scaling of On 3 and memory is scaling of On, where n he number of raining poins, hinder heir more widespread applicaion. Bu recenly, here have been several sparse online algorihms such as 8 and 9, which make Gaussian process more and more popular. he main work of Gaussian processes for regression is done by Williams and Rassmussen5, Neal, we also follow he inroducion made by MacKay0 in his paper. On he oher hand, for a wide variey of machine learning problems, especially he classificaion problems, he boosing echniques have been proven o be an effecive mehod for reducing bias and variance, and improving misclassificaion. Bu he knowledge abou he uiliy of hese echniques in regression is no as much as classificaion. Freund and Schapire45 aacks he regression problem by reducing i o a classificaion problem using heir algorihm AdaBoos.R. Druck and Ridgewary, e al3 sugges an acual implemenaion and erimenaion wih boosing regression in which hey applies an ad hoc modificaion of AdaBoos.R o some regression problems respecively. Friedman, e al6 lore regression using he gradien descen approach. hey lain he AdaBoos as an addiive logisic regression model. hey model he poserior probabiliy as P x P y x e F x / + e F x, and use some AdaBoos algorihms o regress he addiive funcion F x. Bu i is essenially a classificaion model. In he absence of he hypohesis ha he arge value is binary, he algorihm could no be lained in heory. Duffy and Helmbold examine ensemble mehods by ieraively calling hem on modified samples, he presen several gradien descen algorihms and prove AdaBoos-like bounds on heir sample errors using inuiive assumpions on he base learners. hey use a decision sumps as a base learner and he main weakness of heir heoreical resuls is he assumpion ha he base learner can consisenly reurn hypoheses wih a useful edge, even when he daa are re-labelled by he maser algorihm. Zemel and Piassi6 propose an analogous formulaion for adapive boosing of regression problems, uilizing a novel objecive funcion ha leads o a simple boosing algorihm. hey use a back-propagaion neural neworks as he he base learner and prove ha he mehod reduces he raining error. Bu hey only give a resul ha has only 0 ieraions, in his paper, our erimen shows ha, if here is one crierion o sop he algorihm, heir boosing is very likely o exhibi over-fiing. In his paper, mainly based on he work of Zemel and Piassi6, we presen a new objecive funcion according o

2 he characerisic of Gaussian process which can esimae he variance of he given sample. he mehods can also work wih oher regressors if only i can reurn a esimaed value and a variance a he es sample. Resuls show ha our mehods are more sable han he oher mehods and exhibi no overfiing problems a all. he paper is organized as follows. In secion, we give an shor inroducion o Gaussian process for regression ha MacKay0 gives in heir paper. Saring from secion 3, we presen our wo mehods of ensembling he regressors. Experimenal resuls are given in secion 4 and secion 5 we conclude he ex and show he fuure work. II. A SHOR INRODUCION O GAUSSIAN PROCESS Firsly, we define some noaions. Here we only inroduce he basic Gaussian process for regression problems, following he mehods inroduced by MacKay0. Since Gaussian process is very similar o Bayesian neural neworks, we follow he noaion of hem. We denoe he raining se of N daa poins as X } N x n N n, heir arge se is N n } N n. In neural neworks, we have a non-linear funcion yx, wih respec o he parameer w. In parameer approach o regression such as RBF neural neworks, we use H fixed basis funcions. Le us assume ha a lis of N inpu poins X N has been specified and define he N H marix R o be he marix of values of he basis funcions a he poins: R nh φ h x n. And hen define he vecor y N o be he vecor of values of yx a he N poins: y n R nh w h. If he prior disribuion of w is Gaussian h wih zero mean: P w N0, σwi, hen y, being a linear funcion of w, is also Gaussion disribued wih zero mean.he covariance marix is: Q y, y Rw, w R R w, w R σ wrr So he prior disribuion of y is: P y N0, Q N0, σ wrr We call yx is a Gaussian process if for any finie selecion of poins x, x,..., x N, he join densiy P yx, yx,..., yx N is Gaussian. For regression problems, we assume he arge and he corresponding esimaion funcion differ by a Gaussian noise of variance σ v: y + v, P v N0, σ v, hen he prior disribuion of arge vecor is also Gaussian: P N0, Q + σ vi N0, C 3 We denoe he covariance marix C by: C σ wrr + σ vi 4 When he neuron s number of he RBF neural neworks increases o infinie, i is proved0 ha he enry of covariance marix C has he form: Cx, x θ D x i x i ri + θ 5 where x i is he ih componen of x, D is he dimension of he daa space, and Θ θ, θ, r i }} is a se of hyperparameers. We can use he MCMC echnique5 or a MAP esimaion05 o obain hem, in his paper, we adop he laer. We now disinguish beween differen sizes of covariance marix C wih a subscrip, we define sub-marices of C N+ as follow: CN k C N+ k 6 κ According o he pariioned inverse equaions, we have:, M C N + µ mm, C N+ M m m µ µ κ k C N k, m µc N k. 7 So he inference of a new arge is a condiional disribuion which is also Gaussian: P N+ N Z N+ ˆ N+ C ˆ 8 N+ where: ˆ N+ k C N N C ˆ N+ κ k C N k 9 hus we ge he predicive mean value and he variance which define he error bars a he new given poin. Due o he characerisic of esimaion of mean and variance, we develop wo ensemble mehods o boos he Gaussian process o ge more efficien regressors. One is a AdaBoos-like algorihm, he oher is a bias/variance decomposiion based mehod. III. ENSEMBLES OF GAUSSIAN PROCESSES Zemel and Piassi6, in heir paper, presen a gradien based Boosing algorihm for regression problems. he mehod is differen from AdaBoos.R which is proposed by Freund and Schapire45, which is no direcly used o a regression problem. And also, i is differen wih he mehods of Duffy and Helmbold3 which do no resample he raining daa hrough a new disribuion a each ieraion, bu o modify he arge value of he raining daa. hey define a new objecive funcion: J N β β i y x i n 0 his is he cos afer ieraions o minimize he over all oneniaed squared error and can be viewed as minimizing he hypohesis error over disribuion of he raining daa: J N β β i y x i n β β i y x i N ω i β β i y x i

3 Y hey compare he objecive wih a probabilisic ression. hey poin ha he objecive funcion can be viewed as a produc of he reciprocal likelihood a each ieraion, while he likelihood funcion is: gy x i, M πβ β i y x i his can be heoreically inerpreed, bu in pracice, we find ha afer many ieraions, heir mehod will seriously mee a over-fiing problem if wihou a sop crierion. o avoid of his, we presen wo novel mehods ha can sably boos GPs and wihou any over-fiing problems. We firs presen an AdaBoos-like algorihm which modifies he objecive funcion of Zemel and Piassi s and develop a new mehod of updaing he weigh. Furher more, we use a new mehod o combine regressors using he bias/variance based composiion algorihm while mainain our objecive funcion unchanged. A. An AdaBoos-Like Ensemble Algorihm: he wo crucial elemens of boosing algorihm are he way in which a new disribuion is consruced and he way in which hypoheses are combined o produce a new oupu6. We assume ha he mehod of ensembling he GPs follows he way which Zemel and Piassi do: ȳx i β y x i β 3 where x i is he poin which we wan o know he arge value, y is he hypoheses which a GP oupus a each ieraion, and β is he combinaion coefficien of each GP. Since a each ieraion he GP esimaes he value of a given poin is Gaussian, we can rewrie he Gaussian formulaion as: i y x i N0, C x i 4 where i is he arge value of a given poin x i. hus he difference beween arge and he hypoheses of he combined regressor is also Gaussian: i ȳx i β y x i β β y x i β N0, β C x i β 5 We use he reciprocal likelihood, he new objecive funcion is: where: J n ȳ x i N C x i i ȳ x i C x i β y x i β, C x i 6 β C x i β 7 he minimizing of his funcion 6 is equal o maximizing he likelihood funcion of he combined regressor. Since we can no decompose he funcion J o a funcion of y...y and a funcion of y, we need some approximaion. Zemel and Piassi s mehod has an inuiional meaning ha he weigh is proporional o he reciprocal likelihood: when a he former ieraions he likelihood s produc is small a a given raining poin, meaning ha he uncerainy of he regression resul is large, so he weigh of he poin is large which makes more poins are re-sampled in he nex ieraion. Fig. shows ha he variance is a funcion of he raining daa s densiy, he error bars increase where he daa poin densiy decreases. We simply modify he updae rule of he weighs o ge some ineresing resuls. A ieraion, we assume he weigh is proporional o a funcion of he average predicing variance which is esimaed using he former ieraions: ω i C x i C x i 8 When he average variance is varying from 0 o, he weigh is a monoonously increasing funcion wih he variable. In pracice, we find ha he average variance varies no so suddenly as he likelihood funcion, and his mehod will no cause he accumulaion of he weigh which will cause he over-fiing problem. According o he consrain of he objecive funcion, our Boosing can ge a more sable resul. he flow char of he algorihm is shown in able I X a raining daa b regression resul Fig.. A simple funcion regression resul. Lef: he raining daa and he rue funcion; Righ: he GP regression resulinclude mean and variance..inpu: ABLE I ADABOOS-LIKE ENSEMBLE ALGORIHM raining se examples x n, n } N n. Base learner: Learning a Gaussian process produces a hypohesis..choose an iniial disribuion ω i n. 3.Ierae: a Learn a Gaussian process o regress a funcion wih disribuion ω i. b Se 0 β o minimizej. c Updae raining disribuion: ω i + C 4.Esimae he oupu : ȳ x i x i C x i. β y x i β.

4 An EM View: According o our mehod, we can rewrie he objecive funcion as: ω i J n N ω i i ȳ x i 9 is no same he weigh we menion before, bu a funcion of predicive variance, we rewrie formula 8 as: ω i Ĉ x i Ĉxi 0 where Ĉ is he predicive variance a each ieraion. hen he new objecive funcion can be viewed as an onenial squared error over a cerain disribuion L. Hence, we can infer some ineresing resuls from he squared error: i ȳ x i β i ȳ x i + β y x i β + β β i ȳ x i + β y x i β + β We hen use a EM view o examine he objecive funcion 9. A E-sep, we esimae laen variables: he mean value ȳ and he average predicive variance C x i, and hen use a funcion ω i Ĉ x i Ĉxi o generae a disribuion o re-sample he daa se. A M-sep, we wan o find a new regressor ȳ ha have minimum squared error wih a opimal parameer β. he parameer β is o make he objecive funcion 6 minimized, ha is o make he likelihood maximized. According o he righ side of he inequaliy, he M-sep can be viewed as anoher objecive funcion which is generaed afer re-sampling. We can regard i as o opimize he funcion y x i under he consrain of i ȳ x i 0 where he Lagrange facor is : β β B. Bias/Variance Decomposiion Based Boosing In he previous sub-secion we give an ensemble mehod which is similar o Zemel and Piassi s. Bu we do no use a decomposiion form of he weigh and he new regressor wih he objecive funcion 6. Insead, we use an ad hoc echnique ha inuiionally uses a funcion of variance o updae he weighs. Heskes7 in his paper proves an ineresing conclusion: he mean squared error is a special case of he Kullback-Leibler divergence of his bias/variance decomposiion model. If a regressor can esimae he mean value and he variance of he given poin, we can use a new ensemble mehod o obain he average: C x i /C x i ȳ x i C x i y x i C x i hen he decomposiion yields: y x i i E + C x i log C x i ȳx i i + Cx i log Cx i y x i ȳx i + E + C x i log C x i Cx i 3 4 he firs erm beween brackes on he righhand side is he error of he average model, he second erm measures he variance of he differen esimaors. And also he firs erm of he righhand is a logarihmic form of our objecive funcion. We rewrie he formula 4 in he onenial form as: J n n N C x i N E E i ȳ x i C x i i y x i + C x i log C x i y x i ȳx i C x i Jensen s inequaliy N E n + log C x i } Cx i i y x i + C x i log C x i y x i ȳx i C x i + log C x i } Cx i 5 We find ha beween he onenial brackes he firs erm can no be used o esimae he new regressor, herefore we simply use a new regressor o replace he former ieraions. his insan esimae which replaces he ecaion leads o a new boosing algorihm. We develop a new objecive funcion as: J new n N where N C x i E ω i C x i iy x i C x i } y x i ȳx i C x + i log Cxi Cx i i y x i } C x i } 6 ω i E } y x i ȳx i C x + i log Cxi Cx i C x i y x i ȳ x i Cx i C x i 7

5 Noe ha we ress he objecive funcion as a decomposiion form which has a weigh ω i and a new reciprocal likelihood. he weigh is approximae o he likelihood of each regressor o heir average. his indicaes ha he weigh is large a he sable poins of he regressors. he flow char of his algorihm is shown in able II..Inpu: ABLE II BIAS/VARIANCE DECOMPOSIION BASED BOOSING raining se examples x n } N, n n. Base learner: Learning a Gaussian process produces a hypohesis..choose an iniial disribuion ω i n. 3.Ierae: a Learn a Gaussian process o regress a funcion wih disribuion ω i. b Updae raining disribuion: ω i + C j x i Cx i j 4.Esimae he oupu : C x i /C x i, ȳ x i C x i y x i C x i. IV. MAIN RESULS y j x i ȳ j x i C j x i In his secion we show some resuls o see how he mehods above perform. We also implemen wo mehods which are given by Duffy and Helmbold in 3: and, and also Zemel and Piassi algorihm6 naming nips-boosing. We use a MSEmean squared error o be a crierion funcion. In fig. he lnm SE is adoped o be he oupu of each ieraion. We es our algorihm using he following funcions and daa ses: Sinc funcion sinx/x + ε: x U0, 0, ε N0, 0.. sinx + 0.0x + ε: x U0, 0, ε N0, A Plane 0.6x + 0.3x + ε: x U,, ε N0, Friedman: 0 sinπx x + 0x x 4 + 5x 5 + ε. x x 5 U0,, ε N0, Friedman: x + 6 Friedman3: x x 3 x x 4 xx3 + ε. x an x 4 x + ε. For boh 5 and 6: x U0, 00 x U40π, 560π x 3 U0, x 4 U,, ε N0, Boson Housing: his daase conains informaion colleced by he U.S Census Service concerning housing in he area of Boson Mass. I comprises 506 examples wih 4 variables and he resuls are given for he ask of predicing he median house value from he oher 3 variables. We normalize he arge value wihin he range from - o. 8 Abalone: his daa se comes from he UCI reposiory of machine learning daabases. he ask is o predic he age number of rings of abalone from physical measuremens. We rea he oupu as a coninuous variable, even hough i is a posiive ineger wih a maximum value of 9. he inpu variables is 8 dimension vecors, he arge is he normalized value of a fish s age. here are oally 477 examples in he daa se. For funcion 5, 6 and daa se 8, we simply normalize he inpu vecors o he range from - o, while for daa se 7, being oo many variables of 0, is performed PCA o reduce o 5 dimensions. For funcion o 6, we selec 00 random samples for raining and 500 random samples for esing. For daa se 7, we randomly selec 00 samples for raining and he res for esing. For daa se 8, we randomly selec 300 samples for raining and 000 samples for esing. ABLE III MSE RESUL OF 8 DAASES Nips Our Our sinx/x sinx + 0.0x x + 0.3x Friedman Friedman Friedman Boson Housing Abalone able III shows he main resuls of 8 funcions and daa ses, each row has 5 algorihms implemenaion playing on he signed daa se as column shows. he bes MSE value is highlighed wih a bold fon. Fig. shows he 500 ieraions of each algorihm played on each daa se. Noe ha for simple funcions, and will converge o some fixed value which may be no he bes. For complex funcions and real daa ses, hese wo algorihms will diverge o some unecable value. his is an over-fiing problem because hese algorihm will modify he arge which is relaive o he residual of he hypoheses value a each ieraion, hen use an addiive model o represen he final resul, so he learner will learn unil he residual is 0. Nips-boosing rapidly descends wih in 0 or 0 seps, bu i will also over-fi afer 0 or 30 ieraions. he reason is ha, afer several seps of ieraions, he weigh will accumulae a some cerain samples. he produc of he reciprocal likelihood is very large a some samples and will no diminish any more.

6 a sinx/x our BoosingGP our BoosingGP c 0.6x + 0.3x our BoosingGP e Friedman g Boson Housing Fig.. our BoosingGP our BoosingGP b sinx + 0.0x our BoosingGP d Friedman our BoosingGP f Friedman3 our BoosingGP h Abalone 500 ieraions on 8 daa ses of 5 algorihms Compared wih he former hree, our algorihm has no overfiing problems a all, and he descending speed is also comparable o he oher hree. he AdaBoos-like ensemble mehod is he slowes of five since we use a Geneic AlgorihmGA o opimize he objecive funcion 6 o ge a new β. In pracice, afer several ieraions, mos of he parameer β is 0 or, we assume β only o be 0 or, hen we can ge a bagging-like ression. When ends o infiniy, we have: / β y x i Ey 8 β his shows ha, if a ieraion our combined regressor is very nearly o he ecaion Ey, hen he parameer β is very likely o be 0. On he oher hand, he bias/variance decomposiion based Boosing algorihm dose no opimize an objecive funcion over a parameer, bu o updae he weigh and ge a new hypoheses of Gaussian process over he weigh. So his algorihm will rapidly converge o a cerain value. And for a majoriy of he resuls, his mehod is beer o he AdaBoos-like ensemble mehod. V. CONCLUSIONS In his paper, we firs give an AdaBoos-like algorihm which modifies he objecive funcion of Zemel and Piassi s and develop a new mehod of updaing he weigh. Secondly, we adop a new mehod o combine regressors using he bias/variance based composiion algorihm while mainain our objecive funcion unchanged. Experimenal resuls indicae ha our mehods are comparable wih ohers on descend speed and will no exhibi any over-fiing problems. Bu our mehods also have some problems. Firsly, we use some ad hoc echniques o updae he weighs of he raining samples, especially in bias/variance decomposiion based Boosing mehod, i has no inerpreaion in heory; Secondly, we only show he algorihms and he erimenal resuls while do no give any proof of convergence. In he fuure work, we will prove our algorihm in heory. Boosing of regressors is no received as much aenion as he classificaion problems, we give wo Boosing mehods and obain some ineresing resuls. We ec ha he boosing mehods of regression will be paid more aenion in he fuure. REFERENCES E. Bauer and R. Kohavi, An empirical comparison of voing classificaion algorihms: Bagging, boosing, and varians, Machine Learning, vol. 36:/, pp , 999. H. Drucker, Improving regressors using Boosing echniques, Proceedings of he 4h Inernaional Conference on Machine Learning, pp. 07 5, N. Duffy and D. Helmbold, Boosing mehods for regression, Machine Learning, vol. 47, pp , Y. Freund and R. E. Schapire, Experimens wih a new Boosing algorihm, In Proc. 3h Inernaional Conference on Machine Learning pp , San Maco, CA: Morgan Kaufmann, Y. Freund and R. E. Schapire, A decision-heoreic generalizaion of online learning and an applicaion o boosing, Journal of Compuer and Sysem Sciences, vol. 55:, pp. 9 39, J. Friedman,. Hasie and R. ibshirani, Addiive logisic regression: A saisical view of boosing, he Annals of Saisics, vol. 8:, pp , Heskes, Bias-variance decomposiions for likelihood-based esimaors, Neural Compuaion, 0, pp , Lehel Csa o and Manfred Opper, Sparse online Gaussian processes, Neural Compuaion, vol. 4 pp , N. D. Lawrence, M. Seeger and R. Herbrich, Fas sparse Gaussian process mehods: he informaive vecor machine, Advances in Neural Informaion Processing Sysems 5, MI Press, Cambridge, MA, D. MacKay, Inroducion o Gaussian processes, echnical Repor, Cambridge Universiy, UK, 997. R. M. Neal, Bayesian Learning for Neural Neworks, in Lecure Noes in Saisics, Springer, New York, 996 R.M. Neal, Mone Carlo implemenaion of Gaussian process models for Bayesian classificaion and regression, echnical Repor 970, Deparmen of Saisics, Universiy of orono, January, G. Ridgewary, D. Madigan and. Richardson, Boosing mehodology for regression problems, In D. Heckerman, and J. WhiakerEds., Proc. Arificial Inelligence and Saisics, pp. 5 6, V. Vapnik, he Naure of Saisical Learning heory. Springer, C. K. I. Williams and C. E. Rasmussen, Gaussian processed for regression, In Advances in Neural Informaion Processing Sysems 8, MI Press, R. S. Zemel and. Piassi, A gradien-based boosing algorihm for regression problems, In Advances in Neural Informaion Processing Sysems, 3, Cambridge, MA. MI Press, 00.