Prediction Intervals for Neural Network Models

Transcription

1 Predcto Itervals for Neural Network Models ACHILLEAS ZAPRANIS Departmet of Accout ad Face Uversty of Macedoa 56 Eata St. PO ox hessalok GREECE EFSRAIOS LIVANIS Departmet of Appled Iformatcs Uversty of Macedoa 56 Eata St. PO ox hessalok GREECE Abstract: Neural etworks are a cosstet example of o-parametrc estmato, wth powerful uversal approxmato propertes. However, the effectve developmet ad deploymet of eural etwork applcatos, has to be based o establshed procedures for estmat cofdece ad especally predcto tervals. hs holds partcularly true cases where there s a stro culture for test the predctve power of a model, e.., facal applcatos. I ths paper we revew the major state-of-the-art approaches for costruct cofdece ad predcto tervals for eural etworks, dscuss ther assumptos, streths ad weakesses ad we compare them the cotext of a cotrolled smulato. Our prelmary results, whch are be preseted ths paper, dcate a clear superorty of the combato of the bootstrap ad maxmum lkelhood approaches costruct predcto tervals, relatve to the aalytcal approaches. Key-Words: eural etworks, cofdece tervals, predcto tervals, bootstrap, maxmum lkelhood. Itroducto he effcet utlzato of eural etworks, especally facal applcatos, requres a cofdece measure of ther predctve behavor the statstcal sese. Neural etwork predctos suffer from ucertaty due to: ) accuraces the tra dataset ad ) lmtatos of the model ad the tra alorthm. he fact that the tra dataset s typcally osy ad complete, whle all the possble realzatos of the depedet varable are ot avalable, cotrbutes to the total predcto varace a compoet kow as data ose varace, σ ε. Moreover, the lmtatos of the model ad the tra alorthm troduce further ucertaty to the etwork s predctos. It s called model ucertaty ad ts cotrbuto to the total predcto varace s called model ucertaty varace, σ m. hese two ucertaty sources are assumed to be depedet ad the total predcto varace σ p s ve by the sum of ther varaces,.e., σ ε ad σ m []. If the above varace estmates σ m ad σ p are avalable, we ca form cofdece ad predcto tervals. I the case of cofdece tervals we focus o σ m, sce we are terested the dfferece betwee the predcted output ad the ukow fucto φ(x), whch eerated the avalable observatos (x, y ). I the case of predcto tervals we focus o σ p, sce we are terested the dfferece betwee the predcted output ad the realzed observato y. I secto of ths paper, we exame more detal the dfferece betwee cofdece ad predcto tervals. I secto 3, we exame the aalytcal approach costruct cofdece ad predcto tervals, whch bascally exteds the olear reresso theory the oparametrc sett. he ayesa approach takes a dfferet vew of ths problem, but sce t s bascally approprate for multdmesoal problems, we do ot exame t ay further here. I secto 4, we exame the use of maxmum lkelhood techques for provd local error bars (local estmates of a varable ose varace). I secto 5, we exame the use of bootstrap, as a typcal resampl techque employed by esemble methods (.e., ba ad balac) for costruct cofdece tervals. I yˆ yˆ

2 secto 6, the cotext of a cotrolled smulato we cotrast the syerstc use of bootstrap ad maxmum lkelhood approaches wth the aalytcal approach. Fally, secto 7 we coclude. Cofdece Itervals versus Predcto Itervals Suppose we have a set of observatos D = (x, y ),, that satsfy the olear eural model: ( ; ) y = x w +ε () where y s the output of the eural etwork (x ; w ) ad w represets the true vector of the etwork s parameters w for the ukow fucto φ(x ), whch s be estmated by the etwork. I ths sett, t s true that: ( x ; w ) ϕ ( x ) E[ x ] y ) () Itally, we assume that the error ε s..d. wth zero mea ad costat varace σ ε. he vector wˆ s the least squares estmate of w obtaed by mmz the error fucto: ( ( x; w ) (3) SSE = y = he predcted output of the etwork for the put vector x ad the weht vector w, s: ( ; ˆ ) yˆ = x w (4) I ths framework, a cofdece terval s cocered wth the accuracy of our estmate of the true but ukow fucto φ(x ),.e., t s cocered wth the dstrbuto of the quatty: ˆ ( ) ( ; ˆ ) ϕ( ) ϕ x ˆ x w x y (5) O the other had, the much more mportat oto of a predcto terval s cocered wth the accuracy of our estmate of the predcted output of the F. : Relatoshp betwee the etwork s predcto, the observato y ad the uderly fucto φ(x ), whch has created the observato wth the addto of the stochastc compoet ε. etwork,.e., t s cocered wth the dstrbuto of the quatty: ( ; ˆ ) y x w y yˆ (6) From F. ad equatos (5) ad (6) t follows that: ( ˆ ) ( ( ) ˆ = ϕ ) y y x y + ε (7) As we ca see from equato (7) the cofdece terval s eclosed the predcto terval. 3 Aalytcal Methods Let us deote wth (x, y ) a observato whch has ot bee used for the tra of the etwork (e.., a future observato), that satsfes the follow relatoshp: ( ; ) y = x w + ε (8) Our am s to costruct a predcto terval for y ad a cofdece terval for φ(x ), whch s bascally the codtoal expectato of y ve x. We assume that ε s..d. wth zero mea ad costat varace σ ε. he vector of the etwork parameters s be estmated by mmz the sum of

3 squared errors (3). For a lare umber of tra patters ad for a eural etwork whch provdes a ood approxmato of the uderly fucto φ(x ), the estmated vector w wll be close to the true parameter vector w. he a frst order aylor expaso ca be used order to obta a lear approxmato of the eural etwork fucto aroud x : yˆ where: f ˆ ( x ; wˆ ) ( x ; w) f ( ˆ w w) + ( ) ( ) x ; w x ; w =,..., w w p (9) () he the (-α)% cofdece terval for φ(x ) s ve by []: ( ) yˆ ± t ˆ a, pσ ε f F F f () ( ) yˆ ± t ˆ a, pσ ε + f F F f () De Veaux et al [3] showed that the above method for comput the predcto terval works well whe the tra dataset s lare. However, whe the tra dataset s small ad the etwork s traed to coverece the matrx F Τ F ca be early sular. I ths case, the estmated predcto tervals are ot relable. O the other had, stopp the tra pror to coverece, to avod overftt, reduces the effectve umber of parameters ad ca lead to predcto tervals that are too wde. A soluto to ths problem s ve by employ coecto pru techques, such as the Irrelevat Coecto Elmato scheme (ICE) [3]. After the coverece of the tra alorthm to a soluto, ICE elmates the etwork coectos that ca be presumed redudat. Aother approach s to deactvate rrelevat coectos dur tra us a weht decay method []. I ths case, the error fucto whch s be mmzed has the form: ( ( ; )) = = y x w + c w (3) p he matrx F s the ( p) Jacoba matrx, where s the umber of samples used to estmate w, p s the umber of the etwork parameters ad σ s ˆε the estmate of the stadard devato of the error term. For a etwork wth m put uts ad hdde uts, the umber of etwork wehts () s p = (m + ) +. For a eural etwork wth rrelevat coectos (ueeded coectos for the task at had), the umber of the parameters s ot equal to the umber of the etwork wehts. here s a effectve umber of parameters p effectve < p, whch correspods to a equvalet soluto ( terms of SSE) to the tal oe. Hua ad D [] showed that f the etwork s traed to coverece, the equato () s vald for lare tra samples, eve f we set the umber of the etwork parameters equal to the umber of the coectos. Furthermore, f we assume that the error term s ormally dstrbuted as Ν(, σ ε ), the the (- α)% predcto terval for y s ve by []: ˆ where c > s a weht decay parameter. he predcto terval ths case becomes [3]: σ ε f ( F F I) ( + c ) ] yˆ ± t ˆ [+ + c a, p FFFF I f 4 Maxmum Lkelhood Methods (4) I cotrast wth aalytcal methods, here we do ot assume a costat error varace. Maxmum lkelhood methods do ot mpose ths restrctve codto, but stead they try to estmate σ ε (x) as a fucto of x. Just as the case of aalytcal methods, we assume that the estmated eural etwork provdes a ood approxmato of the ukow uderly fucto, that s the expectato E[y x] see equato (). From ths equato t follows that the varace ca be approxmated by tra a secod eural etwork f v (x;u) (where ν s the umber of hdde uts ad u s the weht vector of the ew etwork), us squared resduals ( (x;w ) 3

4 y) as the taret values. I ths case the error fucto that s be mmzed s: = {( ( ) ) ( )} ; y fν ; x w x u ( ) ( ) ad ˆ σ x f x ; u. ε ν (5) Rather tha us two separate etworks, Nx ad Weed [9] proposed a sle etwork wth oe output for φ(x) = E[y x] ad aother for σ ε (x). Us the sum of squares error fucto to obta u* f v (x;u), ad thus σ ε (x), s equvalet to us maxmum lkelhood estmato, ad for ths reaso these methods are called maxmum lkelhood methods. 5 Esemble Methods he rapd crease comput power of moder computers made realstc the use of eural etwork esemble methods for estmat cofdece ad predcto tervals for eural etworks [6], []. I these techques estmates from a umber of eural etworks are combed to provde eeralzato performace superor to that provded by a sle etwork. Some of the most popular varetes such as ba ad balac [6], use bootstrap [5] to eerate the tra datasets for the esemble approach. oth of these techques attempt to stablze hh-varace predctors such as eural etworks by eerat multple bootstrap versos of the predctor ad the comb the outputs of these dvdual versos to form smoother predctos. However, ba ad balac dffer the way the predctos are combed. ootstrap creates a set Ψ of ew datasets, by repeatedly sampl by replacemet from the oral data set a radom maer: ( ) { ( ˆ ; )} Ψ= xw (6) = hese datasets are be used for tra a set of etworks. he output of the etwork for the put vector x wll be the averae of the etwork outputs:, av ( x) = = (* ) ( ; ˆ x w ) (7) o eerate cofdece ad predcto tervals we assume that the eural etwork provdes a ubased estmato of the true reresso φ(x) E[y x]. hs meas that the dstrbuto P(φ(x),av (x)) s cetred o the estmate,av (x). Our assumpto here s that the bas compoet the cofdece terval s mmal comparso to the varace compoet. If we also assume that the dstrbuto of P(φ(x),av (x)) s Gaussa, the the varace of ths dstrbuto ca be estmated by calculat the varace across the outputs: ˆ σ m ( x) = (* ) ( ( ; ˆ ) ( )) x w, av x = y assum the dstrbuto P(φ(x),av (x)) s Gaussa, the ts verse dstrbuto P(,av (x) φ(x)) s also Gaussa. Whle we do ot kow the dstrbuto of puts ad outputs, the best that we ca do s to estmate the dstrbuto P(,av (x) φ(x)) from the dstrbuto P( (x),av (x)). So ve the observato (x, y ) us bootstrap we ca costruct the follow cofdece terval: ( ) ± t σ ( ) x x (9) ˆ, av a, m where the estmato of the model ucertaty varace σ m s ve from equato (8). However, ths varace estmate wll be based. For most put vectors x wll be over-estmated ad so the cofdece terval (9) wll also be over-estmated. Carey et. al. [] proposed a method to deal wth ths problem. hey dvde the umber of bootstrap etworks for the esemble to M smaller esembles eerat a set of M,av (x) values. From the set of these values we approxmate a more accurate varace measure for the dstrbuto P(,av (x) φ(x)). he varace estmate s ot comput oly from the M esemble outputs, whle ths case the varace measure tself would be hhly varable ad urelable. Istead we form ew bootstrap re-sampled sets of the M,av (x) values. 4

5 -,8,,6,,8,8,4,4,,,3,4,5,6,7,8,9,,,3,4,5,6,7,8,9 -,4 -,4 -,8 F. : he 95% predcto terval for ormal dstrbuto of y, us the alebrac estmato of σ p. he umber of hdde uts s = 3. he sythetc data set was sampled by the Gaussa dstrbuto Ν(.5+.4s(πx),.3 ). he ose varace s costat. he PICP s 89.6%. F. 3: he 95% predcto terval for ormal dstrbuto of y, us the combato of the bootstrap ad maxmum lkelhood estmato of σ p. he umber of hdde uts s = 3. he sythetc data set was sampled by the Gaussa dstrbuto Ν(.5+.4s(πx),.5+.x ). he ose varace s fucto of x. he PICP s 95.9%. We calculate a varace measure for each of these sets ad the calculate a averae of these to provde a smoother, lower varace estmate of the varace of the dstrbuto P(,av (x) φ(x)). hs process s ot computatoally tesve sce there are o etworks to tra. If we assume Gaussa dstrbuto we ca costruct a cofdece terval the usual fasho: ( ) ± z σ ( ) x x () ˆ, av a m where ( α)% s the level of cofdece. o estmate predcto tervals, we must compute a estmate of the predcto varace σ p, whch s ve by the sum of the model ucertaty varace σ m ad the data ose varace σ ε. For the estmato of σ ε we ca use maxmum lkelhood techques or aalytcal methods [], [6]. For the observato (x, y ) whch has ot used for the tra of the etwork, the predcto terval s ve by: ( ) ± t σ ( ) ˆ, av a, p 6 A Cotrolled Smulato We eerated two sythetc datasets: α) wth costat ose term varace ad β) wth ose term varace whch s a fucto of x. he frst dataset was created by employ radom sampl from the Gaussa dstrbuto Ν(.5+.4s(πx),.3 ), whle the secod dataset was created by employ radom sampl from the Gaussa dstrbuto Ν(.5+.4s(πx),.5+,x ). I both cases a eural model wth oe hdde layer ad = 3 hdde uts was selected, o the bass of the Zapras ad Refees framework for eural model detfcato, selecto ad adequacy [4]. I F. we ca see the 95% predcto terval for the sythetc dataset α ad the aalytcal approach (). As we have already dscussed the aalytcal approach ca oly hadle costat error varace. he Predcto Iterval Correct Percetae (PICP) ths case s 89.6%. Sce, ts omal value s 95%, for predcto tervals of ood qualty we expect the value of PICP to be systematcally aroud 95%. x x () I F. 3 we ca see the 95% predcto terval () for the sythetc dataset β. he local estmates of the data ose varace, σ ε (x), were obtaed by us the ML approach, whle the local estmates of the model ucertaty varace, σ m (x), were obtaed by us the bootstrap approach. he total predcto varace, σ p (x), () s smply the I the ext secto we compare the aforemetoed approaches the cotext of a cotrolled smulato. 5

6 sum of σ m (x) ad σ ε (x). As we ca see the PICP ths case s much mproved (95.9%). 7 Summary ad Coclusos Neural etworks are a research feld whch has ejoyed rapd expaso ad creas popularty both the academc ad dustral research commutes. However, ther effcet utlzato requres depedable cofdece ad especally predcto tervals. I ths paper, we examed the state-ofthe-art approaches for cofdece ad predcto terval estmato, ad we compared the aalytcal approach ad the syerstc use of the ML ad bootstrap approaches for costruct predcto tervals, the cotext of a cotrolled smulato. he aalytcal approach we preseted here was based o the frst order aylor expaso of the eural estmator. Other aalytcal approaches are the delta estmator (frst order aylor expaso whch uses the Hessa matrx) ad the sadwch estmator (secod order aylor expaso us the Hessa matrx). he sadwch estmator s cosdered to tolerate better model msspecfcato. ut o the other had, delta ad sadwch estmators requre the computato ad verso of the Hessa matrx, a procedure whch, uder certa crcumstaces, ca be very ustable. I a emprcal vestato [] t s reported that the use of the Hessa matrx does ot mprove the accuracy of the estmato. I ay case, the aalytcal approaches ca ot hadle o costat ose varace. he maxmum lkelhood approach ca be used for estmat local error bars whch are a fucto of x,.e., σ ε (x). However, these caot be used for costruct ether cofdece, or predcto tervals, by themselves. Moreover, the ML approach uderestmates the true ose varace, sce the eural etwork f ν (5) terpolates betwee the errors ad does ot pass throuh all of them. he esemble methods attempt to stablze the hh varace of the eural etwork predctors us bootstrap to eerate multple versos of the model ad the comb the etwork outputs. he bootstrap approach ca be used to obta local estmates of the model ucertaty varace, σ m (x), ad thus for costruct cofdece tervals. y add to σ m (x) the local ose varace estmate, σ ε (x), we ca estmate the total predcto varace, σ p (x), ad thus obta a predcto terval from equato (). As we have see that approach ave as PICP equal to 95.9% for the sythetc dataset wth ose varace whch was a fucto of x. hs compares very favourably to the PICP of 89.6% for the sythetc dataset wth costat varace ad the aalytcal approach. 8 Refereces [] Carey J. G., Cuham P., hawa U., Cofdece ad predcto tervals for eural etwork esembles, Proc. of the Iteratoal Jot Coferece of Neural Networks (IJCNN 99), 999, Washto DC, USA. [] Chryssolours G., Lee M., Ramsey A., Cofdece terval predcto for eural etworks models, IEEE ras. Neural Networks, 7 (), 996, pp [3] De Veaux R. D., Schum J., Schwesber J., Uar L. H., Predcto tervals for eural etworks va olear reresso, echometrcs, 4 (4), 998, pp [4] Dybowsk R., Ass cofdece tervals to eural etwork predctos, Neural Comput Applcatos Forum (NCAF) Coferece, Lodo, 997. [5] Efro., bshra R. J., A Itroducto to the ootstrap, Chapma ad Hall, 993. [6] Heskes., Practcal cofdece ad predcto tervals, I Mchael C. Mozer, Mchael I. Jorda, ad homas Petsche, edtors, Advaces Neural Iformato Process Systems, vol. 9, 997, pp. 76-8, he MI Press. [7] Hwa J.. G., D A. A., Predcto tervals for artfcal eural etworks, Joural of the Amerca Statstcal Assocato, 9 (438), 997, pp [8] Neal R. M., ayesa lear for eural etworks, PhD hess, Dept. of Computer Scece, Uversty of oroto, 994. [9] Nx D. A., Weed A. S., Lear local error bars for o lear reresso, Proceeds of NIPS 7, 995, pp [] Papadopoulos G., Edwards P. J., Murray A. F., Cofdece estmato methods for eural etworks: A practcal comparso, Proc. ESANN, pp [] bshra R., A comparso of some error estmates for eural etwork models, Neural Computato, 8, 996, pp

7 [] Ya L., Kavl., Carl M., Clause S., De Groot P. F. M., A evaluato of cofdece boud estmato methods for eural etworks, ESI, 4 5 September, Aache, Germay. [3] Zapras, A.D. ad G. Harams, "A alorthm for cotroll the complexty of eural lear: he rrelevat coecto elmato scheme", Proc. he Ffth Hellec Europea Research o Computer Mathematcs ad Its Applcatos Coferece (HERCMA),, Athes, - September. [4] Zapras, A.D., Refees, A-P.N., Prcples of Neural Model Idetfcato, Selecto ad Adequacy: Wth Applcatos to Facal Ecoometrcs, Lodo, Sprer-Verla,