Cross-validation estimations of hyper-parameters of Gaussian processes with inequality constraints

Transcription

1 Cross-valdato estmatos of hyper-parameters of Gaussa processes wth equalty costrats Hassa Maatouk, Olver Roustat, Ya Rchet To cte ths verso: Hassa Maatouk, Olver Roustat, Ya Rchet. Cross-valdato estmatos of hyper-parameters of Gaussa processes wth equalty costrats. Spatal Statstcs 015: Emergg Patters commttee, Ju 015, Avgo, Frace. Proceda Evrometal Sceces, 7, pp.38-44, < /.proev >. <hal > HAL Id: hal Submtted o 15 Ju 017 HAL s a mult-dscplary ope access archve for the depost ad dssemato of scetfc research documets, whether they are publshed or ot. The documets may come from teachg ad research sttutos Frace or abroad, or from publc or prvate research ceters. L archve ouverte plurdscplare HAL, est destée au dépôt et à la dffuso de documets scetfques de veau recherche, publés ou o, émaat des établssemets d esegemet et de recherche fraças ou étragers, des laboratores publcs ou prvés.

2 Avalable ole at SceceDrect Proceda Evrometal Sceces 00 (015) Spatal Statstcs 015: Emergg Patters Cross-Valdato Estmatos of Hyper-Parameters of Gaussa Processes wth Iequalty Costrats Hassa Maatouk a, *, Olver Roustat a, Ya Rchet b a Mes de Sat-Etee, 158 Cours Faurel, 403 St-Etee, Frace b Isttut de Radoprotecto et de Sûreté ucléare (IRS, Pars), Frace Abstract I may stuatos physcal systems may be kow to satsfy equalty costrats wth respect to some or all put parameters. Whe buldg a surrogate model of ths system (lke the framework of computer expermets 7 ), oe should tegrate such expert kowledge sde the emulator structure. We proposed a ew methodology to corporate both equalty codtos ad equalty costrats to a Gaussa process emulator such that all codtoal smulatos satsfy the equalty costrats the whole doma 6. A estmator called mode (maxmum a posteror) s calculated ad satsfes the equalty costrats. Here we focus o the estmato of covarace hyper-parameters ad cross valdato methods 1. We prove that these methods are suted to equalty costrats. Appled to real data two dmesos, the umercal results show that the Leave-Oe-Out mea square error crtero usg the mode s more effcet tha the usual (ucostraed) Krgg mea. 015 The Authors. Publshed by Elsever B.V. Peer-revew uder resposblty of Spatal Statstcs 015: Emergg Patters commttee. Keywords: Gaussa process emulator ; equalty costrats ; cross valdato 1. Itroducto I the lterature of corporatg equalty costrats to a Gaussa process (GP) emulator, some methodologes are based o the kowledge of the dervatves of the GP at some put locatos 3,4,8. The methodology preseted (Maatouk ad Bay, 014) 6 s qute dfferet from the methods costructed so far. The * Correspodg author. Tel.: +33(4) ; fax: E-mal address: hassa.maatouk@mes-stetee.fr The Authors. Publshed by Elsever B.V. Peer-revew uder resposblty of Spatal Statstcs 015: Emergg Patters commttee.

3 Hassa Maatouk/ Proceda Evrometal Sceces 00 (015) equalty costrats are forced by costructos. The ma dea s the approxmato of the orgal GP by a ftedmesoal oe. It s doe va corporatg Gaussa radom coeffcets ad determstc bass fuctos. The bass fuctos 6 are chose such that the equalty costrats of the GP are equvalet to costrats o the coeffcets. By ths specal choce of the bass fuctos, the problem s reduced to smulate a trucated Gaussa vector (radom coeffcets) restrcted to covex sets whch s a well-kow problem wth exstg algorthms, see e.g. the algorthm descrbed (Maatouk ad Bay, 014) 5. I ths paper, estmatg covarace hyper-parameters s studed to equalty costrats ad Cross Valdato () methods are used. We focus o the Leave-Oe-Out (LOO) mea square error crtero whch s closely related to tradtoal maxmum lkelhood estmato 1. Let us meto that the case of equalty costrats, the covarace parameters are estmated wthout costrats. Here a suted cross valdato techque to equalty costrats s derved. Addtoally, a real applcato two dmesos to vestgate the performace of the proposed algorthm s cluded.. Gaussa processes wth equalty ad equalty costrats Let (Y(x x R d be a cetered Gaussa Process (GP) wth cotuous covarace fuctos: K : (u,v) R d R d K(u,v) = cov(y(u), Y(v R. (1) I the rug example (Maatouk ad Bay, 014) 6, the Gaussa covarace fucto s cosdered: ( x K e k xk d θ ( x, x' ) ') k=1 = C( x, x' ), () for all x, x R d, where C s the correlato fucto, σ ad θ = (θ 1,, θ d) are parameters that wll be estmated ad cross valdato methods well be used. Wthout loss of geeralty, the put x s [0, 1] d. Let C be the space of cotuous fuctos verfy some propertes such as boudary, mootocty or covexty costrats. The terpolato codto ad the equalty costrats of Y are gve respectvely as follow: ( ) Y( x ) y, 1,..., Y C where the desg pots x () [0,1] d are gve by the row of the matrx X = (x (1),, x () )..1. Fte-dmesoal Gaussa processes I ths secto we recall the model preseted (Maatouk ad Bay, 014) 6. The ma dea s the approxmato of the orgal Gaussa process Y by a fte-dmesoal oe of the form Y ( x) : ( x), 0 x R where ξ = (ξ0,,ξ) s a cetered Gaussa vector wth carefully chose covarace matrx Γ ad determstc bass fucto (φ ), =0,,. The choce of these bass fucto ad Γ deped o the type of the equalty

4 Hassa Maatouk/ Proceda Evrometal Sceces 00 (015) costrats 6. The bass fuctos are chose such that the equalty costrats of Y are equvalet to costrats o the coeffcets ξ. Hece the problem s equvalet to smulate the trucated Gaussa vector ξ restrcted to 0 d where C c R : c C Y C ( ) ( ) ( x ) ( x ) y, 1,..., 0, c=(c0,, c) see (Maatouk ad Bay 014) 6 for detals. 3. Cross valdato estmato of covarace hyper parameters I the framework of estmatg covarace hyper parameters, we fd two types of methods. The frst oe s the Maxmum Lkelhood (ML) estmator ad the secod oe s the o whch we focus o ths paper Cross valdato wthout equalty costrats Let us recall that the Leave-Oe-Out (LOO) mea square error crtero s defed as the followg estmator of the correlato legth hyper parameters θ: ˆ arg m y yˆ (, (3) 1 where ŷ,θ(y - ) = Eθ (y y 1,, y -1, y +1,, y ) ad Θ s a compact set of R d, see (Bachoc, 013) 1. Addtoally, the varace parameter s estmated usg the followg crtero: C LOO 1 1 yˆ, ˆ c,, (4) c var y,..., y, y 1,..., y where, 1 1 ). From (Cresse, 1993), the varace parameter s computed such that crtero (4) s closed to 1 ad the: ˆ 1 1 yˆ, ˆ c,, where ˆ s calculated from Equato (3). 3.. Cross valdato wth equalty costrats Let us meto that the two methods cted above (ML ad ) are ot suted to equalty costrats. Ths s because the usual ucostraed krgg mea defed as the mea of the Gaussa process codtoally to gve

5 4 Hassa Maatouk/ Proceda Evrometal Sceces 00 (015) observato data s ot guarateed to satsfy equalty costrats the whole doma. To ths ed, the dea s to use a ew estmator called mode (maxmum a posteror) whch s defed (Maatouk ad Bay, 014) 6 ad ts aalytcal expresso s equal to: M KI ( x) ( x), 0 (5) where μ = (μ0,, μ) s the soluto of the followg quadratc optmzato problem 1 t arg m ( c ( ) 1 ci C c), wth Γ the covarace matrx of the Gaussa vector ξ. The vector μ ca be see as the mode of the Gaussa vector ξ restrcted to I C, where I : 0. ( ) ( x ) y, 1,..., The postve pot of such estmator s that verfes equalty codto ad equalty costrats the whole doma. Moreover, t does ot deped o the varace parameters σ sce μ ad the bass fuctos do ot deped o t as well. ow, the dea ths paper s to replace the usual ucostraed krgg mea by the mode Equato (3). I that case, the LOO mea square error crtero ca be reformulated as follows: θ arg m θθ (y Mˆ,θ(y 1 Mˆ, ) M KI y1,..., y 1, y 1,..., y; ˆ (6) where ) ad M KI s defed (5). Moreover, the varace parameter s estmated usg the followg crtero: C LOO 1 1 Mˆ, ˆ c,, (7) where c, s obtaed by corporatg equalty costrats to the Krgg varace whch s used Equato (4) ad t s calculated from smulatos. ow, the varace parameter s computed such that crtero (7) s closed to 1 ad the: ˆ 1 1 Mˆ, ˆ c,, where ˆ s calculated from Equato (6).

6 Hassa Maatouk/ Proceda Evrometal Sceces 00 (015) Real Applcato The am of ths secto s to show the performace of the proposed estmator ad to compare t wth the usual krgg mea. We cosder the real data gve Fg. 1 (a). These observato data (=11) defed o [0,0] [10,0] respect mootocty (o-decreasg) costrats for the two put varables. The dea s to fx some desg pots ad test the estmator at the other observato data. (a) (b) Fg. 1. (a) the observato data; (b) our estmator mode uder both equalty codtos ad equalty costrats. (a) (b) Fg.. (a) the usual krgg mea usg the legth parameter θ = (9.05, 9.10) estmated by LOO crtero; (b) the fucto mode usg the legth parameters θ = (5.17, 10.57) estmated by suted LOO crtero to equalty costrats.

7 6 Hassa Maatouk/ Proceda Evrometal Sceces 00 (015) I Fg. (a), we plot the usual krgg mea wth the legth parameters θ = (9.05, 9.10) estmated by LOO crtero 9. The mootocty (o-decreasg) costrat s ot respected the whole doma, cotrarly to Fg. (b), where the estmator mode s llustrated usg the legth parameters θ = (5.17, 10.57) estmated by suted LOO crtero to equalty costrats descrbed ths paper. I Fg. 3, we compare the values estmated versus the real oe. The estmators used are the usual krgg mea Fg. 3 (a) ad mode Fg. 3 (b). To vestgate the performace of the proposed estmator mode, we calculate the crtero Q defed as: Q yˆ ) y), for the two methods. It s equal to 0.98 for the method usg the mode as a estmator (y ) ad equal to 0.69 for oe usg the usual krgg mea. (a) (b) 4. Cocluso Fg. 3. the values estmated versus the real values usg the usual krgg mea as a estmator (a) ad mode (b). I ths paper, we have proposed a ew techque to estmate covarace hyper-parameters of Gaussa processes wth equalty costrats. A suted cross valdato algorthm to equalty costrats s derved. The performace of the proposed method s vestgated by a real applcato two dmesos. Ackowledgemets Ths work has bee coducted wth the frame of the ReDce Cosortum, gatherg dustral (CEA, EDF, IFPE, IRS, Reault) ad academc (Mes de Sat-Etee, IRIA, ad the Uversty of Ber) parters aroud advaced methods for Computer Expermets. The authors also thak Xaver Bay (EMSE) ad Laurece Grammot (ICJ, Lyo1) for costructve commets.

8 Hassa Maatouk/ Proceda Evrometal Sceces 00 (015) Refereces 1. Bachoc F. Cross valdato ad maxmum lkelhood estmatos of hyper-parameters of Gaussa processes wth model msspecfcato. Computatoal Statstcs & Data Aalyss 013; 60(0): Cresse. A. C.. Statstcs for Spatal Data, revsed edto. Joh Wley & Sos, ew York Da Vega S., Marrel A. Gaussa process modelg wth equalty costrats. Aales de la faculté des sceces de Toulouse 01; 1(3):59-555, Golch S., Bgham D. R., Chpma H., Campbell D. A.. Mootoe Emulato of Computer Expermets. ArXv e-prts, September Maatouk H., Bay X. A ew Reecto Samplg Method for Trucated Multvarate Gaussa Radom Varables Restrcted to Covex Sets. hal , September Maatouk H., Bay X. Gaussa Process Emulators for Computer Expermets wth Iequalty Costrats. hal , December Rasmusse C. E., Wllams C. K. I. Gaussa Processes for Mache Learg (Adaptve Computato ad Mache Learg). The MIT Press Rhmak J., Vehtar A. Gaussa processes wth mootocty formato. I AISTATS, volume 9 of JMLR Proceedgs. p JMLR.org, Roustat O., Gsbourger D., Devlle Y. Dcekrgg, dceoptm: Two r packages for the aalyss of computer expermets by krgg-based metamodelg ad optmzato. Joural of Statstcal Software 10 01; 51(1):1-55.