Module 3: Gaussian Process Parameter Estimation, Prediction Uncertainty, and Diagnostics

Transcription

1 Mdule 3: Gaussian Prcess Parameter Estimatin, Predictin Uncertainty, and Diagnstics Jerme Sacks and William J Welch Natinal Institute f Statistical Sciences and University f British Clumbia Adapted frm materials prepared by Jerry Sacks and Will Welch fr varius shrt curses Acadia/SFU/UBC Curse n Dynamic Cmputer Experiments September December 2014 J Sacks and WJ Welch (NISS & UBC) Mdule 3: Estimatin and Uncertainty Cmputer Experiments / 20

2 Outline f Tpics Outline 1 Estimating the Parameters f the GP Mdel 2 Case Study: G-Prtein Cmputer Experiment 3 Measuring Predictin Accuracy 4 GP Diagnstics 5 Summary 6 Appendix J Sacks and WJ Welch (NISS & UBC) Mdule 3: Estimatin and Uncertainty Cmputer Experiments / 20

3 Estimating the Parameters f the GP Mdel Parameters f the Gaussian Prcess (GP) Mdel Recall frm Mdule 2 that the Gaussian prcess prir fr y(x) = y(x 1,, x d ) has hyper-parameters: mean, µ, variance, σ 2 crrelatin parameters, eg, θ 1,, θ d and p 1,, p d fr the pwer-expnential crrelatin functin, R(x, x ) = d exp( θ j x j x j p j ) j=1 Their values will be chsen t be cnsistent with the cmputer-mdel runs J Sacks and WJ Welch (NISS & UBC) Mdule 3: Estimatin and Uncertainty Cmputer Experiments / 20

4 Estimating the Parameters f the GP Mdel Maximum Likelihd Recall als that y(x) is assumed t be Gaussian Hence, y = [y(x (1) ),, y(x (n) )] T, the data frm the cmputer mdel, are a sample frm a multivariate-nrmal distributin The likelihd, L(y µ, σ 2, θ 1,, θ d, p 1,, p d ), is 1 (2πσ 2 ) n/2 det 1/2 (R) exp( 1 2σ 2 (y µ1)t R 1 (y µ1)) Maximum likelihd estimatin (MLE) chses the hyper-parameters t maximize this Or use Bayes rule t get a psterir distributin fr the hyper-parameters and fr predictins f y(x) (see Appendix A) J Sacks and WJ Welch (NISS & UBC) Mdule 3: Estimatin and Uncertainty Cmputer Experiments / 20

5 Estimating the Parameters f the GP Mdel Maximum Likelihd: Cmputatin Fr fixed crrelatin parameters, and ˆµ = 1T R 1 y 1 T R 1 1 σ 2 = 1 n (y ˆµ1)T R 1 (y ˆµ1) The likelihd functin (with ˆµ and σ 2 substituted) has t be numerically maximized wrt the crrelatin parameters J Sacks and WJ Welch (NISS & UBC) Mdule 3: Estimatin and Uncertainty Cmputer Experiments / 20

6 Case Study: G-Prtein Cmputer Experiment G-Prtein Cmputer Mdel Bisystems mdel fr s-termed ligand activatin f G-prtein in yeast d = 4 input variables x is cncentratin f ligand u 1,, u 8 is a vectr f 8 kinetic parameters (nly u 1, u 6, and u 7 are varied) Output variable y is the nrmalized cncentratin f part f the cmplex J Sacks and WJ Welch (NISS & UBC) Mdule 3: Estimatin and Uncertainty Cmputer Experiments / 20

7 Case Study: G-Prtein Cmputer Experiment G-Prtein System Dynamics: Differential Equatins 1 η 1 = u 1 η 1 x + u 2 η 2 u 3 η 1 + u 5 2 η 2 = u 1 η 1 x u 2 η 2 u 4 η 2 3 η 3 = u 6 η 2 η 3 + u 8 (G tt η 3 η 4 )(G tt η 3 ) 4 η 4 = u 6 η 2 η 3 u 7 η 4 5 y = (G tt η 3 )/G tt where η 1,, η 4 are cncentratins f 4 chemical species and η 1 η 1 t, etc G tt = (fixed) ttal cncentratin f G-prtein cmplex after 30 secnds J Sacks and WJ Welch (NISS & UBC) Mdule 3: Estimatin and Uncertainty Cmputer Experiments / 20

8 Case Study: G-Prtein Cmputer Experiment Inputs and Cde Runs Input variables d = 4 variables Wrk with lg(x), lg(u 1 ), lg(u 6 ), lg(u 7 ) ie, what we called the x vectr befre is lg(x), lg(u 1 ), lg(u 6 ), and lg(u 7 ) here All input variable ranges are nrmalized t [0, 1] n the lg scale Number f runs n = 33 (this chice and the design fr the 33 runs is described in Mdule 4) J Sacks and WJ Welch (NISS & UBC) Mdule 3: Estimatin and Uncertainty Cmputer Experiments / 20

9 Case Study: G-Prtein Cmputer Experiment Cmputer Mdel Data ymd ymd lgu lgu6 ymd ymd lgu lgx J Sacks and WJ Welch (NISS & UBC) Mdule 3: Estimatin and Uncertainty Cmputer Experiments / 20

10 Case Study: G-Prtein Cmputer Experiment Gaussian Prcess (GP) Mdel y(x) is a realizatin f a Gaussian prcess with: mean µ variance σ 2 crrelatins given by Cr(y(x), y(x )) R(x, x ) = The parameters in red need t be estimated 4 j=1 e θ j x j x j p j J Sacks and WJ Welch (NISS & UBC) Mdule 3: Estimatin and Uncertainty Cmputer Experiments / 20

11 Case Study: G-Prtein Cmputer Experiment Maximum Likelihd Estimates ˆµ = 036 ˆσ 2 = 051 Variable ˆθ ˆp lg(x) lg(u 1 ) lg(u 6 ) lg(u 7 ) It is difficult t interpret the magnitudes f the estimates (we will revisit this example in Mdule 5 and d a sensitivity analysis) J Sacks and WJ Welch (NISS & UBC) Mdule 3: Estimatin and Uncertainty Cmputer Experiments / 20

12 Measuring Predictin Accuracy Plug-In Predictin and Standard Errr Replace all hyper-parameters by their MLEs in the cnditinal mean and variance frmulas: and predictin f y(x) = ŷ = ˆm(x) = ˆµ + r T (x)r 1 (y ˆµ1) estimated variance f predictin = ˆv(x) = σ 2 (1 r T (x)r 1 r(x)) (R and r(x) are als estimates) The plug-in estimated variance ignres uncertainty in estimating the hyper-parameters It can be adapted t include uncertainty frm estimating µ: ˆv(x) = σ 2 ( 1 r T (x)r 1 r(x) + [1 1T R 1 r(x)] 2 1 T R 1 1 This plug-in frmula is ften used t give a standard errr, ie, s(x) = ˆv(x) J Sacks and WJ Welch (NISS & UBC) Mdule 3: Estimatin and Uncertainty Cmputer Experiments / 20 )

13 Measuring Predictin Accuracy Measures f Accuracy We culd rely n the standard errr, ˆv(x) If we have m test data bservatins, the rt mean squared errr (RMSE) f predictin is 1 RMSE = (ŷ y(x)) m 2 But rarely available Crss validatin (CV) test pts J Sacks and WJ Welch (NISS & UBC) Mdule 3: Estimatin and Uncertainty Cmputer Experiments / 20

14 Crss Validatin (CV) GP Diagnstics Let x (i) dente x fr run i in the data (i = 1,, n) Fr run i: The crss validated predictin f y(x (i) ) is ŷ i (x (i) ), ie, ŷ(x) = ˆm(x) cmputed frm the n 1 runs excluding run i The crss validated standard errr f ŷ i (x (i) ) is s i (x (i) ), ie, s(x) = ˆv(x) cmputed frm the n 1 runs excluding run i The crss-validated residual fr run i is y(x (i) ) ŷ i (x (i) ) The standardized crss-validated residual fr run i is y(x (i) ) ŷ i (x (i) ) s i (x (i) ) J Sacks and WJ Welch (NISS & UBC) Mdule 3: Estimatin and Uncertainty Cmputer Experiments / 20

15 Diagnstic Plts GP Diagnstics Plt the crss-validated residuals t assess the verall magnitude f errr Plt the standardized crss-validated residuals t assess the validity f the standard errr fr individual predictins J Sacks and WJ Welch (NISS & UBC) Mdule 3: Estimatin and Uncertainty Cmputer Experiments / 20

16 GP Diagnstics G-Prtein Diagnstic Plts ymd Standardized residual Predicted ymd Predicted ymd J Sacks and WJ Welch (NISS & UBC) Mdule 3: Estimatin and Uncertainty Cmputer Experiments / 20

17 GP Diagnstics Crss Validatin: Numerical Summaries Magnitude f errr The crss-validated rt mean squared errr is 1 CVRMSE (y(x n (i) ) ŷ i (x (i) )) 2 = 020 Maximum crss-validated residual is 044 Fairly accurate relative t a range f abut 07 in y Standard errrs? y(x(i) ) ŷ i (x (i) ) fr i = 1,, n are rughly in ( 2, 2) s i (x (i) ) Standard errrs lk reliable J Sacks and WJ Welch (NISS & UBC) Mdule 3: Estimatin and Uncertainty Cmputer Experiments / 20

18 Fast and Slw CV GP Diagnstics When run i is remved, the hyper-parameters shuld be re-estimated Fr cmputatinal reasns the crrelatin parameters are ften nt updated (it is cheap t update the estimates f µ and σ 2 ), prducing a fast CV Fr slw CV, d say 10-fld crss-validatin, re-estimating all hyper-parameters The agreement between fast CVRMSE and slw CVRMSE is ften gd The agreement between fast CVRMSE and the RMSE frm test pints has been gd in examples J Sacks and WJ Welch (NISS & UBC) Mdule 3: Estimatin and Uncertainty Cmputer Experiments / 20

19 Mdule Summary Summary The GP mdel has t be tuned t data s that its prperties match thse f the cmputer mdel Tuning (fitting) the GP by maximum likelihd is cmputatinally feasible fr up t abut n = 1000 runs and d = 50 input variables GP mdel gives an apprximatin and a measure f accuracy The measure f accuracy (standard errr) can be checked fr validity by crss validatin J Sacks and WJ Welch (NISS & UBC) Mdule 3: Estimatin and Uncertainty Cmputer Experiments / 20

20 Appendix Appendix A: Bayesian Treatment f the Hyper-parameters Psterir distributin f the hyper-parameters ( hyper belw), µ, σ 2, θ 1,, θ d, etc, f the GP Frm Bayes rule, given the data y p(hyper y) π(hyper)l(y hyper), π(hyper) is the prir fr hyper L(y hyper) is the multivariate nrmal likelihd Predictive distributin fr y(x) at a new x p(y(x) y) = p(y(x) y, hyper)p(hyper y) dhyper Usually, the integratin is nt carried ut explicitly Rather, prperties such as the psterir predictive mean and variance f p(y(x 0 ) y) are btained by MCMC sampling f the psterir distributin fr the hyper-parameters, p(hyper y) J Sacks and WJ Welch (NISS & UBC) Mdule 3: Estimatin and Uncertainty Cmputer Experiments / 20