Kriging and Alternatives in Computer Experiments

Transcription

1 Kriging and Alternatives in Computer Experiments C. F. Jeff Wu ISyE, Georgia Institute of Technology Use kriging to build meta models in computer experiments, a brief review Numerical problems with kriging Alternatives to kriging: Regularized kriging, Hybrid kriging Overcomplete basis surrogate model (OBSM) 1

2 2

3 3

4 4

5 Kriging Models Ordinary Kriging Y( x) = μ + Z( x) Correlation function ( 2 ) ( 2 ) n σ ϕ σ ϕ Z( x)~ N 0, ( h) GP 0, ( h) ϕ ( 0) = 1 ϕ( h) = ϕ( h), (symmetric function) ϕ is a positive semi definite function 5

6 Matérn Correlation function 1 ϕ ( h ) = 2 νθ 2 1 ( )2 h K νθ h Γ ν ν K ν ( ) ν ( ) ν where is the modified Bessel function of order ν 2 ν, ϕ ( h ) exp( θh ) Power exponential correlation ( q ) ϕ ( h ) = exp θ h, 0 < q 2, 0 < θ q =2 Gaussian correlation function (infinitely differentiate) q =1 Ornstein Uhlenbeck process (ν = 1 in Matern) Linear, Cubic correlation 6

7 Kriging predictor Best Linear Unbiased Predictor (BLUP),, ) y ( x ) i = y i, an interpolating property* *required for deterministic simulations 7

8 Recent work in kriging Calibration of computer model, Kriging with calibration parameters (Kennedy O Hagan, 2001), with tuning parameters (Santner et al., 2009) Computer simulations with different levels of accuracy (Kennedy O Hagan,2000; Qian et al., 2006; Qian Wu, 2008) construction of nestedspace filling space (e.g., Latin hypercube) designs (Qian Ai Wu, 2009, various papers pp by Qian and others, 2009 date) 8

9 Recent work in kriging (cont.) Kriging gfor multiple utpeoutputs and functional cto response (Conti et al., 2009; Conti and O Hagan, 2010) Treed Gaussian Process model (Gramacy and Lee, 2008). Kriging (i.e., GP model) with quantitative and qualitative factors (Qian Wu Wu, 2008, Han et al., 2009) construction of sliced space filling (e.g., Latin hypercube) designs (Qian Wu Wu, 2009, Qian, 2010) 9

10 Maximum Likelihood Estimation Profile log likelihood likelihood approach 10

11 Numerical Instability in R 1 (θ) R(θ) is an nxn matrix, n=sample l size Its condition number (max e.v./min e.v.) as I. Sample size n II. Dimension of input vectors (Peng Wu, 2010) 11

12 Branin function (Andre, Siarry and Dognon, 2001) 12

13 Log likelihood function 2 (Regular grid: n = 7 2 ) 13

14 Regularized Kriging Introducing a regularizing constant λ into the predictor Peng and Wu (2010, submitted) Similar modification in estimation: maximizing a regularized likelihood 14

15 Kriging with nugget effects Model from spatial statistics BLUP 15

16 Algorithm (Ridge Trace) Root Mean Squared Prediction Error (RMSPE) N 1 RMSPE = ( Y( x ) ˆ i Y( xi)) N i =

17 Log likelihood function, Branin function, (Regular grid: n = 7 2 ) Log-likelihood n = 49, lambda = 1e θ θ 1 17

18 Regularized Kriging x x x

19 Overcomplete Basis Surrogate Model Use an overcomplete dictionary of basis functions Use linear combinations of basis functions to approximate unknown functions Use Matching Pursuit for fast (i.e. greedy) computations Choice of basis functions to mimic the shape of the surface Chen, Wang, and Wu (2010, IIE Tran. Q&R) 19

20 Surrogate Representation Surrogate model: use a linear combination of pre specified basis functions, i.e., f (x) = j c j φ j (x), x χ unitary norm ǁφ j ǁ = 1 basis dictionary, {φ j, j = 1,, M} no unknown parameter in φ j, only unknown are the linear c j Overcomplete : M much larger than data size 20

21 Surrogate Model (continued) Explored poedpoint set: P exp = {x 1,, x p }. Current responses: V = ( f ( x ),..., f ( x ) ) T P exp ( 1 p ) Use to approximate V Pexp, ~ φ = j ( ) φ T c φ ~ j j j j ( x1 ),..., φ j ( x p ) Two interesting questions: Choice of the basis functions? Estimation of the linear coefficients c i? 21

22 Coefficient Inference Matching Pursuit Algorithm (Mallat and Zhang, 1993): ~ V c φ Infercoefficients by minimizing P exp j j j A greedy algorithm: at the jth iteration, Let R (j 1) be the current residual vector. ~ ( j 1) ~ Selected a basis by φ( j ) = arg max i R, φi : c R ~ φ ( j 1) ( j ) = c( j ) + R, ( j ) ( j ) = R ( j 1) R ~ φ, ~ φ ( j 1), ( j ) ( j ). 22

23 Response Surface for Bistable Laser Diodes The true surface over a pre specified grid: Search all positive Lyapunov exponents (PLE) (red area) PLE corresponds to chaotic light output. 23

24 Gabor Functions Basis functions: n dimensional Gabor function T x Mx x g ( x ) = exp( )exp(2 πia x ), x = ( x,..., x 2 ( 1 n Two dimensional Gabor function, i.e. n = 2 ) T 24

25 Plots of 2 D Gabor Function 25

26 Overall Comparisons 26

27 Summary Computer experiments/simulations have becomepopular in engineering and science Kriging is the most common method for statistical meta model building but is more oe limited for large or complex problems Alternatives to kriging gare being sought: Tweaking of kriging to achieve stability (regularized, hybrid, tapering, reduced rank) Approximations with fast computations (OBSM, RIDW): but lacking inferential capability 27

28 Covariance Mti Matrix Tapering Covariance tapering g( (Kaufman et al., 2008) Covariance matrix is tapered or multiplied element wise by a sparse matrix, to approximate the likelihood. Advantages: Significant computational gains/stability. Retain interpolating property. Asymptotic convergence of the tamper estimator. But: The tapering function is isotropic: OK for spatial statistic problems, but not applicable to engineering problems. The tapering radius needs to be determined. 28

29 Rank krd Reduction Fixed rank kriging g( (Cressie-Johannesson, 2008) A flexible family of non-stationary covariance function is defined by using a set of basis functions that are fixed in number (smaller than the data size n). Advantage: Reduce the computational cost of kriging g to O(n). () But: How to choose the appropriate p basis functions. Not an interpolator. 29

30 Upper bound Upper bound exp( θ ) = 1 θ = 0 The worst case of a correlation matrix k k 30

31 31

32 32

33 33

34 Lower bound on λ Lower bound where Machine accuracy (or unit round off) 34