Thomas Santner Ohio State University Columbus OH. Joint Statistical Meetings Seattle, WA August 2006

Transcription

1 Introductory Overview Lecture on Computer Experiments - The Modeling and Analysis of Data from Computer Experiments Thomas Santner Ohio State University Columbus OH Joint Statistical Meetings Seattle, WA August 2006

2 Outline 1. Experiments 2. Experimentation using Computer Codes 3. A Taxonomy of Problems 4. Gaussian Stochastic Process (GaSP) Models &Þ Prediction based on the GaSP Model 6. Conclusions-Take Home Messages

3 1. Experiments Physical Experiments Gold standard for establishing cause and effect relationships Mainstay of Agriculture, Industry, Medicine Principles of randomization, blocking, choice of sample size, and stochastic modeling of response variables all developed in response to needs of physical experiments

4 1. Experiments Physical Experiments Gold standard for establishing cause and effect relationships Mainstay of Agriculture, Industry, Medicine Principles of randomization, blocking, choice of sample size, and stochastic modeling of response variables all developed in response to needs of physical experiments Simulation Experiments Complex physical system each of whose parts interact in a known stochastic manner but whose ensemble is not understood analytically. Heavily used in Industrial Engineering--e.g., compare job shop set ups

5 1. Experiments Physical Experiments Gold standard for establishing cause and effect relationships Mainstay of Agriculture, Industry, Medicine Principles of randomization, blocking, choice of sample size, and stochastic modeling of response variables all developed in response to needs of physical experiments Simulation Experiments Complex physical system each of whose parts interact in a known stochastic manner but whose ensemble is not understood analytically. Heavily used in Industrial Engineering--e.g., compare job shop set ups Computer Experiments relatively new (below)

6 1. Experiments Physical Experiments Gold standard for establishing cause and effect relationships Mainstay of Agriculture, Industry, Medicine Principles of randomization, blocking, choice of sample size, and stochastic modeling of response variables all developed in response to needs of physical experiments Simulation Experiments Complex physical system each of whose parts interact in a known stochastic manner but whose ensemble is not understood analytically. Heavily used in Industrial Engineering--e.g., compare job shop set ups Computer Experiments relatively new (below) Combinations of the above

7 2. Experimentation using Computer Codes In some situations performing a physical experiment is not feasible 1. Physical process is technically too difficult to study 2. Number of variables is too large 3. Too expensive to study directly (it's all money) 4. Ethical considerations

8 2. Experimentation using Computer Codes In some situations performing a physical experiment is not feasible 1. Physical process is technically too difficult to study 2. Number of variables is too large 3. Too expensive to study directly (it's all money) 4. Ethical considerations When physical experiments are not possible, it may still be feasible to conduct a computer experiment

9 2. Experimentation using Computer Codes In some situations performing a physical experiment is not feasible When physical experiments are not possible, it may still be feasible to conduct a computer experiment IF the physical process relating the inputs B to the response(s) a. Can be described by a mathematical model relating the output, CÐBÑ, to B b. Numerical methods exist for solving the mathematical model c. The numerical methods can be implemented with computer code (in finite time!) THEN one can run the computer code to produce a "response'' CÐBÑ at any input B k., i.e., one can conduct a computer experiment B Ò Code Ò CÐBÑ The computer code is a proxy for the physical process.

10 Features of Computer Experiments B Ò Code Ò CÐBÑ CÐBÑ is deterministic

11 Features of Computer Experiments B Ò Code Ò CÐBÑ CÐBÑ is deterministic CÐB Ñ may be biased (hence the need for "calibration")

12 Features of Computer Experiments B Ò Code Ò CÐBÑ CÐBÑ is deterministic CÐBÑ may be biased Traditional principles used in designing physical experiments to balance the effects of non-identical experimental units (randomization, blocking, etc) are irrelevant.

13 Features of Computer Experiments B Ò Code Ò CÐBÑ CÐBÑ is deterministic CÐBÑ may be biased Traditional DoE principles are irrelevant Sometimes output from a physical experiments is also available. Usual philosophy physical experiment is a noisy measurement of the true input- output relationship ÐB Ò. X ÐBÑÑÞ Model : X c X C B œ. ÐBÑ % ÐBÑ and C B œ. ÐBÑ $ ÐBÑ e% ÐBÑf B are measurement errors (usually, white noise) e$ ÐBÑfB is computer model bias Warning Sometimes physical experiments are available only for components of the ensemble process, eg, code that emulates an auto crash test. In other cases, only experiments that approximate reality are available, e.g., a knee simulator

14 Features of Computer Experiments B Ò Code Ò CÐBÑ CÐBÑ is deterministic CÐBÑ may be biased Traditional DoE principles are irrelevant Sometimes output from a physical experiments is also available. Our interest in settings where Few computer runs are possible - complex codes Ðfine-grid FEA codesñ Highqdimensional input B

15 (1) Design of VLSI circuits Examples of Computer Experiments (2) Design automobile engines, and other components Kai-Tai Fang, Runze Li, and Agus Sudjianto (2005) Design and Modeling for Computer Experiments (3) Determine optimum operating conditions for a compression molding process (4) Determine the performance of controlled nuclear fusion devices (5) Describe the temporal evolution of contained and wild fires (6) Design of jet engines, helicopter rotor blades (7) Biomechanics Explain behavior of (or even design) prosthetic devices

16 Example Zone Computer Models used to predict the evolution of a fire in an enclosed room (eg, the computer code ASET = Available Safe Egress Time) In particular, ASET-B describes the of a fire in a single room temporal evolution with closed doors and windows that contains an object at some point below the ceiling that has been ignited.

17

18

19

20 Mathematical Model Inputs to ASET-B Room ceiling height ÐB Ñ " Room floor area ÐB Ñ # Height of the fire source (the burning object) above the floor ÐB Ñ $ Heat loss fraction for the room (depends on room insulation) ÐB Ñ % (etc, material-specific heat release rate) Computed Response CÐB", B# ßB$ ßB% Ñ œ time required by smoke layer to reach 5 ft above the ground

21 Objective Predict the time for the fire plume to reach 5 ft above the ground for untried combinations

22 3. A Taxonomy of Problems Setup 1. Inputs BœÐBßB - /, B7Ñwhere B - œ control ( manufacturing, engineering design) variables B / œ noise ( field, enviromental) variables B7 œ model variables (Not all types of inputs need be present in every application Þ)

23 3. A Taxonomy of Problems Setup 1. Inputs BœÐBßB - /, B7Ñwhere B - œ control ( manufacturing, engineering design) variables B / œ noise ( field, enviromental) variables B7 œ model and calibration variables (Not all types of inputs need be present in every application Þ) 2. Outputs Real-valued : CÐBÑ or Multivariate: C" ÐBÑß C# ÐBÑß á ß C5ÐBÑ or Functional: (>ß CÐ>ß BÑÑ

24 3. A Taxonomy of Problems Setup 1. Inputs BœÐBßB - /, B7Ñwhere B - œ control ( manufacturing, engineering design) variables B / œ noise ( field, enviromental) variables B7 œ model variables (Not all types of inputs need be present in every application Þ) 2. Outputs Real-valued : CÐBÑ or Multivariate: C" ÐBÑß C# ÐBÑß á ß C5ÐBÑ or Functional: (>ß CÐ>ß BÑÑ 3. X / µjðñ describes "target field conditions'' X 7 µ 1( Ñ describes prior knowledge about the model variables

25 3. A Taxonomy of Problems Setup 1. Inputs BœÐBßB - /, B7Ñwhere B - œ control ( manufacturing, engineering design) variables B / œ noise ( field, environmental) variables B7 œ model variables (Not all types of inputs need be present in every application Þ) 2. Outputs Real-valued : CÐBÑ or Multivariate: C" ÐBÑß C# ÐBÑß á ß C5ÐBÑ or Functional: (>ß CÐ>ß BÑÑ 3. X / µjðñ describes "target field conditions'' X 7 µ 1( Ñ describes prior knowledge about the model specifications 4. Summary of the CÐBß\ Ñ distribution (assuming no X ) - / 7.ÐB-Ñ œ IJeCÐB- ß\ / Ñfà 0( B-): P JeCÐB- ß\ / Ñ Ÿ 0( B-) f œ α (median) à ( B ) œ Var ÐCÐB ß\ ÑÑ 5 # - J - /

26 3. A Taxonomy of Problems Setup 1. Inputs BœÐBßB - /, B7Ñwhere B - œ control ( manufacturing, engineering design) variables B / œ noise ( field, environmental) variables B7 œ B7ß7+>2ß B7ß8?7 œ model variables (Not all types of inputs need be present in every application Þ) 2. Outputs Real-valued : CÐBÑ or Multivariate: C" ÐBÑß C# ÐBÑß á ß C5ÐBÑ or Functional: (>ß CÐ>ß BÑÑ 3. X / µjðñ describes "target field conditions'' X 7 µ 1( Ñ describes prior knowledge about the model specifications 4. Summary (assuming no X7 ) of CÐBß\ - / Ñ distribution.ðb-ñ œ IJeCÐB- ß\ / Ñfà 0( B-): P JeCÐB- ß\ / Ñ Ÿ 0( B-) f œ α (median) à ( B ) œ Var ÐCÐB ß\ ÑÑ 5 # - J - / > > 5. "Design" of a computer experiment choice of B" ßá, B8 at which to > evaluate computer code, whereß eg, B œ ˆ > > B ßB, 3 = ", á, 8 3 -ß3 /ß3

27 3. Taxonomy of Problems Problem 1 Interpolation/Prediction Given computer code output at a set of training inputs, ˆ > > B B ˆ > > ßCÐ Ñ ßáß B ßCÐB Ñ " " 8 8 predict CÐ Ñ at a new input B! (predictor metamodel)

28 3. Taxonomy of Problems Problem 1 inputs, Interpolation/Prediction Given computer code output at a set of "training" ˆ > > B B ˆ > > ßCÐ Ñ ßáß B ßCÐB Ñ " " 8 8 predict CÐ Ñ at a new input B! (predictor metamodel) Problem 2 Experimental design Determine a set of inputs at which to carry out the sequence of code runs (a "good" design of a physical or computer experiment depends on the scientific objective of the research) Exploratory Designs ("space-filling") Prediction-based Designs Optimization-based Designs (e.g., find B 9:> - œ argmin CÐBÑ)

29 3. Taxonomy of Problems Problem 1 Interpolation/Prediction Problem 2 Experimental design Exploratory Designs ("space-filling") Prediction-based Designs Optimization-based Designs (e.g., find B 9:> - œ argmin CÐBÑ) Problem 3 Uncertainty/Output Analysis Determine the distribution of the random variable CÐB -, \/ Ñ. (Determine the variability in the performance measure C( ) for design B - when applied to the population defined by the distribution of \ /, eg, patient specific variables (patient weight or bone material properties) or surgeon specific variables (measuring surgical skill)

30 Example In his Cornell PhD thesis, Kevin Ong conducted an uncertainty analysis of the effect of Engineering Cup design, Surgical, Patient, and Fluid Effects on the Stability of Uncemented Acetabular Components

31 3. Taxonomy of Problems Problem 1 Interpolation/Prediction Problem 2 Experimental design Problem 3 Uncertainty/Output Analysis Problem 4 Sensitivity Analysis Determine how variation in CÐBÑ can be apportioned to the different inputs of B (which inputs is CÐBÑ not sensitive to? which sets of inputs is CÐBÑ most sensitive to?) Philosophy Inputs that have relatively little effect on the output can be set to some nominal value; additional investigation can be restricted to determining how the output depends on the active inputs

32 <œroom ceiling height +œ Room floor area 0œHeight of the fire source above the floor 2œHeat loss fraction for the room

33 3. Taxonomy of Problems Problem 1 Interpolation/Prediction Problem 2 Experimental design Problem 3 Uncertainty/Output Analysis Problem 4 Sensitivity Analysis Problem 5 Calibrate the computer code Use outputs from both a physical experiment and an associated computer code that represents the physical process to set the computer code calibration variables. Calibration variables are those B 7 variables that can be used to eliminate the bias in a computer code's representation of a physical input-output relationship Typical Numerical Variables (1) FEA Mesh Density œ? ß (2) Load Discretization œ? ß (3) Solution tolerances œ?, etc

34 3. Taxonomy of Problems Problem 1 Interpolation/Prediction Problem 2 Experimental design Problem 3 Uncertainty/Output Analysis Problem 4 Sensitivity Analysis Problem 5 Calibrate the computer code Problem 6 Prediction Accuracy Using the data from both a physical experimental data and a (calibrated) computer experiment, give predictions (including uncertainty bounds) for an associated physical system.

35 3. Taxonomy of Problems Problem 1 Interpolation/Prediction Problem 2 Experimental design Problem 3 Uncertainty/Output Analysis Problem 4 Sensitivity Analysis Problem 5 Calibration Problem 6 Prediction Problem 7 Find Robust Inputs In experiments with engineering design and patient-specific environmental variables, determine robust choices of the engineering design variables. If B -. B œi ecðb, \ Ñf - J - / then a robust set of inputs B - is an engineering "design" whose output is minimally sensitive to the assumed distribution JÐ Ñ of \ /

36 Bottom Line Many of the problems above have "natural" solutions obtained by approximating CÐB-, B/ Ñ by a fast (linear in the training data) predictorß a metamodel

37 4. Gaussian Stochastic Process (GaSP) Models (used as basis for both prediction and some design choices) Idea Regard CÐBÑ as a realization, a "draw ß " of a random function ] (Bayesian Thinking) The simplest possible (prior) model for ] B is ]ÐBÑœ " ÐBÑ î^ B ðóóñóóò 4 " large scale trends" " smooth deviations" œ " ð 0ÐBÑ ^ B where 0Ð " BÑáß0Ð, 5 BÑare known regression functions, " is an unknown regression vector, and ^ B is a stationary Gaussian Stochastic Process (GaSP) B

38 ^ÐBÑß B k satisfies Ee^ÐBÑf œ!ðzero meanñ ð ð Ê I e]ðbñf œ " 0ÐBÑ! œ " 0ÐBÑ Var ^ÐBÑ œ 5 #^ Correlation Function: symmetric VÐ Ñwith VÐ!Ñ œ ", # Cov( ^ÐB" Ñ, ^ÐB# Ñ) œ 5^ VÐB" B# Ñ 5 #^ œ Var ^Ð B Ñ Typically VÐÑœVÐl 0Ñis a function of a finite number of unknown parameters GaSP: For any B", á, B= ß ^ÐB" Ñá,, ^ÐB= Ñ has the multivariate normal distribution Usually, taking " ð 0ÐB Ñ œ "! with a data-selected parametric correlation function R l0

39 4. GaSP Models GaSP Models Are Flexible Four draws from ^ÐBÑß a zero mean, unit variance GaSP with input B [0,1] and having correlation function for ) e%ß "'ß "!! f VÐ2Ñ œ /B: ˆ ) 2 #

40 Some draws from a GaSP with inputs B Ò!ß "Ó #

41 4. GaSP Models GaSP Models Are Flexible GaSP Models are computationally "simple"

42 4. GaSP Models GaSP Models Are Flexible GaSP Models are computationally "simple" GaSP Models are Bayesian in character

43 &Þ Prediction based on the GaSP Model Given ( training) data ˆ > > B B ˆ > > ßCÐ Ñ ßáß B ßCÐB Ñ " " 8 8 predict C B ß where B!! is an untried new input.

44 &Þ Prediction based on the GaSP Model Given ( training) data predict C B ß where ˆ > > B B ˆ > > ßCÐ Ñ ßáß B ßCÐB Ñ B!! " " 8 8 is an untried new input. 8 > > 8 > > " 8 " 8 Notation ] œð]ðb Ñßáß]ÐB ÑÑ and C œðcðb Ñß áßcðb ÑÑ

45 &Þ Prediction based on the GaSP Model Given ( training) data predict C B ß where ˆ > > B B ˆ > > ßCÐ Ñ ßáß B ßCÐB Ñ B!! " " 8 8 is an untried new input. 8 > > 8 > > " 8 " 8 Notation ] œð]ðb Ñßáß]ÐB ÑÑ and C œðcðb Ñß áßcðb ÑÑ The minimum MSPE predictor of CÐ Ñ is B! scðb Ñ œ I e]ðb Ñl] œ C!! 8 8 f

46 &Þ Prediction based on the GaSP Model Given ( training) data predict C B ß where ˆ > > B B ˆ > > ßCÐ Ñ ßáß B ßCÐB Ñ B!! " " 8 8 is an untried new input. 8 > > 8 > > " 8 " 8 Notation ] œð]ðb Ñßáß]ÐB ÑÑ and C œðcðb Ñß áßcðb ÑÑ The minimum MSPE predictor of CÐ Ñ is B! scðb Ñ œ I e]ðb Ñl] œ C!! 8 8 Example If ]ÐB Ñfollows the GaSP( " ß5 ßVÐ Ñß ) then scðb Ñ I ]ÐB Ñl] 8 œ C 8 œ s ð " 8 e f " < ÐB ÑV Š C " s "!!!!!! 8 where R œ ˆ > > VÐB B Ñ 3 4 is 8 8 ð > > ð <ÐBÑœ! ÐVÐB! B" Ñß á ß VÐB! B 8 ÑÑ is " 8 " s " œ ð " " ð " 8 WLSE of " R " " R C!! # ^ f

47 &Þ Prediction based on the GaSP Model Given ( training) data predict C B ß where ˆ > > B B ˆ > > ßCÐ Ñ ßáß B ßCÐB Ñ B!! " " 8 8 is an untried new input. 8 > > 8 > > " 8 " 8 Notation ] œð]ðb Ñßáß]ÐB ÑÑ and C œðcðb Ñß áßcðb ÑÑ The minimum MSPE predictor of CÐ Ñ is B! scðb Ñ œ I e]ðb Ñl] œ C!! 8 8 Example If ]ÐB Ñfollows the GaSP( " ß5 ßVÐ Ñß ) then scðb Ñ I ]ÐB Ñl] 8 œ C 8 œ s ð " 8 e f " < ÐB ÑV Š C " s "!!!!!! 8 IF only the moment assumptions holds, CÐ Ñ Best Linear Unbiased Predictor of B! s B! ]Ð Ñ # ^ f

48 &Þ Prediction based on the GaSP Model Model Prediction uncertainty at B! 5 # ( B! ) œ E Ð]ÐB! Ñ CÐ s B! ÑÑ # l] 8 œ C 8

49 &Þ Prediction based on the GaSP Model Model Prediction uncertainty at B! 5 # ( B! ) œ E Ð]ÐB! Ñ CÐ s B! ÑÑ # l] 8 œ C 8 Empirical BLUP If the correlation is unknown (the usual case Ñ, Ê R, <ÐBÑ ð, and " s are also unknown : (!! ß VÐ Ñ œ VÐ l0ñis parametric, and we estimate 0 by If further, we can predict using the corresponding empirical BLUP s 0, say, 8 8 scðb Ñ I ]ÐB Ñl] œ C ß s œ s s< ÐB ÑVs 8 C s! š 0 " Š " " ð "!!!! 8 s0 œ MLE, REML, penalized likelihood ß or other estimator of 0

50 &Þ Prediction based on the GaSP Model Model Prediction uncertainty at B! 5 # ( B! ) œ E Ð]ÐB! Ñ CÐ s B! ÑÑ # l] 8 œ C 8 Empirical BLUP If the correlation is unknown (the usual case Ñ, X Ê R, < ÐB Ñ, and " s are also unknown : (!! Ifß further, VÐ Ñ œ VÐ l0ñis parametric, and we estimate 0 by s0, say, we can predict using the corresponding empirical BLUP "!!!!! 8 scðb Ñ I ]ÐB Ñl] 8 œ C 8 ß s œ s s<ðb ÑVs 8 š 0 " Š C s " " s0 œ MLE, REML, penalized likelihood ß or other estimator of 0 Fully Bayesian Predictor (giving all parameters priors) 8 8 e B ] C f œðóóóóóóóóóóóñóóóóóóóóóóóò # 8! B! ßC scðb! Ñ œ I ]Ð Ñl œ œ I I ]Ð Ñl"! ß ß0 5^ 8 Need Ò"! ß5 #^ ß0lC Ó known

51 Properties of s " 8 CÐB! Ñ œ " s ð < ÐB ÑV Š C " s " Simple to compute (linear in C 8 )!!! 8 > scðbñ œ - B - ÐBÑCÐB Ñ 8! 4 4œ" 4

52 Properties of s " 8 CÐB! Ñ œ " s ð < ÐB ÑV Š C " s " Simple to compute (linear in C 8 )!!! 8 > scðb Ñ œ - B - ÐB ÑCÐB Ñ!!! 4! 4œ" but, unfortunately, not the Empirical BLUP :-( GASP (W. Welch) PErK (B. J. Williams) SAS Proc Mixed BACCO (Hankin) and others... R -1 can be computationally demanding 8 4

53 Properties of s " 8 CÐB! Ñ œ " s ð < ÐB ÑV Š C " s " Simple to compute (linear in C 8 ) Viewed as a function of B!,!!! 8 > scðb Ñ œ - B - ÐB ÑCÐB Ñ!!! 4! 4œ" scðb Ñ œ.. Vˆ > B B!! 4! 4œ"

54 Properties of s " 8 CÐB! Ñ œ " s ð < ÐB ÑV Š C " s " Simple to compute (linear in C 8 ) Viewed as a function of B, scðb Ñ interpolates data, i.e.,!!! 8 > scðb Ñ œ - B - ÐB ÑCÐB Ñ!!! 4! 4œ" scðb Ñ œ.. Vˆ > B B!! 4! 4œ" 4 > > 4 8 scðb Ñ œ CÐB Ñß 4 œ "ßáß

55 Properties of s " 8 CÐB! Ñ œ " s ð < ÐB ÑV Š C " s " Simple to compute (linear in C 8 ) Viewed as a function of B, scðbñ interpolates data, i.e.,!!! 8 > scðb Ñ œ - B - ÐB ÑCÐB Ñ!!! 4! 4œ" scðb Ñ œ.. Vˆ > B B!! 4! 4œ" 4 > > 4 8 scð B Ñ œ CÐB Ñß 4 œ "ß á ß 8 Splines, neural networks and other well-known interpolators correspond to specific choices of regressors and correlation function VÐ Ñ 8 4 4

56 Illustration True Curve ( solid); five training data points ( diamonds); REML-EBLUP with exponential correlation function ( dotted)

57

58 5. Conclusions-Take Home Messages ". An increasing number of phenomenon that could previously be studied only by physical experiments, can now be investigated using "computer experiments'' or combinations of computer and physical experiments #. Modeling the responses from computer experiments must account for the (highly) correlated nature of the output CÐBÑ over the input space. $. Prediction of the output function CÐBÑ based on Gaussian (or other) stochastic processes can be used to interpolate training data %. GaSP models are the basis for Determining interpolating predictors of CÐBÑ Assessing error bands of the predictor due to model uncertainty, Targeted experimental design, and Solving calibration problems

59 Related Sessions 335 Multivariate Functional ANOVA for Kriging Model in Computer Experiments (Zhe Zhang) 345 Statistical Design of Computer Experiments for a 3D Chemical Microanalysis Imaging System (Juan Soto, James J. Filliben, and John H. Scott) 363 Design and Analysis of Experiments for Complex Computer Simulators (Aug 9 Session) Designs for Integrated Computer and Physical Experiments (C. Shane Reese, Derek Bingham, and Wilson Lu) Sequential Experiment Design for Contour Estimation from Complex Computer Codes (Pritam Ranjan) Uncertainty Quantification for Combining Experimental Data and Computer Simulations from Multiple Data Sources (Brian J. Williams, Dave Higdon,Jim Gattiker) 474 Network Analysis and Spatial Applications-Contributed (Aug 9 Session) Calibration and Prediction for Computer Experiment Output Having Qualitative and Quantitative Input Variables (Gang Han, Thomas Santner, and William Notz) Validity of Likelihood and Bayesian Inference for Gaussian Process Regression (Bela Nagy, Jason Loeppky, and William J. Welch) Questions?