The Pennsylvania State University The Graduate School A METHODOLOGY FOR EVALUATING SYSTEM-LEVEL UNCERTAINTY

Transcription

1 The Pennsylvania State University The Graduate School A METHODOLOGY FOR EVALUATING SYSTEM-LEVEL UNCERTAINTY IN THE CONCEPTUAL DESIGN OF COMPLEX MULTIDISCIPLINARY SYSTEMS A Thesis in Mechanical Engineering by Jay D. Martin c 2005 Jay D. Martin Submitted in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy May 2005

2 The thesis of Jay D. Martin was reviewed and approved by the following: Timothy W. Simpson Associate Professor of Mechanical Engineering and Industrial Engineering Chair of Committee Thesis Advisor Kon-Well Wang William E. Diefenderfer Chaired Professor in Mechanical Engineering Russell R. Barton Professor of Supply Chain and Information Systems Mark T. Traband Research Associate Runze Li Assistant Professor of Statistics Richard Benson Professor of Mechanical Engineering Head of the Department of Mechanical & Nuclear Engineering Signatures are on file in the Graduate School.

3 Abstract Conceptual design is the early stage of system design, when little is precisely known about the physical description of a new system. One of the goals in conceptual design is to aggregate all current corporate knowledge about the new design and exhaustively search the feasible design space to find potential designs that best meet the design s requirements and satisfies its constraints. In the conceptual design stage, simplified models are often created in preference to more complex models to permit the rapid assessment of many designs that cover the entire feasible design space. Uncertainty in the assessment of a potential design may result from uncertainty in the inputs to the design, such as sizes, weights, efficiencies, or costs. The use of simplified models may also introduce additional uncertainty, termed model uncertainty, due to the reduction in the number of parameters used to describe the system or due to incorrect relationships between parameters. A metamodel is a model of a model and can be used as a computationally efficient approximation to a computer model such as a finite element analysis. Kriging models are a type of metamodel that can interpolate their observations and provide a probability distribution of the output that quantifies the model uncertainty. Kriging models are created as simplified models from observations of detailed subsystem models. A Monte Carlo simulation (MCS) based methodology is developed to permit the specification of arbitrary probability distributions of the inputs to the system design using a hierarchy of kriging models. Through the use of kriging models, the model uncertainty introduced can also be quantified along with the input uncertainties impact on the system performance measurements. This methodology is demonstrated on a satellite design problem composed of three subsystems. These results are compared to those found using original computer models in the MCS system uncertainty assessment. This methodology enables the computationally efficient use of MCS with simple random sampling to estimate the resulting uncertainty of the system s performance parameters given the probability distribution of the system inputs and the uncertainty introduced by using approximations to the original deterministic computer models. iii

4 Table of Contents List of Figures List of Tables Acknowledgments viii x xiii Chapter 1 Introduction Motivation Review of Multidisciplinary Design Optimization Importance of Uncertainty Assessment During Design System Uncertainty in Design Current Uses of Kriging Metamodels in Design Research Objective and Tasks Thesis Roadmap Chapter 2 Background on Metamodels Input and Output Data Polynomial Regression Model Derivation of the Polynomial Regression Model Requirements for the Linear Regression Model Kriging Models Derivation of the Kriging Model Requirements for the Kriging Model Kriging Model Parameter Estimation Kriging Model Assessment Summary Remarks iv

5 Chapter 3 Parameter Estimation Methods for Deterministic Kriging Models Test Functions One-Dimensional Test Function Two-Dimensional Test Functions Five-Dimensional Test Function Graphical Methods for Parameter Estimation Experimental Variogram Correlation Plots Advantages and Deficiencies of the Graphical Method Cross-Validation Derivation of Equations Comparison of MLE and CV Methods Importance of Residual Distribution Computational Issues with MLE and CV Trend Function Importance to CV and MLE Recommendations Chapter 4 Parameter Estimation Methods for Probabilistic Kriging Models Maximum Likelihood Estimation Observed Fisher Information Matrix Estimation of Prediction Mean Square Error Bayesian Analysis Comparison of Bayesian Analysis and MLE Markov Chain Monte Carlo Derivation of Equations Convergence Issues Estimate of Marginal Distributions Simulated Output Distribution Recommendations Chapter 5 Kriging Model Assessment Methods Estimation of Root Mean Square Error Validation Data Method Cross-Validation Method Integrated Mean Square Error Method Simulation Method Comparison of RMSE Estimators v

6 5.2 Coefficient of Multiple Determination Information Criteria Likelihood Ratio Test Penalized Model Selection Criteria Akaike Information Criterion Bayesian Information Criterion Information Criteria Summary Comparisons of Model Selection Methods Coefficients of Multiple Determination Akaike Information Criterion Recommendations Chapter 6 Methodology and Case Study Overview of the Methodology Step 1: Establish a System Model Rocket Engine Analysis Fuel and Oxidizer Tank Analysis Orbit Mechanics Step 2: Create Probabilistic Kriging Models for Subsystems Design of Experiments Establish Kriging Model Inputs and Outputs Estimate Kriging Model Parameters Assess Model Quality Step 3: Specify a Design Step 4: Define Input Uncertainty for the Design Step 5: Perform Monte Carlo Simulation of System Model Step 6: Analyze the Results Uncertainty Assessment Sensitivity Analysis Application of Proposed Methodology to RBDO Recommendations Chapter 7 Conclusions Summary of Thesis Outline of Research Contributions Benefits of Research Contributions Future Work vi

7 Appendix A Data Used in Test Functions 161 A.1 One-Dimensional Test Function A.2 Two-Dimensional Functions A D Test Function A D Mystery Function A D Branin Function A D Six-hump Camelback Function A.3 Five-Dimensional Test Function Appendix B Derivatives of the Gaussian Spatial Correlation Function 186 B.1 Definitions of Covariance Functions B.2 First Partial Derivatives B.3 Second Partial Derivatives Appendix C Data and Results of Satellite Design Problem 190 C.1 Observations of the Analyses C.2 Kriging Models of the Observations Bibliography 199 vii

8 List of Figures Figure: 1.1 Representations of Reality and a Model Figure: 2.1 Plots of Four Spatial Correlation Functions Figure: 3.1 Output Temp. vs. Oxidant Fuel Ratio Figure: 3.2 Plots of the Four 2-D Test Functions Figure: D Latin Hypercube Sampled Points Figure: 3.4 Variogram of One-Dimensional Test Function Figure: 3.5 Correlation Plots for the One-Dimensional Kriging Models Figure: 3.6 Correlation Plots for the 2-D Test Function 1 Kriging Models Figure: 3.7 Plot of standard error vs. p-value of the Shapiro-Wilk s test for normality Figure: 3.8 Contour Plots of MLE and CV Parameter Estimation Methods Figure: 3.9 Plots of R 2 actual vs. Trend Function for the Six Test Functions Figure: 3.10 Plots of the Geometric Mean of the Correlation Ranges vs. Trend Function for the Six Test Functions Figure: 4.1 Likelihood Plots of Model Parameters Figure: 4.2 Approximated and Actual Likelihood Plots of Model Parameters.. 62 Figure: 4.3 Plots of Marginal Probabilities and Likelihoods for π ( β, σ 2, θ ) = Figure: 4.4 Plots of Marginal Probabilities and Likelihoods for π ( β, σ 2, θ ) = 1 σ 2 69 Figure: 4.5 Plots of Marginal Probabilities and Likelihoods for π ( β, σ 2, θ ) = 1 70 σ 3 Figure: 4.6 Plots of the Mean of the Model Parameters vs. Length of the Chain. 75 Figure: 4.7 Plots of the Simulated pdfs of the Model Parameters for the 1-D Test Function Figure: 4.8 Plots of the Simulated pdfs of the Model Parameters for the 2-D Six- Hump Camelback Function Figure: 4.9 Plots of Simulated pdfs of the Model Parameters for 2-D Test Function Figure: 4.10 Simulation Results of 1-D Kriging Model Output at x = Figure: 4.11 Simulation Results of 1-D Kriging Model Output viii

9 Figure: 4.12 Simulation Results of 2-D Test Function 1 Kriging Model Output.. 84 Figure: 5.1 RMSE Estimates vs. Trend Function Using the IMSE Method Figure: 5.2 RMSE Estimates vs. Trend Function Using the Simulation Method. 95 Figure: 5.3 Comparison of RMSE Estiamtes for the Six Test Functions Figure: 5.4 Probability Distribution of the Validation Errors for the 5-D Test Function Figure: 5.5 Plots of all R 2 prediction vs. R2 actual for the Six Test Functions Figure: 5.6 Combined Plot of MLE and MCMC R 2 prediction vs. R2 actual for the Six Test Functions Figure: 5.7 Plots of AIC c vs. R 2 actual for the Six Test Functions Figure: 6.1 Flowchart of Proposed Methodology Figure: 6.2 Diagram of System and Subsystem Models Figure: 6.3 Diagram for Design of a Tank Figure: 6.4 Diagram for Finite Burn Loss Maneuver [121] Figure: 6.5 Distribution of Scaled Total Satellite Masses for the Burn Time Model129 Figure: 6.6 Burn Time as a Function of Mass Flow Rate and Final Mass Figure: 6.7 Logarithm of Burn Time as a Function of Mass Flow Rate and Final Mass Figure: 6.8 Block Diagram of the Method Used to Create Probabilistic Kriging Models Figure: 6.9 Revised System Diagram that Uses Kriging Subsystem Models Figure: 6.10 Histograms for Outputs of Probabilistic Kriging Monte Carlo Simulation Figure: 6.11 Bar Charts of the System Input/Output Correlations Figure: 6.12 Convergence Charts for Simulated Annealing Algorithm ix

10 List of Tables Table: 2.1 Common Spatial Correlation Functions Table: 2.2 Examples of Transformations Used for Kriging Models Table: 3.1 Rank Correlations of R 2 actual and the Number of Parameters in the Trend Function Table: 3.2 Abbreviations for the Trend Functions Table: 3.3 Results of Tests to Determine if MLE and CV Result in Equivalent R 2 actual Table: 3.4 Rank Correlations of Correlation Ranges and the Number of Parameters in the Trend Function Table: 3.5 Test Results for the Comparison of the Estimated Correlation Ranges 54 Table: 4.1 Covariance Matrix, MLE Selected Values, and Standard Deviations of the Model Parameters for the 1-D Test Function Table: 4.2 Correlation Matrix of the Model Parameters for the 1-D Test Function 62 Table: 4.3 Properties for the Gamma Distribution Table: 4.4 Model Parameter Estimates for Different Prior Assumptions Table: 4.5 Mean and Variances of the Model Parameters for Many Acceptance Table: 4.6 Rates Mean and Variances of the Model Parameters for Different Sampling Rates Table: 4.7 Model Parameter Estimates for 2-D Test Function Table: 4.8 Shape Correction Factor, λ, as a Function of Student-t Shape Factor, ν 82 Table: 5.1 Comparisons of RMSE Estimates Table: 5.2 Rank Correlations of R 2 Metrics Table: 5.3 Rank Correlations of AIC c Metrics Table: 6.1 Satellite System Design Parameters and Point Design Table: 6.2 Rocket Engine Design Parameters Table: 6.3 Tank Analysis Design Parameters Table: 6.4 Orbit Mechanics Design Parameters x

11 Table: 6.5 Input Parameter Ranges for the Subsystem Models Table: 6.6 Predictive Capability of the Kriging Models Table: 6.7 Input Parameter Distributions Table: 6.8 System Uncertainty Results of Selected Design Table: 6.9 Sources of Uncertainty for the Remaining Oxidizer as Measured by r Table: 6.10 Sources of Uncertainty for the Remaining Fuel as Measured by r Table: 6.11 Sources of Uncertainty for the Total Mass as Measured by r Table: 6.12 Sources of Uncertainty for burntime as Measured by r Table: 6.13 Results of Final Satellite Design Using Deterministic Kriging Models 149 Table: 6.14 Results of Final Satellite Design Using Probabilistic Kriging Models 150 Table: A.1 Data Used to Create One-dimensional Test Function Table: A.2 Results of Fitting the Kriging Models with MLE Table: A.3 Results of Fitting the Kriging Models with CV Table: A.4 Results of Fitting the Kriging Models with MCMC Table: A.5 Data Used to Create 2-D Test Function Table: A.6 Results of Fitting the Kriging Models with MLE for 2-D Test Function Table: A.7 Results of Fitting the Kriging Models with CV for 2-D Test Function Table: A.8 Results of Fitting the Kriging Models with MCMC for 2-D Test Function Table: A.9 Data Used to Create 2-D Mystery Function Table: A.10 Results of Fitting the Kriging Models with MLE for 2-D Mystery Function Table: A.11 Results of Fitting the Kriging Models with CV for 2-D Mystery Function Table: A.12 Results of Fitting the Kriging Models with MCMC for 2-D Mystery Function Table: A.13 Data Used to Create 2-D Branin Function Table: A.14 Results of Fitting the Kriging Models with MLE for 2-D Branin Function Table: A.15 Results of Fitting the Kriging Models with CV for 2-D Branin Function176 Table: A.16 Results of Fitting the Kriging Models with MCMC for 2-D Branin Function Table: A.17 Data Used to Create 2-D Six-Hump Camelback Function Table: A.18 Results of Fitting the Kriging Models with MLE for 2-D Six-Hump Camelback Function Table: A.19 Results of Fitting the Kriging Models with CV for 2-D Six-Hump Camelback Function xi

12 Table: A.20 Results of Fitting the Kriging Models with MCMC for 2-D Six-Hump Camelback Function Table: A.21 Data Used to Create 5-D Test Function Table: A.22 Results of Fitting the Kriging Models with MLE for 5-D Test Function 183 Table: A.23 Results of Fitting the Kriging Models with CV for 5-D Test Function 184 Table: A.24 Results of Fitting the Kriging Models with MCMC for 5-D Test Function Table: C.1 Observations of the Engine Analysis Table: C.2 Observations of the Tank Analysis Table: C.3 Observations of the Orbits Analysis Table: C.4 Results of Rocket Engine Isp Model Table: C.5 Results of Rocket Engine Combustion Chamber Pressure Model Table: C.6 Results of Rocket Engine Power Model Table: C.7 Results of Tank Mass Model Table: C.8 Results of Log[burn time] Model xii

13 Acknowledgments It has been over thirteen years since I passed my Ph.D. candidacy exam, and as a result there are many people played roles in my completion of this degree. Completing a Ph.D. while working full-time with a wife and three kids is not something I would recommend to anyone, but I am thankful to finally complete my degree. I would first like to acknowledge Dr. Kam Ng of the Office of Naval Research, Code 333 for his encouragement and support of my research. Without his commitment to support my research for the past three years, this work would have been impossible. I would next like to thank my thesis advisor, Dr. Timothy Simpson, for his advice and patience with me as I found my way through this research and helping me find the conclusion to it. Writing has never been my strongest ability, but with his guidance, I have improved. I need to thank my supervisors at the Applied Research Laboratory for shifting work away from me while I finished this work. Special thanks go Dr. Mark Traband for his work on my committee and his commitment to helping me finishing my degree. I also want to thank Dr. Michael Yukish for the discussions that helped focus my work when things were going nowhere. Dr. Russell Barton, who also served on my committee, deserves a special word of thanks for seeing the real issues in my research and for pointing them out to me. I always appreciated our talks and felt invigorated by them. Finally, I need to thank my wife and family for putting up with me always working on the computer. It has been very hard on them when most evenings daddy comes home and has to work rather than go outside and play baseball or work in the woodshop. Thank you Monica for not getting angry with me when I would get home and say, I need to get a few more hours of work done tonight, when you had a tough day with the kids and only want to go to bed or at least get away from them for a few hours. Without your help and support this degree would never have been possible. xiii

14 Chapter 1 Introduction 1.1 Motivation The conceptual design of complex multidisciplinary systems can be considered a decision making process [1]. As a result, the design process becomes a task of selecting a candidate design from a set of possible feasible designs. The possible designs are compared to each other using a valid measure of value or an objective function (a term from the optimization community). This is not a trivial task since the desired metric must quantify multiple and frequently competing objectives, thus requiring the creation of a preference weighting between the various desired objectives of the design, such as cost, performance, size, and risk. In addition, the values of these objectives are rarely known precisely during the conceptual design process, introducing uncertainty into the decision between possible design alternatives. This research develops a methodology for evaluating system-level uncertainty in the conceptual design of complex multidisciplinary systems. A multidisciplinary system is made up of a hierarchy of subsystems or disciplines that influence each other and the overall system performance. The methodology introduced in this work quantifies the uncertainty present in a design through the use of probabilistic models, specifically kriging models. A hierarchy of probabilistic kriging models is used to evaluate systemlevel uncertainties. This methodology is intended to include information about each subsystem s contribution of uncertainty to the combined system s uncertainty as well as quantify the uncertainty that exists in using models to represent the many subsystem s performance.

15 Review of Multidisciplinary Design Optimization The need to design complex systems rather than just independent subsystems has motivated interest in Multidisciplinary Design Optimization (MDO). Sobieski and Haftka [2] provide an excellent review of MDO, and the important issues that they raise that relate to this research will be briefly summarized here. They defined MDO as a methodology for the design of systems in which strong interaction between disciplines motivates designers to simultaneously manipulate variables in several disciplines. The multidisciplinary optimization problem is more complicated than the singlediscipline optimization problem because of the coupling that exists between subsystems or disciplines. The primary cause for this increase in complexity is due to the increase in dimensionality of the optimization problem. The combined subsystems are dependent upon each other and therefore must be considered simultaneously. A secondary cause for the increase in complexity is due to each discipline typically having its own analysis model that may be dispersed geographically and whose interfaces may not be compatible with each other. There are three general approaches to MDO. The first is to centralize all system design expertise into one location and design the entire system at once. This method is often called the all-at-once approach (AAO). In this approach, high fidelity models of each subsystem are manipulated simultaneously to satisfy the system-level performance requirements (see, e.g., [3, 4]). The second approach is to use simplified subsystem models, such as metamodels, to design the system and then increase the fidelity of the analyses as the design process progress (see, e.g., [5, 6]). The final approach is to optimize a system model that only uses current sensitivity information from the subsystem models. This permits the decoupling of the system analysis from the subsystems during optimization. As the design process progresses, the current subsystem sensitivity information is updated within the system model. Some examples of this approach include concurrent subspace optimization (CSSO) [7] and collaborative optimization (CO) [8]. These three MDO approaches are based upon: 1) using deterministic models to find a single solution that satisfies system requirements and 2) finding a feasible system that is composed of subsystems that are compatible with each other (satisfies subsystem constraints). Lewis and Mistree [9], in their review of the state of the art in MDO, identify that the world is not deterministic and that design decisions need to be made with limited amounts of information, introducing uncertainty in all aspects of the design of real

16 3 systems. Uncertainty exists in the specifications of the system and in the models used to estimate system-level performance. In deterministic approaches, these uncertainties are not quantified. The need to include uncertainties has lead to the introduction of robust design, the selection of a design that is insensitive to design uncertainties, and reliability-based design optimization (RBDO), the selection of a design that satisfies system and subsystem constraints with a specified probability. Examples of each are given in Section Importance of Uncertainty Assessment During Design Uncertainty in conceptual design arises from having imperfect knowledge about the results of making a design decision. Models are used in design to support the decisionmaking process by predicting the results of a decision [10]. Scientists use models to explain the behavior of observed phenomena (reality), and term them descriptive or explanatory models. Scientists validate their models against data and events that have occurred in the past. Designers or decision-makers, on the other hand, use models to predict the behavior of a potential system before it is created and term them predictive or exploratory models. Decision-makers use models to answer the following three questions [10]: 1. Is the system feasible? Can it meet the requirements of the design? 2. Which of a set of technologies is best for satisfying the requirements? 3. Do I have enough information about the design to make an intelligent choice? There are two types of models used in design, namely, physical and mathematical models. A physical model represents a design with a simplified and/or scaled object, implicitly including the complexities of nature by taking measurements of the object s performance in nature, but also including measurement errors. Examples of a physical model include a tow tank, water tunnel, and wind tunnel. A mathematical model quantifies the complexity of nature with abstract symbols and expressions. A mathematical model encapsulates the knowledge a designer has about the system and the assumptions that are made in the design process. A mathematical model is typically faster, less expensive, and more flexible than a physical model, but a mathematical model can only predict with the information and relationships explicitly included in it.

17 4 An input of unknown dimension, X 0, to reality, g, produces a scalar response, y, as shown in Figure 1.1. A mathematical model of reality uses a subset of reality s inputs, X X 0, and a relationship, m, between these inputs to approximate the response of reality, y ŷ = m(x, θ). A model has parameters, θ, that are typically invariant of the input to the model, X. As a result of this approximation, it is impossible for models to deterministically predict nature. A deterministic model uses model parameters that are constants, always producing the same result, ŷ, for the same input. Support of the decision-making process in design requires quantifying the uncertainty that exists in the model s prediction, ŷ. A probabilistic model can quantify this uncertainty by treating the model parameters, θ, as random variables and producing a statistical distribution of its output, ŷ, for an input value. X 0 Reality g y (a) Basic Form of Reality X, θ Model m ŷ (b) Basic Form of a Model Figure 1.1: Representations of Reality and a Model There are two types of uncertainty in a probabilistic model: parametric and structural uncertainty [11]. With the model shown in Figure 1.1, the uncertainty in ŷ, induced by the uncertainty of θ, at a given model input, X, can be estimated by sampling values of θ from its probability distribution and performing a Monte Carlo Simulation. The uncertainty in the model output induced by the uncertainty in the model parameters is parametric uncertainty. Structural uncertainty results from a basic lack of knowledge about the underlying physical processes being modeled. It may result from the dimension of X being insufficient to describe the state of the physical processes, or it may result from inaccurate or incomplete mathematical relationships of X and θ being used to describe the physical processes. The probability distribution of model parameters is seldom known during design, requiring the use of alternative methods to quantify a model s uncertainty. The uncertainty in a mathematical model can be quantified through the use of subjective error bounds on the outputs or through the use of objective uncertainty measurements based on observations of reality. One technique, worst case analysis [12, 13],

18 5 applies an error bound around the design parameter s value, for example, ±10%. This type of error bound or uncertainty measurement is frequently the result of past experience with the model s predictive capabilities. It makes no assumptions on the probability distributions of a model s output and can be applied to a deterministic model to make it into a probabilistic model, usable in a worst-case uncertainty analysis. A second technique of uncertainty assessment is to compare the output of a model with observations of reality (validation points) and provide a probability distribution of the model s output to quantify the uncertainty in the model [14]. Since the purpose of a model during conceptual design is to assist the decision-making process, there may be no observations of reality to quantify the model s uncertainty. In order to overcome the lack of observations of reality to quantify the model uncertainty, a validated high-fidelity model (HFM) such as a finite element analysis, having a possibly large set of input dimensions and a high computational expense, is used to take on the role of reality. Using the results of a HFM, the uncertainty in the predictions of a more computationally efficient model with a possibly smaller smaller set of inputs, called a low-fidelity model (LFM), can be quantified. An LFM establishes the dimension of the input space and the form of the model using a priori knowledge of the system being modeled. A metamodel, or a model of a model [15], is frequently used in design optimization and uncertainty assessment when an LFM does not exist. A metamodel is typically created from a set of observations of reality (or a HFM). It takes a general form and draws only upon the observations to select the parameters for the model s form, providing a relationship from the input space to the output space. A kriging model is frequently used as a deterministic interpolating metamodel, but a kriging model can also be used as a probabilistic model by treating its parameters as random variables. This forms the basis for the uncertainty analysis methodology described in this work. The next two subsections provide some background on system uncertainty in design and on the current uses of kriging models in design System Uncertainty in Design Evaluating system-level uncertainty in design can be divided into two separate, yet related, tasks: uncertainty assessment and sensitivity analysis. Uncertainty assessment

19 6 is concerned with quantifying the effect of input and model uncertainties on the output uncertainty. Sensitivity analysis determines the uncertainty factors that most influence the uncertainty of the output. Uncertainty assessment is a major task completed during reliability-based design optimization, where sensitivity analysis is the major task completed during robust design. Review of Methods for Uncertainty Assessment Uncertainty assessment methods can generally be divided into two techniques: 1) firstorder (or second-order) reliability methods referred to as FORM (or SORM) and 2) Monte Carlo (MC) methods. FORM and MC are both reliability-based design optimization (RBDO) methods that find the point in the design space that satisfies all design constraints given the uncertainty of the input variables and optimize the performance function of the system being designed. The first implementation of FORM was the Mean-Value First-Order Second Moment method (MVFOSM) [16, 17]. In this method, a first-order approximation of the resulting mean and variance for the performance function (SORM uses a second-order approximation), given the input uncertainties, are determined at the mean value of the input parameters. The method provides a reliability index, β, which is defined as the ratio of the mean of the performance function to the standard deviation of the performance function. The larger the reliability index, the more likely the selected design will satisfy the design constraints. By assuming a normal probability distribution of the system performance, the probability of the design s failure is approximated by p f Φ( β) where Φ( ) is the standard normal probability function. The MVFOSM was modified by Hasofer and Lind [18] to provide a better understanding of the probability of a design s failure to satisfy its constraints. They proposed finding a most probable point (MPP) of failure given: the current design point, uncertainty in the input parameters to the design, and the constraints on the design. In this technique, correlated input parameters are transformed into an uncorrelated standard normal space. After this transformation, the distance between the current design and the MPP of failure is the reliability index. In this Reliability Index Approach (RIA), the design point is moved until it satisfies a prescribed probability of failure. Unfortunately, RIA tends to converge slowly if it converges at all. As a result, an alternative method that solves the inverse problem was proposed: Performance Measure Approach (PMA).

20 7 In the PMA, the convergence and reliability analysis is inherently more efficient than the RIA. The PMA requires the minimization of a more complicated function subject to simple constraints rather than using a simpler function with more complicated constraints. The PMA occurs in the original function space rather than the space resulting from the transformed, uncorrelated standard normal space. Monte Carlo (MC) methods do not require any transformations of the random variables to an uncorrelated standard normal space like MPP methods. MC methods draw samples directly from the probability distributions of the random variables and generate the probability space of the output variables. From these samples, the probability of failure can be calculated numerically. The methods require a large number of performance function evaluations in order to properly estimate the resulting probability distributions of the system performance. The result is that MC methods are often too computationally expensive to be used with detailed performance models. In this work, the detailed performance models are replaced with kriging metamodels, using the probabilistic form of the kriging model to quantify the model uncertainty introduced by using the approximation to the original model. Review of Methods for Sensitivity Analysis A Sensitivity Analysis (SA) is used to determine the sources of uncertainty that most influence the resulting uncertainty of the design. This information is critical to understanding the robustness of a potential design to uncontrollable variability of the design parameters and to specifying tolerances on those parameters that can be controlled. Additionally, it is important during the design process to identify the design parameters that most directly control the systems overall performance. The large number of design parameters present in most complex system designs frequently makes it computationally infeasible to perform system uncertainty assessment. Identifying the most important design parameters and using only those parameters in system uncertainty assessment enables a more computationally efficient method. A comprehensive study on SA was compiled by Saltelli et al. [19]. In their book, they identify SA as encompassing: design of experiments, specifically screening designs such as orthogonal arrays [20] or Latin Hypercubes [21,22]; local methods that are concerned with the local sensitivities at a specific point in the design space; global methods which are concerned with the sensitivity of the output over the range of possible inputs;

21 8 and the techniques used to evaluate the local or global sensitivity, such as samplingbased methods and variance-based methods. As discussed in Chapter 6, this work is concerned with performing global sensitivity analysis using sampling-based methods with kriging-based response surfaces. The design of experiments used during SA has two purposes: 1) to quantify the sensitivity of the design space to variations and 2) to identify the factors that are most influential to the resulting variations. The earliest and simplest designs were one-at-atime (OAT) experiments where the impact of each factor is determined in turn. These experiments can only find main effects and ignore any correlations between factors. This limitation prompted the development of factorial experimentation where all the factors are perturbed at the same time. Unfortunately, the increase in the number of experiments frequently becomes computationally infeasible. The next development was fractional factorial experiments where some of the higher order interactions are ignored in an effort to balance the number of experiments with the information on the factor interactions that results from the experiments. Myers and Montgomery [23] provide an excellent reference on these many different design of experiments. Orthogonal Arrays (OA) [20] and Latin Hypercube Sampling (LHS) [22] are more recent developments of design of experiments for computer models. They have superior projective properties, i.e., the ability to provide sensitivity information about all of the factors even when some of the factors prove to have little importance. This makes OA and LHS very valuable as screening designs for computer models. The local SA of a design can be accomplished by determining the first-order partial derivatives of the design parameters influence on the system performance. This summary of the sensitivities provides a linearization of the local impact of the design parameters. With this local sensitivity information, the impact of variations in the noise factors, parameters that can not be selected during design, and the control factors, parameters that are selected during design, can be approximated [24]. For small variations or systems that are almost linear, the resulting approximation is quite accurate. Most complex systems do not have sensitivities that are linear throughout the region of variations of the design parameters, thus requiring the need to determine global sensitivities. The global SA of a design can be accomplished through the use of Monte Carlo simulation (MCS) [25] or Response Surface Methodology (RSM) [23]. The MCS method simultaneously samples all of the design parameters as dictated by their probability

22 9 density functions (pdf). After the simulation has executed, the output statistics can be estimated and an analysis of variance completed to rank the impact of each parameter on the distribution of the design s performance. Some desirable properties of MC include: 1) its ability to operate in parallel, 2) its ability to simulate any pdf for the parameters, and 3) its relative simplicity that is achieved at the expense of many model evaluations. The RSM sequentially fits regression models to estimate the system performance over a region of the design near the desired performance. The sequential nature of the process identifies the most and least important factors in the design and modifies its experimentation strategy to exploit this information. RSM is limited in the parameter pdfs it can accommodate (typically only Gaussian distributions) and its ability to be executed in parallel due to its sequential nature, but it is much more efficient than MCS in terms of the numbers of model evaluations required. As discussed in Chapter 6, this work uses a combination of kriging models and MCS to complete the SA for a complex system design by taking advantage of the computational efficiency and accuracy of kriging models along with the flexibility of MCS [26, 27] Current Uses of Kriging Metamodels in Design The use of kriging models as metamodels in design has been rapidly gaining popularity in the recent past. The kriging model was originally introduced by D. Krige in 1951 [28] as a method to determine the best regions to mine for gold given a few samples from the mining area. This concept was extended for use in geostatistics by Matheron [29]. This work was further extended by many other researchers in geostatistics and was used to predict spatially varying properties such as contaminate concentrations based upon samples taken over a region [30, 31]. In the field of statistics, independent from the work of Matheron, Goldberger [32] developed a method to extend a general linear regression model by quantifying the correlation that may still exist in the residuals between the observations and the fitted regression model. Functionally, the two approaches are identical and form the basis of the work for creating kriging models of computer models. One of the first references that used a kriging model to approximate a computer model is by Currin et al. [33], but kriging was made much more popular by the publication of Design and Analysis of Computer Experiments by Sacks et al. [34] in 1989.

23 10 This publication popularized the acronym DACE to refer to the method of using a kriging model to provide an interpolating approximation to a computer simulation. The work by Sacks et al. was primarily concerned with identifying the difference between designing experiments for traditional physical systems which are subject to random errors and that for computer simulations which are not subject to random errors. They demonstrated an information-based design of experiments method that was easily extended into a sequential design of experiments method, using the information available in the current set of observations to direct the future sampling of the deterministic computer model. In order to measure the information available in a model and interpolate the set of observations, the probabilistic version of the kriging model was used. Their design of experiments method would select new locations to sample the deterministic computer model that would minimize the integrated mean square error estimated by the kriging model. More recently, kriging models have been employed to serve as an approximation of computationally expensive computer models for the purpose of optimization [35 38]. Osio and Amon [39] applied Bayesian updating to kriging models to aid in the thermal design of embedded electronic packages. Simpson et al. [40] used kriging models for approximations during the design of an aerospike nozzle. Unfortunately, kriging models do not appear to be used significantly in design. This lack of more prevalent use can be attributed to a lack of available off-the-shelf software to assist in selecting the best forms and model parameters for the kriging models [41]. A significant amount of emphasis in this work is placed on the development and demonstration of using kriging models, first as deterministic approximations (see Chapter 3) and then as probabilistic models (see Chapter 4). These detailed developments of kriging models form the foundation for the proposed methodology introduced in Chapter Research Objective and Tasks The background on uncertainty assessment in design and identification of the sources of uncertainty in design are meant to serve as motivating factors for this research. The detailed research in kriging models is motivated by the lack of basic research in using kriging models as computationally efficient approximations during design. The objective in this research is to:

24 11 Develop a methodology for evaluating system-level uncertainty in the conceptual design of complex multidisciplinary systems through the use of a hierarchy of probabilistic kriging models to permit the rapid assessment of input and model uncertainty in subsystem parameters on the system-level performance measurements. The stated research objective requires the following research tasks to be completed. 1. Investigate the current applications of using kriging models as metamodels in design, including the current methods of parameter estimation and metamodel assessment techniques (see Chapter 2). 2. Compare the computational efficiency and accuracy of the Maximum Likelihood Estimation (MLE), cross-validation (CV), and Bayesian analysis methods for selecting kriging model parameters (see Chapter 3). 3. Quantify the probabilistic nature of a kriging model and its ability to quantify uncertainty (see Chapter 4). 4. Compare the metamodel assessment techniques of cross-validation and Akaike information criterion as metrics for assessing metamodel quality (see Chapter 5). 5. Compare the results of a Monte Carlo simulation of the system performance using probabilistic kriging and deterministic kriging models of subsystems to the those from using the original computer models with arbitrary probability distributions for the inputs to the system and subsystem models (see Chapter 6). These research are addressed as outlined in the next section. 1.3 Thesis Roadmap This thesis is organized in the following manner. Chapter 2 introduces the most widely used methods to approximate deterministic computer models in engineering design, the polynomial regression model and the kriging model. Chapter 3 introduces six test functions that are used throughout Chapters 3-5 to demonstrate most of the parameter estimation and model assessment methods presented in this thesis. Chapter 3 also presents a computationally efficient formulation to perform parameter estimation using leave-oneout cross-validation (CV) and compares the CV method to the traditional Maximum

25 12 Likelihood Estimation (MLE) method. Chapter 4 presents and compares MLE and Bayesian Analysis (BA) as methods to select model parameters for probabilistic kriging models. A significant amount of detail is provided on using Markov Chain Monte Carlo (MCMC) as a computationally efficient BA method. Chapter 5 presents two methods for assessing the quality of deterministic and probabilistic kriging models: 1) estimates for root mean square error of predictions and 2) information-based criteria for model quality comparisons. Chapter 6 presents a methodology for evaluating system-level uncertainty in the conceptual design of complex multidisciplinary systems, briefly introduces the conceptual design of a satellite, and compares the results of using kriging models and original computer models in the uncertainty assessment during the conceptual design of a satellite. The last chapter provides a summary of the thesis, identifies the major contributions of the research, and lists a few items of future work. Appendix A includes the data from the six test functions used to create the kriging models and the results of the created kriging models using MLE, CV, and BA. Appendix B contains derivations of the first and second partial derivatives of the Gaussian spatial correlation function using a product correlation rule, which is employed extensively to computer model parameter uncertainty in Chapter 5. Appendix C includes the data used in Chapter 6 during the demonstration of the proposed methodology in the conceptual design of a satellite.

26 Chapter 2 Background on Metamodels This chapter explains some of the mathematics behind and techniques used to create metamodels. The term metamodel was first coined by Kleijnen [15] to refer to a model of a model. A metamodel is a computationally efficient function that approximates the output of a potentially computationally expensive computer model given its set of inputs and outputs. There are two general classes of functions that can be used to approximate a computer model. The first class is approximation in which the resulting function does not exactly reproduce the observations used to create it. A polynomial regression model is an example of an approximating model. The second class is interpolation in which the resulting function does exactly reproduce the observations used to create the model. Examples of the second class of function include interpolating polynomials, radial basis functions, and kriging models. In order to quantify uncertainty in the subsystem models, this work is concerned with creating probabilistic metamodels. The probabilistic nature of the metamodels quantifies what is known about the original computer model based upon the set of observations, and it quantifies the lack of knowledge or uncertainty in its estimate of the original computer model by generating a probability distribution for the output of the metamodel. In this chapter, background on two types of probabilistic metamodels, namely, polynomial regression models and kriging models, are presented. These types of metamodels are two of the most commonly used deterministic metamodels in engineering design [42]. This chapter highlights the development of the mathematical form of the metamodels, identify the assumptions for their use as deterministic and more restrictively as probabilistic metamodel, detail the parameter estimation method (maximum

27 14 likelihood estimation) used most commonly during the design and analysis of computer experiments [34], and define the most common method of assessing kriging models, namely, root mean square error. 2.1 Input and Output Data A metamodel is defined over its input space, called its domain, Ω. In most situations, Ω is a subset of R d, but it can also include the whole of R d, where d is the dimension of the input space for the metamodel. The data used to create a metamodel has two parts: 1. a finite set, X = {x 1, x 2,..., x n }, of n scattered d-dimensional points in the domain, Ω R d and 2. a set of real numbers, y = {y 1, y 2,..., y n } T, that represent the values of the computer model being estimated at the n given input points. Before a metamodel can be created to reproduce the response of the computer model, the domain, Ω, of the metamodel must be defined and its form must be established. The domain is not restricted for either model, but polynomial regression models are frequently transformed to a hypercube, Ω = [ 1, 1] d, and kriging models are transformed to a unit hypercube, Ω = [0, 1] d. These transformations make interpretation of the model parameters for each dimension more meaningful by giving them the same scale. In the next two sections, forms of the polynomial regression and kriging models are presented. 2.2 Polynomial Regression Model The polynomial regression model approximates a computer model with a linear regression model of k regressor variables, f, and has the form y = β 0 + β 1 f 1 + β 2 f β k f k + ε. (2.1) The p = k + 1 model parameters, β = {β 0, β 1, β 2,..., β k } T, are called the regression coefficients, and ε is the error between the metamodel and the computer model. The metamodel describes a hyperplane in the k-dimensional space of the regressor variables, f = {1, f 1, f 2,..., f k }. Each model parameter in β represents the expected change in

28 15 the response y per unit change in the corresponding regressor variable when all the remaining independent variables are held constant. The linear regression model can be used to approximate many complex functions by transforming the inputs to the computer model to the regressors with any class of function such as powers or logarithms. The polynomial regression model limits the selection of regressors to polynomials of the model inputs. For example y = β 0 + β 1 f 1 + β 2 f 2 + ε becomes a second-order polynomial of a single variable, x, by setting f 1 = x and f 2 = x 2. Given this form for an approximating model and the set of observations of the computer model X, the next issue is selecting the best parameters, β, for the model Derivation of the Polynomial Regression Model The typical method used to select the polynomial regression model parameters is the method of least squares [23]. In this method the model parameters, β, are chosen to minimize the sum of the squared errors, ε, between the metamodel and the computer model. The matrix form of the linear regression model is y = Fβ + ε (2.2) where y is a vector of the n observations, F is an n p matrix of the values of the regressors at each observation, and ε is a n 1 vector of the residuals between the model and the observations. A function for the sum of the squared errors SS E or the loss function L follows. n SS E = L = ε 2 i = ε T ε i=1 = (y Fβ) T (y Fβ) = y T y 2β T F T y + β T F T Fβ (2.3) The function L is minimized with respect to β. The least squares estimators, ˆβ = {ˆβ 0, ˆβ 1,..., ˆβ k } T, must satisfy the first-order optimality condition,

29 16 L β which, solving for ˆβ, simplifies to ˆβ = 2F T y + 2F T Fβ = 0, (2.4) ˆβ = ( F T F ) 1 F T y, (2.5) which is the least squares estimator of β. The fitted linear regression model is then ŷ (x) = f (x) ˆβ. (2.6) where ŷ (x) is the estimate of output of the model at any location, x, within the model s domain, Ω. The model represented by Eq. (2.6) is a deterministic model that best approximates the given observations. It makes no assumptions on the statistics of the data being modeled, but the model parameters, β, are random variables, having probability distributions associated with them. The best values of the model parameters must be determined given the set of observations and the form of the regression model. The method of least squares provides a best linear unbiased estimator of the model parameters β if the estimate is unbiased, i.e., E(ε) = 0. The expected value of β is E [ β ] [( = E F T F ) 1 F y] T [( = E F T F ) 1 F T (Fβ + ε)] [( = E F T F ) 1 F T Fβ + ( F T F ) 1 F ε] T = ˆβ, and the variance, a measurement of uncertainty in the unbiased estimator ˆβ, is expressed by the covariance matrix [( Cov (β, β) = E β ˆβ ) ( β ˆβ ) ] T. The covariance matrix of β is a p p symmetric matrix whose (i, j)th element quantifies the covariance between β i and β j. It is defined as Cov (β, β) = σ 2 ( F T F ) 1, (2.7)

30 17 where σ 2 becomes an additional model parameter to be estimated. It can be shown that an unbiased estimate of σ 2 is given by ˆσ 2 = SS E n p, (2.8) where n p is the number of degrees of freedom associated with the sum of the squared errors, SS E from Eq. (2.3). The results given thus far in this section provide solutions for the best model parameter values given a set of model regressors or a form of the model. A measurement is needed to permit the comparison of different model forms. The R 2 statistic is a measure of the amount of reduction in the variability of y obtained by using the regressor variables f in the metamodel. The coefficient of multiple determination R 2 is defined as where the total sum of squares is R 2 = 1 SS E SS T, (2.9) SS T = y T y n y 2 i i=1 n. (2.10) In general, 0 R 2 1, but for very poor models, where the sum of the squared errors (see Eq. (2.3)) is greater than the total sum of squares, the result obtained if a constant mean were assumed, it is possible for R 2 < 0. A large value of R 2 does not always imply the current model is a good approximation of the observations. It is always possible to improve the R 2 by adding more parameters to model. This process is called overfitting the model. A measurement that can be used to select a more parsimonious model is the adjusted R 2 statistic which is defined as R 2 adj = 1 n 1 ( ) 1 R 2. (2.11) n p This statistic will not always increase as more parameters are added. In general, as unnecessary terms are added, its value will decrease. The adjusted R 2 statistic is frequently

31 18 used to compare different forms of a model that represent the best set of regressors to approximate the given observations [23] Requirements for the Linear Regression Model In order to use a linear regression model as a probabilistic model, enabling the calculation of a probability distribution of the regression model s output for any location in its domain, the following procedures require that the errors ε in the model be normally and independently distributed with a mean zero and a variance σ 2, typically identified as ε N ( 0, σ 2). The result of this assumption is that the observations y are normally and independently distributed with mean f(x) ˆβ and variance ˆσ 2. The least squares estimate of the model parameters β is a linear combination of the normally and independently distributed observations. As a result, β is normally distributed with mean ˆβ and covariance matrix given by Eq. (2.7). This conclusion provides the probability distribution of the model parameters given the form of the model and a set of observations of the computer model being approximated. The final development needed is to quantify the probability distribution of the metamodel output given the observations and form of the metamodel. The output of the linear regression model, given a set of normally and independently distributed observations will have a Student-t distribution with n p degrees of freedom [43]. The expected value of an unobserved point has regressor values of f T (x) = {1, f 1 (x), f 2 (x),..., f k (x)}, (2.12) and is given by Eq. (2.2) as ŷ (x) = f T (x) ˆβ, (2.13) and the variance of the expected value is Var [ ŷ (x) ] = ˆσ (f 2 T (x) ( F T F ) ) 1 f T (x). (2.14) The resulting probability distribution of the model s estimate, located by Eq. (2.13) and scaled by Eq. (2.14), is

32 19 y ŷ (x) Var [ŷ (x) ] t n p. (2.15) The Student-t distribution in Eq. (2.15) quantifies the total uncertainty present in the linear regression model. It can be used to generate confidence intervals on the output of the linear regression model. If the model parameters were known values, then the distribution of the model output would be the normal distribution with mean given by Eq. (2.13) and variance given by Eq. (2.14). The Student-t distribution in Eq. (2.15) includes the uncertainty introduced into the model by estimating the p model parameters with n observations. A property of the Student-t distribution is that as the degrees of freedom (n p) increases, it approaches the Normal distribution. In other words, as the number of observations of the computer model increases, the uncertainty in the estimates of the model parameters decreases. Unfortunately, the assumption of normally and independently distributed observations used in the linear regression model is seldom met with observations of real systems. As a result, linear regression models were extended to generalized linear models [44], which accommodate observations that follow the exponential family of distributions. The exponential family includes the normal, Poisson, binomial, exponential, and gamma distributions. The generalized linear model uses a link function to transform the distribution of the current observations to a space that is normally distributed. The standard linear regression techniques can then be used with the resulting, transformed observations. One of the most common link functions is the logit link, which is used to create logistic regression models. These models are used to characterize the probability of an event; therefore, the output is bounded between 0 and 1. A second common link function is the log link, which is used to transform the Poisson distribution, a common distribution that is used to describe failure rates. This discrete distribution is asymmetric, strictly positive, and extends out to positive infinity. 2.3 Kriging Models As stated earlier, the kriging model was initially developed by geologists to estimate mineral concentrations over an area of interest given a set of sampled sites from the

33 20 area [29]; it was also introduced about the same time in the field of statistics to include the correlations that exist in the residuals of a linear estimator [32]. There are many texts in geostatistics [30,31] and in spatial statistics [45 47] that provide extensive details on the development and use of kriging models in their respective disciplines. This section covers the important details for using kriging models to approximate deterministic computer models with both deterministic and probabilistic metamodels. A kriging model is a linear regression model that explicitly accounts for the correlation in the residuals between the regression model and the observations [32]. Given the mathematical form of kriging (see Section 2.3.1) and satisfaction of the kriging model s requirements (see Section 2.3.2), the process of using kriging first requires the estimation of the best kriging model parameters (see Section 2.3.3), and an assessment of the resulting kriging model s accuracy (see Section 2.3.4) before it can be used as an approximation to a deterministic computer model Derivation of the Kriging Model The mathematical form of a kriging model has two parts as shown in Eq. (2.16). The first part is a linear regression of the data with k regressors that model the drift of the process mean, the trend, over the domain. Most engineering applications only use a constant trend model over the domain and rely upon the second part of the model to pull the response surface through the observed data by quantifying the correlation of nearby points [40]. k ŷ (x) = β i f i (x) + Z (x) (2.16) i=1 The second part, Z(x), is a model of a stationary Gaussian random process with zero mean and covariance Cov (Z (x 1 ), Z (x 2 )) = σ 2 R (x 1, x 2 ). (2.17) The process variance, σ 2, is a scalar parameter of the spatial correlation function (SCF), R(x 1, x 2 ). There are two aspects to choosing the model form within the constraints of Eqs. (2.16) and (2.17): 1) choice of the trend function regressors, f, and 2) choice of the SCF.

34 21 The most common choice of trend function regression, when using a kriging model as a deterministic approximation of a computer model, is a constant trend function, f (x) = {1}. This is due to the ability of the correlation function to effectively model the observed variations of the computer model s output over it s domain [48]. The importance of the trend function in providing a deterministic approximation of a computer model is given in Section The minimum requirements to consider when choosing a potential SCF are that it must be positive semi-definite and finite. The desired properties of an SCF are the ability to control: (1) the range of influence of nearby points, (2) the smoothness of the resulting surface, and (3) the differentiability of that surface. Properties 2 and 3 are closely related to each other. Koehler and Owen [49] provide an insightful overview of four commonly used SCFs used with kriging models: Gaussian, exponential, cubic spline, and Matérn functions (see Table 2.1). Table 2.1: Common Spatial Correlation Functions Name R (x 1, x 2 ) Restrictions ( ) Gaussian e x2 x 1 2 θ θ > 0 ( ) Exponential e x2 x 1 z θ θ > 0, 1 z ( ) x 2 x 1 2 ( θ + 6 x2 ) x 1 3 θ, x2 x 1 < θ 2 Cubic Spline 2 ( ) 1 x 2 x 1 3 θ, θ x 2 2 x 1 < θ θ > 0 0, x 2 x 1 θ ( ) x2 x 1 ν θ Matérn Γ [ν] 2 ν 1 K [ x2 ] x 1 ν θ > 0, ν > 0 θ All of the SCFs listed in Table 2.1 use a parameter, θ, to control the range of influence of nearby points. A graphical representation of the four correlation functions for different parameters is shown in Figure 2.1. Two of the functions, the exponential and the Matèrn, have a second parameter that controls the smoothness of the resulting surface. The Gaussian function results in a very smooth response surface. The exponential function results in a much more jagged surface. The Matérn function is defined using the Gamma function and the modified Bessel function of the second kind. The ν parameter controls the differentiability of the resulting surface, with the resulting surface being Floor(ν) times differentiable. The dashed line in Figure 2.1(d) indicates ν = 1.5,

35 (a) Gaussian SCF, θ = {1, 0.5, 0.1} (c) Cubic SCF, θ = {1, 0.5, 0.1} (b) Exponential SCF, z = 1 and θ = {1, 0.5, 0.1} (d) Matèrn SCF, ν = {1.5, 2.5} and θ = {0.5, 0.1} Figure 2.1: Plots of Four Spatial Correlation Functions and the solid line is ν = 2.5. The Matérn SCF is a superset of the Gaussian and the exponential SCFs, with ν = 0.5 being identical to the exponential function with z = 1 and ν = being identical to the Gaussian distribution. There is little guidance available in the literature on the selection of the best form of the SCF. The choice of a SCF is a balance between the quality of the fit and the computational expense in fitting the parameters of the SCF. For example, the cubic spline SCF has the advantage of a finite range beyond which the correlation between two points is zero, resulting in a possibly sparse correlation matrix that is easier to invert, but use of the cubic spline SCF complicates the determination of the best correlation parameters, as discussed in Chapter 3. Statisticians suggest the use of the Matérn family of SCFs [47] due to its flexibility and ability to evaluate the asymptotic properties of the kriging model when the observations are placed on a lattice (an equally spaced grid). The Gaussian function is the most commonly used SCF in engineering design [42] as it provides a relatively smooth and infinitely differentiable surface, making it a better choice when used with gradient-based optimization algorithms and only requires the selection of a single correlation parameter; consequently, it is the SCF used in this work. The most common form of the Gaussian SCF [40, 49, 50] is as follows:

36 23 R (x 1 x 2 ) = e θ x 2 x 1 2, where θ > 0. (2.18) The correlation function range parameter θ used in Eq. (2.18) has little meaning in a physical sense [51]. The form used in this work is that shown in Table 2.1 where the range parameter θ indicates the distance at which the influence is e 1 = or approximately 37%. To support a multivariate correlation function, a product correlation rule, combining the univariate correlation functions, is used for each of the d input dimensions [37, 52] rather than using the Euclidean norm of the space [51, 53]. The product correlation rule is given as d R (x 1, x 2 ) = R ( x2,i x ) 1,i. (2.19) i=1 This formulation improves the flexibility of modeling the correlation of each input dimension at the expense of requiring the selection of additional correlation function parameters (d parameters instead of just one). As was described in Section 2.1, let the locations of a set of n observations of the computer model be X = {x 1, x 2,..., x n } Ω, where Ω is the set of all possible inputs to the model that result in an output, i.e., the domain of the computer model. The resulting outputs are y = {y (x 1 ), y (x 2 ),..., y (x n )}. Given these sampled outputs of the computer model, consider a linear estimator of the output, ŷ (x) = λ T (x) y, (2.20) at any point x Ω. In a manner very similar to that used with the development of the linear regression model, the kriging approach treats ŷ (x) as a random function and finds the best linear unbiased predictor, λ T (x) y, which minimizes the mean square error of the prediction, subject to the unbiasedness constraint, MSE [ ŷ (x) ] = E [( λ T (x) y y (x) ) 2 ], (2.21) E [ λ T (x) y y (x) ] = 0. (2.22)

37 24 The general form of kriging, Universal kriging, is defined with a set of regressor variables f(x) in the same way as the linear regression model, e.g., Eq. (2.1). A second type of kriging, Ordinary kriging, is a special case of Universal kriging where f (x) = {1}. (2.23) Ordinary kriging is the most commonly used form of kriging employed to approximate computer models [34, 42, 50]. A matrix F is constructed by evaluating the vector f(x) at each of the n known observations, f(x 1 ) f(x 2 ) F =. (2.24). f(x n ) The next definition needed is for the correlation matrix R which is composed of spatial correlation function evaluated at each possible combination of the known points, R (x 1, x 1 ) R (x 1, x 2 ) R (x 1, x n ) R (x 2, x 1 ) R (x 2, x 2 ) R (x 2, x n ) R =. (2.25) R (x n, x 1 ) R (x n, x 2 ) R (x n, x n ) This matrix R is a positive semi-definite matrix since the SCF defining each element is positive semi-definite. It is also symmetric since R ( ) ( ) x i, x j = R x j, x i, and the diagonal consists of all ones since R (x i, x i ) = 1. The last definition needed is a vector to represent the correlation between an unknown point, x Ω, and the n known sample points: r (x) = {R (x, x 1 ), R (x, x 2 ),..., R (x, x n )} T. (2.26) The estimate to λ(x) that solves the minimization problem of Eq. (2.21) subject to the unbiasedness constraint of Eq. (2.22) is

38 25 ˆλ (x) = R 1 F ( F T R 1 F ) 1 f (x) + R 1 [I F ( F T R 1 F ) ] 1 F T R 1 r (x), (2.27) where I is the n-dimensional identity matrix. The best linear unbiased estimator (BLUE) of ŷ (x) results from plugging the the estimate from Eq (2.27) into Eq. (2.20) [32]. The BLUE of ŷ(x) is then given by ŷ (x) = f T (x) ˆβ + r T (x) R 1 ( y F ˆβ ), (2.28) where the generalized least squares estimate of β is ˆβ = ( F T R 1 F ) 1 F T R 1 y. (2.29) The first component of Eq. (2.28) is the generalized least squares estimate of a point, x Ω, given the correlation matrix, R; meanwhile the second component pulls the generalized least squares estimate through the observed data points, providing a deterministic response surface that interpolates all of the observations. The BLUE defined in Eq. (2.28) assumes the correlation parameters θ, used to define the correlation of the observations, are known a priori. This is seldom the case and therefore requires the correlation parameters to also be estimated from the set of observations. The most common process of estimating the best model parameters, maximum likelihood estimation, is covered in Section Requirements for the Kriging Model A kriging model can be used as a probabilistic model, like the linear regression model described in Section 2.2, by assuming that the residuals between the linear model and the observations are normally distributed. The difference between kriging and linear regression models are that the residuals are assumed to be correlated. In most cases, the observations of computer models and their resulting residuals are not normally distributed. Three transformation methods can be used to satisfy the normal distribution of the residuals requirement: 1) a more complex trend function, 2) a continuous link function (as was discussed with the linear regression models), or 3) a piecewise defined Rosenblatt function. Examples of all of these transformation methods are shown in Table 2.2.

39 26 Transformation Method Improved Trend Function Table 2.2: Examples of Transformations Used for Kriging Models Original distribution Transformed distribution Log-Link Function Rosenblatt The form of the trend function for the kriging model can be selected that results in residuals that more closely approximate a normal distribution. The coefficients of the regressors are always selected to remove the bias in the observations, satisfying the zero mean requirements. The choice of the trend function also impacts the distribution of the errors. The most common practice in approximating computer models is to use a constant trend function. In the work completed as part of this research (see Section 3.4.3), it appears that there is little difference in the capability of a kriging model to approximate computer analyses between using a constant and a second-order trend function [54], but it appears that the residuals from the trend function were more likely to follow a normal distribution if a more complex trend function was used. A link function, identical to the those used to transform the linear regression model outputs (see Section 2.2.2), can also be used to transform the probability distribution to be normal. Finally, a Rosenblatt function can be used to transform an arbitrary probability distri-

40 27 bution to a standard normal distribution. This transformation is achieved by ranking the original data in ascending order, with the assumption that each occurrence has an equal probability of occurring, in order to create a sample cumulative frequency distribution. These probabilities are mapped into a standard normal space, N(0, 1). This method does have difficulty in properly specifying the behavior of the mapping of values above and below the observed values, i.e., the tail regions of the distribution [31]. The kriging model is a spatial Gaussian process model. It defines the probability distribution of the model output over the model s domain with a Gaussian or normal distribution with an expected value given by Eq. (2.28) and variance given by Eq. (2.21). The mean square error (MSE), or variance of the estimate ŷ (x) from Eq. (2.21) can be restated as MSE [ ŷ (x) ] = σ ( 2 1 2λ T (x) r (x) + λ T (x) Rλ (x) ), (2.30) where σ 2 is termed the process variance and defined as the variance of the residuals. By substituting the estimate of λ (x) from Eq. (2.27) into Eq. (2.30), the following equation for the variance of the estimate results MSE [ ŷ (x) ] = σ 2 1 [ f T (x) r T (x) ] 0 F T F R 1 f(x) r(x). (2.31) The estimates at the observations are returned exactly because the kriging model still interpolates the observations; the MSE at these points is zero since there is no uncertainty in the observations of a deterministic computer model. As an unobserved point, x, moves away from the observations, the second component of Eq (2.28) approaches zero, yielding the generalized least squares estimate, and the uncertainty in the estimate approaches its maximum value, namely, the process variance σ 2. The normal distribution for the output of a kriging model is defined conditionally on the set of observations and the known model parameters, γ = { β, σ 2, θ }. It assumes that the residuals from the trend function are normally distributed and correlated or dependent upon nearby observations. As was the case with the linear regression model, the model parameters are not known exactly but are random variables that must be estimated from the set of observations given the model form, i.e., the regressors f and spatial correlation function R (, ). Others have determined analytically [55, 56] that if

41 28 the correlation function parameters are known and only the regression parameters are estimated from the observations, the resulting probability distribution of the model output is not Gaussian, but it has a Student-t distribution with n p degrees of freedom, where p is the number of regressors in the trend function. The correlation function parameters are not considered degrees of freedom since they are known values. The distribution is identical to that defined in Eq. (2.15) except that the expected value and variance are provided by Eqs. (2.28) and (2.31), respectively. A study was performed as part of this research to quantify the resulting probability distribution of a kriging model when all of the model parameters are estimated from the set of observations (see Chapter 4) Kriging Model Parameter Estimation The Design and Analysis of Computer Experiments (DACE) [34] primarily uses the statistics-based method of Maximum Likelihood Estimation (MLE) as an objective estimator of the regression function coefficients, process variance, and spatial correlation function (SCF) parameters, γ = { β, σ 2, θ }, that are most consistent with the observed data [52, 56, 57]. MLE assumes that the residuals have a known probability distribution shape, which in most cases is the Gaussian probability distribution. The likelihood of the model parameters given the set of observations and the model s form is defined as the probability of the n observations y given the model parameters γ, is n L (γ y) = p (y γ) = p (y i γ). (2.32) If the output distribution of the computer model comes from a Gaussian distribution, then the likelihood of the model parameters, γ, is defined with the multivariate normal distribution of the n observations of y given the model parameters, γ, and is given as i=1 L (γ y) = p (y γ) = (y Fβ)TR 1 1 θ (y Fβ) ( 2πσ 2 )n e 2σ 2. (2.33) R θ The goal of the MLE method is to maximize the probability of all of the observations, given the model parameters, γ. The multivariate normal likelihood function can be difficult to maximize due to large flat regions of near zero values. To improve the optimization process, the logarithm of the likelihood function is taken, since the maxi-

42 29 mum of the logarithm of the likelihood occurs at the same location as the maximum of the likelihood. The logarithm of the multivariate Gaussian likelihood function is l(γ y) = n 2 ln [ 2πσ 2] 1 2 ln [ R θ ] 1 2σ 2 (y Fβ)T R θ 1 (y Fβ). (2.34) By taking the derivative of the log-likelihood equation (Eq. (2.34)) with respect to β and σ 2 and solving for zero, the closed-form solution for the optimal value of β is ˆβ = ( F T R θ 1 F ) 1 F T R θ 1 y, (2.35) which matches the result of Eq. (2.29) found by solving for the least square error, and the solution for σ 2 is ˆσ 2 = 1 ( y F ˆβ ) T ( 1 Rθ y F ˆβ ). (2.36) n A closed-form solution does not exist for the optimal parameters of most common SCFs, thus requiring numerical optimization. In order to reduce the number of model parameters determined with the numerical optimization, the profile log-likelihood is optimized. The profile log-likelihood substitutes the known optimal values of ˆβ and ˆσ 2 from Eqs. (2.29) and (2.36) back into Eq. (2.34) and optimizes for the unknown correlation parameters, θ, as shown in Eq. (2.37) [58]. maximize n 2 ln [ 2π ˆσ 2] 1 2 ln [ R θ ] n 2 subject to θ > 0 (2.37) Even with the reduction in the dimension of the optimization problem that results from using the profile likelihood function, the optimization process can still be a computationally expensive process. Examples of the difficulties experienced while using MLE to estimate kriging model parameters are provided in Chapter Kriging Model Assessment The quality of a kriging model must be assessed differently than the linear regression model since a kriging model will interpolate all of the observations used to fit it, irre-

43 30 spective of the values of the model parameters. A kriging model is based upon providing an unbiased estimate of the original model. As a result, the expected error in the prediction is zero. This expected value gives no indication of the expected distribution about zero; therefore, the RMSE, equivalent to the standard deviation of the expected error, can provide a more meaningful measurement. Root Mean Square Error The RMSE is unknown and can be estimated from a set of validation observations. The most common method of estimating RMSE requires a large number (m n) of potentially computationally expensive computations to obtain the additional observations of the computer model that are not used to fit the kriging model. The RMSE of prediction for the kriging model ŷ (x) can be estimated empirically with RMSE = 1 m m (ŷ (x i ) y (x i )) 2 i=1 1/2, (2.38) where ŷ (x i ) is the expected value of the kriging model from Eq. (2.28) and y (x i ) is the value at each of the m validation locations, x i. Coefficient of Multiple Determination The coefficient of multiple determination, often termed R 2 (see Eq. (2.9)), is a nondimensional and normalized measurement that quantifies the amount of variability captured by the model. It typically ranges from 0 to 1 with 1 indicating a perfect fit and 0 indicating a poor fit. It is possible for the R 2 of a model to be less than zero in instances where the model is worse than a constant average of the model s domain. The normalizing parameter used to create this statistic is the total sum of squares and is defined as ( ) m 2 y i SS T = y T i=1 y m. (2.39) Given this definition for the total sum of squares for the m observations in the validation data set, the coefficient of determination for the validation set of observations is defined as

44 31 R 2 actual = 1 SS E SS T, where the SS E is the sum of the squared errors and is equivalent to the square of the RMSE from Eq. (2.21) times m, the number of observations in the validation set. This definition is slightly different from that given in Eqs. (2.9) and (5.5). 2.4 Summary Remarks In this chapter, the background on two probabilistic metamodels is provided. Specifically, the current practice employed to create two of the most common types of models used as computationally efficient alternatives to deterministic computer models is given, namely, linear regression and kriging models. Kriging models appears to be better suited for approximating computationally expensive computer models than linear regression models due to: (1) their ability to produce a more complex surface with fewer parameters than the linear regression models and (2) interpolate all of the observations used to create them. The difficulty with using kriging models as metamodels has centered on a lack of commercial off-the-shelf software and the computational expense that must be invested to estimate the best model parameters, a numerical optimization of a potentially multi-modal function. The next two chapters detail the research completed to thoroughly investigate different methods to estimate kriging model parameters. Chapter 3 describes the methods to estimate kriging model parameters when the kriging model is to be used as a deterministic approximation of a computer model. It introduces a new formulation to efficiently calculate the leave-one-out cross-validation for a kriging model and compares the resulting models created with those constructed using MLE and cross-validation. Chapter 4 explores probabilistic aspects of kriging models created using MLE, details a Bayesian analysis of the kriging model and describes the method of Markov Chain Monte Carlo (MCMC) as a more computationally efficient Bayesian analysis method to estimate kriging model parameters. A comparison of the resulting models created using MCMC and MLE is included.

45 Chapter 3 Parameter Estimation Methods for Deterministic Kriging Models Parameter estimation is the process of selecting the regression function coefficients, process variance, and spatial correlation function (SCF) parameters, γ = { β, σ 2, θ }, for the kriging model that shows the best possible predictive performance. In this chapter, two alternative methods to maximum likelihood estimation (MLE) for the estimation of parameters for deterministic kriging models are described. The first, the graphical method, is included since it was the first method used by geostatisticians to select correlation and process variance parameters, but it will not be used due to its inability to be automated and difficulty with visually selecting unbiased values. The second method, cross-validation (CV), is based on the optimization of an objective function like MLE. This chapter provides a description of these two alternative deterministic kriging model parameter estimation methods. The CV method is compared, detailing the advantages and deficiencies of the method, to the MLE method. The comparison between the CV and MLE methods is made through the use of six test functions that are described in the next section. 3.1 Test Functions In order to demonstrate and compare the four parameter estimation methods (MLE, CV, and the graphical method in this chapter and MLE and Bayesian Analysis in the next) investigated in this research, a simple one-dimensional test function is used. Addition-

46 33 ally, four two-dimensional test functions and a five-dimensional test function are used to demonstrate some of the computational issues that exist in higher dimensions. The observations used to create the kriging models using the different methods are listed in Appendix A One-Dimensional Test Function The 1-D test function (1D) calculates the output temperature of a chemical reaction [59]. The ratio of oxidant to the fuel being burned is increased from no oxidant to an excess of oxidant. In this process, the reaction increases temperature to a maximum and then decreases as excess oxidant is added as shown in Figure 3.1. The sample points, indicated by dots in Figure 3.1, are evenly spaced at 0.1 increments from 0 to 1 for a total of 11 points (see Table A.1 for actual data used). A set of 500 evenly spaced validation points is used to estimate the root mean square error (RMSE) by using Eq. (2.38) for each kriging model output temp K oxidant fuel ratio Figure 3.1: Output Temp. vs. Oxidant Fuel Ratio Two-Dimensional Test Functions Four different two-dimensional test functions are used in this work. They have all been used by other researchers while investigating kriging models. Two of the test functions, Test Function 1 and the Branin Function, are relatively smooth and well-behaved functions and demonstrate situations where the kriging model provides a good approximation of the original function. The other two functions, the Mystery Function and the Six-Hump Camelback Function, are much more complicated and can not be well approximated using the 21 observations sampled for the demonstrations (see Figure 3.3).

47 34 x x y 2 y x (a) Test Function x1 4 5 (b) Mystery Function 15 1 x x y x1 10 (c) Branin Function 4 y x1 1 2 (d) Six-Hump Camelback Function Figure 3.2: Plots of the Four 2-D Test Functions Plots of the response surface for the four two-dimensional functions are shown in Figure 3.2. The first 2-D test function (2D-TF1) is highly non-linear in one dimension and is linear in the second dimension. This was introduced by Osio and Amon [39] and used by Jin et al. [60] as a test problem for sequential experimentation. The function for TF1 is given by y (x) = cos (6 (x 1 0.5)) ( x ) + 2 (x 1 0.5) ( ) sin + 0.5x 2, x [0, 1] 2. x (3.1) The second 2-D test function is a mystery multi-modal function (2D-MF) in two dimensions that comes from Sasena [61]

48 35 Figure 3.3: 2-D Latin Hypercube Sampled Points ) 2 + (1 x1 ) + 2 (2 x 2 ) 2 y (x) = ( x 2 x sin (0.5x 1 ) sin (0.7x 1 x 2 ), x [0, 5] 2. (3.2) The third 2-D test function is the Branin test function (2D-BF) [60], which has less oscillation over its domain than 2D-MF; it is defined as y (x) = ( x π 2 x π x 1 6 ) ( 1 1 8π) cos (x1 ) + 10, x 1 [ 5, 10], x 2 [0, 15]. (3.3) The final 2-D test function is the six-hump camelback function (2D-SHCF) [61], which is defined as ( ) y (x) = 4 2.1x x4 1 3 x x 1x 2 + ( 4 + 4x 2 2) x 2 2, x 1 [ 2, 2], x 2 [ 1, 1]. (3.4) For all of these 2-D test functions, the same set of 21 sample points is used to fit each kriging model. These points are generated using a Latin Hypercube [22] (see Figure 3.3). The domain of each example is mapped to [0, 1] 2. A set of 900 validation points placed on a grid is used to evaluate the RMSE for the metamodels.