A Multivariate Input Uncertainty in Output Analysis for Stochastic Simulation

Transcription

1 A Mutivariate Input Uncertainty in Output Anaysis for Stochastic Simuation WEI XIE, Rensseaer Poytechnic Institute BARRY L. NELSON, Northwestern University RUSSELL R. BARTON, The Pennsyvania State University When we use simuations to estimate the performance of stochastic systems, the simuation is often driven by input modes estimated from finite rea-word data. A compete statistica characterization of system performance estimates requires quantifying both input mode and simuation estimation errors. The components of input modes in many compex systems coud be dependent. In this paper, we characterize the distribution of a random vector by its margina distributions and a dependence measure: either productmoment or Spearman rank correations. To quantify the impact from dependent input mode and simuation estimation errors on system performance estimates, we propose a metamode-assisted bootstrap framework. Specificay, we estimate the key properties of dependent input modes with rea-word data and construct a joint distribution by using the fexibe NORma To Anything (NORTA representation. Then, we empoy the bootstrap to capture the estimation error of the joint distributions, and an equation-based stochastic kriging metamode to propagate the input uncertainty to the output mean, which can aso reduce the infuence of simuation estimation error due to output variabiity. Asymptotic anaysis provides theoretica support for our approach, whie an empirica study demonstrates that it has good finite-sampe performance. Categories and Subject Descriptors: I.6.6 [Simuation and Modeing: Simuation Output Anaysis Genera Terms: Agorithms,Experimentation Additiona Key Words and Phrases: bootstrap, confidence interva, Gaussian process, mutivariate input uncertainty, NORTA, output anaysis. INTRODUCTION Stochastic simuation is used to estimate the behavior of compex systems that are driven by random input modes. The distributions of these input modes are often estimated from finite rea-word data. Therefore, a compete statistica characterization of stochastic system performance requires quantifying both input and simuation estimation error. Ignoring either source of uncertainty coud ead to unfounded confidence in the system performance estimate. The choice of input modes directy impacts the system performance estimates. A prevaent practice is to mode the input processes as a coection of independent and identicay distributed (i.i.d. univariate distributions. However, considering that components of rea inputs coud be dependent, these simpe modes do not aways faithfuy This paper is based upon work supported by the Nationa Science Foundation under Grant No. CMMI Author s addresses: W. Xie (corresponding author, Department of Industria and Systems Engineering, Rensseaer Poytechnic Institute, Troy, NY ; emai: xiew3@rpi.edu; B. L. Neson, Department of Industria Engineering and Management Sciences, Northwestern University, Evanston, IL 60208; emai: nesonb@northwestern.edu; R. R. Barton, Smea Coege of Business, Pennsyvania State University, University Park, PA 6802; emai: rbarton@psu.edu. Permission to make digita or hard copies of part or a of this work for persona or cassroom use is granted without fee provided that copies are not made or distributed for profit or commercia advantage and that copies show this notice on the first page or initia screen of a dispay aong with the fu citation. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to repubish, to post on servers, to redistribute to ists, or to use any component of this work in other works requires prior specific permission and/or a fee. Permissions may be requested from Pubications Dept., ACM, Inc., 2 Penn Paza, Suite 70, New York, NY USA, fax + ( , or permissions@acm.org. c YYYY ACM /YYYY/-ARTA $5.00 DOI:

2 A:2 W. Xie et a. represent the physica processes. For exampe, in a project panning network, the activity durations for different tasks coud be correated if they are affected by the same nuisance factors, e.g., weather conditions. In a suppy chain system, the demands of a customer over different products, e.g., ow-fat and whoe mik, coud be reated. Ignoring such dependence can ead to poor estimates of system performance measures. Thus, it is desirabe to buid input modes that can faithfuy capture the dependence. In this paper we account for input modes with random-vector distributions and do not consider time-series input processes. Bier and Ghosh [2006 reviewed various approaches to construct joint input distributions. Considering the amount of information needed to specify the joint distribution, amost a these methods suffer from some serious drawbacks. In ight of this difficuty, most input-modeing research focuses on methods that match ony certain key properties of the input modes incuding the margina distributions and some dependence measure. We assume that dependent input modes are characterized by their margina distributions and a dependence measure. Specificay, the margina distributions have known parametric famiies with parameter vaues unknown. The dependence between different components of input modes can be measured by various criteria [Bier and Ghosh We focus on product-moment and Spearman rank correations in this paper. Product-moment correation is widey used in engineering appications. The definition of product correation needs the variances of the components to be finite. Thus, we aso incude the use of Spearman rank correation as a dependence measure, which finds wide appications in business studies, e.g., decision and risk anaysis [Cemen and Reiy 999. Instead of measuring inear dependence, the Spearman rank correation is a nonparametric method for capturing the monotonic reationship between different components of input modes. It does not require the variances of the components to be finite. Further, since it is based on ranks, this measure is not sensitive to observation outiers. Notice that in genera dependence may be more than just pairwise, and may be noninear, in which case more compex characterization is needed [Wu and Mieniczuk 200. Since margina distribution parameters and dependence measures are estimated from rea-word data, their estimation error is caed input uncertainty. When we use the simuation outputs to estimate system performance, there are two sources of uncertainty: input and simuation error. To quantify the overa uncertainty about the system performance estimate, we buid on Xie et a. [204b, which proposed a metamodeassisted bootstrapping approach to form a confidence interva (CI accounting for the impact of input and simuation uncertainty. Further, a variance decomposition was proposed to estimate the reative contribution of input to overa uncertainty. However, our previous study in Xie et a. [204b was based on the assumption that the input distributions are univariate and mutuay independent. The independence assumption does not hod in genera for input modes in many stochastic systems. This paper is a significant enhancement of Xie et a. [204b. To efficienty and correcty account for uncertainty in mutivariate input modes, we introduce a more genera metamode-assisted bootstrapping framework; it can quantify the impact of dependent input uncertainty and simuation estimation error on system performance estimates whie aso reducing the infuence of simuation estimation error due to finite simuation effort as compared with direct simuation methods. Specificay, we estimate margina distribution parameters and dependence measures of input modes with rea-word data, and construct the joint distributions by using the fexibe NORma To Anything (NORTA representation [Cario and Neson 997. Then, the bootstrap is used to quantify the estimation error of these joint distributions, and an equation-based stochastic kriging (SK metamode propagates the input uncertainty

3 Mutivariate Input Uncertainty in Output Anaysis for Stochastic Simuation A:3 to output mean. We can derive a CI that accounts for both simuation and input uncertainty by using this generaized metamode-assisted bootstrapping approach. Therefore, our approach aows us to do statistica uncertainty anaysis for stochastic systems with dependence in the input modes. Notice that our metamode-assisted bootstrapping framework does not require NORTA. Other approaches that characterize dependent input modes by their parametric margina distributions and a dependence matrix coud be easiy empoyed in our framework. There are two centra contributions of this paper: First, we generaize the metamodeassisted bootstrap framework [Xie et a. 204b to stochastic simuation with dependent input modes; second, we propose a rigorous anaysis for cases where the dependence is measured by product-moment or Spearman rank correation. The next section describes other research on dependent input modeing and input uncertainty anaysis. This is foowed by a forma description of the probem of interest in Section 3. In Section 4, we propose a generaized metamode-assisted bootstrapping framework and provide a procedure to buid a CI accounting for both input and simuation estimation error on system mean performance estimates. Our approach is supported by asymptotic anaysis. We then report resuts of finite sampe behavior from an empirica study in Section 5 and concude the paper in Section 6. A proofs are in the Appendix. 2. BACKGROUND For stochastic simuations, various approaches to account for input uncertainty have been proposed; see Barton [202 and Song et a. [204 for reviews. The methods can be divided into Bayesian and frequentist approaches, which have their underying merits and imitations [Xie et a. 204a. To faithfuy capture the dependence between different components of input modes, Cario and Neson [997 proposed a fexibe NORTA distribution to represent and generate random vectors with amost arbitrary margina distributions and productmoment correation matrix. Cemen and Reiy [999 used NORTA to represent the dependent input modes for decision and risk anaysis with dependence measured by Spearman s and Kenda s rank correations. Bier and Coru [20 proposed a Bayesian approach to account for the parameter uncertainty for dependent input modes. Correated inputs are modeed with NORTA and the dependence is measured by product-moment correation. The uncertainty around the NORTA distribution parameters estimated from rea-word data is quantified by posterior distributions. For compex stochastic systems with a arge number of correated inputs, a fast agorithm draws sampes from these posterior distributions to quantify the input uncertainty. Then, the direct simuation method is used to propagate the input uncertainty to output mean by running simuations at each sampe point, which coud be computationay expensive for compex simuated systems. Further, the direct simuation method does not incorporate the simuation uncertainty into the Bayesian formuation. Direct bootstrapping uses bootstrap resamping of the rea-word data to measure the input uncertainty and propagates it to output mean by direct simuation [Barton and Schruben 993; 200; Barton 2007; Cheng and Hoand 997. Compared with the Bayesian approaches, the direct bootstrap can be adapted to any input modes without additiona anaysis and it does not need to resort to computationay expensive approaches to draw sampes to quantify the input uncertainty. However, direct simuation cannot efficienty use the computationa budget to reduce the impact from simuation estimation error. Further, since the statistic that is bootstrapped is the random output of a simuation, it is not a smooth function of input data; this vioates the asymptotic vaidity of the bootstrap.

4 A:4 W. Xie et a. The metamode-assisted bootstrapping approach was introduced by Barton et a. [204. The input uncertainty is measured by bootstrapping and an equation-based SK metamode propagates the input uncertainty to the output mean. This approach addresses some of the shortcomings of the direct bootstrap. Specificay, the metamode can reduce the impact of simuation estimation error. Further, metamodeing makes the bootstrap statistic a smooth function of the input data so that the asymptotic vaidity concerns faced by the direct bootstrap method disappear. However, Barton et a. [204 assumed that the simuation budget is not tight and the metamode uncertainty can be ignored. If the true mean response surface is compex, especiay for high-dimensiona probems with many input distributions, and the computationa budget is tight, the impact of metamode uncertainty can no onger be ignored. The metamode-assisted bootstrapping approach was improved in Xie et a. [204b to buid a CI accounting for the impact from both input and metamode uncertainty on the system mean estimates. Further, a variance decomposition was proposed to estimate the reative contribution of input to overa uncertainty, which is very usefu for decision makers to determine where to put more effort to reduce the estimator error. The metamode-assisted bootstrapping approach demonstrates robust performance even when there is a tight computationa budget and simuation estimation error is arge. However, Xie et a. [204b is based on the assumption that input modes are a coection of mutuay independent univariate distributions. The success of metamode-assisted bootstrapping for stochastic simuations with independent univariate input distributions in Xie et a. [204b motivates us to extend it to more compex cases with dependence in the input modes. 3. PROBLEM STATEMENT The stochastic simuation output is a function of random numbers and the input mode denoted by F. For notation simpification, we do not expicity incude the random numbers. The output from the jth repication of a simuation with input mode F can be written as Y j (F = µ(f + ɛ j (F where µ(f = E[Y j (F denotes the unknown output mean and ɛ j (F represents the simuation error with mean zero. Notice that the simuation output depends on the choice of input mode. Let F = {F, F 2,..., F L } with F, F 2,..., F L mutuay independent; F coud be composed of univariate and mutivariate joint distributions. Let F be a d dimensiona distribution having margina distributions denoted by {F,, F,2,..., F,d } with d. For the distributions F with d >, we focus on the continuous margina distributions with stricty increasing cumuative distribution functions (cdf {F,, F,2,..., F,d }. For the th distribution F with d >, we suppose that it is characterized by margina distributions and a dependence measure: either product-moment or Spearman rank correation matrix. Specificay, et a d random vector X F having d d product-moment and Spearman rank correation matrix denoted, respectivey, by ρ X and R X ρ X (i, j = corr(x, X,j = Cov(X, X,j Var(X Var(X,j, R X (i, j = corr(f (X, F,j (X,j = E[F (X F,j (X,j E[F (X E[F,j (X,j Var(F (X Var(F,j (X,j with i, j =, 2,..., d. Suppose these correation matrices are positive definite. Since the correation matrices are symmetric and their diagona terms are, we can view a

5 Mutivariate Input Uncertainty in Output Anaysis for Stochastic Simuation A:5 d d correation matrix as an eement of d d (d /2 dimensiona space. Therefore, the product-moment and Spearman rank correation matrix can be uniquey specified by d vectors denoted by Vρ X and VX R, respectivey. For F with d =, these correation vectors are empty and d = 0. For F with =, 2,..., L, we assume that the famiies of margina distributions {F,, F,2,..., F,d } are known, but not their parameter vaues. Let an h vector θ denote the unknown parameters for the ith margina distribution F. By stacking θ with i =, 2,..., d together, we have a d dimensiona parameter vector θ (θ,, θ,2,..., θ,d with d d i= h. Input modes characterized by margina distributions and correation matrices can be specified by ϑ {(θ ; V X, =, 2,..., L} that incudes d L = d + L = d eements, where V X = V ρ X or VX R. We ca ϑ the input mode parameters. By abusing notation, we can rewrite µ(f as µ(ϑ. The true input parameters ϑ c are unknown and estimated from finite sampes of rea-word data. Thus, our goa is finding a ( α00% CI [Q L, Q U such that Pr{µ(ϑ c [Q L, Q U } = α. ( Given finite rea-word data, the samping distribution of estimators for input mode parameters, denoted by ϑ m, can be used to quantify the input uncertainty. Specificay, et denote the number of i.i.d. rea-word observations avaiabe from the th input process X,m { X (, X (2,..., X ( } with a d random vector X (i i.i.d F c for i =, 2,...,. Let X m = {X,m, =, 2,..., L} be the coection of sampes from a L input distributions in F c, where m = (m, m 2,..., m L. The rea-word data are a particuar reaization of X m, say x (0 m, and the estimator of input mode parameters ϑ m = ( θ m ; V ρ X,m or ( θ m ; V X,m R is a function of X m. Under the assumption that the first h margina moments are finite for i =, 2,..., d and =, 2,..., L, we can use the moment estimators for margina distribution parameters, denoted by θ m [Xie et a. 204b. The usua estimators for the product-moment and Spearman rank correations are m ( X ρ X, (i, j = R X, (i, j = [ m X ( X,j X,j /(m (2 S S,j ( m r(x r(x ( r(x,j r(x,j ( [ 2 r(x r(x m ( (3 2 r(x,j r(x,j for i, j =, 2,..., d and =, 2,..., L with d >, where X = X / and S 2 = ( X X 2/(m. We denote the rank function by r( rank(. In this ( paper, we use the uprank to estimate the Spearman rank correations. It is defined by r X ( j= I X (j X and r(x = ( r X / with I( denoting an indicator function. Then, based on Equations (2 and (3, we can find corresponding correation estimators V ρ X,m and V X,m R. The impact of input uncertainty on the system mean performance estimate is quantified by the samping distribution of µ( ϑ m. Further, since the underying response surface µ( is unknown, at any θ, et µ( ϑ denote the corresponding mean response

6 A:6 W. Xie et a. estimator. Thus, there are both input and simuation estimation errors in the system mean performance estimates. For stochastic systems with dependent input modes, our objective is to propose an approach to quantify the overa impact of both input and simuation estimation error on system mean performance estimates and then buid a CI satisfying Equation (. Further, since each simuation run coud be computationay expensive and we have tight computationa budget, we want to reduce the infuence of simuation estimation error. 4. METAMODEL-ASSISTED BOOTSTRAPPING FRAMEWORK For probems with parametric input distributions that are univariate and mutuay independent, the metamode-assisted bootstrapping framework was used to account for the impact of both input and simuation estimation errors on the system performance estimates in Xie et a. [204b. In this section, we generaize the metamode-assisted bootstrapping approach for stochastic simuations with dependence in the input modes. The procedure to buid a CI quantifying the overa uncertainty of system mean performance estimates is described. Further, asymptotic anaysis provides theoretica support for our approach. To make this section easy to foow, we start with an overa description of the generaized metamode-assisted bootstrapping framework. We assume that dependent input modes can be characterized by the margina distributions and a dependence measure, either product-moment or Spearman correation matrix. Since the partia characterization by these key properties does not uniquey determine the joint distributions except in specia cases, we use the NORTA representation to construct the joint distributions. Notice that uness the true distribution is NORTA, there is unmeasured error due to incorrect input modes. This error is not addressed in this paper. Further, we suppose that the parametric margina distributions can be specified by either parameters or moments. Since the margina moments/parameters and correation matrix are estimated from finite rea-word data, the metamode-assisted bootstrapping framework is proposed to quantify the overa uncertainty of system performance estimate from input and simuation estimation errors. Specificay, we empoy the bootstrap to capture the estimation error of the joint distributions and propagate the input uncertainty to the output mean by using an equation-based stochastic kriging metamode that is buit based on the simuation outputs at a few we-chosen design points. Both simuation and metamode uncertainty can be estimated using properties of the metamode. Then, we derive a CI that accounts for both simuation and input uncertainty. 4.. Bootstrap for Input Uncertainty In this section, we describe how to empoy the bootstrap to quantify the input uncertainty. The way we choose to represent input modes pays an important roe in the impementation of metamode-assisted bootstrapping. For the th input distribution F, instead of using the natura parameters θ to characterize the margina distributions, we can use moments; see Barton et a. [204 for an expanation. Suppose that the parametric margina distribution F can be uniquey characterized by its first h finite moments denoted by the h vector ψ for i =, 2,..., d. By stacking ψ with i =, 2,..., d together, we have a d dimensiona vector of margina moments ψ (ψ,, ψ,2,..., ψ,d. Therefore, the input modes can be characterized by the coection of moments M {M, =, 2,..., L} with M = (ψ ; V X and V X = V ρ X or VX R. Notice that there is a one-to-one mapping between the input parameters ϑ and moments M. Abusing notation, we rewrite µ(ϑ as µ(m.

7 Mutivariate Input Uncertainty in Output Anaysis for Stochastic Simuation A:7 The true moments of dependent input modes, denoted by M c, are unknown and estimated based on a finite sampe X m. Specificay, we use standardized sampe moments as estimators for margina distributions, denoted by ψ m ; see Xie et a. [204b. The correation estimator V ρ X,m or V X,m R is obtained by using Equation (2 or (3. The estimation error of input modes can be quantified by the samping distribution of M m = ( ψ m ; V ρ X,m or ( ψ m ; V X,m R, denoted by F M c m. Therefore, the impact of input uncertainty on the system mean performance estimate can be measured by the samping distribution of µ( M m with M m FM c m. Since it is hard to derive the samping distribution FM c m, we use bootstrap resamping to approximate it [Shao and Tu 995. Specificay, denote index sets by A {, 2,..., } for the th distribution with =, 2,..., L. Notice that since F, F 2,..., F L are mutuay independent, we can do bootstrapping for each distribution separatey. Impementation of bootstrap resamping in metamode-assisted bootstrapping is as foows. ( For the th distribution F with =, 2,..., L, draw sampes with repacement from set A and obtain bootstrapped indexes {i, i 2,..., i m }; choose corresponding sampes from rea-word data x (0 m and get x (, {x (i, x (i2,..., x (i }. Denote the coection of bootstrap sampes by x ( m = { x ( cacuate the bootstrapped moment estimate, denoted by or ( ψ ( m ; (ṼR X,m. ( (2 Repeat the previous step B times to generate x (0 m x (0 m,, =, 2,..., L} and use them to ( M m ( ψ ( m ; (Ṽρ X,m ( M (b m with b =, 2,..., B. The bootstrap resamped moments are drawn from the bootstrap distribution denoted by F ( Mm with M m F ( Mm. For estimation of a CI, B is recommended to be a few thousand; see Xie et a. [204a for an expanation. In this paper, a denotes a quantity estimated from rea-word data, whie a denotes a quantity estimated from bootstrapped data. Theorem 4. shows that when the amount of rea-word data increases to infinity, the bootstrap provides a consistent estimator for the true input moments M c. THEOREM 4.. conditions hod. Suppose F, F 2,..., F L are mutuay independent and the foowing ( We have X i.i.d F c with k =, 2,..., and =, 2,..., L; (2 The margina distribution F c is uniquey characterized by first h moments and it has finite first 4h moments for i =, 2,..., d and =, 2,..., L. (3 E(X 4 X4,j < for i, j =, 2,..., d and =, 2,..., L with d >. (4 As m, we have /m c with =, 2,..., L for a constant c > 0. Then, as m, the bootstrap moment estimator M m converges a.s. to the true moments M c. The detaied proof of Theorem 4. is provided in the onine appendix NORTA Representation In this section we describe how to empoy the NORTA representation for constructing the joint distributions and generating sampes of random vectors, given the partia

8 A:8 W. Xie et a. characterization specified by margina distributions and a dependence measure either product-moment or Spearman rank correations. Since F = {F, F 2,..., F L } with F, F 2,..., F L mutuay independent, we ony need to appy NORTA separatey for the distributions F with d >. Specificay, we represent X as a transformation of a d -dimensiona standard mutivariate norma (MVN vector Z = (Z,, Z,2,..., Z,d with product-moment correation matrix denoted by ρ Z, X = ( F, [Φ(Z,; θ,, F,2 [Φ(Z,2; θ,2,..., F,d [Φ(Z,d ; θ,d (4 where Φ( denotes the cdf for the standard norma distribution. If the margina distribution famiies are given, as we assume here, then the NORTA representation for F can be specified by (θ, ρ Z. Let R Z denote the d d Spearman rank correation matrix for Z. There is a cosed form reation between product-moment and Spearman rank correations for the standard norma distribution [Cemen and Reiy 999 R Z (i, j = 6 π sin ( ρz (i, j 2 with i, j =, 2,..., d. Since the NORTA impementations for cases where the dependence is measured by Spearman rank and product-moment correations are different, we describe them separatey Spearman Rank Correation. If the dependence in the input modes is measured by the Spearman rank correation, we have R X = R Z since it is invariant under monotone one-to-one transformation F [Φ( for i =, 2,..., d. Therefore, given key properties of the input modes characterized by ϑ = (θ; VX R, the procedure to find NORTA representations and generate sampes for X is as foows. ( From VX R, get the Spearman rank correation matrix for Z, R Z = R X. (2 Cacuate ρ Z (i, j = 2 sin(πr Z (i, j/6 for i, j =, 2,..., d. i.i.d. (3 Generate Z MVN(0, ρ Z and obtain X by using Equation (4. (4 Appy Steps -3 to a F with d >. For F with d =, use a standard approach [Neson 203, such as the inverse cdf method, to generate sampes for X. By repeating this procedure, we generate sampes for X and then use them to drive simuations to estimate the mean response µ(ϑ. Notice that when we use Spearman rank correation to measure the dependence between the components of input modes, the choice of margina distributions and correation is separabe. Thus, for F specified by any combination of feasibe θ and positive definite R X, we can find corresponding NORTA representations. However, this may not be true when we use the product-moment correation for the dependence measure Product-Moment Correation. If the dependence between the components of input modes is measured by product-moment correation, then the procedure to find NORTA representations becomes more compex because the choice of margina distributions infuences the feasibiity of correation matrices. Specificay, for F with d >, there is a pairwise reation between product-moment correation matrices of X and Z ρ X (i, j = C j [ρ Z (i, j; θ F [Φ(z ; θ F,j [Φ(z,j; θ,j ϕ ρz (i,j(z, z,j dz dz,j E(X E(X,j (6 Var(X Var(X,j (5

9 Mutivariate Input Uncertainty in Output Anaysis for Stochastic Simuation A:9 where, ϕ ρz (i,j denotes the standard bivariate norma density with correation ρ Z (i, j and C j denotes a pairwise transformation from ρ Z (i, j to ρ X (i, j for i, j =, 2,..., d. Given θ and ρ X, we sove d moment matching Equations (6 to find ρ Z. Unike Spearman rank correation, the margina distributions characterized by parameters θ pay an important roe in determining the vaue and feasibiity of correation matrix ρ Z. Therefore, given key properties of input modes characterized by ϑ = (θ; V ρ X, if their NORTA representations exist, the procedure to find them and generate sampes for X is as foows. ( Given V ρ X, sove Equations (6 for correation matrix ρ Z. (2 Generate Z i.i.d. MVN(0, ρ Z and obtain X by using Equation (4. (3 Appy Steps -2 for a F with d >. For F with d =, use a standard approach [Neson 203, such as the inverse cdf method, to generate sampe for X. By repeating this procedure, we generate sampes for X and then use them to drive simuations to estimate µ(ϑ. When we sove for ρ Z in Step, typicay, there is no cosed form anaytica soution except for some specia margina distributions, e.g., uniform distribution, and we need resort to numerica search to obtain ρ Z. Notice that this procedure ony works under the condition that the correation matrix ρ Z obtained in Step ( is positive semidefinite, which may not hod in genera [Ghosh and Henderson That means a NORTA representation may not exist for some combination of feasibe θ and positive definite ρ X. If ρ Z obtained by soving d Equations (6 is positive semidefinite, (θ, ρ X is caed NORTA feasibe; Otherwise, it is caed NORTA infeasibe. Therefore, input modes with dependence measured by product-moment correation are NORTA feasibe if (θ, ρ X is NORTA feasibe for a =, 2,..., L with d >. Based on the study by Ghosh and Henderson [2002, NORTA infeasibiity is more ikey in high-dimensions and with correations cose to ±. For a NORTA infeasibe (θ, ρ X, we can find a feasibe correation matrix that is cose to ρ X. Since F, F 2,..., F L are mutuay independent, we can consider each F separatey. Theorem 4.2 gives a property of the NORTA feasibe region: For any F with d >, if the true moment combination M c has a feasibe NORTA representation with positive definite ρ c Z, then there exists a sma open neighborhood centered at M c in the d + d space such that any moment combination M in the neighborhood has a feasibe NORTA representation. This property is usefu when we show the asymptotic consistency of the CI buit by the metamode-assisted bootstrapping to quantify both input and simuation uncertainty in Section 4.4. THEOREM 4.2. Let Θ R d be the feasibe domain for margina distribution parameters θ and suppose θ c is an interior point in Θ. Suppose there is one-to-one continuous mapping between margina moments ψ and parameters θ for i =, 2,..., d. For F with d >, suppose the foowing conditions hod. ( F c has a NORTA representation (θc, ρc Z with ρ c Z positive definite. (2 At any x, suppose the margina distributions F (x; θ, density functions f (x; θ and inverse distributions F (x; θ are continuousy differentiabe over θ for i =, 2,..., d on Θ. (3 For any θ Θ, suppose the margina cdfs F, (x; θ,, F,2 (x; θ,2,..., F,d (x; θ,d are continuous and stricty increasing in x.

10 A:0 W. Xie et a. Then the true moment vector M c = (ψc ; (Vρ X c is an interior point of the NORTA feasibe region: In the d + d dimensiona space, there exists a constant δ > 0 such that any moment combination M in the open ba B δ (M c is NORTA feasibe. The detaied proof of Theorem 4.2 is provided in the onine appendix Stochastic Kriging Metamode In the metamode-assisted bootstrapping framework, after quantifying the input uncertainty with the bootstrap as described in Section 4., an equation-based SK metamode introduced by Ankenman et a. [200 is used to propagate this uncertainty to the output mean. The succinct review of SK in this section is based on Xie et a. [204b. Dependent input modes characterized by moments M can be interpreted as a ocation x in a d = L = (d + d dimension space. Suppose that the underying true (but unknown response surface is a continuous funtion of moments of input modes and it can be thought of as a reaization of a stationary Gaussian Process (GP. We mode the simuation output Y by Y j (x = β 0 + W (x + ɛ j (x. (7 This mode incudes two sources of uncertainty: the simuation output uncertainty ɛ j (x and mean response uncertainty W (x. For many, but not a, simuation settings the output is an average of a arge number of more basic outputs, so a norma approximation can be appied: ɛ(x N(0, σɛ 2 (x. Since stochastic systems with dependent input modes having simiar key properties tend to have cose mean responses, a zero-mean, second-order stationary GP W ( is used to account for this spatia dependence. Therefore, the uncertainty about the unknown true response surface µ(x is represented by a GP M(x β 0 + W (x (note that β 0 can be repaced by a more genera trend term f(x β. Its spatia dependence is characterized by the covariance function, Σ(x, x = Cov[W (x, W (x = τ 2 γ(x x, where τ 2 denotes the variance and γ( is a correation function that depends ony on the distance x x. Based on prior information about smoothness of µ(, we can choose the form of correation function [Xie et a Considering that mean response surfaces for most system engineering probems have a high order of smoothness, we use the product-form Gaussian correation function ( d γ(x x = exp ζ j (x j x j 2 (8 for the empirica evauation in Section 5. Let ζ = (ζ, ζ 2,..., ζ d represent the correation parameters. Before having any simuation resut, the uncertainty about µ(x can be represented by a Gaussian process M(x GP(β 0, τ 2 γ(x x. To reduce the uncertainty about µ(x we choose an experiment design consisting of pairs D {(x i, n i, i =, 2,..., k} with (x i, n i denoting the ocation and the number of repications at the ith design point. The simuation outputs at D are Y D {(Y (x i, Y 2 (x i,..., Y ni (x i ; i =, 2,..., k} and the sampe mean at design point x i is Ȳ (x i = n i j= Y j(x i /n i. Let the sampe means at a k design points be Ȳ D = (Ȳ (x, Ȳ (x 2,..., Ȳ (x k T and its variance is represented by a k k diagona matrix C = diag { } σɛ 2 (x /n, σɛ 2 (x 2 /n 2,..., σɛ 2 (x k /n k because the use of common random numbers is detrimenta to prediction [Chen et a The simuation outputs Y D and spatia dependence characterized by the covariance function Σ(, can be used to improve system mean prediction at any fixed point x. Specificay, et Σ be the k k spatia covariance matrix of the design points and et j=

11 Mutivariate Input Uncertainty in Output Anaysis for Stochastic Simuation A: Σ(x, be the k spatia covariance vector between each design point and x. If the parameters (τ 2, ζ, C are known, then the metamode uncertainty can be characterized by a refined GP M p (x that denotes the conditiona distribution of M(x given a simuation outputs, M p (x GP(m p (x, σ 2 p(x (9 where m p ( is the minimum mean squared error (MSE inear unbiased predictor and the corresponding variance is m p (x = β 0 + Σ(x, (Σ + C (ȲD β 0 k, (0 σ 2 p(x = τ 2 Σ(x, (Σ + C Σ(x, + η [ k (Σ + C k η ( where β 0 = [ k (Σ + C k k (Σ + C Ȳ D and η = k (Σ + C Σ(x, [Ankenman et a Since in reaity the spatia correation parameters τ 2 and ζ are unknown, maximum ikeihood estimates are typicay used for prediction, and the sampe variance is used as an estimate for the simuation variance at design points C [Ankenman et a By substituting parameter estimates ( τ 2, ζ, Ĉ in Equations (0 and ( we can obtain the estimated mean m p (x and variance σ p(x. 2 Thus, the metamode we use is µ(x = m p (x with variance estimated by σ p(x. 2 This is commony done in the kriging iterature because incuding the parameter estimation error is intractabe Procedure to Buid CI Since there are both input and simuation estimation errors in the system mean performance estimates, in this section, we propose a procedure to buid a CI quantifying the overa uncertainty for µ(m c. We show that as m, B, the CI has asymptoticay consistent coverage. Based on a hierarchica samping approach, we propose the foowing procedure to buid a ( α bootstrap percentie CI. ( Identify a design space E for the moments of input modes M over which to fit the metamode. Since the metamode is used to propagate the input uncertainty measured by the bootstrapped moments M m to the output mean, the design space is chosen to be the smaest eipsoid covering most ikey bootstrapped moments. See Barton et a. [204 for more detaied information. (2 Use a maximin distance Latin hypercube design to embed k design points into the design space E. Assign equa repications to k design points to exhaust the simuation budget N and obtain an experiment design D = {(M (i, n i, i =, 2,..., k}. (3 At k design points, generate sampes of X by using NORTA representations for F with d > as described in Section 4.2 and using a standard approach [Neson 203 for F with d =, =, 2,..., L. Use these sampes to drive simuations and obtain outputs y D. Compute the sampe average ȳ(m (i and sampe variance s 2 (M (i of the simuation outputs, i =, 2,..., k. Fit a SK metamode to obtain m p ( and σ p( 2 using ( ȳ(m (i, s 2 (M (i, M (i, i =, 2,..., k. (4 For b = to B (a Generate bootstrap moments (b Draw M ( (b b N m p ( M m, σ p( 2 Next b (5 Report CI: [ M( B α 2, M ( B( α vaues. 2 M (b m by foowing the procedure in Section 4... M (b m where, M ( M (2 M (B are the sorted

12 A:2 W. Xie et a. Remark 4.3. Unike direct bootstrapping that runs simuation at a sampes generated to quantify the input uncertainty, M(b m with b =, 2,..., B, the metamodeassisted bootstrapping buids a metamode based on the simuation resuts at wechosen design points and uses the metamode to propagate the input uncertainty to the output mean. To buid joint distributions to drive the simuation runs, design points need to be NORTA feasibe. Since some bootstrapped moments may be NORTA infeasibe, the design space E buit to cover the most ikey bootstrapped sampes coud incude moment combinations M that are aso NORTA infeasibe. However, when the dimension of correated random vector is reativey ow, say d 5, and the pairwise correation is not so extreme or cose to ±, the NORTA infeasibe probem is rare for the sampe sizes of rea-word data encountered in many appications, as shown by the empirica study in Section 5. Therefore, we remove any NORTA infeasibe design points in the design space E in Step (2 and assign equa repications to the remaining points. Then, we construct a SK metamode and use it to estimate mean responses (b at a bootstrapped sampes M m with b =, 2,..., B even incuding the NORTA infeasibe trias because there typicay exists a cose NORTA approximation for the infeasibe bootstrapped moments [Ghosh and Henderson 2002 and µ( for most system engineering probems has a high order of smoothness. Since F, F 2,..., F L are mutuay independent, by appying Theorem 4.2, we have: If the true moments M c are NORTA feasibe having positive definite ρc Z for =, 2,..., L with d >, as the amount of rea-word data increases, a the moments M in the design space E eventuay become NORTA feasibe. Specificay, if M c has a NORTA representation with positive definite ρ c Z, it is an interior point of NORTA feasibe region as described in Theorem 4.2. By Theorem 4., as m, we have a consistent moment estimator M a.s. m M c. Since the design space E is buit to cover most ikey bootstrap sampes, it automaticay shrinks to smaer and smaer region around M c as the amount of rea-word data increases. Therefore, E woud eventuay be incuded in the NORTA feasibe region and a moment combinations in the design space have feasibe NORTA representation. That means asymptoticay we can ignore the effect of NORTA infeasibe probem in the metamode-assisted bootstrap framework. The CI [ M( B α 2, M ( B( α characterizes the impact from both input and metamode uncertainty on system performance estimate. A variance decomposition in Xie et a. [204b can be used to assess their reative contributions and guide a decision maker as to where to put more effort: If the input uncertainty dominates, then get more rea-word data; if the metamode uncertainty dominates, then run more simuations; if neither dominates, then do both activities to improve the estimation accuracy of µ(m c. If SK parameters (τ 2, ζ, C are known and we repace M b in Step (4.b of the CI procedure with M b N ( (b m p ( M m, σp( 2 (b M m, we can show that the CI obtained [M ( B α 2, M ( B( α 2 is asymptoticay consistent by Theorem 4.4. In practice, (τ 2, ζ, C must be estimated. Based on the sensitivity study in Xie et a. [204b, SK can provide robust inference without accounting for parameter-estimation error when we empoy an adequate experiment design, e.g., space-fiing design used in this paper. 2 THEOREM 4.4. Suppose conditions of Theorems 4. and 4.2 and the foowing additiona assumptions hod.

13 Mutivariate Input Uncertainty in Output Anaysis for Stochastic Simuation A:3 Fig.. A stochastic activity network. ( ɛ j (x i.i.d. N(0, σɛ 2 (x for any x, and M(x is a stationary, separabe GP with a continuous correation function satisfying γ(x x c og( x x 2 for a x x 2 δ 2 (2 +δ d for some c > 0, δ > 0 and δ 2 <, where x x 2 = j= (x j x j 2. (2 The input processes, simuation noise ɛ j (x and GP M(x are mutuay independent and the bootstrap process is independent of a of them. Then the interva [M ( B α 2, M ( B( α 2 is asymptoticay consistent, meaning the iterated imit im m im Pr{M ( Bα/2 M p (M c M ( B( α/2 } = α. (3 B The detaied proof of Theorem 4.4 is provided in the onine appendix. 5. EMPIRICAL STUDY Since F, F 2,..., F L are mutuay independent and we addressed the case of independent univariate distributions in Xie et a. [204b, here we consider an exampe with L = mutivariate distribution and suppress the subscript for the th input distribution. We use the stochastic activity network shown in Figure to examine the finitesampe performance of our metamode-assisted bootstrapping approach. Suppose that the time required to compete task (arc i is denoted by X i for i =, 2,..., 5 and X = (X, X 2,..., X 5. We wish to compute the time to compete the project, which is the ongest path through the network, Y = max{x + X 2 + X 5, X + X 4, X 3 + X 5 }. We are interested in the mean response E[Y. We assume that F c is NORTA. The margina distributions are X i exp(θi c for i =, 2,..., 5 with means θ c = (0, 5, 2,, 5. We consider two cases with dependence measured by either Spearman rank or product-moment correations. The true correation matrices RX c or ρc X are equa to A

14 A:4 W. Xie et a. Therefore, the number of parameters characterizing the input mode F is d = d + d = 5 + (5 4/2 = 5. Since the true mean response µ(ϑ c is unknown, we run 0 7 repications and obtain the foowing resuts: Case : If we use the Spearman rank correation with RX c = A, the estimated true mean response is with standard error Case 2: If we use the product-moment correation with ρ c X = A, the estimated true mean response is with standard error To evauate our metamode-assisted bootstrapping approach, we pretend that the input-mode parameters (θ c, RX c or (θc, ρ c X are unknown and they are estimated by m i.i.d. observations from F c ; this represents obtaining rea-word data. The goa is to buid a CI quantifying the impact of both input and simuation estimation error on the system mean response estimate. We compare metamode-assisted bootstrapping to the conditiona CI and direct bootstrapping. For the conditiona CI, we fit the input distribution to the rea-word data by moment matching and aocate the entire computationa budget of N repications to simuating the resuting system. In direct bootstrapping, we run N/B repications (b of the simuation at each bootstrap moment M m, record the average simuation output Ȳb (b = Ȳ ( M m, and report the percentie CI [Ȳ( B α 2, Ȳ( B( α 2. In metamodeassisted bootstrapping, we eveny assign N repications to k design points, run simuations, buid a SK metamode and record the percentie CI [ M( B α 2, M ( B( α 2 by foowing the procedure in Section 4.4. For the input distribution with dependence characterized by either Spearman rank or product-moment correations, Tabes I and II show the statistica performance of conditiona and direct bootstrapping CIs and metamode-assisted bootstrapping with k = 80 design points, m = 00, 500, 000 rea-word observations, and computationa budget of N = 0 3, 0 4, and 0 5 repications. We ran 000 macro-repications of the entire experiment. In each macro-repication, we first generate m mutivariate observations by using NORTA with parameters (θ c, RX c or (θc, ρ c X. Then, for the conditiona CI, we run N repications at the estimated parameters ( θ m, R X,m or ( θ m, ρ X,m and buid CIs with nomina 95% coverage of the response mean. For direct bootstrapping and metamode-assisted bootstrapping, we use bootstrapping to generate B = 000 sampe moments to quantify the input uncertainty. Since µ( is unknown, we use the fixed computationa budget N to propagate the input uncertainty either via direct simuation or via the SK metamode to buid percentie CIs with nomina 95% coverage. When we use the product-moment correation to measure the dependence, the probabiity of NORTA infeasibe bootstrapped moments M m is ow with mean 0.% and (b standard deviation 0.34% for m = 00, and 0% for m = 500, 000. In metamode-assisted bootstrapping, we remove any NORTA infeasibe design points and eveny assign N repications to the remaining points to buid the SK metamode. In direct bootstrapping, for the sma percentage of NORTA infeasibe bootstrap resamped moments M m, an approximation is used to estimate the system mean re- (b (b sponse. Specificay, for each M m, we sove the moment-matching equations (6 for ρ Z. If ρ Z is positive definite, we can use the NORTA representation to generate sampes of X and run simuations. Otherwise, we find a cose positive semi-definite approximation for ρ Z, denoted by ρ Z, [Higham 2002 and et ρ a Z ρ Z + δ I 5 5, where I 5 5 denotes a 5 5 identity matrix and δ is a sma positive vaue. We use δ = 0 5 in the empirica study. Then, we set ρ a Z as the correation matrix for NORTA and generate sampes of X for simuation runs.

15 Mutivariate Input Uncertainty in Output Anaysis for Stochastic Simuation A:5 Tabe I. Resuts of nomina 95% CIs when m = 00, 500, 000 for Case when the dependence is characterized by Spearman rank correations. m = 00 N = 0 3 N = 0 4 N = 0 5 conditiona CI coverage 43% 4.% 4.4% CI width (mean CI width (SD direct bootstrap coverage 00% 00% 99.4% CI width (mean CI width (SD metamode-assisted coverage 97.6% 96.% 94.8% bootstrap CI width (mean CI width (SD m = 500 N = 0 3 N = 0 4 N = 0 5 conditiona CI coverage 76.9% 32.% 2% CI width (mean CI width (SD direct bootstrap coverage 00% 00% 00% CI width (mean CI width (SD metamode-assisted coverage 96.3% 98.7% 97.2% bootstrap CI width (mean CI width (SD m = 000 N = 0 3 N = 0 4 N = 0 5 conditiona CI coverage 83.4% 42.2% 6.7% CI width (mean CI width (SD direct bootstrap coverage 00% 00% 00% CI width (mean CI width (SD metamode-assisted coverage 97.% 94% 96.6% bootstrap CI width (mean CI width (SD From Tabes I and II, the resuts for Cases and 2 with dependence measured by either product-moment or Spearman rank correations are simiar. We observe that under the same computationa budget N, the conditiona CIs that ony account for the simuation uncertainty tend to have undercoverage. The CIs obtained by direct bootstrapping are much wider and they typicay have obvious over-coverage. The CIs obtained by metamode-assisted bootstrapping have coverage much coser to the nomina eve of 95%. As N increases and simuation estimation error decreases, the undercoverage probem for the conditiona CI becomes worse. Since direct bootstrap and metamode-assisted bootstrap use the same set of bootstrapped sampes to quantify input uncertainty, the overcoverage for the direct bootstrap represents the additiona simuation uncertainty introduced whie propagating the input uncertainty to the output mean. Tabes I and II show that the metamode can effectivey use the computationa budget and reduce the impact from simuation estimation error. Further, as the computationa budget increases, the difference between the CIs obtained by the two methods diminishes. Figure 2 shows scatter pots of conditiona CIs and CIs obtained by direct bootstrapping and metamode-assisted bootstrapping with m = 500, k = 80 and N = 0 4 when we use either Spearman-rank or product-moment correations. They incude resuts from 00 macro-repications. The horizonta axis represents (Q L + Q U /2, the center of the CI, where Q L and Q U are the ower and upper bounds of the CIs. The vertica axis is (Q U Q L /2, the haf width of the CIs. Region contains points that correspond to CIs having underestimation and Region 3 contains points corresponding to overestimation, whie Region 2 contains CIs that cover µ(f c ; see Kang and Schmeiser [990.

16 A:6 W. Xie et a. Tabe II. Resuts of nomina 95% CIs when m = 00, 500, 000 for Case 2 when the dependence is characterized by product-moment correations. m = 00 N = 0 3 N = 0 4 N = 0 5 conditiona CI coverage 44.6% 4.3% 4.7% CI width (mean CI width (SD direct bootstrap coverage 00% 00% 99.% CI width (mean CI width (SD metamode-assisted coverage 97.5% 97.5% 96.3% bootstrap CI width (mean CI width (SD m = 500 N = 0 3 N = 0 4 N = 0 5 conditiona CI coverage 74% 32.5% 9.2% CI width (mean CI width (SD direct bootstrap coverage 00% 00% 00% CI width (mean CI width (SD metamode-assisted coverage 97.9% 97.4% 96.5% bootstrap CI width (mean CI width (SD m = 000 N = 0 3 N = 0 4 N = 0 5 conditiona CI coverage 82.6% 44.8% 4.4% CI width (mean CI width (SD direct bootstrap coverage 00% 00% 00% CI width (mean CI width (SD metamode-assisted coverage 97.5% 95.6% 98.4% bootstrap CI width (mean CI width (SD The concusions obtained for both Cases and 2 are simiar. Conditiona CIs have width too short and their centers have arge variance. Therefore, they have serious undercoverage. The variance for centers of CIs comes from the impact of input uncertainty. Since metamode-assisted bootstrapping accounts for both input and simuation uncertainty, its CI width is arge enough to avoid undercoverage. The proportion of CIs in Region 2 is cose to 95% and CIs outside tend to have underestimation based on resuts from 000 macro-repications. The width of CIs obtained by the direct bootstrap is too arge; a CIs are ocated in Region 2 and they have serious overcoverage. 6. CONCLUSIONS In this paper, we extended the metamode-assisted bootstrapping framework of Xie et a. [204b to stochastic simuation with dependent input modes. The input modes are characterized by their margina distribution parameters and dependence measured either by Spearman rank or product-moment correations, which are estimated from rea-word data. The joint distributions are constructed by using the fexibe NORTA representation. Metamode-assisted bootstrapping uses the bootstrap to quantify the estimation error of these joint distributions and propagates it to the output mean by using an equation-based SK metamode. We proposed a procedure to deiver a CI quantifying the overa uncertainty of the system performance estimate. The asymptotic consistency of this interva is proved under the assumption that the true mean response surface is a reaization of a GP. An empirica study using a stochastic activity network demonstrates that for the input distribution with dependence measured by either Spearman rank or product-