Proceedings of the 2014 Winter Simulation Conference A. Tolk, S. Y. Diallo, I. O. Ryzhov, L. Yilmaz, S. Buckley, and J. A. Miller, eds.

Transcription

1 Proceedings of the 2014 Winter Simuation Conference A. Tok, S. Y. Diao, I. O. Ryzhov, L. Yimaz, S. Buckey, and J. A. Mier, eds. STATISTICAL UNCERTAINTY ANALYSIS FOR STOCHASTIC SIMULATION WITH DEPENDENT INPUT MODELS Wei Xie Rensseaer Poytechnic Institute Troy, NY , USA Barry L. Neson Northwestern University Evanston, IL 60208, USA Russe R. Barton The Pennsyvania State University University Park, PA 16802, USA ABSTRACT When we use simuation to estimate the performance of a stochastic system, ack of fideity in the random input modes can ead to poor system performance estimates. Since the components of many compex systems coud be dependent, we want to buid input modes that faithfuy capture such key properties. In this paper, we use the fexibe NORma To Anything (NORTA) representation for dependent inputs. However, to use the NORTA representation we need to estimate the margina distribution parameters and a correation matrix from rea-word data, introducing input uncertainty. To quantify this uncertainty, we empoy the bootstrap to capture the parameter estimation error and an equation-based stochastic kriging metamode to propagate the input uncertainty to the output mean. Asymptotic anaysis provides theoretica support for our approach, whie an empirica study demonstrates that it has good finite-sampe performance. 1 INTRODUCTION Stochastic simuation is used to characterize the behavior of compex systems that are driven by random input modes. The choice of input modes directy impacts the system performance estimates. It is common practice to treat the input modes as a coection of independent univariate distributions. However, these simpe input modes do not aways faithfuy represent the physica input processes. For exampe, in a manufacturing system the processing times for a singe workpiece at a series of machining stations coud be dependent due to characteristics of that particuar workpiece; and in a suppy chain system the customer demands over different products from muti-product warehouses coud be reated. Ignoring such dependence can ead to poor estimates of system performance measures (Livny et a. 1993). Thus, it is desirabe to buid input modes that faithfuy capture the dependence. Considering the amount of information needed to construct fu joint distributions, in this paper we focus on input modes characterized by margina distributions and correation matrix. The margina distributions have known parametric famiies with unknown parameter vaues. The dependence between different components of the input modes can be measured by various criteria (Bier and Ghosh 2006); we focus on the Spearman rank correation. Even though product-moment correation is widey used in engineering appications, it cannot capture compex noninear dependence and its definition needs the variances of the components to be finite. The Spearman rank correation can avoid these probems. Since the input modes are estimated from finite sampes of rea-word data, a compete statistica characterization of stochastic system performance requires quantifying both simuation and input estimation error. We buid on Xie et a. (2014a), which used a metamode-assisted bootstrapping approach to form /14/$ IEEE 674

2 Xie, Neson, and Barton a confidence interva (CI) accounting for the impact of input and simuation uncertainty on the system s mean performance estimates. However, their study assumes that the input distributions are univariate and mutuay independent. To efficienty and correcty account for the dependence between various components of input modes, we introduce a more genera metamode-assisted bootstrapping framework that can quantify the impact of dependent input modes and simuation estimation error on system performance estimates, whie simutaneousy reducing the infuence of simuation estimation error compared with direct simuation methods. Specificay, we estimate the key properties of a fexibe input mode with rea-word data. The bootstrap is then used to quantify the estimation error of input margina distributions and dependence measures. An equation-based stochastic kriging metamode propagates the input uncertainty to the output mean. From this, we can derive a CI that accounts for both simuation and input uncertainty. Notice that we are interested in the dependence between different components of the input distributions instead the dependence among the estimated input-mode parameter vaues. There are two centra contributions of this paper: First, we generaize the metamode-assisted bootstrap framework of Xie et a. (2014a) to stochastic simuations with dependent input modes. Second, we propose a rigorous anaysis for cases where the dependence is measured by Spearman rank correation. The next section reviews some previous work for input uncertainty in simuation. This is foowed by a forma description of the probem of interest in Section 3. In Section 4, we propose a generaized metamodeassisted bootstrapping framework and provide an agorithm to buid a CI accounting for both input and simuation estimation error on the system mean performance estimates. Our approach is supported with asymptotic anaysis. We then report resuts of finite-sampe behavior from an empirica study in Section 5 and concude the paper in Section 6. 2 BACKGROUND Various approaches to account for input uncertainty in stochastic simuations have been proposed, see Barton (2012) for a review. The metamode-assisted bootstrapping approach was introduced in Barton et a. (2014). In that paper the input modes are a coection of independent univariate parametric distributions with known parametric famiies and unknown parameters vaues. The input parameters are estimated by rea-word data and the bootstrap is used to quantify the input-parameter uncertainty. Based on the simuation outputs at a few design points, a fexibe stochastic kriging (SK) metamode is buit to propagate the input uncertainty to the output mean. A CI is derived to quantify the impact of input uncertainty. Compared with the direct simuation approach, the metamode can reduce the impact of simuation estimation error. However, Barton et a. (2014) assumed that the simuation budget is not tight and the metamode uncertainty can be ignored. If the true mean response surface is compex, especiay for high-dimensiona probems with many input distributions, and the computationa budget is tight, then the impact of metamode uncertainty can no onger be ignored. The metamode-assisted bootstrapping approach was improved in Xie et a. (2014a) to buid a CI accounting for the impact from both input and metamode uncertainty on the system mean estimates. Further, a variance decomposition was proposed to estimate the reative contribution of input to overa uncertainty, which is very usefu for decision makers to determine where to put more effort to reduce the system uncertainty. The metamode-assisted bootstrapping approach demonstrated robust performance even when there is a tight computationa budget and simuation estimation error is arge. However, Xie et a. (2014a) aso assumed that the input modes are mutuay independent univariate distributions. To faithfuy capture the dependence between different components of input modes, Cario and Neson (1997) proposed a fexibe NORma To Anything (NORTA) distribution to represent and generate random vectors with amost arbitrary margina distributions and correation matrix. Since NORTA representations are estimated from finite sampes of rea-word data, Bier and Coru (2011) proposed a Bayesian approach to account for parameter uncertainty. Specificay, the uncertainty about the NORTA distribution parameter estimates is quantified by posterior distributions. For compex stochastic systems with a arge number of correated inputs, a fast agorithm was proposed to draw sampes from these posterior distributions 675

3 Xie, Neson, and Barton to quantify the input uncertainty. Then, the direct simuation method was used to propagate the input uncertainty to the output mean. However, when the simuated system is compex and the computationa budget is tight, the direct simuation method can not efficienty use the computationa budget and incurs substantia simuation estimation error. The good performance of metamode-assisted bootstrapping for stochastic simuations with independent univariate input distributions in Barton et a. (2014) and Xie et a. (2014a) motivates us to extend it to more compex cases with dependence in the input modes. Therefore, in this paper, we use fexibe NORTA representations for unknown dependent inputs. Since the NORTA method is based on estimating margina distribution parameters and a correation matrix from rea-word data, metamode-assisted bootstrapping is generaized to quantify the impact of both NORTA parameters and simuation estimation error whie simutaneousy reducing the infuence of simuation estimation error due to output variabiity. 3 PROBLEM STATEMENT The stochastic simuation output is a function of random numbers and the input modes denoted by F, where F is composed of a coection of input distributions used to drive the simuation. For notation simpification, we do not expicity incude the random numbers. The output from the jth repication of a simuation with input modes F can be written as Y j (F) = µ(f) + ε j (F) where µ(f) = E[Y j (F)] denotes the unknown output mean and ε j (F) represents the simuation error with mean zero. Notice that the simuation output depends on the choice of input modes. Let F c denote the true correct input modes, which are unknown and estimated from finite sampes of rea-word data. Our goa is to quantify the impact of the statistica error on system mean performance estimates by finding a (1 α)100% CI [Q L,Q U ] such that Pr{µ(F c ) [Q L,Q U ]} = 1 α. Let F = {F 1,F 2,...,F L } with F 1,F 2,...,F L mutuay independent; F coud be composed of univariate and mutivariate joint distributions. In this paper, we do not consider time-series input processes. Let F be a d dimensiona distribution having margina distributions denoted by {F,1,F,2,...,F,d } with d 1. For the th distribution F with d > 1, et X F be a d 1 random vector having a d d Spearman rank correation matrix denoted by R X with eements R X (i, j) = cor(f,i (X,i ),F, j (X, j )) = E[F,i(X,i )F, j (X, j )] E[F,i (X,i )]E[F, j (X, j )] Var(F,i (X,i ))Var(F, j (X, j )) with i, j = 1,2,...,d. Since the correation matrix is symmetric and diagona terms are 1, we can view a d d correation matrix as an eement of a d d (d 1)/2 dimensiona vector space. Then, the Spearman rank correation matrix can be uniquey specified by a d 1 vector denoted by V X. For F with d = 1, V X is empty and d = 0. For F with = 1,2,...,L, we assume that the famiies of margina distributions {F,1,F,2,...,F,d } are known, but not their parameter vaues. Let an h,i 1 vector θ,i denote the unknown parameters for the ith margina distribution F,i. By stacking θ,i with i = 1,2,...,d together, we have a d 1 dimensiona parameter vector θ (θ,1,θ,2,...,θ,d ) with d d i=1 h,i. Suppose the input mode is characterized by margina distributions and correation matrix, which are specified by ϑ {(θ ;V X ), = 1,2,...,L} that incudes d L =1 d + L =1 d eements. We ca ϑ the input mode parameters. By abusing notation, we can rewrite µ(f) as µ(ϑ). The true input parameters ϑ c are unknown and estimated from finite sampes of rea-word data. Thus, our goa can be restated as finding a (1 α)100% CI [Q L,Q U ] such that Pr{µ(ϑ c ) [Q L,Q U ]} = 1 α. (1) 676

4 Xie, Neson, and Barton { Let m denote the } number of i.i.d. rea-word observations avaiabe from the th input process X,m X (1),X (2),...,X (m ) with d 1 random vector X (i) i.i.d F c for i = 1,2,...,m. Let X m = {X,m, = 1,2,...,L} be the coection of sampes from a L input distributions in F c, where m = (m 1,m 2,...,m L ). The estimators for the input mode parameters are denoted by ϑ m {( θ,m ; V X,m ), = 1,2,...,L} and are a function of X m. The rea-word data are a particuar reaization of X m, say x (0) m. Input uncertainty is quantified by the samping distribution of µ( ϑ m ). Further, since the underying response surface µ( ) is unknown, for any ϑ et µ(ϑ) denote the corresponding mean response estimator. Thus, there are both input and simuation estimation errors in the system mean performance estimates. This paper focuses on estimating the performance of compex stochastic systems with input modes incuding mutivariate distributions. Our objective is to propose an approach to quantify the overa impact of both input and simuation estimation error on system mean performance estimates and then buid a CI satisfying Equation (1). Further, since each simuation run coud be computationay expensive and we have computationa budget denoted by N, we want to reduce the infuence of simuation estimation error due to output variabiity. 4 METAMODEL-ASSISTED BOOTSTRAPPING FRAMEWORK For simuations with parametric input distributions that are univariate and mutuay independent, a metamode-assisted bootstrapping framework was used to account for the impact of both input and simuation estimation errors in Xie et a. (2014a). In this section, we generaize the metamode-assisted bootstrapping approach to aow NORTA input modes. A procedure to buid a CI for the system mean performance that accounts for overa uncertainty is described and supported by asymptotic anaysis. 4.1 Bootstrap for Input Uncertainty The way we choose to represent input modes pays an important roe in the impementation of metamodeassisted bootstrapping. For the th input distribution F, instead of using the natura parameters θ to characterize the margina distributions, we can use moments; see Barton et a. (2014) for an expanation. Suppose that the parametric margina distribution F,i can be uniquey characterized by its first h,i finite moments denoted by the h,i 1 vector ψ,i for i = 1,2,...,d. By stacking ψ,i with i = 1,2,...,d together, we have a d 1 dimensiona vector of margina moments ψ (ψ,1,ψ,2,...,ψ,d ). Therefore, the input modes can be characterized by the moments M {(ψ ;V X ), = 1,2,...,L}. Abusing notation, we rewrite µ(ϑ) as µ(m ). The true moments of dependent input modes, denoted by M c, are unknown and estimated using X m. We use moment estimators for margina distributions, denoted by ψ,m. The estimation error of input modes can be quantified by the samping distribution of Mm {( ψ,m ; V X,m ), = 1,2,...,L}, denoted by FM c m. Therefore, the input uncertainty can be measured by the samping distribution of µ( M m ) with M m FM c m. Since it is hard to derive the samping distribution FM c m, we use bootstrap resamping to approximate it (Shao and Tu 1995). Since F 1,F 2,...,F L are mutuay independent, we can do bootstrapping for each distribution separatey. For distribution F, et A {1,2,...,m } be the index set of the rea-word observations. The procedure to quantify the input uncertainty by the bootstrap is as foows: 1. For the th distribution F with = 1,2,...,L, draw m sampes with repacement from set A and obtain indexes {i 1,i 2,...,i m }; choose corresponding sampes from rea-word data x (0) m and get x (1),m {x (i 1),x (i 2),...,x (i m ) }. Denote the coection of bootstrap sampes by x (1) m = 677

5 { x (1) Xie, Neson, and Barton,m, = 1,2,...,L} and use it to cacuate the bootstrapped moment estimates, denoted by { } M m (1) ( ψ (1),m ;Ṽ (1) X,m ), = 1,2,...,L. 2. Repeat the previous step B times to generate M m (b) for b = 1,2,...,B. The bootstrap resamped moments are drawn from the bootstrap distribution denoted by F Mm ( x (0) m ) with M m (b) F Mm ( x (0) m ). For estimation of a CI, B is recommended to be a few thousand. In this paper, a denotes a quantity estimated from rea-word data, whie a denotes a quantity estimated from bootstrapped data. 4.2 NORTA Representation In this paper, mutivariate input modes are characterized by their margina distributions and correation matrix. Since this partia characterization does not uniquey determine the joint distributions except in specia cases, we use the NORTA representation introduced by Cario and Neson (1997); NORTA vectors are competey specified by their margina distributions and correation matrix. In this section we describe how to appy NORTA in our metamode-assisted bootstrapping approach. Since F = {F 1,F 2,...,F L } with F 1,F 2,...,F L mutuay independent, we ony need to appy NORTA separatey for the distributions F with d > 1. Specificay, we represent X as a transformation of a d dimensiona standard mutivariate norma (MVN) vector Z = (Z,1,Z,2,...,Z,d ) with product-moment correation matrix ρ Z, ( ) X = F,1 1 [Φ(Z,1);θ,1 ],F,2 1 [Φ(Z,2);θ,2 ],...,F,d 1 [Φ(Z,d );θ,d ]. (2) If the margina distribution famiies are given, as we assume here, then the NORTA representation for F is (θ,ρ Z ). Let R Z denote a d d Spearman rank correation matrix for Z. Notice that there is a cosed form reationship between product-moment and Spearman rank correations for the standard mutivariate norma distribution (Cemen and Reiy 1999): For any i, j = 1,2,...,d, we have R Z (i, j) = 6 π sin 1 ( ρz (i, j) 2 ). (3) In this paper, we focus on continuous margina distributions with stricty increasing cdfs {F,1,F,2,...,F,d } for = 1,2,...,L. Since the Spearman rank correation is invariant under the monotone one-to-one transformation F,i 1 [Φ( )], we have R X = R Z. Given ϑ = {(θ ;V X ), = 1,2,...,L}, the procedure to generate the NORTA random variates is as foows: 1. From V X, get the Spearman rank correation matrix R Z = R X. 2. Cacuate ρ Z (i, j) = 2sin(πR Z (i, j)/6) for i, j = 1,2,...,d. 3. i.i.d. Generate Z MVN(0,ρ Z ) and obtain X by using Equation (2). 4. Repeat Steps 1 3 for a F with d > 1. Notice that when we use Spearman rank correation to measure the input mode dependence, the choice of margina distributions and correation is separabe. Thus, for any feasibe ϑ, we can find the corresponding NORTA representations. 4.3 Stochastic Kriging Metamode Instead of running simuations for systems with input modes represented by bootstrap moment sampes { M m (1), M (2) m,..., M (B) m }, we can choose a sma number of design points and run simuations there to buid a metamode; see Xie et a. (2014a). We then propagate the input uncertainty to the output mean 678

6 Xie, Neson, and Barton by using the metamode. In this paper, we use a fexibe stochastic kriging (SK) metamode introduced by Ankenman et a. (2010). We briefy review it in this section. For more detaied information, see Ankenman et a. (2010). Suppose that the underying true (but unknown) response surface is a reaization of a stationary Gaussian Process (GP). We mode the simuation output Y by Y j (x) = β 0 +W(x) + ε j (x). (4) This mode incudes two sources of uncertainty: the simuation output uncertainty ε j (x) and the mean response uncertainty W(x). Since stochastic systems with dependent input modes having simiar key properties tend to have mean responses cose to each other, SK uses a mean-zero, second-order stationary GP W( ) to account for this spatia dependence of the response surface. The uncertainty about the unknown true response surface µ(x) is represented by a GP M(x) β 0 +W(x) (note that β 0 can be repaced by a more genera trend term f(x) β). For many, but not a, simuation settings the output is an average of a arge number of more basic outputs, so a norma approximation can be appied: ε(x) N(0,σε 2 (x)). In SK, the covariance between W(x) and W(x ) quantifies how knowedge of the surface at one ocation affects the prediction at another ocation. In this paper, a parametric form of the covariance is used to capture this spatia dependence, Σ(x,x ) = Cov[W(x),W(x )] = τ 2 r(x x ), where τ 2 denotes the variance and r( ) is a correation function that depends ony on the distance x x. Using prior information about the smoothness of µ( ), we can choose the form of correation function. Based on previous study (Xie et a. 2010), we use the product-form Gaussian correation function ( r(x x d ) ) = exp φ j (x j x j) 2 (5) j=1 for the empirica evauation in Section 5. Let φ = (φ 1,φ 2,...,φ d ) represent the correation function parameters. So in summary, before having any simuation resuts the uncertainty about µ(x) is represented by a Gaussian process M(x) GP(β 0,τ 2 r(x x )). To reduce the uncertainty about µ(x) we choose an experiment design consisting of pairs D {(x i,n i ),i = 1,2,...,k} at which to run simuations, where (x i,n i ) denotes the ocation and the number of repications, respectivey, at the ith design point. The simuation outputs at D are Y D {(Y 1 (x i ),Y 2 (x i ),...,Y ni (x i ));i = 1,2,...,k} and the sampe mean at design point x i is Ȳ (x i ) = n i j=1 Y j(x i )/n i. Let the sampe means at a k design points be Ȳ D = (Ȳ (x 1 ),Ȳ (x 2 ),...,Ȳ (x k )) T. Since the use of common random numbers is detrimenta to prediction (Chen et a. 2012), the simuations at different design points are independent and the variance of Ȳ D is represented by a k k diagona matrix C = diag { σε 2 (x 1 )/n 1,σε 2 (x 2 )/n 2,...,σε 2 } (x k )/n k. Let Σ be the k k spatia covariance matrix of the design points and et Σ(x, ) be the k 1 spatia covariance vector between each design point and a fixed prediction point x. If the parameters (τ 2,φ,C) are known, then the metamode uncertainty can be characterized by a refined GP M p (x) that denotes the conditiona distribution of M(x) given a simuation outputs, M p (x) GP(m p (x),σ 2 p(x)) (6) where m p ( ) is the minimum mean squared error (MSE) inear unbiased predictor and the corresponding variance is m p (x) = β 0 + Σ(x, ) (Σ +C) 1 (Ȳ D β 0 1 k 1 ), (7) σ 2 p(x) = τ 2 Σ(x, ) (Σ +C) 1 Σ(x, ) + η [1 k 1(Σ +C) 1 1 k 1 ] 1 η (8) 679

7 Xie, Neson, and Barton where β 0 = [1 k 1 (Σ +C) 1 1 k 1 ] 1 1 k 1 (Σ +C) 1 Ȳ D and η = 1 1 k 1 (Σ +C) 1 Σ(x, ) (Ankenman et a. 2010). Since in reaity the spatia correation parameters τ 2 and φ are unknown, maximum ikeihood estimates are typicay used for prediction, and the sampe variance is used as an estimate for the simuation variance at design points C (Ankenman et a. 2010). By pugging these into Equations (7) and (8) we can obtain the estimated mean m p (x) and variance σ 2 p(x). Thus, the metamode we use is µ(x) = m p (x) with variance estimated by σ 2 p(x). Notice that since accounting for the parameter estimation error is intractabe, this pug-in estimator is commony used in the kriging iterature. In the metamode-assisted bootstrapping approach, the dependent input mode characterized by moments M can be interpreted as a ocation x in a d dimensiona space. 4.4 Procedure to Buid CI Since there are both input and simuation estimation errors in the system mean performance estimates, in this section, we propose a procedure to buid a CI quantifying the overa uncertainty by using our generaized metamode-assisted bootstrapping approach. Asymptotic anaysis shows that as m, B, the CI is consistent. Based on a hierarchica approach, we propose the foowing procedure to buid a (1 α)100% bootstrap percentie CI: 1. Identify a design space E for input mode moments M over which to fit the metamode. Since the metamode is used to propagate the input uncertainty measured by the bootstrapped moments Mm to the output mean, the design space is chosen to be the smaest eipsoid covering most ikey bootstrapped moments. See Barton et a. (2014) for more detaied information. 2. To obtain an experiment design D = {(M i,n i ),i = 1,2,...,k}, use a Latin hypercube sampe to embed k design points into the design space E and assign equa repications to these points to exhaust N. 3. For i = 1 to k (oop through k design points) (a) Use moment matching to cacuate the margina parameters θ (i) for = 1,2,...,L. (b) By foowing the description in Section 4.2, find the NORTA representation with parameters ( θ (i) ),ρ(i) Z for F with = 1,2,...,L and d > 1. Next i 4. At a design points, generate sampes of X by using NORTA representations for F with d > 1 and using standard approaches (Neson 2013) for F with d = 1, = 1,2,...,L. Use these sampes to drive simuations (see Section 4.2) and obtain outputs y D. Compute the sampe average ȳ(m i ) and sampe variance s 2 (M i ) of the simuation outputs, i = 1,2,...,k. Fit a SK metamode to obtain m p ( ) and σ p( ) 2 using ( ȳ(m i ),s 2 ) (M i ),M i, i = 1,2,...,k. 5. For b = 1 to B (a) Generate bootstrap moments M (b) ( m by foowing the procedure in Section 4.1. (b) Draw M b N m p ( M m (b) ), σ p( 2 M ) m (b) ). Next b [ ] 6. Report CI: M ( B α 2 ), M ( B(1 α 2 ) ) where, M (1) M (2) M (B) are the sorted vaues. If SK parameters (τ 2,φ,C) are known and we repace M b in Step 5(b) of the CI procedure with M b N ( m p ( M m (b) ),σp( 2 M m (b) ) ), then we can show that the CI obtained [M ( B α 2 ),M ( B(1 α 2 ) )] is asymptoticay consistent by Theorem 1. Further, this CI characterizes the impact from both input and metamode uncertainty on the system performance estimate. Theorem 1 Suppose that the foowing assumptions hod. 680

8 Xie, Neson, and Barton Figure 1: A stochastic activity network. 1. We have i.i.d. observations X ( j) i.i.d F c for j = 1,2,...,m and = 1,2,...,L. 2. The ε j (x) i.i.d. N(0,σε 2 (x)) for any x, and M(x) is a stationary, separabe GP with a continuous correation function satisfying 1 r(x x c ) og( x x 2 ) 1+γ for a x x 2 δ for some c > 0, γ > 0 and δ < 1, where x x 2 = d j=1 (x j x j )2. 3. The input processes, simuation noise ε j (x) and GP M(x) are mutuay independent and the bootstrap process is independent of a of them. As m, we have m /m c with = 1,2,...,L for a constant c > 0. Then the interva [M ( B α 2 ),M ( B(1 α 2 ) )] is asymptoticay consistent, meaning the iterated imit im m im B Pr{M ( Bα/2 ) M p (M c ) M ( B(1 α/2) ) } = 1 α. (9) Proof: By Theorem 1 in Xie et a. (2014b), we have M a.s. m M c as m. Then, foowing simiar steps as in proving Theorem 1 in Xie et a. (2014a), we can show Equation (9). 5 EMPIRICAL STUDY Since F 1,F 2,...,F L are mutuay independent, without oss of generaity we consider an exampe with L = 1 and suppress the subscript for the th input distribution. We use a stochastic activity network as shown in Figure 1 to examine the finite-sampe performance of our metamode-assisted bootstrapping approach. Suppose that the time required to compete task (arc) i is denoted by X i for i = 1,2,...,5 and X = (X 1,X 2,...,X 5 ). We wish to compute the time to compete the project, which is the ongest path through the network, Y = max{x 1 + X 2 + X 5,X 1 + X 4,X 3 + X 5 }. We are interested in the mean response E[Y ]. We assume that F c is NORTA. The margina distributions are X i exp(θi c ) for i = 1,2,...,5 with means θ c = (10,5,12,11,5). The Spearman rank correation matrix is R c X =

9 Xie, Neson, and Barton Therefore, the number of parameters characterizing the input mode F is d = d +d = 5+(5 4)/2 = 15. Since the true mean response µ(ϑ c ) is unknown, we run 10 7 repications and obtain the estimated true mean response with standard error To evauate our metamode-assisted bootstrapping approach, we pretend that the input-mode parameters (θ c,r c X ) are unknown and they are estimated by m i.i.d. observations from Fc ; this represents obtaining rea-word data. The goa is to buid a CI quantifying the impact of both input and simuation estimation error on the system mean response estimate. We compare metamode-assisted bootstrapping to the conditiona CI and direct bootstrapping. For the conditiona CI, we fit the input distribution to the rea-word data by moment matching and aocate a computationa budget N repications to simuating the resuting system. In direct bootstrapping, we run N/B repications of the simuation at each bootstrap moment Ȳ b = Ȳ ( M (b) ), and report the percentie CI m [Ȳ( B α 2 ),Ȳ ( B(1 α 2 ) ) ] m. M (b), record the average simuation output Tabe 1 shows the statistica performance of conditiona and direct bootstrapping CIs and metamodeassisted bootstrapping with k = 80 design points, m = 100, 500, 1000 rea-word observations, and computationa budget of N = 10 3, 10 4, and 10 5 repications. We ran 1000 macro-repications of the entire experiment. In each macro-repication, we first generate m mutivariate observations by using NORTA with parameters (θ c,r c X ). Then, for the conditiona CI, we run N repications at the estimated parameters ( θ m, R X,m ) and buid CIs with nomina 95% coverage of the response mean. For direct bootstrapping and metamode-assisted bootstrapping, we use bootstrapping to generate B = 1000 sampes moments to quantify the input uncertainty. Since µ( ) is unknown, we use the fixed computationa budget N to propagate the input uncertainty either via direct simuation or the SK metamode to buid percentie CIs with nomina 95% coverage. From Tabe 1 we observe that under the same computationa budget N, the conditiona CIs that ony account for the simuation uncertainty tend to have undercoverage. The CIs obtained by direct simuation are much wider and they typicay have obvious over-coverage. The CIs obtained by metamode-assisted bootstrapping have coverage much coser to the nomina eve of 95%. As N increases and simuation estimation error decreases, the undercoverage probem for the conditiona CI becomes worse. Since direct bootstrapping and metamode-assisted bootstrapping use the same set of bootstrapped sampes to quantify input uncertainty, the overcoverage for the direct bootstrap represents the additiona uncertainty introduced whie propagating the input uncertainty to the output mean. Tabe 1 shows that the metamode can effectivey use the computationa budget and reduce the impact from simuation estimation error. Further, as the computationa budget increases, the difference between the CIs obtained by the two methods diminishes. Figure 2 shows a scatter pot of conditiona CIs and CIs obtained by direct simuation and metamodeassisted bootstrapping when m = 500, k = 80 and N = It incudes resuts from 100 macro-repications. The horizonta axis represents (Q L +Q U )/2 that gives a point estimate of system mean performance, where Q L and Q U are the ower and upper bounds of the CIs. The vertica axis is (Q U Q L )/2 that gives haf width of the CIs. Region 1 contains points that correspond to CIs having underestimation and Region 3 contains points corresponding to overestimation, whie Region 2 contains CIs that cover µ(f c ) (Kang and Schmeiser 1990). Conditiona CIs have short width and their centers have arge variance. Therefore, they have serious undercoverage. For resuts from metamode-assisted bootstrapping, the proportion of CIs in Region 2 is cose to 95% and CIs outside tend to have underestimation. 6 CONCLUSIONS In this paper, we extended the metamode-assisted bootstrapping approach to cases with dependence in the input modes. The input modes are characterized by their margina distributions and Spearman rank correation, which are estimated from rea-word data. Metamode-assisted bootstrapping uses the bootstrap to quantify the input uncertainty and propagates it to the output mean by using a SK metamode. This 682

10 Xie, Neson, and Barton Tabe 1: Resuts for CIs when m = 100,500,1000. m = 100 N = 10 3 N = 10 4 N = 10 5 conditiona CI coverage 43% 14.1% 4.4% CI width (mean) CI width (SD) direct simuation coverage 100% 100% 99.4% CI width (mean) CI width (SD) metamode-assisted coverage 97.6% 96.1% 94.8% CI width (mean) CI width (SD) m = 500 N = 10 3 N = 10 4 N = 10 5 conditiona CI coverage 76.9% 32.1% 12% CI width (mean) CI width (SD) direct simuation coverage 100% 100% 100% CI width (mean) CI width (SD) metamode-assisted coverage 96.3% 98.7% 97.2% CI width (mean) CI width (SD) m = 1000 N = 10 3 N = 10 4 N = 10 5 conditiona CI coverage 83.4% 42.2% 16.7% CI width (mean) CI width (SD) direct simuation coverage 100% 100% 100% CI width (mean) CI width (SD) metamode-assisted coverage 97.1% 94% 96.6% CI width (mean) CI width (SD)

11 Xie, Neson, and Barton Figure 2: Scatter pot of conditiona CIs and CIs obtained by direct simuation and metamode-assisted bootstrapping when m = 500, k = 80 and N = approach deivers a CI quantifying the overa uncertainty of the system performance estimate. Compared with the direct bootstrap, the metamode can make effective use of the simuation budget. An empirica study demonstrated that our metamode-assisted bootstrap has good finite-sampe performance under various quantities of rea-word data and simuation budget. ACKNOWLEDGMENTS This paper is based upon work supported by the Nationa Science Foundation under Grant No. CMMI REFERENCES Ankenman, B. E., B. L. Neson, and J. Staum Stochastic Kriging for Simuation Metamodeing. Operations Research 58: Barton, R. R Tutoria: Input Uncertainty in Output Anaysis. In Proceedings of the 2012 Winter Simuation Conference, edited by C. Laroque, J. Himmespach, R. Pasupathy, O. Rose, and A.M. Uhrmacher, 67 78: IEEE. Barton, R. R., B. L. Neson, and W. Xie Quantifying Input Uncertainty via Simuation Confidence Intervas. Informs Journa on Computing 26: Bier, B., and C. G. Coru Accounting for Parameter Uncertainty in Large-Scae Stochastic Simuations with Correated Inputs. Operations Research 59: Bier, B., and S. Ghosh Mutivariate Input Processes. In Handbooks in Operations Research and Management Science: Simuation, edited by S. Henderson and B. L. Neson, Chapter 5. Esevier. 684

12 Xie, Neson, and Barton Cario, M. C., and B. L. Neson Modeing and Generating Random Vectors with Arbitrary Margina Distributions and Correation Matrix. Technica report, Department of Industria Engineering and Management Sciences, Northwestern University. Chen, X., B. E. Ankenman, and B. L. Neson The Effect of Common Random Numbers on Stochastic Kriging Metamodes. ACM Transactions on Modeing and Computer Simuation 22:7. Cemen, R. T., and T. Reiy Correations and Copuas for Decision and Risk Anaysis. Management Science 45: Kang, K., and B. Schmeiser Graphica Methods for Evauation and Comparing Confidence-Interva Procedures. Operations Research 38 (3): Livny, M., B. Meamed, and A. K. Tsiois The Impact of Autocorreation on Queueing Systems. Management Science 39: Neson, B. L Foundations and Methods of Stochastic Simuation: A First Course. Springer-Verag. Shao, J., and D. Tu The Jackknife and Bootstrap. Springer-Verag. Xie, W., B. L. Neson, and R. R. Barton. 2014a. Statistica Uncertainty Anaysis for Stochastic Simuation. Technica report, Department of Industria Engineering and Management Sciences, Northwestern University. Xie, W., B. L. Neson, and R. R. Barton. 2014b. Statistica Uncertainty Anaysis for Stochastic Simuation with Dependent Input Modes. Technica report, Department of Industria Engineering and Management Sciences, Northwestern University. Xie, W., B. L. Neson, and J. Staum The Infuence of Correation Functions on Stochastic Kriging Metamodes. In Proceedings of the 2010 Winter Simuation Conference, edited by B. Johansson, S. Jain, J. Montoya-Torres, J. Hugan, and E. Yucesan, : IEEE. AUTHOR BIOGRAPHIES WEI XIE is an assistant professor in the Department of Industria and Systems Engineering at Rensseaer Poytechnic Institute. Her research interests are in computer simuation, appied statistics and data anaytics. Her emai address is weixie2013@u.northwestern.edu. BARRY L. NELSON is the Water P. Murphy Professor and Chair of the Department of Industria Engineering and Management Sciences at Northwestern University. He is a Feow of INFORMS and IIE. His research centers on the design and anaysis of computer simuation experiments on modes of stochastic systems, and he is the author of Foundations and Methods of Stochastic Simuation: A First Course, from Springer. His e-mai address is nesonb@northwestern.edu. RUSSELL R. BARTON is Professor of Suppy Chain and Information Systems and Professor of Industria Engineering at the Pennsyvania State University. He currenty serves as senior associate dean for research and facuty in the Smea Coege of Business. He received a B.S. degree in Eectrica Engineering from Princeton University and M.S. and Ph.D. degrees in Operations Research from Corne University. Before entering academia, he spent tweve years in industry. He is a past president of the INFORMS Simuation Society, and currenty serves as the I-Sim representative on the INFORMS Subdivisions Counci and chairs the QSR Advisory Board. He is a senior member of IIE and IEEE and a Certified Anaytics Professiona. His research interests incude appications of statistica and simuation methods to system design and to product design, manufacturing and deivery. His emai address is rbarton@psu.edu. 685