Proceedings of the 2016 Winter Simulation Conference T. M. K. Roeder, P. I. Frazier, R. Szechtman, E. Zhou, T. Huschka, and S. E. Chick, eds.

Transcription

1 Proceedings of the 2016 Winter Siulation Conference T. M. K. Roeder, P. I. Frazier, R. Szechtan, E. Zhou, T. Huschka, and S. E. Chick, eds. THE EMPIRICAL LIKELIHOOD APPROACH TO SIMULATION INPUT UNCERTAINTY Henry La Departent of Industrial and Operations Engineering University of Michigan 1205 Beal Ave. Ann Arbor, MI 48109, USA Huajie Qian Departent of Matheatics University of Michigan 2074 East Hall 530 Church Street Ann Arbor, MI 48109, USA ABSTRACT We study the epirical likelihood ethod in constructing statistically accurate confidence bounds for stochastic siulation under nonparaetric input uncertainty. The approach is based on positing a pair of distributionally robust optiization, with a suitably averaged divergence constraint over the uncertainput distributions, and calibrated with a χ 2 -quantile to provide asyptotic coverage guarantees. We present the theory giving rise to the constraint and the calibration. We also analyze the perforance of our stochastic optiization algorith. We nuerically copare our approach with existing standard ethods such as the bootstrap. 1 INTRODUCTION Stochastic siulation often relies onput distributions that are not fully known but only observed via a finite saple of input data. The quantification of the errors in perforance analyses due to the isspecification of input odels is known as input uncertainty. This paper focuses particularly on situations where there are no specific assuptions on the paraetric for of the input odels, or nonparaetric input uncertainty. One ajor goal in this context is to construct confidence interval (CI) for the target perforance easure that accounts for input odel errors on top of siulation errors. This paper proposes an optiization-based ethod to construct such a CI. The upper and lower bounds of the CI are obtained as the optial values of a pair of optiization progras posited over the space of input probability distributions. These progras can be interpreted as finding the upper and lower worst-case perforance easures, subject to a constraint on the unknownput distributions within a neighborhood of the epirical distributions. We show that, by choosing the right size of this neighborhood and the appropriate nonparaetric distance to easure its size, these worst-case optiizations lead to asyptotically exact confidence bounds for covering the true perforance easure. We develop these statistical guarantees by using the epirical likelihood (EL) ethod, which can be viewed as a nonparaetric analog of the axiu likelihood theory. Our approach resebles, but also should be contrasted with, the literature of distributionally robust optiization (DRO) (e.g., Delage and Ye 2010, Ben-Tal et al. 2013, Goh and Si 2010). This literature ais to find worst-case stochastic perforance easures (or optiizes decisions over the worst-case) subject to so-called uncertainty sets that represent the inforation or uncertainty about an underlying probability distribution. One coon constraint is a nonparaetric neighborhood ball around a baseline odel, typically corresponding to the odeler s best guess of the truth, easured by soe nonparaetric distance such as φ-divergence (Pardo 2005). When this ball contains the true distribution, the optial values of the worst-case optiizations will cover the true easure. Thus, it is argued that one should calibrate the ball size by estiating the divergence between the data and the baseline distribution, and consequently translate the confidence of this estiationto a confidence bound for the perforance easure. In practice, this

2 iplies calibration using a goodness-of-fit χ 2 -quantile, with the degree of freedo corresponding to the effective nuber of categories in a discrete distribution, or divergence estiation that involves estiating a functional of density. These ethods could runto the statistical challenges in divergence estiation, and also potentially a loss of coverage accuracy in translating the uncertainty set into a CI. One ain contribution of this paper is thus a new way of interpreting these DROs via the EL ethod. Through this connection we locate the precise nonparaetric distance one should use in constructing valid CIs under nonparaetric input uncertainty. Our distance deviates fro those in the literature as it consists of a weighted average of a collection of divergences, each applied on andependent input odel needed in the siulation. We also show that the size of the nonparaetric neighborhood can be taken as a χ 2 -quantile, with degree of freedo exactly one, independent of the nuber of input odels and the continuity of these rando variates. This is in sharp contrast with the proposed calibration ethods in the DRO literature. In general, the optiization progras we ipose is siulation-based. As another contribution, we investigate the properties of irror descent stochastic approxiation (MDSA) (Neirovski et al. 2009) used to locally solve these progras. This siulation-based optiization fraework provides an alternate approach to quantifying nonparaetric input uncertainty copared to the current ajor technique of bootstrap resapling. The latter is a sapling approach that repeatedly generates new epirical distributions to drive the siulation runs. On a high level, our approach trades the coputational load in resapling with the optiization routine needed in the iteration of the SA algorith. We investigate this tradeoff, and copare the perforances of the obtained CIs and their vulnerability to the siulation noise. The rest of this paper is organized as follows. Section 2 lays out our optiization progras and explains the using the EL ethod. Section 3 presents and analyzes our MDSA algorith. Section 4 show soe nuerical results and coparison with the bootstrap. 2 EMPIRICAL-LIKELIHOOD-BASED CONFIDENCE INTERVAL We consider a perforance easure in the for Z(P 1,...,P ) = E P1,...,P [h(x 1,...,X )], (1) where each X i = (X i (1),...,X i (T i )) is a sequence of T i i.i.d. rando variables under andependent input odel (or distribution) P i, and T i is a deterinistic run length. The function h apping fro X T 1 1 XT to R is assued coputable given the inputs X i s. Our preise is that each P i is unknown but i.i.d. data X i,1,...,x i,ni are available. The true value of (1) is therefore unknown even under abundant siulation runs. Our goal is to find a (1 α) level CI for the true perforance easure. Our approach is the following. For odel i, we consider the probability siplex of the weights w i = (w i,1,...,w i,ni ) over the support set {X i,1,...,x i,ni }. We consider the pair of optiization progras L α /U α :=in/ax Z(w 1,...,w ) subject to 2 log w i, j X 2 1,1 α w i, j = 1, for all i w i, j 0, for all i, j (2) where X 2 1,1 α is the 1 α quantile of the chi-square distribution with degree of freedo one, and each w i in Z(w 1,...,w ) should be interpreted as the distribution putting weight w i, j on X i, j, for j = 1,...,. The forulation (2) can be interpreted as the worst-case optiizations over the independent input distributions subject to a weighted-averaged divergence. Note that the quantity (1/ ) logw i, j is

3 the Burg-entropy divergence (Ben-Tal et al. 2013) between the probability weights w i and the unifor weights. Thus, letting n = be the total saple size, we have ( ) 1 n log w i, j = 1 n log w i, j which can be viewed as a weighted average of the individual Burg-entropy divergences iposed on different input odels, each weight being /n. The first constraint in (2) iposes a bound X1,1 α 2 /(2n) on this averaged divergence. 2.1 Epirical Likelihood Theory for Su of Means We justify the forulation (2) using the EL ethod. First proposed by Owen (1988), this ethod can be viewed as a nonparaetric counterpart of axiu likelihood theory. Analogous to Wilks Theore (Cox and Hinkley 1979) in axiu likelihood theory that states the convergence of the so-called logarithic likelihood ratio to a χ 2 -distribution, the nonparaetric profile likelihood ratio in EL converges siilarly. This will be the key to obtaining our forulation (2). We describe the EL ethod. Given independent sets of data {X i,1,...,x i,ni }, we define the nonparaetric likelihood, in ters of the probability weights w i for the i-th input odel, to be w i, j. The ulti-saple likelihood is w i, j. It can be shown, by a siple convexity arguent, that unifor weights w i, j = 1/ for each odel axiize (1/). Moreover, unifor weights still axiize evef one allows putting weights outside the support of data, in which case w i, j < 1 for soe i, aking w i, j even saller. Therefore, (1/) can be viewed as the nonparaetric axiu likelihood estiate. To proceed, we need to define a paraeter of interest that is deterined by the input odels. Inference about this paraeter is carried out based on the so-called profile nonparaetric likelihood ratio, which is the axiu likelihood ratio between all weights that epirically produce a given value of the paraeter, and the unifor weights (i.e. the MLE). In our case the paraeter of interest is the perforance easure Z(P 1,...,P ), which is however possibly nonlinear in P i s and hence not easy to work with. Fortunately, it turns out that there isn t uch loss to deal with the special case h(x 1,...,X ) = h i(x i (1)) for soe h i : R R and T i = 1 for all i, i.e. the paraeter of interest is siply the su of eans of rando variables h i (X i (1)). We will focus on this case for now. To siplify further, let us consider estiating EX i, where for convenience we write X i in place of X i (1). The profile nonparaetric likelihood ratio is defined as R(µ) = ax { w i, j w i, j X i, j = µ, w i, j = 1,w i, j 0, for all i, j }, (3) and is defined to be 0 if the optiization proble in (3) is infeasible. Before getting into details of the asyptotic theory, we note that usually the logarithic profile nonparaetric likelihood ratio at the true value, i.e. 2logR( EX i(1)), asyptotically follows soe chisquare distribution whose degree of freedo is equal to the nuber of non-probability-siplex constraints, e.g. w i, jx i, j = µ in (3). As another interpretation, treating the weights w i as paraeters, the degree of freedo is the difference in diensions of the full and constrained paraeter space, and typically d constraints result in a loss of d diensions of the paraeter space. Now we state our theore. Theore 1 Let X i be a rando variable distributed under P i. Assue 0 < Var(X i) <, and i I c i I always holds for soe constant c > 0, where I = {i Var(X i ) > 0}. Then 2logR( EX i) converges in distribution to X1 2, the chi-squared distribution with degree of freedo one, as for i I. In this theore only saple sizes of those having positive variance are required to grow to infinity. To see the reason for this, note that data of inputs with zero variance are always equal to the true ean so we

4 can always assign the the unifor weights in (3). Other than these zero-variance inputs, the condition i I c i I basically forces the saple sizes to grow at the sae rate. Theore 1 is a ulti-saple generalization of the well known epirical likelihood theore for single-saple ean, which can be found in Chapter 2 of Owen (2001). We state it as a special case where = 1. Theore 2 Let Y 1,Y 2,...,Y n be i.i.d. rando variables distributed under soe distribution P, 0 < Var(Y 1 ) <. Then 2logR(EY 1 ) converges in distribution to X1 2, as n. The function R( ) here is the sae as that in (3) with = 1,n 1 = n,x 1, j = Y j. 2.2 Epirical-likelihood-based Confidence Interval We discuss how to construct confidence interval for the quantity Z(P 1,...,P ) based on the EL theory presented in the last section, and thus justify the validity of the forulation (2). We will first study the linear output case, and then discuss the general perforance easure in (1). As pointed out in the last section, linear output takes the for h(x 1,...,X ) = h i(x i ), where X i is distributed under P i. For output of this particular for, Theore 1 iplies that the optiization pair (2) gives an asyptotically correct confidence interval with coverage probability 1 α, naely Theore 3 Assue 0 < Var(h i(x i )) <, and i I c i I for soe constant c > 0, where I = {i Var(h i (X i )) > 0}. Then ( where P L α Eh i (X i ) U α ) 1 α, as for i I, L α /U α :=in/ax and X 2 1,1 α is the 1 α quantile of X 2 1. subject to 2 w i, j h i (X i, j ) log w i, j X 2 1,1 α w i, j = 1, for all i w i, j 0, for all i, j Proof. By applying Theore 1 to h i (X i, j ), we know that the set {µ R 2logR(µ) X 2 1,1 α } contains the true su Eh i(x i ) with probability 1 α asyptotically. Note that this set can be identified as U = { w i, j h i (X i, j ) 2 log w i, j X 2 1,1 α, w i, j = 1,w i, j 0 It is obvious that L α /U α = in/ax{µ µ U}. So if the set U is convex, then U = [L α,u α ], and we conclude the theore. To show convexity, it s enough to notice that the feasible set of (4) is convex, and the objective is linear in w i, j. We now extend our result to the general perforance easure (1). We show that, by decoposing Z(w 1,...,w ) into linear and nonlinear parts via a Taylor-like expansion, applying Theore 3 to the linear part, and controlling the agnitude of the nonlinear part, we can construct a CI that has an asyptotic guarantee siilar to Theore 3. We assue that Assuption 1 Var(G i(x i )) > 0, where G i (x) = T i E P 1,...,P [h(x 1,...,X ) X i ( j) = x]. }. (4)

5 Assuption 2 Denote the index I i = (I i (1),...,I i (T i )) {1,2,...,T i } T i, and X i,ii = (X i (I i (1)),...,X i (I i (T i ))). Assue that there exists soe positive integer k such that h(x 1,I1,...,X,I ) has finite 2k-th oent for all possible choices of I i s. It is not difficult to see that Assuption 1 is the counterpart of the condition Var(h i(x i )) > 0 in Theore 3, which is needed in dealing with the linear part of Z(w 1,...,w ), and Assuption 2 is used for controlling the nonlinear part. We state our result for the general perforance easure below. Theore 4 If Assuptions 1, 2 hold, and i c always holds for soe constant c > 0, then ( P ( 1 L α + O p n k 1 k ) Z(P 1,...,P ) U α + O p ( 1 n k 1 k )) 1 α, as, where n = and L α,u α are defined as in (2). This theore shows that if at least sixth oents (k 3) of the function h are finite, then the difference O p (1/n k 1 k ) is asyptotically negligible copared with O p (1/ n), the length of the CI. This holds trivially if, for instance, h is a bounded function. 3 MIRROR DESCENT STOCHASTIC APPROXIMATION This section focuses on solving optiization proble (2), using the irror descent (MD) algorith. The MD algorith was first proposed by Neirovsky et al. (1982) for deterinistic convex optiization general nored space, otivated by the challenge that the standard gradient descent in the Euclidean space ay not generally ake sense because the prial space, where the solution lies on, can be different fro its dual, where the gradient is defined on. It resolves the issue by apping the solution to the dual space and conducts gradient descent therein. For optiizations in R n, one does not necessarily need to use MD since the prial and the dual space are the sae. However, with a given set of constraints, a judicious choice of a prial-dual ap can result iteration subroutine that is coputationally efficient. Such observations extend to the setting where gradient can only be siulated. In this case, the algorith becoes a constrained SA procedure, and leads to MDSA. Here we outline the MDSA procedure for the following generic version of proble (2) with η > 0 in Z(w 1,...,w ) subject to 2 log w i, j η w i, j = 1, for all i w i, j 0, for all i, j (5) The feasible set here, denoted by A, is convex and satisfies A P R n, where P denotes the probability siplex associated with a support set of size, and n =. The nor on R s chosen to be the 1-nor (w 1,...,w ) 1 = w i, j. First we take the entropy distance generating function ω on P (used in the so-called entropic descent algorith (Beck and Teboulle 2003)) ω(w 1,...,w ) = w i, j logw i, j. Strong convexity of ω is guaranteed by the following proposition, whose proof is siilar to that of the special case = 1 in Neirovski et al. (2009).

6 Proposition 1 ω(w 1,...,w ) is strongly convex with paraeter 1/ w.r.t. 1-nor in the relative interior of P, i.e. ω(u 1,...,u ) ω(w 1,...,w ) ω w i, j (w1,...,w ) (u i, j w i, j ) ( u i w i 1 ) 2. Corresponding to ω is the prox-function responsible for projecting the solution back to the prial decision space ω V (u 1,...,u ;w 1,...,w ) = ω(u 1,...,u ) ω(w 1,...,w ) i, j w i, j ) (w1,...,w )(u = w i, j log w i, j u i, j. w i, j In each step of MDSA, the current solution (w k 1,...,wk ) is updated via the prox-apping prox(w k 1,...,w k ) = argin (w1,...,w ) A γ k i, j where γ k is the step size, and 3.1 Gradient Estiation Z (w i, j w k i, w j) +V (w k 1,...,w k ;w 1,...,w ), (6) i, j Z/ wi, j is an estiate for each coponent of the gradient. Here we discuss how to estiate the gradient of the objective Z(w 1,...,w ), which is needed in the prox-apping (6). It is not hard to see that Z(w 1,...,w ) is a ultivariate polynoial in w i, j, and hence its gradient is also a polynoial. However, a closer exaination reveals that the nuber of ters in this polynoial grows as fast as n T, where s the order of the saple size and T the tie horizon. Thus it is generally ipossible to copute the gradient by naively suing up all the ters, and siulation is needed. We adopt the proposal by Ghosh and La (2015a) and Ghosh and La (2015b) of using the directional derivative ψ i, j = d dε Z(w 1,...,(1 ε)w i + εe i, j,...,w ) ε=0 instead of the standard derivative Z/ w i, j, where e i, j is the j-th coordinate vector of R. It s shown that substituting ψ i, j in place of Z/ w i, j retains all properties needed in the prox-apping (6), and appealingly ψ i, j is siulable. Here is an extension of their result. Proposition 2 For any (w 1,...,w ) P with each w i, j > 0, let ψ i, j = d dε Z(w 1,...,(1 ε)w i + εe i, j,...,w ) ε=0, and Z/ w i, j be the standard derivative. Then and where ψ i, j (u i, j w i, j ) = Z (u i, j w i, j ), for any (u 1,...,u ) w i, j ψ i, j = E w1,...,w [h(x 1,...,X )S i, j (X i )], S i, j (X i ) = T i t=1 1 { X i (t) = X i, j } w i, j T i. This proposition suggests the following unbiased estiator for ψ i, j ˆψ i, j = 1 R R r=1 P h(x r 1,...,X r )S i, j (X r i ), (7) where X r 1,...,Xr,r = 1,...,R are R independent replications of the input processes generated under weights (w 1,...,w ), and are used siultaneously in all ˆψ i, j s.

7 3.2 Coputing the Prox-apping We discuss how to solve the iniization proble in prox-apping (6) with Z/ wi, j replaced by ˆψ i, j. Let s work with the generic for in subject to 2 ξ i, j w i, j +V (u 1,...,u ;w 1,...,w ) log w i, j η w i, j = 1, for all i w i, j 0, for all i, j. (8) where ξ i, j s are put in place of the gradient estiators. Seeing that the decision space of (8) has diension, which can be significantly larger than the nuber of functional constraints +1, our strategy is to first solve the Lagrangian dual proble, then use the dual optial solution to recover the prial optial solution. Consider the Lagrangian = L(w 1,...,w, λ,β) [ ni ( ξ i, j w i, j + w i, j log w i, j u i, j The dual function defined for λ R,β 0 is g(λ,β) = in L(w 1,...,w, λ,β) w 1,...,w 0 = = in w i, j 0 ) ( )] ( ni + λ i w i, j 1 β 2 ( (ξ i, j + λ i )w i, j + w i, j log w i, j u i, j 2β log w i, j ( (ξ i, j + λ i ) w i, j + w i, j log w i, j u i, j 2β log w i, j ) log w i, j + η ) ) λ i βη (9) λ i βη, (10) where w i, j is the iniizer of the inner optiization (9). This inner optiization can be solved efficiently using Newton s ethod. In other words, g(λ,β) = L( w 1,..., w, λ,β) can be easily evaluated. Moreover, it can be shown that w i, j is always strictly positive, hence L/ w i, j wi, j = w i, j = 0. By this observation and the chain rule, the first derivatives of the dual function are n g i = λ i w i, j 1, g β = 2 log w i, j η. (11) To obtain the second derivatives, note that w i, j is deterined by ξ i, j + λ i + log w i, j µ i, j + 1 2β w i, j = 0. Iplicit differentiation gives w i, j λ i = w2 i, j w i, j + 2β, w i, j λ s = 0, for s i, w i, j β = 2 w i, j w i, j + 2β.

8 So the second derivatives are 2 g λi 2 = w 2 i, j w i, j + 2β, 2 g λ i λ s = 0 for s i, 2 g β 2 = 2 g λ i β = ni 4 w i, j + 2β, 2 w i, j w i, j + 2β. (12) Therefore we have shown (10), (11) and (12) that the dual function g(λ,β), its gradient and its Hessian can all be efficiently evaluated. This allows us to solve the dual proble ax λ,β g(λ,β) subject to λ = (λ 1,...,λ ) R,β 0 (13) efficiently using, e.g., the Interior Point ethod. The following proposition helps justify our proposal of solving the dual (13) instead. Proposition 3 There is a unique optial solution (λ1,...,λ,β ) to the dual (13). Moreover, the prial optial solution w i, j to (8) can be obtained by solving ξ i, j + λi + log w i, j + 1 2β u i, j w = 0,i = 1,...,, j = 1,...,. i, j The full MDSA algorith is described in Algorith 1, which possesses the following convergence guarantee. Theore 5 Suppose there exists a unique optial solution (w 1,...,w ) for (5) such that for any (w 1,...,w ) in the feasible set Z (w 1,...,w )(w i, j w i, w j) = 0 if and only if w i = w i for all i. (14) i, j Then the sequence ( w k 1,...,wk ) generated by Algorith 1 with step size γk such that k=1 γ k =, k=1 γ 2 k <, converges to (w 1,...,w ) alost surely. It is easy to verify that the condition (14) ust hold if the objective is convex, provided the optial solutios unique. Besides convexity, condition (14) also applies to objectives that along any ray starting fro the optial solutios a strictly increasing function. 4 NUMERICAL EXPERIMENT In this section we present soe siulation study on the validity of the proposed ethod, and copare our ethod with bootstrap resapling (Barton and Schruben 1993, Barton and Schruben 2001). We will show that our ethod produces statistically valid CIs that have siilar coverage, and in soe sense are ore stable copared with the bootstrap. We consider the canonical M/M/1 queue with arrival rate 0.8 and service rate 1. The syste is epty when the first custoer coes in. We are interested in the probability that the 20-th custoer waits for

9 Algorith 1 MDSA for solving (5) Input: a paraeter η > 0, anitial feasible solution (w 1 1,...,w1 ), a step size sequence γ k, and nuber of replications per iteration R. Iteration: set k = 1 1. Estiate ˆψ k i, j = ˆψ i, j(w k 1,...,wk ),i = 1,...,, j = 1,..., using ˆψ k i, j = 1 R R r=1 h(x r 1,...,X r )S i, j (X r i ) where X r 1,...,Xr,r = 1,...,R are R independent replications of the input processes under distributions (w k 1,...,wk ) 2. Copute the optial solution (λ1 k+1,...,λ k+1,β k+1 ) to ax g(λ,β) subject to λ = (λ 1,...,λ ) R,β 0 as discussed in Section 3.2, with ξ i, j = γ k ˆψ k i, j,u i, j = w k i, j. 3. Solve the equations γ k ˆψ k i, j + λ k+1 i for (w k+1 1,...,w k+1 ), then go back to 1. + log wk+1 i, j w k i, j + 1 2β k+1 w k+1 i, j = 0 longer than 2 units of tie to get served. To put it in the for of (1), let A t be the inter-arrival tie between the t-th and (t + 1)-th custoers, S t be the service tie of the t-th custoer, and h(a 1,A 2,...,A 19,S 1,S 2,...,S 19 ) = 1 {W20 >2}, (15) where the waiting tie W 20 is calculated via the Lindley recursion W 1 = 0,W t+1 = ax{w t + S t A t,0}, for t = 1,...,19. (16) To test the ethod, we pretend that both the inter-arrival tie distribution and service tie distribution are unknown, but data for both inputs are accessible. Specifically, we generate two saples of size n 1 and n 2 fro exponential distribution with rate 0.8 and 1 respectively, and plug in the data into (2) to copute a 95% CI using Algorith 1. The paraeters of Algorith 1 are epirically chosen based on several test runs. In all the following experients, the step size is set to be γ k = 1/(2 n 1 k), and the nuber of replications for gradient estiation R = 30. The algorith is terinated if the difference easured in 1-nor between the average of the last 50 iterates and that of the last 51 to 100 iterates is less than soe sall ε > 0, which shall be specified in each of the experients below. The average of the last 50 iterates also serves as the final output of the algorith. Apart fro the stochastic nature of the algorith, another source of uncertainty that affects the CI is the final evaluation of the objective at the confidence bounds, i.e. coputing L α,u α. The bounds are coputed by taking the average of a nuber of replications, called R e, of (15). Since the length of the CI with only input uncertainty is of order 1/ n 1, to ake the stochastic error negligible, R e is chosen to be oderately larger than n 1,n 2. The value of R e for each experients is also to be specified below.

10 4.1 Accuracy of the EL confidence interval We draw 100 saples of sizes n 1 = n 2 = 30,n 1 = n 2 = 50,n 1 = n 2 = 100, respectively, and construct one CI based on each saple. Table 1 shows that the CIs have accurate coverage probabilities aong all the considered saple sizes, and can be coputed in a reasonably short tie. Here R e = 1000 is large enough because the saple size is no ore than 100. To verify that the algorith has indeed converged under these stopping criteria ε, trace plots of coponents of the weight vector are extracted. See Figure 1 for soe of the. Table 1: Perforance of EL ethod under various saple sizes. # of replications for final evaluation R e = 1000 for all, stopping criterion paraeter ε are specified in each case. Saple size 30,ε = ,ε = ,ε = Run tie per CI(seconds) Coverage probability estiate % CI for coverage probability [0.893, 0.987] [0.867, 0.973] [0.893, 0.987] # of replications of h per CI # of iterations per CI(in+ax) w 1, w 2, iteration iteration (a) w 1,1 (b) w 2,1 Figure 1: Trace plot of the first coponent of the weight vectors for inter-arrival tie and service tie. 4.2 Coparison with bootstrap resapling The bootstrap has been widely used to approxiate sapling distribution of a statistic fro which CIs can be constructed. One ethod of constructing CIs using the bootstrap is the percentile bootstrap used in Barton and Schruben (1993) and Barton and Schruben (2001). Given a pair of saples, A 1,A 2,...,A n 1 for the inter-arrival tie and S 1,S 2,...,S n 2 for the service tie, the percentile bootstrap proceeds as follows. First choose B, the nuber of bootstrap saples and N, the nuber of siulation replications for each bootstrap saple. Then for each b = 1,2,...,B, (1) draw a siple rando saple of size n 1 with replaceent fro {A 1,...,A n 1} and a saple of size n 2 fro {S 1,...,S n 2}, denoted by {A 1 b,...,an 1 b }, and {S1 b,...,sn 2 b } respectively; (2) generate N replications of h with the inter-arrival distribution being unifor over A 1 b,...,an 1 b and the service tie distribution being unifor over Sb 1,...,Sn 2 b, and take the average Z b as the estiate of Eh. Finally output the 0.025(B + 1)-th and 0.975(B + 1)-th order statistics of {Z b } B b=1, Z ( 0.025(B+1) ) and Z ( 0.975(B+1) ), as the lower and upper liits of the CI.

11 To copare our ethod with bootstrap resapling, we study two cases. In the first case (Table 2), we draw 100 saples of size n 1 = n 2 = 50, and copute CIs for each of the. In the second case (Table 3), we draw only one saple of size n 1 = n 2 = 50, and repeat coputing 50 CIs based on a single saple. To ake the coparison fair, we appropriately set the paraeters B,N for the bootstrap ethod, and ε,r e for the EL ethod so that the run ties are coparable. The result shows that the EL ethod generates coparable but slightly narrower CIs than the bootstrap, while still keeps siilar coverage probabilities. In the first case, ost of the uncertainty in either ethod coes fro the input data, so the CIs they generate exhibit siilar variation. In the second case, the input data is fixed, and hence the fluctuations of the CIs are solely due to the resapling and the stochastic noises. Table 3 shows that the EL ethod gives ore stable CIs than the bootstrap, eaning that the EL is less vulnerable to these noises. This can be attributed to the fact that EL is an optiization-based ethod and hence does not succub to the additional resapling noise introduced in the bootstrap. Table 2: EL ethod versus bootstrap, 100 confidence intervals for 100 saples. Bootstrap EL ethod B = 1000,N = 500 B = 500,N = 1000 ε = ,R e = 500 Run tie per CI(seconds) Coverage probability estiate % CI for coverage probability [0.841, 0.959] [0.893, 0.987] [0.880, 0.980] Mean CI length Standard deviation of CI length Mean lower liit Standard deviation of lower liit Mean upper liit Standard deviation of upper liit # replications of h per CI Table 3: EL ethod versus bootstrap, 50 replications of confidence intervals for a single saple. Bootstrap,B = 500,N = 1000 EL,ε = ,R e = Run tie per CI(seconds) Standard deviation of CI length Standard deviation of lower liit Standard deviation of upper liit # replications of h per CI CONCLUSION We have proposed an optiization-based ethod to quantify nonparaetric input uncertainty for siulation perforance easures. The ethod coputes CIs that account for input errors by positing a pair of optiizations subject to a weighted average of Burg-entropy divergence constraints on the collection of epirical input odels. We have argued the statistical accuracy of these optiizations via the EL ethod. We have also investigated an MDSA algorith to locally solve the optiizations, and copared its nuerical perforances with bootstrap resapling. In future work, we will investigate the generalization of this approach to other types of perforance easures, and ore efficient algoriths to solve the involved class of optiizations.

12 ACKNOWLEDGMENTS We gratefully acknowledge support fro the National Science Foundation under grants CMMI / and CMMI / REFERENCES Barton, R. R., and L. W. Schruben Unifor and Bootstrap Resapling of Epirical Distributions. In Proceedings of the 1993 Winter Siulation Conference, edited by G. Evans, M. Mollaghasei, E. Russell, and W. Biles, ACM. Barton, R. R., and L. W. Schruben Resapling Methods for Input Modeling. In Proceedings of the 2001 Winter Siulation Conference, edited by B. A. Peters, J. S. Sith, D. J. Medeiros, and M. W. Rohrer, Volue 1, Piscataway, New Jersey: Institute of Electrical and Electronics Engineers, Inc. Beck, A., and M. Teboulle Mirror Descent and Nonlinear Projected Subgradient Methods for Convex Optiization. Operations Research Letters 31 (3): Ben-Tal, A., D. Den Hertog, A. De Waegenaere, B. Melenberg, and G. Rennen Robust Solutions of Optiization Probles Affected by Uncertain Probabilities. Manageent Science 59 (2): Cox, D. R., and D. V. Hinkley Theoretical statistics. CRC Press. Delage, E., and Y. Ye Distributionally Robust Optiization Under Moent Uncertainty with Application to Data-Driven Probles. Operations Research 58 (3): Ghosh, S., and H. La. 2015a. Coputing Worst-Case Input Models in Stochastic Siulation. arxiv preprint arxiv: Ghosh, S., and H. La. 2015b. Mirror Descent Stochastic Approxiation for Coputing Worst-Case Stochastic Input Models. In Proceedings of the 2015 Winter Siulation Conference, edited by S. Jain, R. R. Creasey, J. Hielspach, K. P. White, and M. Fu, Piscataway, New Jersey: IEEE Press: Institute of Electrical and Electronics Engineers, Inc. Goh, J., and M. Si Distributionally Robust Optiization and its Tractable Approxiations. Operations Research 58 (4-part-1): Neirovski, A., A. Juditsky, G. Lan, and A. Shapiro Robust Stochastic Approxiation Approach to Stochastic Prograing. SIAM Journal on Optiization 19 (4): Neirovsky, A.-S., D.-B. Yudin, and E.-R. Dawson Proble Coplexity and Method Efficiency in Optiization. John Wiley & Sons, Inc., Panstwowe Wydawnictwo Naukowe (PWN). Owen, A. B Epirical Likelihood Ratio Confidence Intervals for a Single Functional. Bioetrika 75 (2): Owen, A. B Epirical Likelihood. CRC Press. Pardo, L Statistical Inference Based on Divergence Measures. CRC Press. AUTHOR BIOGRAPHIES HENRY LAM is an Assistant Professor in the Departent of Industrial and Operations Engineering at the University of Michigan, Ann Arbor. His research focuses on stochastic siulation, risk analysis, and siulation optiization. His eail address is khla@uich.edu. HUAJIE QIAN is a Ph.D. student in Applied and Interdisciplinary Matheatics at the University of Michigan, Ann Arbor. His research interest lies in applied probability and siulation optiization. His eail address is hqian@uich.edu.