Proceedings of the 2014 Winter Simulation Conference A. Tolk, S. Y. Diallo, I. O. Ryzhov, L. Yilmaz, S. Buckley, and J. A. Miller, eds.

Transcription

1 Proceedings of the 2014 Winter Simuation Conference A. Tok, S. Y. Diao, I. O. Ryzhov, L. Yimaz, S. Buckey, and J. A. Mier, eds. ADVANCED TUTORIAL: INPUT UNCERTAINTY QUANTIFICATION Eunhye Song Barry L. Neson Dept. of Ind. Engr. & Mgmt. Sci. Northwestern University Evanston, IL USA C. Dennis Pegden Simio LLC 504 Beaver Street Sewickey, PA USA ABSTRACT Input uncertainty refers to the (often unmeasured) effect of not knowing the true, correct distributions of the basic stochastic processes that drive the simuation. These incude, for instance, interarriva-time and service-time distributions in queueing modes; bed-occupancy distributions in heath care modes; distributions for the vaues of underying assets in financia modes; and time-to-faiure and time-to-repair distributions in reiabiity modes. When the input distributions are obtained by fitting to observed reaword data, then it is possibe to quantify the impact of input uncertainty on the output resuts. In this tutoria we carefuy define input uncertainty, describe various proposas for measuring it, contrast input uncertainty with input sensitivity, and provide and iustrate a practica approach for quantifying overa input uncertainty and the reative contribution of each input mode to overa input uncertainty. 1 INTRODUCTION The stochastic in stochastic simuation is produced by input modes that drive the simuation output. In the simpest case the input modes are fuy specified univariate probabiity distributions from which the simuation generates independent and identicay distributed (i.i.d.) random variates. More generay, the input modes are stochastic processes described probabiisticay (e.g., their joint distribution), rather than ogicay (customers are served in the order they arrive). The fideity with which the input modes capture the uncertainty in the system of interest directy affects the fideity of the concusions that can be drawn from the simuation experiment. In this tutoria we focus on input modes that are fit to representative sampes of rea-word data. Because the quantity of data is necessariy finite, the resuting input modes are amost surey imperfect representations of reaity. Our interest is in quantifying this uncertainty in the input modes that is propagated to the simuation performance estimates. Let F denote the coection of input modes in the simuation (we wi be more specific about this coection ater). We wi represent the simuation output on repication j as Y j (F) = η(f)+ε j (F) where η(f) = E[Y j (F)] is the expected vaue of the simuation output random variabe when we use input distributions F, and ε j (F) is a mean 0 random variabe representing the stochastic noise that is a consequence of the input modes. Since the output coud be an average, an indicator variabe, a sampe variance, a sampe quantie or any number of other quantities, this is a very genera representation that emphasizes the dependence of the system performance measure η(f) and the simuation output Y j (F) on the input modes F. To execute a simuation experiment we need a specific choice or choices for F, and the natura approach when we have rea-word input data is to use a fitted distribution F, which is a stand-in for the unknown /14/$ IEEE 162

2 true rea-word distribution denoted by F c (the c indicates correct ). Therefore, the simuation resuts we observe are Y j ( F) = η( F)+ε j ( F). Given the choice F, there is a vast iterature on simuation design and anaysis that provides guidance on how many repications we need, and what estimator to use, to estimate η( F). Ceary this anaysis is conditiona on F, and therefore ignores a potentiay arge source of error due to using F F c. The input uncertainty probem is to quantify how much uncertainty in the simuation estimator of η(f c ) is caused by the uncertainty in estimating F c. In this tutoria we formay define input uncertainty, describe various proposas for measuring it, contrast input uncertainty with input sensitivity, and provide and iustrate a practica approach for quantifying overa input uncertainty and the reative contribution of each input mode to it. We start with a high-eve discussion of input modeing itsef. 2 INPUT MODELING For ease of exposition, suppose (temporariy) that the system of interest has ony one source of randomness represented by a univariate input distribution F c. Later we wi extend to the case of mutipe input distributions. Generay F c is unknown; however, we can observe a reaization z m = {z 1,z 2,...,z m } of the i.i.d. input process Z 1,Z 2,...,Z m F c. Notice that we use capita etters such as Z to denote a random variabe, and owercase etters such as z for a reaization. We wi aso use Z for rea-word input data from F c, and X for simuation-generated input data from F. The estimated distribution F depends on any prior information we have about F c and the observed data z m. Choosing F is caed input modeing. What constitutes good input modeing? The goa of the simuation experiment is to precisey estimate the true system performance measure, which is η(f c ) in our set up. Therefore, estimating F c precisey is a sufficient, but not a necessary condition; to achieve our goa we need η( F) to be cose to η(f c ), not F = F c. In some speciaized situations we can figure out what specific properties of F need to be cose to the corresponding properties of F c for η( F) to be cose to η(f c ), but this is certainy not possibe in genera because η( ) is unknown. Therefore, the focus of input modeing is typicay on the fideity of F with respect to F c. The two main phiosophies of input modeing are frequentist and Bayesian, and whie certainy different, they share some common features when it comes to input uncertainty. Frequentists assume that there exists a true distribution F c, and the goa is to infer F c from the rea-word data using methods that have good performance when averaged over the possibe sampes Z 1,Z 2,...,Z m. For instance, if F c is beieved to have a known distribution famiy but unknown parameters θ c that is, F c (x) = F(x θ c ) with F known then the parameters may be estimated using maximum ikeihood estimators (MLEs), east squares, generaized method of moments, or other methods. Typicay these estimators are asymptoticay consistent for θ c as m. Prior information such as the physica basis of the input process may infuence the choice of distribution famiy. There is commercia input modeing software that fits the parameters of a arge coection of distribution famiies and aso provides ways to assess the fit, incuding goodness-of-fit tests such as Chi-square, Anderson-Daring and Komogorov-Smirnov. Errors in fitting are characterized by the samping distribution of the parameter estimator θ; since this distribution is often hard to derive, approximations based on arge-sampe asymptotics or bootstrapping are often used. Bayesians are ess interested in discovering the true distribution F c and more interested in the ikeihood of possibe choices given the data. Bayesian inference starts by capturing any prior knowedge avaiabe about F c in the form of a prior probabiity distribution; this distribution may be on the famiy of distributions, the parameters of a specific famiy of distributions, or both. For instance, we might beieve that F c is one of the famiies in the set {F 1,F 2,...,F k } with prior probabiities π 1,π 2,...,π k. More often, we assume that the famiy of F c is known and provide a prior π Θ (θ) on the vaue of its parameter vector Θ, which is treated as a random vector. 163

3 After coecting the rea-word data z m, the prior distribution π Θ (θ) is updated to a posterior distribution p Θ (θ z m ) according to Bayes rue p Θ (θ z m ) L(z m ;θ)π Θ (θ), where L(z m ;θ) is the ikeihood function of z m given Θ = θ. Notice that the posterior distribution is conditiona on the particuar reaization z m that was obtained. The posterior distribution p Θ (θ z m ) pays the roe for Bayesians that the samping distribution of θ pays for frequentists by quantifying the uncertainty about θ c. Under some reguarity conditions, the posterior distribution of Θ converges to a degenerate distribution at θ c independent of the choice of the prior distribution as the sampe size m ; this is the Bayesian concept of consistency. Returning to the main purpose of the simuation study to estimate η(f c ) both frequentist and Bayesian input modeers face a simiar probem: to actuay execute a simuation experiment requires a specific choice of distribution and distribution parameters. When ony the parameters are unknown, we often use the singe best choice, such as the MLE for frequentists or the maximum a posteriori (MAP) estimator for Bayesians, but a singe choice assumes away the very rea uncertainty about the input mode. However, since η( ) is unknown, characterizing uncertainty about the input mode (via a samping distribution or posterior) is not enough since the goa is to measure the uncertainty in estimating η(f c ). Therefore, to quantify input uncertainty, some way to propagate the uncertainty about F c through to the estimate of η(f c ) is required. In the remainder of the tutoria we define some specific measures of input uncertainty and iustrate one practica approach to estimate them. 3 MEASURES OF INPUT UNCERTAINTY One difficuty in describing input uncertainty is that there are many types of input modes that may arise in a computer simuation. These incude univariate inputs such as customer service times; time-series inputs such as the weeky demands for mik; mutivariate inputs such as the age, income, gender and occupation of a customer; and time-dependent inputs such as the arriva times of cas to a ca center. For this tutoria we wi assume that the input modes for the simuation consist of L independent, univariate distributions F = {F 1,F 2,...F L } each of whose roe is to mode an i.i.d. input process. For instance, in a stationary queueing system with server faiures we might have F = {F 1,F 2,F 3,F 4 }, the distributions of interarriva times, service times, time unti server faiure, and server repair time, respectivey. Let Y(F) be a random variabe that represents the generic output of the simuation such as the average customer deay in the queueing system with faiures when the input distributions are F. We assume that there are unknown, but correct, rea-word distributions F c = {F1 c,f 2 c,...,fc L }. From the th distribution we are abe to observe m i.i.d. rea-word observations Z 1,Z 2,...,Z m from which to fit F c, giving the coection of fitted input modes F = { F 1, F 2,..., F L }. We then run the simuation and obtain outputs Y 1 ( F),Y 2 ( F),...,Y n ( F) across n i.i.d. repications. The goa of the simuation experiment is to estimate η(f c ) = E[Y(F c )]. For ease of exposition here we assume that the η(f c ) is estimated by the sampe mean across n repications Ȳ( F) = 1 n n j=1 Y j ( F) = η( F)+ 1 n n j=1 ε j ( F). 164

4 The variance of the point estimator is [ ] [ ] Var[Ȳ( F)] = Var E(Ȳ( F) F) +E Var(Ȳ( F) F) [ = Var η( F) ]+ 1 [ ] n E Var(ε 1 ( F) F), (1) where outer variance and expectation on the right hand side is with respect to the samping distribution of F (frequentist) or the posterior distribution of F (Bayesian). Notice that the second term in (1) accounts for the inherent variabiity in the simuation error (given a choice of input distributions), and it can be driven to 0 by increasing the simuation effort n. The size of the first term, however, depends on a compex interaction between the rea-word sampe sizes m = {m 1,m 2,...,m ] L } and the structure of η( ). Our forma definition of input uncertainty is σi [η( F) 2 = Var, the variance in the system mean due to having estimated F c. Athough this definition is straightforward, how to estimate it is not immediatey cear as it requires 1) the samping distribution of F (frequentist case) and 2) the functiona form of η( ), neither of which is known in genera. This iustrates why assessing input uncertainty is difficut and motivates the methods that have been suggested in the iterature and are reviewed beow. Conceptuay, if we had B independent rea-word sampes of input data of size m, yieding B independent fitted input distributions F 1, F 2,..., F B, and we knew the true mean response functiona η( ), then we coud estimate σi 2 via the unbiased estimator B σ I 2 = 1 b ) η) B 1 b=1(η( F 2,where η = 1 B B b=1 η( F b ). However, if we actuay had mutipe rea-word data sets then we woud certainy poo them to obtain a better estimator of F c ; thus, no matter how much data we have it is best to treat it as one sampe. In addition, η( ) is unknown or we woud not be simuating. In the next two subsections we consider these probems (singe sampe, unknown response function) in turn, both from frequentist and Bayesian points of view. It is worth mentioning that one other aspect of input uncertainty that is rarey noted, even in the input-uncertainty iterature, is bias. Specificay, E [ Y( F) η(f c ) ] [ ] = E η( F) η(f c ) 0 in genera. This is the case even when E[ F] = F c because η( ) is typicay a noninear transformation. We might hope that η( F) η(f c ) in some sense as the rea-word sampe sizes m, = 1,2,...,L, but this is the most we can hope. 3.1 Variabiity of F For a frequentist, the variabiity of F as an estimator of F c is quantified by its samping distribution; that is, we try to infer the distribution of possibe F s that coud be reaized from the singe sampe of data that we actuay have. There have been two primary approaches in the iterature: arge-sampe asymptotic distributions as in Cheng and Hoand (1998) and bootstrapping as in Barton and Schruben (2001), Ankenman and Neson (2012), and Song and Neson (2013). To simpify the exposition, suppose again that there is ony L = 1 input distribution, and we beieve F c (z) = F(z θ c ) is a parametric distribution with known famiy (e.g., Poisson), but unknown parameter vector θ c. The famiy might be known because of process physics (e.g., an arriva process consisting of the superposition of many customers making independent but infrequent decisions about when to arrive tends to be Poisson), or from fitting a arge number of possibe parametric famiies and ranking their goodness of fit. 165

5 For the parametric case there is a we-deveoped theory for parameter estimators θ = θ(z 1,Z 2,...,Z m ) such as the MLE. The samping distributions for these estimators are known, at east asymptoticay as m. For instance, MLEs are often asymptoticay normay distributed. Thus, for parametric distributions, input mode uncertainty becomes input parameter uncertainty which is quantified by the samping distribution of θ. Cheng and Hoand (1998) provide an expression for σi 2 by appying a first-order Tayor series approximation of η( θ) to obtain σ 2 I = Var[η( θ)] (η(θ c )) Var[ θ] (η(θ c )) (2) where Var[ θ] is the variance-covariance matrix of θ and (η(θ c )) is the gradient of η( ) evauated at θ c. The approximation (2) shows that input uncertainty depends both on how we the input mode has been estimated, and how sensitive the system response is to the input mode. The samping distribution (or an approximation to it) can provide an estimator of Var[ θ], but that sti eaves the question of how to estimate (η(θ c )). In bootstrapping, we assume that the fitted distribution F is a good stand-in for F c. An empirica cumuative distribution function (ecdf) is often used as the fitted distribution, athough this is not required. Then instead of gathering B distinct rea-word sampes of input data, we simuate B bootstrap sampes from the empirica distribution: X (b) 1,X (b) 2,...,Xm (b) i.i.d. F for b = 1,2,...,B. Each bootstrap sampe yieds a fitted input distribution F (b). Therefore, an estimator of σ 2 I is σ I 2 = 1 B 1 B b=1 (η( F (b) ) η ) 2 where η = B b=1 η( F (b) )/B. The vaidity of the bootstrap can aso be estabished as m. However, this estimator requires a stand-in for η( ). Bayesian input modeing typicay emphasizes the choice of parameters for a parametric distribution, athough as noted earier it is possibe to extend the definition of parameter to incude the distribution famiy as we. The posterior distribution of the parameters p Θ (θ z m ) pays the roe of the samping distribution of θ for the frequentist. However, in a Bayesian treatment there is no need for an asymptotic approximation because the posterior is vaid for any quantity of rea-word data. In fact, the posterior expicity accounts for the quantity of data that is avaiabe: the more data there is, the more concentrated the posterior distribution wi be. From a Bayesian perspective, the corresponding overa measure of input uncertainty is σ 2 I = Var[η(Θ)] where the variance is with respect to Θ p Θ (θ z m ). One difficuty in computing this quantity arises when the posterior distribution is not simpe and can ony be evauated approximatey via simuation methods, such as Markov chain Monte Caro. And, of course, η( ) is not known. Instead of using the posterior distribution of Θ, Ng and Chick (2001, 2006) empoy the asymptotic norma distribution of the MAP estimator Θ m. Simiar to Approximation (2), they combine the asymptotic norma approximation with a first-order Tayor series expansion of η( Θ m ) to derive σ 2 I : σ 2 I (η( θ m )) Σ m (η( θ m )), where θ m is the MAP of Θ and Σ m is the inverse of the Bayesian observed information given z m. Likewise, they need an assumption on the structure of the unknown η( ) to propagate the uncertainty in Θ to the simuation estimator. 166

6 3.2 Propagating Input Mode Uncertainty The second reason that σ I 2 is not reaizabe is that η( ) is unknown, and it is this mapping that propagates the input mode uncertainty to the simuation performance measure uncertainty. Whether frequentist or Bayesian, there is reay no choice but to estimate η( ) using simuation, and we can ony estimate η(f) given specific choices of F. The key questions are, what shoud we estimate, and how shoud we expend the simuation effort to do it? Direct simuation chooses a coection of distributions F 1,F 2,...,F B, runs simuations at each choice, and estimates η(f i ) by the average of the simuation resuts Ȳ(F i ). The distributions F 1,F 2,...,F B are usuay randomy chosen from the samping distribution of θ (frequentist) or the posterior distribution of Θ (Bayesian) so as to represent the input mode uncertainty. 1. For b = 1 to B (a) If frequentist: i. Generate θ b from the samping distribution of θ (e.g., using the asymptotic distribution or bootstrapping). ii. Using θ b, run n repications to obtain the sampe mean Ȳ b ( θ b ). (b) Ese if Bayesian: i. Generate Θ b p Θ (θ z m ) ii. Using Θ b, run n repications to obtain the sampe mean Ȳ b (Θ b ). Next b 2. Use the data Ȳ b ( θ b ) or Ȳ b (Θ b ) for b = 1,2,...,B to estimate measures of input uncertainty. Notice that in either case input uncertainty is confounded with stochastic simuation noise so they may need to be separated, which is the chaenge. Metamodeing chooses distributions F 1,F 2,...,F d to cover a reevant design space, and then uses the simuation resuts to fit a metamode η( ) to stand in for η( ). Barton et a. (2014) take a frequentist approach to input modeing and use Gaussian processes to buid a metamode at a chosen set of parameters. Given the metamode, σ I 2 or a number of other measures can be estimated inexpensivey. But there is sti a need to separate input uncertainty from metamode error. Gradients of η(θ) are needed if we use parametric input modes and approximate Var[η( θ)] using a Tayor series expansion of η( θ) around θ c as in (2). Since η(θ) is unknown, its gradients are aso unknown. Cheng and Hoand (2004) suggest severa methods to estimate (η(θ c )). The expansion in (2) faciitates propagating uncertainty about θ (as represented by the samping distribution of θ or the posterior distribution of Θ) to the performance measure η( θ). Ceary this is another form of metamodeing. 3.3 Additiona Measures of Input Uncertainty Athough we focused[ on estimating] σi 2, there are other usefu measures depending on the objective. In (1) et σ 2 = E Var(ε 1 ( F) F), the simuation output variance. Then Ankenman and Neson (2012) standardize σi 2 as γ = σ I σ/ n. Thus, the ratio γ is the standard deviation due to input uncertainty in units of the standard error of the simuation point estimator. For instance, γ = 0.5 means that the error due to input uncertainty is haf as arge as that due to estimating η( F) via simuation. If the repication number n is sufficienty arge, sma γ indicates that the input uncertainty is insignificant. 167

7 Interva estimates are aso vauabe. The typica simuation confidence interva (CI) guarantees (1 α)100% coverage of η( F). Instead, an interva [C L,C U ] that guarantees Pr{η(F c ) [C L,C U ]} 1 α accounts for input mode uncertainty, simuation error and bias. The corresponding Bayesian credibe interva (CrI) woud assert Pr{η(Θ) [Q L,Q U ]} = 1 α. This interva contains 1 α of the probabiity content of the induced posterior distribution on η(θ) when Θ p Θ (θ z m ). Many papers have proposed generating CIs or CrIs from direct simuation, as described above, using empirica quanties of Ȳ b ( θ b ) or Ȳ b (Θ b ), respectivey, for b = 1,2,...,B. See, for instance, Xie et a. (2014) and Zouaoui and Wison (2003, 2004). So far we have ony considered the measures of input uncertainty as a function of F, the coection of a input modes used in the simuation. We next turn our attention to assessing which input modes among F make the greatest contributions to input uncertainty, and from which woud we most benefit from coecting more rea-word data. 3.4 Measures of Contribution and Sampe-Size Sensitivity Contribution measures try to decompose σi 2 in a meaningfu way. This is difficut, and in fact is currenty an area of active research. The utimate goa is to identify from which distributions it woud be most beneficia to reduce uncertainty further, or even to specify how to spend a budget for additiona data coection. Ceary, the contribution of th input mode F depends on the rea-word sampe size m ; as m becomes infinitey arge, F converges to F c (frequentist) or to a degenerate posterior distribution(bayesian). Therefore, sma m causes bigger contribution. However, the contribution aso depends on how sensitive η( ) is to F. To make the point cear, assume that F 1 is in fact a dummy distribution so that the functiona η( ) does not depend on F 1. Then, no matter how sma m 1 is, the contribution of F 1 shoud be 0. On the other hand, if η( ) is very sensitive to F 1, the uncertainty in F 1 is ampified as it is propagated to the uncertainty in η( F). Song and Neson (2013) defined the contribution of F to input uncertainty as [ ] V (m ) Var E[Y(F1 c,f2 c,...,f 1, c F,F+1,...,F c L) F c ]. (3) In words, V (m ) is the variabiity in the simuation s expected vaue when a of the true input distributions except F c are known and F c is estimated by F. Notice that V is a function of the sampe size m ; the arger m is, the smaer V (m ) becomes as F approaches F c. Unfortunatey, L =1 V (m ) σi 2 in genera. Song and Neson (2013) suggest ooking instead at the reative contribution of the th input mode, V (m ) L i=1 V i(m i ). A reasonabe heuristic to reduce σi 2 is to obtain more data for the distributions with the argest reative contribution. Song and Neson (2013) propose a measure that does not decompose σi 2, but instead quantifies the impact of additiona data for each input distribution: the sampe-size sensitivity of Var[Ȳ( F)] with respect to th input mode. If we approximate m as rea vaued then this is Var[Ȳ( F)] m. (4) m =m 168

8 The sampe-size sensitivity quantifies how much the estimator variance woud be reduced by observing one more rea-word sampe from the th input process given we aready have m observations. The input distributions with the argest (most negative) sensitivities are targets for more rea-word data. We iustrate one method for estimating the reative contribution and sampe-size sensitivity in Section 4. Unfortunatey, sampe-size sensitivities are ony oca gradients, and therefore are not idea for optimay aocating a substantia amount of additiona effort. Freimer and Schruben (2002) and Ng and Chick (2001, 2006) attempt to go further. If the budget for rea-word data is effectivey unimited, then it makes sense to coect enough data so that Var[Ȳ( F)] E[Var(Ȳ( F) F)]; i.e., no input uncertainty. Freimer and Schruben (2002) achieve this objective by sequentiay adding rea-word data unti the effect of input mode uncertainty is statisticay negigibe reative to simuation output variabiity. To formaize this they use a random-effects metamode where the random effects are due to distribution parameter estimates obtained from bootstrapping the avaiabe data. In the case of L = 2 parameters, their mode for the simuation output on repication h is Y i jh = η +A i +C j +AC i j +ε i jh, for i, j = 1,2,...,B, h = 1,2,...,n (5) where A i, C j and AC i j are the random effects of bootstrapped parameters θ1i, θ 2 j and their interaction, respectivey, B is the number of bootstrap sampes, and ε i jh is the stochastic variabiity of the simuation. They stop coecting input data when the hypothesis that these three variance effects are 0 is no onger rejected. If the budget for coecting rea-word data, and perhaps aso for running the simuation itsef, is imited, then it makes sense to try to aocate effort to minimize the variance of the point estimator within the budget. This requires an approximation for the variance as a function of m and n. Ng and Chick (2001, 2006) obtain such an approximation by combining a Bayesian treatment of input mode uncertainty with a inear metamode for the simuation output Y(Θ) = η(θ)+ε λ 0 + P p=1 λ p (Θ p θ p )+ε (6) where P is the tota number of individua parameters of a input distributions, and θ p is the MAP of the pth parameter Θ p. Estimation of the metamode parameters is aso treated in a Bayesian manner. Using this mode they derive an expression for the posterior variance of η(θ) as a function of m and n, which they use to optimay aocate the budget for additiona input data or additiona simuation repications. 3.5 Connections to Sensitivity Anaysis Sensitivity anaysis, as it is usuay construed in stochastic simuation, means estimating terms ike g(θ) at given vaues of θ. As we noted in approximation (2), input uncertainty is more than just sensitivity. As iustrated in Section 3.4, input uncertainty is a function of both the size of the rea-word data set from which we estimate the input distribution parameters and the sensitivity of the response to the parameter. There is at east a superficia connection between input uncertainty and goba or probabiistic sensitivity. Key references incude Wagner (1995), Homma and Satei (1996), Oakey and O Hagan (2004), and Pischke et a. (2013). Expressing the goba sensitivity probem in our notation, it is concerned with a response y = η(θ) where η( ) is a function that can be evauated without noise, but usuay at great computationa expense. The probem arises because the parameter θ is either unknown or uncontroabe (e.g., it contains environmenta factors). Therefore, its vaue is modeed as a random variabe Θ with known distribution F Θ ( ). The response can aso be treated a random variabe Y = η(θ) with Θ F Θ. Goba sensitivity anaysis tries to decompose or expain Var[Y] by attributing portions of it to each component parameter of Θ = (Θ 1,Θ 2,...,Θ L ). Research is driven by the fact that direct evauation of η( ) is too expensive to do for many vaues of θ, rather than by ack of knowedge of F Θ. Our input uncertainty probem, on the other hand, is characterized by ack of knowedge of the input distributions themseves and the presence of simuation noise in the evauation of η( ). 169

9 4 A PRACTICAL APPROACH AND ILLUSTRATION In this section, we summarize a method suggested by Song and Neson (2014) to estimate the input uncertainty measures introduced in Section 3, aong with an iustration of its appication on a practica probem. This approach takes a frequentist s view of input modeing and input uncertainty quantification. We continue to assume that the coection of input modes F consist of L independent, univariate distributions. The goa of the experiment is to estimate η(f c ). Song and Neson (2014) describe a sequence of experiments: nomina, diagnostic and foow-up. Typicay, a simuationanayst performs a nomina experiment by fitting the input modes F = { F 1, F 2,..., F L } from a rea-word sampe z m = {z m1,z m2,...,z ml } and conducting mutipe, say n, simuation repications to obtain a point estimator Ȳ( F) and its CI conditiona on F. As noted earier, this estimator and CI do not take into account input uncertainty caused by having estimated F c by F. To address this probem, Song and Neson (2014) introduce the diagnostic and foow-up experiments, described in the foowing subsection. 4.1 The diagnostic and foow-up experiments The purpose of the diagnostic experiment is to quantify the input uncertainty in the estimator from the nomina experiment using the measures defined in Section 3. Song and Neson (2014) empoy the foowing mean-variance effects metamode as a stand-in for Y j (F): Y j (F) = η(f)+ε j = β 0 + L =1 β µ(f )+ L =1 ν σ 2 (F )+ε j, (7) where µ(f ) and σ 2 (F ) denote the mean and variance, respectivey, of F ; β and ν are constant coefficients; and ε j is an independent mean-zero random variabe with constant variance σ 2. The idea behind Mode (7) is that the effect of F on the mean response can be captured by some summary properties of F ; in particuar, the reaized center (mean) and spread (variance) of F. Aso, β and ν represent how sensitive η( ) is to the th input distribution. We do not expect Mode (7) to fit we over the entire samping space of F; rather, it is expected to have a good oca fit for F near F c. Under Mode (7), the simuation uncertainty σi 2 can be represented as σi 2 L { } = Var[η( F)] = β 2 Var[µ( F )]+ν 2 Var[σ 2 ( F )]+2β ν Cov[µ( F ),σ 2 ( F )]. (8) =1 Likewise, the contribution of F to the input uncertainty can be derived from the definition in (3) as [ ] V (m ) = Var β 0 + β i µ(fi c )+ ν i σ 2 (Fi c )+β µ( F )+ν σ 2 ( F ) L i=1,i L i=1,i = β 2 Var[µ( F )]+ν 2 Var[σ 2 ( F )]+2β ν Cov[µ( F ),σ 2 ( F )]. (9) Notice that under Mode (7) the overa input uncertainty is the sum of each distribution s contribution, i.e., σi 2 = L =1 V (m ). Hence, the sampe-size sensitivity defined as in (4) becomes Var[Ȳ( F)] m = V (m ) m. m =m m =m To estimate the contributions from (9), we need to estimate the parameters β and ν, and the variance components of the sampe statistics, Var[µ( F )],Var[σ 2 ( F )] and Cov[µ( F ),σ 2 ( F )], for = 1,2,...,L. If we had B coections of rea-word data, then the parameters β and ν coud be obtained by fitting 170

10 Mode (7) using the corresponding fitted distribution F 1, F 2,..., F B. However, as discussed in Section 3.1, we prefer to poo a rea-word data to obtain a better estimate of F c. Instead, Song and Neson (2014) bootstrap F to obtain ecdfs F 1, F 2,..., F B from bootstrapped sampes and fit Mode (7) via east-squares regression to estimate the β s and ν s. Another advantage of the bootstrap approximation is that it yieds simpe expressions for the variance components in (9). Bootstrapping impies that we assume Var[µ( F ) F ],Var[σ 2 ( F ) F ], and Cov[µ( F ),σ 2 ( F ) F ] are stand-ins for Var[µ( F )],Var[σ 2 ( F )], and Cov[µ( F ),σ 2 ( F )]. Song and Neson (2014) derive Var[µ( F ) F ] = M2 m Var[σ 2 ( F ) F ] = (m 1) 2 m 3 M 4 (m 3)(m 1) m 3 (M) 2 2 M4 (M2 )2 m Cov[µ( F ),σ 2 ( F ) F ] = (m 1) 2 m 3 M 3 M3 m where M k is kth centra moment of F. Therefore, the contribution and sampe-size sensitivity estimator can be written as V (m ) = 1 } 2 { β m M 2 + ν 2 (M 4 (M) 2 2 )+2 β ν M 3, V (m ) m = V (m ). (10) m m =m The agorithm for the diagnostic experiment is as foows: Diagnostic Experiment 1. Given the estimated input modes F (either parametric or non-parametric), do the foowing: 2. For bootstrap sampe b = 1 to B (a) For input mode = 1 to L i. Generate X (b) 1 ii. Let F (b),x (b) 2 be the ecdf of X (b) 1,...,X (b) m by samping m times from F.,X (b),...,x (b) 2 m and cacuate the mean µ( F (b) ) and variance σ 2 ( F (b) ). Next } (b) Using input modes F (b) = { F (b) 1, F (b) 2,..., F (b) L, simuate R i.i.d. repications Y j ( F (b) ), j = 1,2,...,R and cacuate the sampe mean Ȳ( F (b) ). Next b 3. Fit the mode Y j ( F ) = β 0 + L =1 β µ( F )+ L =1 ν σ 2 ( F )+ε j (11) via east-squares regression with dependent variabe Ȳ( F (b) ) from Step 2b and independent variabes µ( F (b) 1 ), µ( F (b) 2 ),..., µ( F (b) L ) and σ 2 ( F (b) 1 ),σ 2 ( F (b) 2 ),...,σ 2 ( F (b) L ) from Step 2a for b = 1,2,...,B to estimate the coefficients β 0,β 1,...,β L,ν 1,ν 2,...,ν L. 4. Estimate the contribution V (m ) and the sensitivity for = 1,2,...,L using (10). 171

11 5. Estimate the overa input uncertainty σ I 2 = L V =1 (m ) and the estimated ratio γ = σ I σ/ n = n L V =1 (m ). σ Notice that σ 2 in Step 5 is an estimator of the simuation error variance from the nomina experiment and n is the number of repications in the nomina experiment. Because the diagnostic experiment requires additiona simuation effort, Song and Neson (2014) aso study how to break the given simuation budget N = BR into B bootstraps of R repications each. For the estimation quaity, arge B is beneficia because it reduces the variance of the estimated β s and ν s. However, from a computationa perspective, sma B is preferred. Therefore, finding a baance between these two is the key to the experiment design. One suggestion made by Song and Neson (2014) is to aocate N according to 2L+2 B > 2L+2+ (12) δ with R = N/B. This choice makes B just as big enough so that the incrementa variance reduction of increasing B by 1 is ess than δ100%, where they recommend δ = As mentioned in Section 3.3, the estimated ratio γ can be used to decide the significance of the input uncertainty in the simuation estimator provided that the number of repications n for the nomina experiment is sufficienty arge. If γ is arge, e.g., γ = 20, then we can concude that there is substantia input uncertainty and move on to the probem of deciding from which input processes we shoud coect additiona rea-word data (if possibe). This question can be answered by the sampe-size sensitivities of the input modes. See Section 4.2 for detais. Song and Neson (2014) aso propose a heuristic way to adjust the nomina CI using γ to incude the input uncertainty σi 2. Roughy, the adjusted 100(1 α)% CI is given as Ȳ( F) ±q α/2 σ n 1+ γ 2, (13) where q α/2 is the 100(α/2)-th quantie of the standard norma distribution. Even if γ is arge, it may be infeasibe to coect additiona data from any of the input processes (for instance, the system no onger exists). In this case, Song and Neson (2014) suggest to improve the estimator from the nomina experiment by combining it with the repications obtained from the diagnostic experiment. The combined estimator Ỹ is Ỹ = ωȳ( F)+(1 ω)ȳ( F), where Ȳ( F) = B b=1ȳ( F (b) )/B and ω [0,1]. The optima weight ω given F can be obtained by minimizing the conditiona mean squared error (MSE) of Ỹ given F; an estimator of the optima weight is ω = ( b ) 2 + σ 2 I /B+ σ 2 /BR ( b ) 2 + σ 2 I /B+ σ 2 /BR+ σ 2 /n, where b = Ȳ( F) Ȳ( F) is an estimator of bias. By combining Ȳ( F) and Ȳ( F), the bias is increased as bootstrapped distributions add more bias to the estimator Ȳ( F), but the variance is decreased because we have more repications. The optima weight ω baances these two. When input uncertainty is significant, and it is possibe to coect additiona rea-word data on the most offending input modes, then a foow-up experiment is performed using the refined input modes formed 172

12 from m = {m 1,m 2,...,m L } observations, where m m. Song and Neson (2014) investigated whether it is worthwhie to combine simuation resuts from the nomina and foow-up experiments and concuded that typicay it is not. Because of the increased rea-word sampe, Ȳ( F m ) is ess biased than Ȳ( F m ). In addition, the variance of Ȳ( F m ) is smaer as it has ess input uncertainty. Therefore, combining the two estimators is unikey to reduce the MSE. 4.2 Iustration: Remote Order-taking System In this section, we summarize an iustration in Song and Neson (2014) borrowed from Neson (2013) to describe a practica use of the three-phase experiments. A resuts in this iustration are performed using Simio and its input uncertainty anaysis add-in that impements the diagnostic experiment procedure in Section 4.1. The probem is to evauate repacing the current order-taking system at the drive-though window of a chain of fast-food restaurants by a remote order-taking system. In the current system, there is a server at each store who takes orders from the incoming customers when they reach the order board. Under the proposed system, the server at the store is repaced by a poo of agents at a remote ca center. The hope is that many fewer agents are needed than one per store. The fast food chain has high standards for service quaity and in particuar wants the average time that a customer waits unti being greeted by an agent to be ess than 1.5 seconds. A simuation study of this new system using the data from 7 stores is undertaken. Tabe 1 shows 9 input processes used in the simuation: inter-arriva times of customers at 7 stores; order-taking times for the remote agents; and the time for a car to pu up to the order board after the previous car (if there is one) departs. The inter-arriva times were coected during the busiest 3-hour period over 10 days. With these data, Song and Neson (2014) fit parametric distributions to perform the nomina experiment. Athough these rea-word data were actuay generated from exponentia distributions, they did not take advantage of knowing the famiies of distributions or parameters. The fitted distributions were exponentias, Weibus, and Gammas. To obtain an estimator of the mean customer waiting time, they performed 1,000 repications of a steady-state simuation adopting a repication-deetion experiment design. The resuts showed that with 4 agents the 95% CI for mean customer waiting time was 0.99 ±0.04 seconds, which is ceary ess than 1.5 seconds. However, as mentioned earier, this nomina CI does not take into account the input uncertainty. Therefore, to make a more robust management decision, one can conduct a diagnostic experiment. Given the budget N = 10,000 for the diagnostic experiment, (B = 80,R = 125) is chosen according to (12) where δ = 0.01 is used. The diagnostic experiment gave γ = 17.48, indicating that there is significant input uncertainty in the simuation estimator from the nomina experiment. The heuristic adjustment (13) expands the CI haf width to The resuting CI (0.36,1.62) contains the critica vaue, 1.5 seconds. To further investigate which input mode contributes the most to input uncertainty, Tabe 1 shows the estimated reative contributions and sampe-size sensitivities of input modes using (10). Notice that the contributions are normaized to sum to 1 and sensitivities are scaed reative to the most sensitive input mode (i.e., order taking time). Based on these resuts it makes sense to coect additiona order-taking times to reduce input uncertainty, since it has the argest (most negative) sensitivity. To see the effect of having additiona observations, Song and Neson (2014) conducted foowup/diagnostic experiments with increasing numbers of observations of order-taking time (the observations are generated from an exponentia distribution with mean 90 seconds). Figure 1 shows the point estimate and adjusted CI. The trend in γ, indicated by, and the adjusted 95% CIs, are presented for different tota amounts of data. The number of simuation repications for the foow-up experiments and the number of agents at the remote order-taking system remain n = 1,000 and 4, respectivey. We observe that γ decreases as the amount of rea-word data increases and it aso becomes more cear that the adjusted CI incudes 1.5. Therefore, one might consider increasing the number of agents from 4 to 5, which woud be a different fina decision than what woud have been made by performing the nomina experiment ony. 173

13 Tabe 1: Scaed contribution and sampe-size sensitivity resuts for the remote order taking system. Input data m Average Contribution Sensitivity 1 Interarriva Interarriva Interarriva Interarriva Interarriva Interarriva Interarriva Order taking Moving Average waiting time Order taking time sampe size 10 γ^ Figure 1: The adjusted confidence interva for expected customer waiting time aong with γ (indicated by ) for different rea-word sampe sizes for the order-taking time. 5 CONCLUSIONS In this tutoria, we described the input uncertainty probem and investigated different measures of input uncertainty. See Barton (2012) for a more comprehensive ist of the reevant iterature. We put particuar emphasis on contribution and sensitivity measures proposed by Song and Neson (2014) that can be used to decide from which input processes to coect more data. Whie usefu, chaenges arise when we have mutivariate input modes where two or more input random variabes are correated; see Bier and Coru (2011). Aso, questions remain as to how to quantify input uncertainty when we have time series input modes or input mode parameters that vary over time (e.g., a non-homogeneous Poisson process). Athough input uncertainty quantification for point estimation has been an active area of research, itte has been done on the impications of input uncertainty for other anaysis probems. An exception is Coru and Bier (2013) who consider a subset seection probem in the presence of input uncertainty. We expect that many soved probems in stochastic simuation output anaysis wi be revisited taking input uncertainty into account. 174

14 ACKNOWLEDGMENTS This tutoria is based upon work supported by the Nationa Science Foundation under Grant No. CMMI REFERENCES Ankenman, B. E. and B. L. Neson A Quick Assessment of Input Uncertainty. In Proceedings of the 2012 Winter Simuation Conference, edited by C. Laroque, J. Himmespach, R. Pasupathy, O. Rose and A. M. Uhrmacher, Piscataway, New Jersey: Institute of Eectrica and Eectronics Engineers, Inc. Barton, R. R Tutoria: Input Uncertainty in Output Anaysis. In Proceedings of the 2012 Winter Simuation Conference, edited by C. Laroque, J. Himmespach, R. Pasupathy, O. Rose, and A. M. Uhrmacher, Piscataway, New Jersey: Institute of Eectrica and Eectronics Engineers, Inc. Barton, R. R., B. L. Neson and W. Xie Quantifying Input Uncertainty via Simuation Confidence Intervas. INFORMS Journa on Computing 26: Barton, R. R. and L. W. Schruben Resamping Methods for Input Modeing. In Proceedings of the 2001 Winter Simuation Conference, edited by B. A. Peters, J. S. Smith, D. J. Medeiros, and M. W. Rohrer, Piscataway, New Jersey: Institute of Eectrica and Eectronics Engineers, Inc. Bier, B. and C. G. Coru Accounting for Parameter Uncertainty in Large-scae Stochastic Simuations with Correated Inputs. Operations Research 59: Cheng, R. C. H. and W. Hoand Two-point Methods for Assessing Variabiity in Simuation Output. Journa of Statistica Computation and Simuation 60: Cheng, R. C. H. and W. Hoand Cacuation of Confidence Intervas for Simuation Output. ACM Transactions on Modeing and Computer Simuation 14: Coru, C. G. and B. Bier A Subset Seection Procedure under Input Parameter Uncertainty. In Proceedings of the 2013 Winter Simuation Conference, edited by R. Pasupathy, S.-H. Kim, A. Tok, R. Hi, and M. E. Kuh, Piscataway, New Jersey: Institute of Eectrica and Eectronics Engineers, Inc. Freimer, M. and L. W. Schruben Coecting Data and Estimating Parameters for Input Distributions. In Proceedings of the 2002 Winter Simuation Conference, edited by E. Yücesan, C. Chen, J. L. Snowdon, and J. M. Charnes, Piscataway, New Jersey: Institute of Eectrica and Eectronics Engineers, Inc. Homma, T. and A. Satei Importance Measures in Goba Sensitivity Anaysis of Mode Output. Reiabiity Engineering and System Safety 52:1 17. Neson, B. L Foundations and Methods of Stochastic Simuation: A First Course. New York: Springer. Ng, S. H. and S. E. Chick Reducing Input Parameter Uncertainty for Simuations. In Proceedings of the 2001 Winter Simuation Conference, edited by B. A. Peters, J. S. Smith, D. J. Medeiros, and M. W. Rohrer, Piscataway, New Jersey: Institute of Eectrica and Eectronics Engineers, Inc. Ng, S. H. and S. E. Chick Reducing Parameter Uncertainty for Stochastic Systems. ACM Transactions on Modeing and Computer Simuation 16: Oakey, J. E. and A. O Hagan Probabiistic Sensitivity Anaysis of Compex Modes: A Bayesian Approach. Journa of the Roya Statistica Society 66: Pischke, E., E. Borgonovo, and C. L. Smith Goba Sensitivity Measures from Given Data. European Journa of Operations Research 226:

15 Song, E., and B. L. Neson A Quicker Assessment of Input Uncertainty. In Proceedings of the 2013 Winter Simuation Conference, edited by R. Pasupathy, S.-H. Kim, A. Tok, R. Hi, and M. E. Kuh, Piscataway, New Jersey: Institute of Eectrica and Eectronics Engineers, Inc. Song, E. and B. L. Neson Quicky Assessing Contributions to Input Uncertainty. Working Paper, Department of Industria Engineering and Management Sciences, Northwestern University, Evanston, IL, USA. Wagner, H. M Goba Sensitivity Anaysis. Operations Research 43: Xie, W., B. L. Neson, and R. R. Barton A Bayesian Framework for Quantifying Uncertainty in Stochastic Simuation. Working Paper, Department of Industria Engineering and Management Sciences, Northwestern University, Evanston, IL, USA. Zouaoui, F. and J. R. Wison Accounting for Parameter Uncertainty in Simuation Input Modeing. IIE Transactions 35: Zouaoui, F. and J. R. Wison Accounting for Input-mode and Input-parameter Uncertainties in Simuation. IIE Transactions 36: AUTHOR BIOGRAPHIES EUNHYE SONG is a Ph.D. candidate of the Department of Industria Engineering and Management Sciences at Northwestern University. Her research interests are input uncertainty quantification, design of simuation experiments and simuation optimization. Her e-mai address is eunhyesong2016@u.northwestern.edu. BARRY L. NELSON is the Water P. Murphy Professor and Chair of the Department of Industria Engineering and Management Sciences at Northwestern University. He is a Feow of INFORMS and IIE. His research centers on the design and anaysis of computer simuation experiments on modes of stochastic systems, and he is the author of Foundations and Methods of Stochastic Simuation: A First Course, from Springer. His e-mai address is nesonb@northwestern.edu. C. DENNIS PEGDEN is the founder and CEO of Simio LLC. He was the founder and CEO of Systems Modeing Corporation, now part of Rockwe Software, Inc. He has hed facuty positions at the Univrsity of Aabama in Huntsvie and The Pennsyvania State University. He ed in the deveopment of the SLAM, SIMAN, Arena, and Simio simuation toos. He is the author/co-author of three textbooks in simuation and has pubished papers in a number of fieds incuding mathematica programming, queuing, computer arithmetic, scheduing, and simuation. His emai is cdpegden@simio.com. 176