Customized Sequential Designs for Random Simulation Experiments van Beers, W.C.M.; Kleijnen, J.P.C.

Transcription

1 Tlburg Unversty Customzed Sequental Desgns for Random Smulaton Eperments van Beers, W.C.M.; Klenen, J.P.C. Publcaton date: 25 Lnk to publcaton General rghts Copyrght and moral rghts for the publcatons made accessble n the publc portal are retaned by the authors and/or other copyrght owners and t s a condton of accessng publcatons that users recognse and abde by the legal requrements assocated wth these rghts. - Users may download and prnt one copy of any publcaton from the publc portal for the purpose of prvate study or research - You may not further dstrbute the materal or use t for any proft-makng actvty or commercal gan - You may freely dstrbute the URL dentfyng the publcaton n the publc portal Take down polcy If you beleve that ths document breaches copyrght, please contact us provdng detals, and we wll remove access to the work mmedately and nvestgate your clam. Download date: 14. Aug. 218

2 No CUSTOMIZED SEQUENTIAL DESIGNS FOR RANDOM SIMULATION EXPERIMENTS: KRIGING METAMODELING AND BOOTSTRAPPING By Wm C.M. van Beers, Jack P.C. Klenen March 25 ISSN

3 Customzed Sequental Desgns for Random Smulaton Eperments: Krgng Metamodelng and Bootstrappng Wm C.M. van Beers Department of Informaton Systems and Management Tlburg Unversty (UvT), Postbo 9153, 5 LE Tlburg, The Netherlands Phone: ; Fa: ; E-mal: wvbeers@uvt.nl Jack P.C. Klenen Department of Informaton Systems and Management/ Center for Economc Research (CentER) Tlburg Unversty (UvT), Postbo 9153, 5 LE Tlburg, The Netherlands Phone: ; Fa: ; E-mal: klenen@uvt.nl Ths paper proposes a novel method to select an epermental desgn for nterpolaton n random smulaton, especally dscrete event smulaton. (Though the paper focuses on Krgng, ths desgn approach may also apply to other types of metamodels such as lnear regresson models.) Assumng that smulaton requres much computer tme, t s mportant to select a desgn wth a small number of observatons (or smulaton runs). The proposed method s therefore sequental. Its novelty s that t accounts for the specfc nput/output behavor (or response functon) of the partcular smulaton at hand;.e., the method s customzed or applcaton-drven. A tool for ths customzaton s bootstrappng, whch enables the estmaton of the varances of predctons for nputs not yet smulated. The new method s tested through two classc smulaton models: eample 1 estmates the epected steady-state watng tme of the M/M/1 queueng model; eample 2 estmates the mean costs of a termnatng (s, S) nventory smulaton. For these smulatons the novel desgn ndeed gves better results than Latn Hypercube Samplng (LHS) wth a prefed sample of the same sze. Key words: Smulaton: desgn of eperments, statstcal analyss, Krgng, bootstrappng, regresson, C, C1, C9, C15, C44.

4 1. Introducton In ths paper, we focus on epensve smulatons; that s, we assume that a sngle smulaton run takes much computer tme. Consequently, nterpolaton s needed;.e., from the smulated nput/output (I/O) data, the outputs are predcted for nput combnatons not yet smulated. We devse a method that s meant to mnmze the number of smulaton runs for such nterpolaton. We talor our desgn of eperments (DOE) to the actual smulaton; that s, we do not derve a generc desgn such as a classc desgn (for eample, a 2 k p desgn) or a LHS desgn. The dfferences between customzed and generc desgns are as follows (also see Klenen and Van Beers (24), who focus on determnstc smulaton). A metamodel s a model of the I/O functon (or response functon ) mpled by the underlyng smulaton model. We denote the metamodel by Y () where denotes the k- dmensonal vector of the k nputs (factors) so = ( 1,,,, k ). Classc DOE assumes a smple metamodel. For eample, desgns of resoluton III (ncludng certan 2 k p desgns) assume a frst-order polynomal I/O functon. Composte desgns (CCD) assume a secondorder polynomal. These desgns are dscussed for physcal eperments n (for eample) the well-known tetbook Bo, Hunter, and Hunter (1978) and the recent tetbook Myers and Montgomery (22); for smulaton eperments we refer to Klenen (1987). LHS (much appled n Krgng, descrbed below) assumes that an adequate metamodel s more complcated than a low-order polynomal. LHS, however, does not assume a specfc metamodel. Instead, LHS focuses on the desgn space formed by the k dmensonal unt cube, defned by 1 ( = 1,, k) after standardzng (scalng) the nputs. LHS s one of the space fllng desgns: LHS samples that space accordng to some pror dstrbuton for the nputs, such as ndependent unform dstrbutons on [, 1]; see McKay, Beckman, and Conover (1979), and also Klenen et al. (25), Koehler and Owen (1996), and Santner, Wllams, and Notz (23). Unlke LHS, we eplctly account for the I/O functon. Unlke classc DOE, we assume that a low-order polynomal (estmated through regresson analyss) gves an nadequate appromaton of the I/O functon. In our method we estmate the uncertanty of predcted outputs at unobserved nput combnatons (these combnatons are also called scenaros, desgn ponts, combnatons of factor levels, or smulaton nputs). To estmate the uncertanty of these predctons caused by the nose and the shape of the I/O functon we use bootstrappng;.e., we resample the outputs for each scenaro already smulated (for 1

5 bootstrappng n general see the classc tetbook, Efron and Tbshran 1993; for bootstrappng n the valdaton of regresson metamodels n smulaton see Klenen and Deflandre 25). We make our procedure sequental for the followng two reasons. 1. Sequental statstcal procedures are known to be more effcent ; that s, they requre fewer observatons than fed-sample (one-shot) procedures; see, for eample, the handbook by Ghosh and Sen (1991) and the recent artcle by Park et al. (22). 2. Smulaton eperments proceed sequentally (unless parallel computers are used; our procedure also fts parallel computers). The lterature on determnstc smulaton shows several desgns that lke ours account for the specfc smulaton s I/O functon, and are sequental. For eample, Crary (22) dscusses G-optmal and I-optmal desgns, whch the DOE lterature defnes as follows. G-optmal desgns mnmze the mamum Mean Squared Error (MSE) of the predcted output; I-optmal or Integrated MSE (IMSE) desgns mnmze the average MSE (obvously, the MSE reduces to the varance f the predctor s unbased; see (5) and (6) below). Wllams, Santner, and Notz (2, 22) use a Bayesan approach to derve sequental IMSE desgns. Sasena, Papalambros, and Govaerts (22) derve sequental desgns for the optmsaton of determnstc smulaton models. Klenen and Van Beers (24) derve customzed sequental desgns for determnstc smulatons. We, however, focus on DOE for random smulatons, and we seem to be the frst to apply bootstrappng for ths problem. (Random smulaton ncludes Dscrete Event Dynamc Systems or DEDS smulaton such as M/M/1 smulaton, but also smulaton models consstng of stochastc dfference equatons.) We shall see that our desgns select most of ther nput combnatons n sub-areas that have more nterestng I/O behavor. In our frst eample we spend most of our computer smulaton tme on the challengng eplosve part of the metamodel that estmates the mean steady-state watng tme for varous traffc rates of sngle-server queueng systems wth Markovan (Posson) arrval and servce tmes known as the M/M/1 model. (The reader may take a peek at Fgure 1, dscussed n subsecton 5.1.) In our second eample, we estmate the average total costs n an (s, S) nventory model; there are several varatons on ths model, but we take the specfcaton gven by Law and Kelton (2). Agan, we fnd a concentraton of the nput combnatons n the sub-area where the metamodel shows steep slopes. (See Fgure 7, detaled n subsecton 5.2.) In both eamples, we compare our desgns wth LHS; our desgns gve better predctons. 2

6 The remander of ths paper s organzed as follows. Secton 2 summarzes the bascs of Krgng. Secton 3 summarzes DOE and Krgng. Usng the M/M/1 model, secton 4 eplans our method, whch apples bootstrappng to estmate the varances of the Krgng predctons for canddate nputs not yet smulated and sequentally selects as the net nput to be smulated, the one wth the largest bootstrap varance. Secton 5 demonstrates the procedure through two classc eamples: subsecton 5.1 uses M/M/1 smulatons, and subsecton 5.2 uses an (s, S) nventory model wth two nputs. For both eamples our method gves better results than LHS wth a prefed sample sze. Secton 6 presents conclusons and topcs for further research. 2. Krgng bascs Krgng (named after the South-Afrcan mnng engneer Krge) s an nterpolaton method that predcts unknown values of a random functon or random process; see Journel and Hubregts (1978) and Cresse s (1993) classc Krgng tetbook on spatal (geo)statstcs. Whereas spatal statstcs consders the two-dmensonal locaton as the known nput of ths process, smulaton consders the k dmensonal scenaro as nput; see Sacks et al. s (1989) classc artcle on the Desgn and Analyss of Computer Eperments (DACE) these computer eperments concern determnstc smulaton. Random (stochastc) smulaton ncludng DEDS smulatons s the topc of our paper. More precsely, a Krgng predcton s a weghted lnear combnaton of all output values already observed. The weghts depend on the dstances between the new nput to be predcted and the old nputs already observed. Krgng assumes that the closer the nputs are, the more postvely correlated the outputs are. Mathematcal formulatons follow n equatons (1) through (4). Currently, Krgng s frequently appled n determnstc smulaton, whch s much used n engneerng; agan see Sacks et al. (1989); for an update see Smpson et al. (21). In determnstc smulaton, Krgng has an mportant advantage over regresson analyss: the predcted values at old nputs are eactly equal to the observed (smulated) outputs. In random smulaton, however, ths property dsappears. Now, each scenaro s smulated several tmes wth non-overlappng pseudo-random number (PRN) streams. Van Beers and Klenen (23) show that Krgng nterpolates the average output per scenaro. These averages, however, are stll random, so the property that at scenaros already smulated the Krgng predctons equal the averages, loses ts ntutve appeal. Stll, Krgng may be 3

7 attractve because t may decrease the predcton bas (and hence the MSE) at scenaros close together. Indeed, n the eamples presented by Van Beers and Klenen (23) the Krgng predctons are much better than the regresson predctons (regresson analyss may be useful for other goals such as screenng and valdaton; see Klenen et al. 24). Therefore we do not further dscuss regresson analyss n ths paper. Mathematcally formulated, Krgng assumes the followng metamodel: Y ( ) = µ ( ) + δ ( ) wth δ ( ) ~ IID(, σ 2 ( )) (1) where µ () s the mean of the stochastc process Y (), and δ () s the addtve nose, whch s assumed ndependently and dentcally dstrbuted (IID) wth mean zero and varance σ 2 ( ). Ordnary Krgng to whch we lmt ourselves further assumes a statonary covarance process for Y () n (1);.e., the epected values µ () are a constant µ and the covarances of Y ( + h) and Y () depend only on the Eucldean dstance (lag) h = ( + h) ( ). (The assumpton µ () = µ s standard n Ordnary Krgng, and does not mply a flat response surface; see Sacks et al ) Y ˆ( The Krgng predctor for the unobserved (non-smulated) nput (say) denoted by ) s a weghted lnear combnaton of all the n observed outputs: n / Yˆ( ) = λ Y ( ) = Y (2) = 1 n wth = λ = 1,,, ) 1 = λn ( λ 1 and Y ( y 1,, y ). To select these weghts, Krgng = n derves the Best Lnear Unbased Predctor (BLUP), whch (by defnton) mnmzes the MSE of the predctor: { } 2 { MSE( Yˆ( ))} mn E( Y ( ) Yˆ( ) mn = ). (3) λ λ 4

8 Obvously, ths soluton depends on the output s covarances. It can be proven that the optmal weghts n (2) resultng from (3) are / 1 / / = + / (4) wth the followng symbols: s the vector of covarances between the outputs at the nput to be predcted and at the nputs already observed, so = / ( γ ( 1),, γ ( n )) ; / 1 = ( 1,,1) s the vector wth n ones; s the n n matr whose element (, ) s the (co)varance at the nputs already observed γ ) wth, = 1,, n. ( Note that the weghts n (4) vary wth (nput to be predcted), whereas regresson analyss uses the same estmated metamodel for all nputs. Note further that the lterature on (determnstc) smulaton speaks of covarances and correspondng correlatons, whereas the geostatstcs lterature speaks of the varogram, defned as 2γ ( h) = var( Y( + h) Y ( ) ). Snce we shall use the Matlab Krgng toolbo DACE made avalable free of charge by Lophaven, Nelsen, and Søndergaard (22) we avod the term varogram. (Recent alternatve free software s made avalable va see Santner, Wllams, and Notz 23.) We emphasze that n practce the covarances and n (4) are unknown so they must be estmated. The classcal estmator for ) (h s ˆ( h) ( Y( ) ( )) 2 Y /( 2N( h) ) =, where N ( h) denotes the number of dstnct pars n N ( h ) {(, ) : = h}. N( h) = Consequently, the weghts n (4) become random varables (say) ˆ. These weghts make the Krgng predctor resultng from (2) non-lnear. Ths characterstc s often neglected n the Krgng lterature. In general, non-lnear functons of random varables are hard to analyze a smple computer-ntensve soluton s bootstrappng; see Efron and Tbshran (1993). Ignorng the randomness of the estmated optmal weghts ˆ tends to underestmate the true varance of the Krgng predctor. For eample, n the bvarate normal case ths 2 follows from the formula for the condtonal varance, namely var( Y X ) = (1 ρ ) var( Y ) ; see, for eample, Kreyszg (197, p. 343). To tackle ths problem, Cresse (1993, p. 146) 5

9 proposes cross-valdaton. Cross-valdaton s also used by Klenen and Van Beers (24) for determnstc smulaton. For determnstc smulaton, Den Hertog, Klenen, and Sem (25) apply parametrc bootstrappng assumng normally dstrbuted predcton errors and fnd that gnorng the randomness of the Krgng weghts leads to serous errors. Because random smulaton may have non-normal outputs (for eample, queueng smulatons have dstrbutons wth heavy rght-hand tals), we use dstrbuton-free bootstrappng as we shall eplan n Secton DOE and Krgng By defnton, an epermental desgn s a set of n combnatons of k factor values. These combnatons are usually bounded by bo constrants: a b wth a, b R and = 1,, k. The set of all feasble combnatons s called the epermental regon (say) H. We suppose that H s a k-dmensonal unt cube, after rescalng the orgnal rectangular area (see Secton 1). Our goal s to fnd the best desgn for Krgng predctons wthn H; the Krgng lterature proposed several crtera (see Sacks et al. 1989, p. 414). Most of these crtera are based on the predctor s MSE. Most progress has been made for the IMSE (see Bates et al. 1996): ( Yˆ( ) ) φ( ) d IMSE = MSE (5) H where MSE follows from (3), and φ () s a gven weght functon usually assumed to be a constant. To evaluate a desgn, Sacks et al. (1989, p. 416) compare the predctons wth the known output values of a test set consstng of (say) N nputs. Assumng a constant φ () n (5), the IMSE can then be estmated by the Emprcal IMSE (EIMSE): EIMSE = 1 N N = 1 2 ( ˆ ( ) ( ) ). y y (6) Besdes ths EIMSE, we wll also study the mamum MSE; that s, we also consder rsk-averse users (also see Van Groengen, 2). So IMSE defned n (5) s replaced by 6

10 { MSE( Yˆ( ) )} MaMSE = ma (7) H and EIMSE n (6) by EMaIMSE = ma {1,..., m} 2 {( yˆ ( ) y ( ) ) }. (8) 4. Sequental DOE We devse the followng sequental DOE procedure wth eght steps, whch we llustrate through the M/M/1 model wth epermental regon H = { ρ :.1 ρ.9} where ρ denotes the traffc rate. Step 1. We start wth a small plot desgn wth (say) n nput combnatons; for eample, n = 5. We select the specfc n values such that they are equally spread over the epermental regon. There are varous space fllng desgns; for eample, LHS desgns. In the frst eample n Secton 5 namely the M/M/1 we use a mamn desgn, whch (by defnton) mamzes the mnmum dstance between any two ponts of the desgn; see Koehler and Owen (1996, p. 288). So n ths eample, we select the traffc rates {.1,.3,.5,.7,.9} ( = 1,, 5 ). Step 2: For each nput value, we ntally generate (say) m IID replcates because bootstrappng requres IID observatons; see Efron and Tbshran (1993). To obtan IID observatons n our M/M/1 smulaton eample, we apply renewal (regeneratve) analyss (see, for eample, Klenen and Van Groenendaal 1992, and Law and Kelton 2). As the renewal state, we choose the dle (empty) state. We therefore start the smulaton run n the empty state for each traffc rate. Net we observe m cycles each wth (random) cycle lengths (say) L (the hgher, the hgher L tends to be). Besdes the m cycle lengths ( = 1,, m ) per traffc rate, we observe the sum of the watng tmes over that cycle: L ; L ; sw ; = w ; t t = 1 ; ( = 1,, n ; = 1,, m ). (9) 7

11 To reduce the varance when comparng the (random) outputs for dfferent nputs (.e., to mprove the sgnal/nose rato), we use common random numbers (CRN). Ths s a popular varance reducton technque (VRT). It s well known that n M/M/1 smulaton the varance decreases substantally f the PRN (say) r t are manpulated as follows: successve PRN are used alternatvely to smulate the arrval tme (say) a and the servce tme s ; n other words, a t = ln r2 t 1E( a) and s t = ln r 2t E( s) (t = 1, 2, ). The correlaton coeffcents for the average watng tmes of two neghborng traffc rates turn out to be very hgh, namely roughly.99. To generate the PRN, we use the Matlab command rand. To ntalze the PRN, we set the Matlab generator (rather arbtrarly) to ts ntal state s =. The Matlab web ste further states: The unform random number generator n MATLAB 5 (and above) uses a lagged Fbonacc generator, wth a cache of 32 floatng pont numbers, combned wth a shft regster random nteger generator. The nteger generator uses shfts and eclusve OR s. ; see ( and also Moler (1995). For further detals on CRN, VRT, and PRN we refer to Law and Kelton (2). Step 3. Based on these m bvarate IID outputs ( L ;, sw ; ) ( = 1,, m ) per nput value, we estmate the mean watng tmes through m sw ; = 1 y ( m) = m. (1) L = 1 ; Ths rato estmator s consstent; for references see agan Klenen and Van Groenendaal (1992) and Law and Kelton (2). We do not try to mprove the small-sample performance of ths estmator (for eample, through ackknfng whch s closely related to bootstrappng), because ths estmator suffces for our Krgng metamodel. To estmate the precson of the estmate defned n (1), we use the followng probablty statement that holds asymptotcally per nput value : σˆ σˆ m m P y ( m ) t E( w ) y ( m ) + t = α α α 1 m 1; 1 2 m 1; 1 2 (11) L L 8

12 2 2 m where σ ˆ = vâr( sw ) + y vâr( L ) 2y côv( sw, L ) and L = L m ; agan see Klenen / = 1 ; and Van Groenendaal (1992). Note that ths nterval does not have an asymptotc ont (or epermentwse) probablty (1- α ) over all smulated nput values. Net, we add replcates one-at-a-tme sequental samplng untl the desred halfwdth of the nterval n (11) has reduced to a prefed relatve error (say) δ ; for eample, δ =.15 (agan see Klenen and Van Groenendaal 1992 and Law and Kelton 2). We denote the fnal number of replcates per nput by nput based on m replcates; see (1) wth m replaced by m. Ths gves the average output y m ) per m. ( Step 4. Based on these n average outputs y ( m ) for the n nputs, we compute the Krgng predctors for the epected outputs of a new set of (say) c n canddate nput values c g (g = 1,, c n ). We agan select these canddates n a space-fllng way; n the M/M/1 eample, we choose the canddate nputs halfway between two old neghborng nputs c so we avod etrapolaton: ( + ) 1 2 (wth g = 1,, n 1 ). g = g g + By defnton, the Krgng predctor s a weghted lnear combnaton of all outputs already observed; see (2). So now Krgng weghts the n values already observed n steps 1 through 3: yˆ( c g n ) = λ y( ) (12) = 1 n wth = 1 λ = 1. To estmate the weghts λ n (12), Krgng uses the old data set (, y ( m )) ( = 1,, n ). To estmate the varance of ths non-lnear predctor, we use bootstrappng as follows. Step 5. Per nput, we bootstrap the m bvarate IID outputs ( L ;, sw ; );.e., we resample wth replacement the outputs resultng from steps 1 through 3. We denote these bootstrap observatons by the superscrpt * (as s tradtonal n the bootstrap lterature): * * * * {( sw, L ),, ( sw, L )}. (13) ; 1 ; 1 ; m ; m 9

13 Usng these bootstrapped observatons and (1), we compute the bootstrap averages: m = 1 m = 1 sw * ; * y ( m ) =. (14) * L ; * Usng the bootstrapped I/O data (, y ( m )) ( = 1,, n ) and (12), we compute the bootstrapped Krgng predctor: * c yˆ ( g n * * ) = λ y ( ). (15) = 1 * We agan estmate the bootstrap weghts λ n (15) through the Matlab Toolbo DACE; see Secton 2. Note that DACE ams to obtan the mamum lkelhood estmator (MLE) of the Krgng weghts * λ n (15). For the numercal search that leads to ths MLE, DACE uses startng values. As startng values, we use the MLE for λ based on the orgnal I/O data n (12). Step 6. The resamplng per nput n step 5 s repeated (say) B tmes (ths B s called the bootstrap sample sze). Hence, (13) through (15) gve y ( ) wth b = 1,, B. ˆ * b c g For each of the c n canddate nputs c g, we compute the bootstrap varance of the Krgng predctor y ˆ c* g at c g : B c* 1 c* c* 2 v âr( yˆ g ) = ( yˆ g; b yˆ g ) (16) B 1 b = 1 where * ˆ c; g b y s the predcted value at canddate nput c g based on the bootstrapped I/O data * c* B c* (, y ( m )) ( = 1,, n ) ; b and yˆ = g b = yˆ B 1 g; b. varance (16): Step 7. We determne whch canddate nput has the largest bootstrap predcton 1

14 c* { vâr( yˆ )} v = arg ma g, (17) c g {1,..., n } and we add ths wnnng nput c v to the old desgn. Now, we run the smulaton model wth ths nput c v untl we have m replcates for ths nput. We stll apply CRN (so we ntalze the PRN wth the seed s ). Furthermore, we agan start wth the empty system as the renewal state. We contnue the smulaton untl the confdence nterval reaches the threshold δ ; see (11). Step 8. We repeat the steps 4 through 7 untl we have reached a stoppng crteron. In other words, we bootstrap the old I/O set augmented wth the canddate selected n step 7. We select a new set of canddates. For these canddates, we compute the Krgng predctors and ther bootstrap varances. Alternatve stoppng crtera may be: () the computer budget has been ehausted, () the proect has reached ts deadlne, () the precson of the Krgng metamodel s acceptable. We observe that addng one pont at a tme as we do n our sequental DOE s not necessarly optmal. However, t s a smple albet myopc heurstc; also see Banevc and Swtzer (22), who refer to Ferr and Pccon (1992). 5. Two eamples We test our customzed sequental desgn (CSD) through two classc academc smulaton models, namely the M/M/1 model and an (s, S) model M/M/1 model An M/M/1 has as true I/O functon the hyperbole y = wth < < 1 (18) 1 where y denotes the epected steady-state watng tme assumng a unt servce rate, and denotes the traffc rate. We apply the procedure descrbed n secton 4, selectng the followng parameters. 11

15 Step 1: We select a plot desgn of sze n = 5. Step 2: We obtan m 1 replcates to get ntal estmates of the varances; we select as the = ntal PRN seed s =. Step 3: We eperment wth two values for the precson, namely =.5 and =.15, and two values for the type-i error rate, namely =.1 and.5 so (11) gves four confdence ntervals. For hgher traffc rates (say, >.7), the numbers of cycles and the cycle lengths may be very large. To lmt computer tme, we lmt the number of cycles ( lmt preserves the renewal property, but may decrease the precson. Step 6: We eperment wth the bootstrap sample szes: B = 5 and B = 1. L ; ) to 1. Ths Step 8: We eperment wth a stoppng crteron that specfes that the total desgn sze s ether n = 1 or n = 5. Fgure 1 dsplays smulaton results for both our desgn and a LHS desgn. Ths fgure s based on the confdence ntervals n (11) wth =.5 and =.15. The bootstrap sample sze s only B = 5. The stoppng crteron s that n = 1 traffc rates have been smulated. Ths fgure corresponds wth one scenaro (labeled 1) of the eght scenaros n our eperment; see Table 1b below. LHS turns out to smulate fewer challengng nputs;.e., hgh traffc rates. Insert Fgure 1 To evaluate our procedure, we use a test set wth N = 32 equdstant traffc rates, namely {.1125,.1375,,.8875} (Sacks et al also use test sets to evaluate ther procedure). We compare the Krgng predctons of the two desgns wth the true outputs of the test set, computed from (18). (The two desgns may contan some members of the test set, but we gnore ths phenomenon.) Fgure 2 llustrates the 32 predctons for replcate 1 of scenaro 1. Insert Fgure 2 To compare the predctons of our desgn and LHS, we mght use the EIMSE crteron, defned n (6). However, the fnal numbers of replcates n the two desgns may dffer, so we calculate the corrected EIMSE, denoted by e later on: 12

16 1 e = CEIMSE = C n t n t t t 2 ( yˆ( ) ) y( ) = 1, (19) where C s the rato of the total number of replcates n the LHS desgn and n our desgn, n t s the number of I/O combnatons n the test set (so n = 32 ), and test set. t t s the th nput of the We compute ths crteron for eght scenaros;.e., eght combnatons of values of the type-i error rate α, the relatve error δ, the bootstrap sample sze B, and the fnal desgn sze n. These scenaros are specfed through a k p 2 desgn wth k = 4 and p = 1. Ths desgn s epressed n standardzed values n Table 1a (see Klenen and Van Groenendaal 1992); note that all columns are orthogonal. The orgnal values are dsplayed n Table 1b. Insert Tables 1a and 1b To decrease the randomness of CEIMSE n (19), we replcate each scenaro n Table 1 R = 5 tmes. To ensure that the PRN streams do not overlap, we start Matlab s PRN generator n the ntal state s = (usng the command RAND('state', )) n the frst replcaton of each scenaro. Net we save the generator s state of the scenaro that requres the largest number of smulaton runs; we use that state as the ntal state for each of the eght scenaros n the net replcaton, and so on. Table 2a shows the R = 5 CEIMSEs per scenaro, denoted by e r ( r = 1,, R), for the Customzed Sequental Desgn; Table 2b shows e r for LHS. Insert Tables 2a and 2b We analyze the results n Table 2 as follows. Comparng Tables 2a and 2b shows that our desgns do not have smaller CEIMSE than LHS desgns, n all cases (scenaros and replcates). More precsely, our desgns gve better results only f the desgn sze n s small ; see the scenaros 1, 5, 6, and 7. But t s eactly these cases that we are nterested n, snce (as we stated n Secton 1) we focus on epensve smulatons, whch mply that bg desgn szes are nfeasble. So, we compute the dfferences 13

17 d e e ; r ; r; LHS ; r; CSD = wth = 1,, 8; r = 1,, 5. (2) Lumpng all scenaros together, the Student t test does not gve sgnfcant dfferences at a type-i error rate of 5% (the varaton of the dfferences d ; s large). However, Fgure 3 r suggests that each of the four scenaros wth small n (desgn sze) gves sgnfcantly postve dfferences. We therefore nvestgate whch factors eplan the performance of our desgn relatve to LHS, as follows. Insert Fgure 3 Remember that we have the k = 4 factors correspondng wth α, δ, B, and n. So we estmate the frst-order polynomal, whch has the man effects β : k + β + ε r = 1 d r = β. (21) We wsh to account for varance heterogenety: 2 var( ε ) σ. Moreover we use CRN, so d r and ( = 1,, 8) are not ndependent. Therefore we compute the OLS estmator of the d r parameters n (21) per replcaton: ˆ = ( X X ) X (22) 1 r d r where X s the 8 5 matr followng from (21) and Table 1a. Ths gves the average OLS estmator based on all R = 5 replcatons: R ˆ 1 = ˆ r. (23) R 1 r= Hence the standard error for the th man effect s R ˆ ˆ 2 R s ˆ ( β ; r β ) ( 1) ( β ) r s( ˆ = 1 β ) = =, (24) R R 14

18 VRWKH6WXGHQWVWDWLVWLFZLWK R 1 degrees of freedom s t ; ν ˆ β β =. (25) s( ˆ β ) Ths statstc assumes normalty, whch probably holds because the Central Lmt Theorem may be appled. The classc null-hypothess s that β = ( = 1,, 4) n (21). We dsplay the correspondng t-statstcs defned by (25) n Table 3 for three values of the type-i error rate, namely.1,.5, and.1. Insert Table 3 Table 3 shows that the desgn sze n (factor 4) has a sgnfcant negatve effect on the dfference d (for any of the three type-i error rates);.e., the advantage of our desgn becomes smaller as the desgn sze n ncreases. Further, the bootstrap sample sze B (factor 3) has no sgnfcant effect: our procedure uses the bootstrap only to estmate whch canddate nput has the largest varance of the Krgng predctor; see (17). So n practce the smaller sze, B = 5, may be used. (Most bootstrap applcatons requre the estmaton of the whole dstrbuton functon, so B s much hgher than 5; for eample, B = 1.) Changes n and (factors 1 and 2) affect the number of replcates, but ths effect s ncorporated n CEIMSE va the factor C; see (19). Rsk-averse users may be guded by EMaIMSE, defned n (8). Agan, our desgns outperform LHS desgns for the smaller desgn szes n. Table 4a shows the fve EMaIMSE values for scenaro, denoted by values for LHS; Fgure 4 shows the dfferences, d ma e for our desgn, and Table 4b shows the analogous ;r = e e. ma ma ma ; r ; r; LHS ; r; CSD Insert Tables 4a and 4b, and Fgure 4 Note that m (number of requred cycles) ndeed ncreases wth (traffc rate). For eample, for the precson requrements =.5 and =.15, =.1 requres 489 cycles, 15

19 whereas =.9 requres the mamum number of cycles, namely 1; see Fgure 5. Moreover, a cycle s lkely to be longer as the traffc rate ncreases. For eample, f =.1 then the average cycle length s L = 4. 8 for m 1 replcates; f =.9 then L = =. For a hgh traffc rate, the mamum number of cycles (1) s reached, n ths fgure. For hgher accuracy ( =.5) ths mamum s also reached for moderate traffc rates. Insert Fgure 5 A queston about our desgn mght be: s the concentraton of the smulaton runs n the nput range wth hgh traffc rates caused by the hgh sgnal (E(y)) or the hgh nose (var(y)) (both the mean and the varance of the M/M/1 s steady-state watng tme ncrease wth the traffc rate)? To answer ths queston, we run some Monte Carlo eperments nspred by the M/M/1 model. In these eperments we use the relatve precson δ =.15, the type-i error rate α =.5, and the fnal desgn sze n = 15. We use the same PRN seed for the same macro-replcate of the four eperments. We run s macro-replcates; the results across the s macro-replcates look very much alke, so to save space we do not dsplay the fgures for all macro-replcates; Fgure 6 gves results for one macro-replcate. Insert Fgure 6 (a) Increasng sgnal and constant nose: y = /( 1 ) + r wth.1. 9 and r U ( 1, 1) ; n other words, the sgnal follows (18), but the nose s unformly dstrbuted between 1 and 1, for any nput value. Fgure 6(a) shows that our desgn allocates ts runs to the area wth rapdly changng sgnal as our desgn dd for the M/M/1 n Fgure (b) Constant sgnal and ncreasng nose: y = 5 + 1r. Fgure 6(b) shows that our desgn agan allocates ts runs to the hgh nput values wth hgh nose. (c) Constant sgnal and constant nose: y = 5 + r. Fgure 6(c) shows that now our desgn spreads ts runs unformly across the epermental area. (d) Increasng sgnal and decreasng nose: y = /( 1 ) + r /(1). Fgure 6(d) shows that now our desgn allocates most of ts runs to the mddle of the epermental area. Our eplanaton s that the ncreasng sgnal pulls the runs to the hgh nput values, whereas the decreasng nose pulls them to the low values so that the net result s a compromse. 16

20 5.2. (s, S) nventory model In an (s, S) model (wth s < S) wth random demand D, the nventory I s replenshed to the order up-to level S whenever the nventory decreases to a value smaller than the reorder level s;.e., the order quantty Q s S I f I < s Q = f I s. There are several varatons on ths basc model, but we smulate Law and Kelton (2, p. 6, 651) s eample 12.9 whch has the followng features. Tmes between demands are IID eponental random varables wth a mean of.1 month. If a demand arrves, ts sze s gven by the probablty functon D Pr{D} The nventory s revewed at the begnnng of each month. Law and Kelton defne an aulary varable d = S s to estmate the optmal values for s and S; the (re)order quantty, however, s not a fed quantty (the order quantty Q vares wth the actual nventory poston, defned as stock on hand, mnus customer backorders, plus outstandng suppler orders; see Bashyam and Fu (1998)). The lead-tme of an order s unformly dstrbuted between.5 and 1 month. Demand s satsfed mmedately f the nventory level I s at least as large as the demand sze D. Otherwse, the demand s possbly partly backlogged and delvered as soon as the nventory s replenshed. The backlog costs are $5 per month per tem backlogged. Holdng costs per tem per month are $1. Orderng costs consst of a setup cost of $32 per order plus ncremental costs of $3 per tem. Law and Kelton smulate the system for 12 months, startng wth an ntal nventory I ( ) = 6 ;.e., ths smulaton model s termnatng (eample 1 estmates a steady-state mean of an M/M/1). Law and Kelton obtan fve replcates for each of the 36 combnatons formed by s =, 2, 4, 6, 8, 1 and d =, 2, 4, 6, 8, 1. Based on these 18 I/O data, they ft the followng second-order polynomal regresson (meta) model for the average monthly total costs called R: 17

21 R ˆ( s, d) = s 1.24d +.14sd +.7s.1d. (26) They compare ths model s predctons wth the true E (R) estmated from 1 replcates for each of 42 new and old combnatons formed by s =, 5, 1,, 1, and d = 5, 1, 15,, 1. We, however, replace (26) by a Krgng model, ftted to the same I/O data (mplyng 36 average outputs), and compare our Krgng predctons wth the true outputs. We fnd that our Krgng model gves more accurate predctons than the regresson model (26); see the Append for detals. Net, we change the desgn from Law and Kelton s grd (wth 16 combnatons of the two nputs s and d wth ( s, d) [2, 8] [2, 8] ) nto our desgn (wth the same fnal desgn sze, namely 16); see Fgure 7. Insert Fgure 7 Lke Law and Kelton, we obtan 5 replcatons per nput combnaton. Net, we ft a Krgng model, and predct 81 true outcomes for the test set ( s, d) {1, 2,, 9} {1, 2,, 9} (a subset of Law and Kelton s true set). Agan, we calculate EIMSE and EMaIMSE defned n (6) and (8). To reduce nose, we repeat ths procedure 5 tmes (usng non-overlappng PRN streams) for our desgns and LHS. Our desgns gve substantal better EIMSE and EMaIMSE; see Table 5. Insert Table 5 We conclude that n ths eample, our sequental desgn also gves more accurate Krgng predctons than LHS wth a fed desgn sze. 6. Conclusons and future research In practce, smulaton often requres much computer tme per run (or replcate) so t s desrable to have an effcent epermental desgn for nterpolaton. It s well known n mathematcal statstcs that sequental desgns are more effcent than fed-sample desgns. Our specfc sequental desgns add as the net nput to be smulated, the nput wth the 18

22 mamum estmated varance for the output predcted at specfc canddate nputs. To obtan such predctons, we use Krgng; to estmate the varances of the Krgng predctors, we use bootstrappng. We appled ths procedure to estmate () the epected steady-state watng tme n M/M/1 smulaton, and () the epected cost n termnatng nventory (s, S) smulaton. We compared the Krgng predcton errors of our sequental desgns and those of fed-sample LHS. Our results show that our procedure gves ndeed smaller predcton errors. In future research, (asymptotc) proofs of the performance of our procedure mght be derved. More epermentaton and analyses may be done to derve rules of thumb for our procedure s parameters, such as the ntal desgn sze n and the ntal number of replcates m. Our procedure may be appled to eamples more complcated than the M//M/1 queueng model or the (s, S) nventory model. Stoppng rules based on a measure of accuracy or precson may be nvestgated. Besdes LHS, other desgns wth prefed szes may be eplored; for eample, mn-ma desgns. Besdes Ordnary Krgng, other metamodels may be used to analyze the I/O data. For eample, the optmal weghts n Ordnary Krgng assume that the predctors equal the average outputs at the nputs already observed; droppng ths constrant mples that new Krgng software must be developed. New Krgng weghts may be derved, replacng the IMSE crteron by the mamum squared error crteron. Besdes Krgng, other nterpolaton models may be used; for eample, lnear or nonlnear regresson metamodels. We focus on senstvty analyss; searchng for the optmal nput of the smulaton model requres further research. Append Law & Kelton s (2, p. 651) data set conssts of 5 replcates for each of the 16 nput combnatons formed by s {2, 4, 6, 8} and d {2, 4, 6, 8} (ths set s a subset of the one n the man tet). Based on ths nput set, we fnd the followng estmates ˆ = ( ,.263,.533,.88,.52,.38), whch agrees wth ther values up to two decmals. As a test set (used to compare regresson and Krgng metamodels), we use ther true I/O set, whch conssts of 1 replcates of each of 42 = 21 2 nput combnatons wth 19

23 s {, 5, 1,, 1} and d { 5, 1, 15,, 1}. For the regresson model we fnd an EIMSE of 145.5, whereas for the Krgng model we fnd an EIMSE of So Krgng does result n a smaller EIMSE. Ths EIMSE, however, s stll rather large, because we have to etrapolate the data outsde the regon [ 2,8] [2,8]. In general, we strongly recommend avodng etrapolaton when fttng a metamodel; ndeed, n smulaton t s easy to avod etrapolaton because we can select our own nput combnatons. Law and Kelton also use a data set consstng of 18 I/O combnatons, namely 5 replcates for each of 36 nput combnatons wth s {, 2, 4, 6, 8, 1} and d {, 2, 4, 6, 8, 1}. We use ther computer program (mported from ther web page to generate the output. Agan, we ft both a second-order regresson model and a Krgng model. We compare the two ftted models va the true data set. For the regresson model, we fnd an EIMSE of 152., whereas for the Krgng model we fnd an EIMSE of only 14. (n ths case etrapolaton s ndeed avoded. Acknowledgements We thank Dck den Hertog (Tlburg Unversty) for hs comments on an earler verson, whch lead to the addtonal Monte Carlo eperments reported n Secton 5, and Ruud Brekelmans (Tlburg Unversty) for helpng us to mport Law and Kelton s C-program codes nto our Matlab program. References Banevc, M. and P. Swtzer (22), Bayesan network desgns for varance as a functon of the locaton. Proceedngs of the 22 JSM Conference, Secton on Statstcs and the Envronment, New York, NY Bashyam, S. and M.C. Fu (1998), Optmzaton of (s, S) nventory systems wth random lead tmes and a servce level constrant. Management Scence. 44, no. 12, pp Bates, R.A., R.J. Buck, E. Rccomagno and H.P. Wynn (1996), Epermental desgn and observaton for large systems. Royal Statstcal Socety. 58, no. 1, pp Bo, G.E.P., W.G. Hunter and J.S. Hunter (1978), Statstcs for epermenters: an ntroducton to desgn, data analyss and model buldng. John Wley & Sons, Inc., New York 2

24 Crary, S.B. (22), Desgn of computer eperments for metamodel generaton, Analog Integrated Crcuts and Sgnal Processng, 32, pp Cresse, N.A.C. (1993), Statstcs for spatal data. John Wley & Sons, Inc., New York Efron, B. and R.J. Tbshran (1993). An ntroducton to the bootstrap. Chapman & Hall, New York Ferr, M. and M. Pccon (1992), Optmal selecton of statstcal unts. Computatonal Statstcs & Data Analyss, 13, pp Ghosh, B.K. and P.K. Sen (edtors), 1991, Handbook of sequental analyss. Marcel Dekker, Inc., New York Den Hertog, D., J.P.C. Klenen, and A.Y.D. Sem (25), The correct Krgng varance estmated by bootstrappng. Journal of the Operatonal Research Socety (accepted; preprnt: Journel, A.G. and C.J. Hubregts (1978), Mnng geostatstcs, Academc Press, London Klenen, J.P.C. (1987), Statstcal tools for smulaton practtoners. Marcel Dekker, Inc., New York Klenen, J.P.C. and D. Deflandre (25), Valdaton of regresson metamodels n smulaton: Bootstrap approach. European Journal of Operatonal Research (n press) Klenen, J.P.C., S.M. Sanchez, T.W. Lucas and T.M. Coppa (25), A user s gude to the brave new world of desgnng smulaton eperments. INFORMS Journal on Computng (accepted as State-of-the-Art Revew) Klenen, J.P.C. and W.C.M. van Beers (24), Applcaton-drven sequental desgns for smulaton eperments: Krgng metamodelng. Journal of the Operatonal Research Socety, no. 55, pp Klenen, J.P.C. and W. van Groenendaal (1992), Smulaton: a statstcal perspectve. John Wley, Chchester (England) Koehler, J.R. and A.B. Owen (1996), Computer eperments. Handbook of statstcs, by S. Ghosh and C.R. Rao, vol. 13, pp Kreyszg, E. (197), Introductory mathematcal statstcs: prncples and methods. John Wley & Sons, Inc., New York Law, A.M. and W.D. Kelton (2), Smulaton modelng and analyss, thrd edton, McGraw-Hll, Boston Lophaven, S.N., H.B. Nelsen and J. Søndergaard (22), A Matlab Krgng toolbo. Techncal report IMM-TR-22-12, Techncal Unversty of Denmark 21

25 McKay, M.D., R.J. Beckman and W.J. Conover (1979), A comparson of three methods for selectng values of nput varables n the analyss of output from a computer code. Technometrcs, 21, no. 2, pp (reprnted n 2: Technometrcs, 42, no. 1, pp Moler, C. (1995), Random thoughts. MATLAB News & Notes, pp Myers, R.H. and D.C. Montgomery (22). Response surface methodology: process and product optmzaton usng desgned eperments; second edton. Wley, New York Park, S., J.W. Fowler, G.T. Mackulak, J.B. Keats, and W.M. Carlyle (22), D-optmal sequental eperments for generatng a smulaton-based cycle tme-throughput curve. Operatons Research, 5, no. 6, pp Sacks, J., W.J. Welch, T.J. Mtchell and H.P. Wynn (1989), Desgn and analyss of computer eperments. Statstcal Scence, 4, no. 4, pp Santner, T.J., B.J. Wllams, and W.I. Notz (23), The desgn and analyss of computer eperments. Sprnger-Verlag, New York Sasena, M.J, P. Papalambros, and P. Goovaerts (22), Eploraton of metamodelng samplng crtera for constraned global optmzaton. Engneerng Optmzaton 34, no.3, pp Smpson, T.W., T.M. Mauery, J.J. Korte, and F. Mstree (21), Krgng metamodels for global appromaton n smulaton-based multdscplnary desgn optmzaton. AIAA Journal, 39, no. 12, 21, pp Van Beers, W. and J.P.C. Klenen (23), Krgng for nterpolaton n random smulaton. Journal of the Operatonal Research Socety, no. 54, pp Van Groengen, J.W. (2), The nfluence of varogram parameters on optmal samplng schemes for mappng by Krgng. Geoderma, no. 97, pp Wllams, B.J., T.J. Santner, and W.I. Notz (2), Sequental desgn of computer eperments to mnmze ntegrated response functons, Statstca Snca, 1, Wllams, B.J., T.J. Santner, and W.I. Notz (22), Sequental desgn of computer eperments for constraned optmzaton of ntegrated response functons, Workng Paper. Oho State Unversty 22

26 ---: True I/O functon * : Smulated output for Customzed Sequental Desgn O: Smulated output for Latn Hypercube Desgn y Fgure 1: Two desgns for M/M/1 wth 1 traffc rates and average smulaton outputs y 23

27 ---: True I/O functon * : Customzed Desgn predcton O: LHS predcton ŷ t Fgure 2: Predctons ŷ for the test set for M/M/1, for two desgns n replcate 1 of scenaro 1 24

28 .4.3 d scenaro Fgure 3: Dfferences d ; r e ; r; LHS e ; r; CSD = for scenaro = 1,, 8 and replcate r = 1,, d scenaro Fgure 4: Dfferences d = e e for scenaro = 1,, 8 and replcate r = 1,, 5 ma ma ma ; r ; r; LHS ; r; CSD 25

29 m Fgure 5: Number of cycles m per traffc rate for M/M/1, gven DQG 26

30 (a) constant nose (b) ncreasng nose y, yˆ y, ˆ y y (c) constant nose (d) decreasng nose y, yˆ y, yˆ y y Fgure 6: Monte Carlo eperments wth four combnatons of sgnal and nose functons; --- denotes sgnal and *** denotes I/O of Customzed Sequental Desgn 27

31 R Fgure 7: I/O smulaton data for (s, S) nventory model wth 16 scenaros denoted by { 28

32 Table 1a: A desgn epressed n standardzed factor values factor B n scenaro = Table 1b: Eght scenaros or combnatons of type-i error rate α, relatve error δ, bootstrap sample sze B, and fnal desgn sze n scenaro B n

33 Table 2a: CEIMSE e r for Customzed Sequental Desgns n 8 scenaros replcated 5 tmes, computed from test set wth 32 values scenaro e 1 e 2 e 3 e 4 e Table 2b: CEIMSE e r for LHS desgns n 8 scenaros replcated 5 tmes, computed from test set wth 32 values scenaro e 1 e 2 e 3 e 4 e

34 Table 3: Sgnfcance of estmated man effects t-statstc two-sded sgnfcance level t α =. 1 α =. 5 α =. 1 ;ν β sgnfcant sgnfcant not sgnf. β sgnfcant sgnfcant not sgnf. β not sgnf. not sgnf. not sgnf. β sgnfcant sgnfcant sgnfcant 4 βˆ 31

35 ma Table 4a: EMaIMSE e r for Customzed Sequental Desgns n 8 scenaros replcated 5 tmes, computed from test set wth 32 values EMaIMSE e for Customze Sequental Desgns scenaro e 1 e 2 e 3 e 4 e ma Table 4b: EMaIMSE e r for LHS desgns n 8 scenaros replcated 5 tmes, computed from test set wth 32 values EMaIMSE e for LHS scenaro e 1 e 2 e 3 e 4 e

36 Table 5: EIMSE and EMaIMSE for CSD and LHS for (s, S) nventory smulaton, based on test set wth 81 true values CSD LHS replcate EIMSE EMaIMSE EIMSE EMaIMSE