ANALYSIS OF SIMULATION EXPERIMENTS BY BOOTSTRAP RESAMPLING. Russell C.H. Cheng

Transcription

1 Proceedngs of the 00 Wnter Sulaton Conference B. A. Peters, J. S. Sth, D. J. Mederos, and M. W. Rohrer, eds. ANALYSIS OF SIMULATION EXPERIMENTS BY BOOTSTRAP RESAMPLING Russell C.H. Cheng Departent of Matheatcs Unversty of Southapton Southapton, SO7 BJ, U.K. ABSTRACT Ths tutoral consders soe very general procedures for analysng the results of a sulaton experent usng bootstrap resaplng. Bootstrappng has coe to be recognsed n statstcs as beng far rangng and effectve. However t s not so well known n sulaton despte beng deally suted for use n such a context. We dscuss aspects rangng fro the eleentary to the advanced. We descrbe the ratonale and the sple steps needed to pleent bootstrappng n () estaton of the dstrbutonal propertes of the output and ts dependence on factors of nterest; () odel fttng; () odel selecton; (v) odel valdaton; (v) senstvty analyss. INTRODUCTION Ths tutoral consders soe very general procedures for analysng the results of a sulaton experent The procedures to be descrbed and dscussed ake use of resaplng. Such ethods have coe to be recognsed n statstcs as far rangng and effectve. However they are not so well known n sulaton despte beng deally suted for use n such a context. Though the underlyng ethodology s essentally well-known t s possble to take advantage of partcular features of sulaton experents to splfy or to prove the accuracy or the effcency of the analyss. We dscuss aspects rangng fro the eleentary to the advanced. We wll dscuss () estaton of the dstrbutonal propertes of the output and ts dependence on factors of nterest; () odel fttng; () odel selecton; (v) odel valdaton; (v) senstvty analyss. Although t s not strctly necessary, t s convenent to conduct ths dscusson n a unfed fraework by adoptng a etaodellng approach whch uses a regresson odel coprsng a deternstc regresson functon and a stochastc error process to descrbe the behavour of the output. All the aspects prevously entoned can be convenently dscussed wthn ths fraework. In Secton we descrbe the classcal bootstrap and also a paraetrc verson based on a etaodellng approach. Secton 3 descrbes the sulaton experental layout. Secton 4 descrbes calculaton of dstrbutonal propertes of the sulaton output. Secton 5 descrbes goodness-of-ft probles and odel selecton. Secton 6 descrbes odel valdaton, Secton 7 descrbes senstvty analyss and Secton 8 descrbes the assessent of sequental procedures,such as subset selecton. BOOTSTRAP METHOD. The Basc Bootstrap Bootstrappng s an deal ethod, well atched to sulaton ethodology, for estatng accuracy of sulaton output and for carryng out senstvty analyss. It provdes a coputer-based soluton to the fundaental queston of all statstcal ethodology: A statstc, T, s calculated fro a rando saple, y = (y, y,..., y ). What s the dstrbuton of T? The basc process s llustrated n Fgure where a rando saple y s drawn fro a (null) dstrbuton F(.), and a statstc of nterest, T, s then calculated fro ths saple. F(.) y T Null Dstrbuton Saple Test Stat. Fgure : Basc Process If we could repeat the basc process a large nuber of tes, B say, (typcally B = 00 at least, see Chernck, 999) then we have a large saple {T (), T (),..., T (B) } of test statstcs, and the (sapled) eprcal dstrbuton functon (EDF) of these estates the requred dstrbuton. Ths de- 79

2 sred process s llustrated n Fgure and the resultng EDF fored fro the (ordered) saple {T (), T (),..., T (B) } s depcted n Fgure 3. Ths converges to the true dstrbuton of T as B becoes ndefntely large. Cheng y *() T *() y () T () F (.) y *() T *() F(.) y () T () y (B) Fgure : Desred Process T (B) y *(B) Fgure 4: Bootstrap Process T *(B) Also depcted n Fgure 5 s the test statstc, T, calculated fro the orgnal saple. Its p-value * ( ) # of T p = B T can readly be read off fro the bootstrap EDF, as ndcated n the Fgure. p-value T () T () T (B) Fgure 3: EDF of the Saple of Test Statstcs Obtanng the saples y (), y (),..., y (B) s usually too expensve to do. The bootstrap resaplng ethod replaces F(.) by the best estate we have for t. Ths s sply the EDF fored fro the orgnal saple y. We denote ths by F (.). The bootstrap verson of the desred process s llustrated n Fgure 4. Here and n what follows we use an astersk to denote a bootstrap value. Thus y *() denotes the th bootstrap saple and T *() the test statstc calculated fro ths saple. The EDF fored fro the (ordered) bootstrap saple {T *(), T *(),..., T *(B) } estates the (true) CDF of T. Ths s llustrated n Fgure 5. T *() T *() T T *(B) (OrgnalValue) Fgure 5: EDF of the Saple of Bootstrap Test Statstcs There s a huge lterature both theoretcal and practcal showng how well the ethod works. Chernck (999) gves soe 600 references. Hjorth (994) gves a good ntroducton. See also Davson and Hnkley (997). Expressed nforally, for the ethod to work, we requre two thngs. Frstly we requre that F (.) converges to F(.). When y s a rando saple ths s guaranteed by the Glvenko-Cantell theore that ensures that F (.) F(.) unforly wth probablty one (See Chernck 999 for detals). Secondly f we regard the statstc T as estatng soe characterstc property of the dstrbuton (techncally ths property s a functonal of the populaton dstrbutons) then ths characterstc ust satsfy certan soothness propertes. Most characterstcs such as oents 80

3 and percentles are sooth and so ther estators have dstrbutons that can be estated by bootstrap resaplng. Bootstrap resaplng ost often fals when T s a statstc estatng a functonal that s not sooth. Thus for exaple axa or na of a saple have dstrbutons that cannot usually be estated by bootstrap resaplng.. The Paraetrc Bootstrap In practce, as θ 0 s unknown, use s ade of one of the asyptotcally equvalent versons: ˆ ˆ θ ~ N (ˆ, θ (ˆ) θ ) or θ ~ N (ˆ, θ I(ˆ) θ ). (3) y *() θˆ *() T *() A varaton of the Basc Process s depcted n Fgure 6. The only dfference s that the statstcal quantty of nterest depends on a vector of paraeters θ 0 that has to be estated fro the saple y. F (., θˆ ) y *() *() θˆ T *() F(.,θ 0 ) y θˆ T( θˆ ) y *(B) θˆ *(B) T *(B) Null Saple Estate Test Stat. Dstrbuton of θ 0 Fgure 6: Basc Paraetrc Process Fgure 7: Paraetrc Bootstrap Process, Frst Verson θ *() T *() The saple s assued to have a jont dstrbuton wth probablty ncreent DF(y, θ) dependng on θ. If the saple s a rando saple then the probablty ncreent becoes a product of unvarate ncreents: DF(y, θ) = j = DF( y j, θ ), but n general we do not necessarly have to assue ths. A good way of obtanng θˆ s to use axu lkelhood (l) estaton (see for exaple Efron and Tbshran, 993). The lkelhood s sply the jont dstrbuton probablty eleent evaluated at the observed values, and then treated as a functon of θ. It s usually easer to work wth ts logarth (loglkelhood): L( θ ) = log DF(, θ). () = The l estate, θˆ, s the value of θ whch axzes ths loglkelhood. When the loglkelhood s axzed at an nteror pont of the paraeter space then θˆ satsfes the lkelhood equaton L (θ)= 0. The observed and expected nforaton are defned as I( θ) = L ( θ) / θ and (θ) = E[I(θ)]. Under general regularty condtons the l estate has the portant property that ts dstrbuton s asyptotcally noral as the saple sze tends to nfnty: y ˆ 0 0 θ ~ N ( θ, ( θ ) ). () 8 N (ˆ, θ (ˆ) θ ) θ *() θ *(B) Fgure 8: Paraetrc Bootstrap Process, Second Verson Bootstrappng can be done n two ways. The nonparaetrc ethod s sply to consder the estaton of θ as part of the process of calculatng T. Bootstrappng can be done n two ways. The non-paraetrc Bootstrappng can be done n two ways. The nonparaetrc ethod s sply to consder the estaton of θ as part of the process of calculatng T. Bootstrappng then proceeds precsely as before by resaplng of y. Ths ethod s depcted n Fgure 7. However an alternatve s possble f a ethod lke l s used to estate θ. Ths s llustrated n Fgure 8. In ths alternatve ethod we bypass the saplng of y, and nstead resaple values of θˆ usng (3) drectly. 3 THE SIMULATION MODEL We assue that y, the output of nterest fro the sulaton run, depends on a strea of unfor rando varates u = (u, u,...), on a vector θ = (θ, θ,..., θ p ) of p paraeters and on a vector x = (x, x,... x q ) of q desgn factors,.e. y=y(u, θ, x). T *() T *(B)

4 The desgn factor values are assued known, but the paraeters ay not all be known. Typcally the unfors are transfored nto rando varates that nvolve soe of the paraeter values. Thus actual rando varates used ght take the for w = ϕ (u, θ) where u denotes soe subset of the unfors used. In what follows we do not need to consder the w s explctly. We regard the objectve as beng to estate the expected value of y, and note that ths s a functon of θ and x only: η ( θ, x) = E( y, θ, x) = y( u, θ, x) du. (4) The above forulaton requres one further refneent. We need to dstngush between two types of paraeter. In the above forulaton θ denotes those paraeters that depend purely on processes that are exogenous to the sulaton odel. Exaples of such paraeters are arrval rates dependng on arrval processes external to the sulaton under consderaton. Such paraeters are part of the descrpton of how the process behaves, and ther values wll have to be fxed before a sulaton can be carred out. For ths reason we call the coponents of θ nput paraeters and θ tself the nput paraeter vector. However there s another way that paraeters can enter the sulaton. We let β denote those paraeters whch are added subsequent to the sulaton experent and whch have been ntroduced to descrbe the dstrbutonal propertes of the output y. Typcally β as part of a regresson etaodel. In fttng such a odel, β wll need to be estated fro the output y values obtaned fro the sulatons. We shall call the vector β the output paraeter vector, and ts coponents, output paraeters. We consder two cases, dependng on whether θ s known or not. In ether case β s unlkely to be known, so we shall only consder the case where t s unknown. Case A: Here θ s regarded as fxed and known. We consder the overall sulaton experent as beng ade up of runs ade at n desgn ponts wth runs ade at x the th desgn pont. The responses or outputs fro these runs wll be wrtten as: y j =y(u j, θ 0, β, x ) = η( θ 0, β, x ) + e(u j, θ 0, β, x ), =,,..., n, j =,,...,, (5) where θ 0 s the gven value of θ. The error varable e s the rando dfference between the sulaton run output and η(θ 0, x, β). We shall assue E(e) = 0. We have E[y j ] = η(θ 0, β, x ) and the ean of the outputs y = j= y / s an unbased estator of η(θ 0, β, x ). To fully dentfy the dstrbutonal propertes of the y j the unknown β wll have to be estated. We can regard the estaton of β as part of the sulaton process wth the estate βˆ forng part of the output of the sulaton. In status βˆ s lke the output y tself. Case B: Here θ s not known but has to be estated. As already rearked θ denotes those paraeters that depend purely on processes that are exogenous to the sulaton odel. We suppose therefore that there exst eprcal data: z = (z, z,...z l ) drawn fro a dstrbuton G(., θ 0 ) where θ 0 s the unknown true paraeter value. Ths value wll thus have to be estated fro ths data. Agan we assue that runs are ade at x, the th desgn pont. The responses or outputs fro these runs wll be wrtten as: y j =y(u j, θˆ, β, x ) = η( θˆ, β, x ) + e(u j, θˆ, β, x ), j =,,...,n, j =,,...,. (6) Here the unknown θ has been replaced by an estate θˆ. We have ncluded the vector β n (5) and (6), but t s best regarded as part of the output process. It can only be estated after the sulatons and ths wll need to be done n order to copletely deterne the dstrbuton of y. The ean of the outputs y = j= y j / s stll an unbased estator of η(θ, β, x ). For notatonal splcty, when the y are assued to have the for (5) or (6), we wrte the dstrbuton of y as F(.,θ, β, x). We now consder each of the probles lsted n the ntroducton. 4 DISTRIBUTIONS AT A FIXED DESIGN POINT The stuaton here s as gven n (5) or (6) dependng on whether θ s known or not. If all sulaton runs are conducted at the sae desgn pont then n =, and the saple s y = (y, y,..., y ). As there s just one saple we suppress the frst subscrpt and wrte t sply as y = y = (y, y,..., y ). We also wrte x 0 = x. Of lkely nterest s the estaton of typcal propertes of the dstrbuton of y such as ts ean, varance and selected percentles. We llustrate wth the case of the ean. Here T = y. 8

5 Consder frst Case A, where θ s known. Let θ = θ 0. In ths case there are two possble ways of estatng the dstrbuton of T. The frst s the non-etaodellng approach where we gnore the structure assued n the rght-hand sde of equaton (5) or (6) and deal wth the y j drectly. We therefore estate the dstrbuton of T by the bootstrap process of Fgure 4. The other s where we adopt the structure assued n the rght-hand sde of (5) or (6). We wrte the dstrbuton of y as F(.,θ 0, βˆ, x 0 ), where βˆ has been estated fro the saple y, by l, say. The dstrbuton of T can thus stll be estated by the bootstrap process of Fgure 4 except that the bootstrap saples y* () are each drawn fro F(.,θ 0, βˆ, x 0 ). The only pont of note s that the calculaton of each bootstrap T *() requres calculaton of a correspondng estate, βˆ *(), of β. Consder now Case B, where θ s not known. In ths case the sulaton runs ust use an estate, θˆ, of θ. The non-etaodellng approach cannot be used as the orgnal saple y does not nclude the varaton arsng fro ths estaton process. We can however use the etaodellng approach. To ncorporate the added uncertanty arsng fro the estaton of θ, each bootstrap saple y* () s now drawn fro a separate dstrbuton, naely F(.,θˆ* (), βˆ, x 0 ), n whch θˆ * () has been sapled by the bootstrap process of Fgure 7 or 8 (except that n Fgure 7 y needs to be replaced by z, and F by G l ) so that t vares fro saple to saple. 5 GOODNESS-OF-FIT AND MODEL SELECTION 5. Goodness-of-Ft The dscusson of Secton 4 extends naturally to the study of how well the sulaton output s represented by soe gven paraetrc odel. We consder agan the stuaton where n = wth the saple wrtten as y = y = (y, y,..., y ). and where we wrte x 0 = x. The stuaton s precsely that of equaton (5) or (6), wth n =, but where we are uncertan whether we have the correct for for η(θ, β, x 0 ) or for e(u, θ, β, x 0 ). Ths s a queston of goodness-of-ft of the odel and s a proble that can be dffcult to handle usng tradtonal technques. Resaplng ethods offer a partcularly good soluton to goodness-of-ft probles (See Cheng et al. 996). The stuaton s bascally that of Fgure 6 but where we are uncertan f the null dstrbuton of y s F(.,θ 0, βˆ, x 0 ) when we have Case A or s F(., θˆ, βˆ, x 0 ) when we have Case B. In ether case we sply take T to be a goodness of ft statstc. A good choce s the Anderson - Darlng statstc (Anderson and Darlng, 95) A = j= ( j ){ln( z j ) + ln( z j+ where z j = F(y (j), θ, βˆ, x 0 ), and y (j) denotes the jth observaton of the ordered y saple. We shall use T = A n ths case. The ethod now precsely follows that of Secton 4, but usng the etaodellng approach. The bootstrap process of Fgure 7 (except that F (., θˆ ) n the Fgure s replaced by F(.,θ 0, βˆ, x 0 ) n Case A and by F(., θˆ * (), βˆ, x 0 ) n Case B) s appled to obtan an estate of the dstrbuton of the test statstc T under the assupton that the ftted odel s the correct one. Thus, n calculatng A* (), the bootstrap value of A obtaned fro the th bootstrap saple, we use z j * () = F(y (j) * (), θ 0, βˆ, x 0 ) n Case A and z j * () = F(y (j) * (), θˆ * (), βˆ, x 0 ) n Case B. We then exane the p-value - as easured by ths estated dstrbuton - of the T statstc calculated fro the orgnal saple y,.e. # of T * p = B ( ) T. If ths p-value s too sall the odel s rejected as beng a poor ft. The ethod above s descrbed for just one desgn pont but extends n an obvous way f we wsh to consder the goodness-of-ft of the odel as a whole, ncludng all desgn ponts. All that s needed s soe approprate portanteau goodness-of-ft test statstc coverng all desgn ponts sultaneously. 5. Model Selecton Model selecton s easly handled f we regard t sply as an extenson of goodness-of-ft to the stuaton where we ay be unsure as to whch of a nuber of alternatve odels ay be the best representaton of the output. Assue that we have a nuber, k say, of contendng odels, each lke (5) or (6). We wrte these odels as: F j (.,θ, β, x), j =,,..., k We calculate the p-value of the goodness-of-ft statstc for each of these contendng odels n exactly the way descrbed n the above goodness-of-ft procedure. Models wth p-values that are too sall can be ruled out as beng )} 83

6 not good fts. If just one odel has to be chosen, then that correspondng to the largest p-value s the natural choce. 6 MODEL VALIDATION We consder only the basc stuaton where the sulaton output s beng copared wth exstng output fro soe real or prevously valdated odel. The stuaton s slar to that of odel selecton when just two odels are beng copared so that essentally the sae technques can be appled here. We consder the case where y 0 = (y 0, y 0,... y 0 ) s the observed output fro a real syste and we wsh to copare ths wth y = (y, y,... y ) the observed output fro a set of sulaton runs. We use a ethod proposed by Klejnen et al. (00). We consder a statstc T=T(y 0, y ) that easures the dfference between the two saples n soe way. Ths ght sply be T = y y0 or a quantty such as T = ( F0 F ) df0 that easures the average squared dfference between the EDF s of the two saples. Here F 0 (.) and F (.) denote the EDF s of the two saples y 0 and y respectvely. We now ake a second set of sulaton runs, y, under the sae condtons used for obtanng y. Thus f we used θ = θ 0 n Case A or θ = θˆ n Case B when obtanng y then we use the sae value for θ n the sulatons that yeld y. The key pont s that T(y 0, y ) and T(y, y ) wll be dentcally dstrbuted f all three saples are drawn fro the sae dstrbuton. The dstrbuton of T(y 0, y ) or of T(y, y ) can be obtaned by bootstrappng. The dstrbuton of T(y, y ) s possbly the easer to use as t can be estated by bootstrappng usng ether the nonetaodellng or the etaodellng approach. In the non-etaodellng approach we can obtan bootstrap versons of y, y by resaplng wth replaceent to yeld: y * (), y * (), for =,,..., ; and fro these pars we can calculate bootstrap T* () = T( y * (), y * () ), for =,,...,. The p-value of T( y 0, y ), as gven by the EDF fored fro the T* (), s a easure of the slarty between the output of the real syste and that of the sulaton, wth a sall value of p ndcatng dsslarty. In the etaodellng approach we can obtan bootstrap versons of y, y by saplng fro the ftted etaodel, ths beng F(.,θ 0, βˆ, x 0 ) n Case A and F(., θˆ, βˆ, x 0 ) n Case B. The procedure s then exactly as for the nonetaodellng case. 7 SENSITIVITY ANALYSIS We consder Case B only. The study of the dstrbuton of paraeter estates, and the way they affect estates of the sulaton output, can be regarded as a proble of senstvty analyss. The effect of randoness (.e. that due to u) bult nto the odel ay also be regarded as a senstvty ssue. In fact usng the regresson etaodel approach, wth the output regarded as beng of the for assued n equaton (6), these two probles can be consdered as two parts of the sae proble. More explctly, we are nterested n how overall varablty depends respectvely on θˆ and on the u j. A nuber of authors have dscussed ths ssue, see for exaple Helton (993, 994). Cheng and Holland (995, 997) show that, to frst order, Var(y) s ade up of two separate contrbutons, one dependng purely on the varaton arsng fro θˆ beng rando, the other due to varaton n u j. We call these two coponents the paraeter and sulaton uncertantes respectvely. Assessng the paraeter uncertanty usng classcal statstcal technques requres estaton of the senstvty coeffcents: η( θ, x) β =, =,,..., p θ Ths can be expensve when p s large. We can overcoe the proble usng bootstrappng. We consder the case n = and wrte y = y as before. Consder frst the non-etaodellng approach. We begn by assessng the sulaton uncertanty on ts own (.e. that due just to the u j ). Ths s done by akng the basc set of sulaton runs usng θ = θˆ n all runs. The saple varance s = ( ) ( y j y) (7) j= calculated fro the runs estates the sulaton uncertanty (at the gven desgn pont) only. We then ake a second set of sulaton runs but now wth θ resapled separately n each run usng the saplng schee of ether Fgure 7 or 8 (Except that y needs to be replaced by z, and F by G l n Fgure 7). The varance estated fro these runs, usng exactly the sae forula of equaton (7), and whch we denote by v, easures the paraeter and sulaton uncertantes cobned. The dfference t = v s now estates the paraeter uncertanty at the th desgn pont. A etaodellng approach s also possble and follows along the sae lnes as dscussed for other probles. Detals are left as an exercse. 84

7 If the varablty of soe quantty other than the ean s of nterest then the entre set of subruns at the desgn pont ay be needed to calculate t. For exaple f the y s are not noral and an estate of a percentage pont s requred, then ths can be calculated fro the EDF of the y values observed at the desgn pont. In ths case no separate estate of the varablty of ths estate s avalable as there was for the ean y. The varance of such an estate wll have to be carred out by bootstrappng of the entre set of runs B tes and calculatng the estate of nterest fro each bootstrap set of runs. 8 SEQUENTIAL PROCEDURES 8. Selecton of the Best Syste Out of n Suppose that the output (6) arses fro a coparson of n dfferent systes where the objectve s to choose the best syste accordng to a gven perforance ndex, assued to be y. Thus the n desgn ponts are actually dstnct systes beng copared. Even f the runs have arsen fro a sequental procedure, then, provdng there are a reasonable nuber of runs for each syste, we can assess the robustness of the process that has produced a gven best selecton. Suppose for exaple that the sequental procedure has resulted n selecton of the frst syste as beng the best. We can test the relablty of ths selecton sply by repeatng the selecton process, only usng bootstrap resaplng each te, rather than actual sulatons to generate the sequence of y s used. Thus, whenever a new y value fro the th desgn pont s requred by the selecton procedure, we sply saple ths fro the set of y values ntally obtaned. Saplng s done wth replaceent of course, wth values sapled each te fro the full saple y. The entre sequental process s repeated B tes. The proporton of tes that the orgnally selected best syste coes out on top provdes an estate of the confdence that ths syste s actually the best. The above ght be regarded as beng the nonetaodellng approach because resaplng s fro the orgnal observatons. An alternatve resaplng strategy can be eployed that akes explct use of the ftted etaodel. Ths would be especally advantageous n stuatons where soe of the are too sall for resaplng to be done wth confdence. In ths case each bootstrap experent s stll a repeat of the entre sequental procedure, the only dfference beng that, when we are called to saple fro the dstrbuton of the th odel, we sply saple fro the ftted dstrbuton: F(., θˆ, βˆ, x ), =,,..., n. 8. Iportant Factors The ethodology of Secton 8. apples to any other sequental procedures. For exaple suppose the regresson functon η(θ, β, x) ncorporates the effects of a nuber dfferent factors and we use a sequental procedure to select those that are consdered to be portant. We can use bootstrappng to assess how stable s the choce of those factors deeed to be portant. We sply bootstrap the entre sequental procedure B tes. In each bootstrap, the y s that are used can agan be generated n ether of the two ways ndcated n Secton 8.. In the frst way resaplng s sply carred out usng the y saples obtaned fro the orgnal sulatons. The resaplng s done wth replaceent. In the second way we resaple fro the ftted dstrbutons F(., θˆ, βˆ, x ), =,,..., n. 9 SUMMARY The above dscusson ndcates the flexblty of the bootstrap procedure and how t can be readly appled to any probles of practcal nterest n sulaton odellng and experentaton. For reasons of space the dscusson has focused on descrbng the procedures n suffcent detal to enable each to be easly and unabguously pleented by the nterested reader. It s hoped to llustrate these procedures wth nuercal exaples n the Conference presentaton. REFERENCES Anderson, T.W. and D.A. Darlng. 95. Asyptotc theory of certan goodness of ft crtera based on stochastc processes, Annals of Matheatcal Statstcs, 3: 93-. Cheng, R.C.H. and W. Holland The effect of nput paraeters on the varablty of sulaton output. In UKSS95, Proceedngs of the nd Conference of the UK Sulaton Socety, eds. R.C.H. Cheng and R.B. Pooley, Ednburgh: Ednburgh Unversty Reprographcs Press. Cheng, R.C.H Bootstrap Methods n Coputer Sulaton Experents. In Proceedngs of the 995 Wnter Sulaton Conference ed. W.R. Llegdon, D. Goldsan, C. Alexopoulos and K. Kang, Pscataway, New Jersey: Insttute of Electrcal and Electroncs Engneers.. Cheng, R.C.H., W. Holland, and N.A. Hughes Selecton of nput odels usng bootstrap goodness-offt. In Proceedngs of the 996 Wnter Sulaton Conference, ed. J.M. Charnes, D.J. Morrce, D.T. Brunner and J.J. Swan, Pscataway, New Jersey: Insttute of Electrcal and Electroncs Engneers. 85

8 , R.C.H. and W. Holland Senstvty of coputer sulaton experents to errors n nput data. Journal of Statstcal Coputaton and Sulaton, 57: 9-4. Chernck, M.R Bootstrap Methods A Practtoner s Gude. New York, Wley. Davson, A.C. and D.V. Hnkley Bootstrap Methods and Ther Applcaton. Cabrdge: Cabrdge Unversty Press. Efron, B. and R.J. Tbshran An Introducton to the Bootstrap. New York and London: Chapan & Hall. Helton, J.C Uncertanty and Senstvty Analyss Technques for Use n Perforance Assessent for Radoactve Waste Dsposal. Relablty Engneerng and Syste Safety, 4: Helton, J.C Treatent of Uncertanty n Perforance Assessents for Coplex Systes. Rsk Analyss, 4: Hjorth, J.S.U Coputer Intensve Statstcal Methods. London: Chapan & Hall. Klejnen, J.P.C., R.C.H. Cheng and B. Bettonvl. 00. Valdaton of Trace-Drven Sulaton Models. Manageent Scence. To appear. AUTHOR BIOGRAPHY RUSSELL C. H. CHENG s Professor and Head of Operatonal Research at the Unversty of Southapton. He has an M.A. and the Dploa n Matheatcal Statstcs fro Cabrdge Unversty, England. He obtaned hs Ph.D. fro Bath Unversty. He s a forer Charan of the U.K. Sulaton Socety, a Fellow of the Royal Statstcal Socety, Meber of the Operatonal Research Socety. Hs research nterests nclude: varance reducton ethods and paraetrc estaton ethods. He s Jont Edtor of the IMA Journal of Manageent Matheatcs. Hs eal and web addresses are <R.C.H.Cheng@aths. soton.ac.uk> and < staff/cheng>. Cheng 86