OPTIMAL COMPUTING BUDGET ALLOCATION FOR MULTI-OBJECTIVE SIMULATION MODELS. David Goldsman

Proceedng of the 004 Wnter Smulaton Conference R.G. Ingall, M. D. Roett, J. S. Smth, and B. A. Peter, ed. OPTIMAL COMPUTING BUDGET ALLOCATION FOR MULTI-OBJECTIVE SIMULATION MODELS Loo Hay Lee Ek Peng Chew Suyan Teng Dept of Indutral and Sytem Engneerng Natonal Unverty of Sngapore 0 Kent Rdge Crecent, 960, SINGAPORE Davd Goldman School of Indutral and Sytem Engneerng Georga Inttute of Technology Atlanta, GA 3033-005, U.S.A. ABSTRACT Smulaton play a vtal role n dentfyng the bet ytem degn from among a et of competng degn. To mprove mulaton effcency, rankng and electon technque are often ued to determne the number of mulaton replcaton requred o that a pre-pecfed level of correct electon guaranteed at a modet poble computatonal expene. A mot real-lfe ytem are mult-objectve n nature, n th paper, we conder a mult-objectve rankng and electon problem, where the ytem degn are evaluated n term of more than one performance meaure. We ncorporate the concept of Pareto optmalty nto the rankng and electon cheme, and try to fnd all of the non-domnated degn rather than a ngle bet one. A mple equental oluton method propoed to allocate the mulaton replcaton. Computatonal reult how that the propoed algorthm effcent n term of the total number of replcaton needed to fnd the Pareto et. INTRODUCTION Smulaton commonly ued to dentfy the bet ytem degn from among a et of propoed alternatve, where bet defned n term of the maxmum (or mnmum) expected value of ome functon of the mulaton output. However, nce mulaton can be both expenve and tme conumng, effcency tll a key concern n th area. Therefore, to evaluate the relatve worth of the competng degn, rankng and electon technque are often ued to determne the number of mulaton replcaton requred for each degn o that a pre-pecfed level of correct electon guaranteed at the leat poble computatonal expene. Th area of reearch ha ganed popularty n mulaton output analy and optmzaton n the lat decade. There are qute a number of revew paper avalable n th feld (Bechhofer, Santner, and Goldman 995; Goldman and Nelon 998; Swher, Jacobon, and Yücean 003). Rankng and electon procedure are tattcal method pecally developed to elect the bet ytem degn or a ubet that contan the bet ytem degn from a et of n competng alternatve (Goldman and Nelon 994). Several dfferent approache to the problem have been propoed. The dfference manly le n how to allocate replcaton to certan degn. For ntance, the commonly ued two-tage ndfference-zone procedure propoed by Rnott (978) determne the number of addtonal mulaton replcaton for each degn baed on the ample varance etmated from the frt tage of amplng. Th procedure baed on a leat-favorable confguraton formulaton to allocate addtonal replcaton. Alternatvely, an average cae analy can be ued to allocate addtonal replcaton. Th dea ha engendered two dtnct approache, outlned below. Chen, Chen, and Da (996) and Chen et al. (997) followed a Bayean methodology, makng ue of nformaton on both ample mean and ample varance. The ratonal here to only mulate lkely compettor for the bet, thu leadng to gnfcant mprovement n computng effort n the mulaton. Chck (997) propoed Bayean decon theoretc method, whch attempt to elect an addtonal number of replcaton for each ytem o that the expected value of nformaton ganed from thoe replcaton maxmzed, rather than ung the thought experment a n Chen, Chen, and Da (996) and Chen et al. (997). Informaton gan for the probablty of correct electon are meaured wth repect to the 0- lo functon. Mot of the tude n the rankng and electon area focu on a ngle meaure of ytem performance, or put another way, the ytem evaluated wth repect to a ngle objectve. However, mot real-lfe ytem and degn often have multple objectve. For example, n productdegn optmzaton, the cot and the qualty of product are two conflctng objectve. In evaluatng arlne flght chedule, we may want to elect flght chedule n term of mnmal FTC (flght tme credt) and mnmal percentage of late arrval (Tan 003). In th ettng, the problem

of electng the bet degn from a et of alternatve through mulaton become a mult-objectve rankng and electon (MORS) problem. One common way to addre the MORS problem to weght everal parameter of nteret to form a ngle meaure of effectvene by applyng multple attrbute utlty (MAU) theory (Butler, Morrce, and Mullarkey 00; Morrce, Butler, and Mullarkey 998; Swher and Jacobon 00). The problem reduce to a ngle-objectve model, and extng method can be appled. Dudewcz and Taneja (978) propoed a multvarate procedure by defnng a multvarate normal vector compoed of c > component varate wth an unknown and unequal varance-covarance matrx. They redefne the ndfference-zone parameter a the Eucldean dtance from a mean vector to the bet mean vector. In both approache, the author try to determne a ngle bet oluton. In the former (weghted parameter) approach, the decon maker not only need to cot out performance n one crteron for performance n another, but he alo need to pecfy the relatve mportance of the performance meaurer. A a reult, the bet oluton elected would be trongly dependent on thee preference. In cae another decon maker ha dfferent preference wth repect to the performance meaure, or f the decon maker change h preference, the oluton may become nferor. In the latter (multvarate) approach, t may not be eay to fnd the bet mean vector due to the mult-objectve nature of the problem. In the cae of problem that are mult-objectve n nature, there may not ext a ngle bet oluton, but rather a et of non-domnated oluton. The complete et of nondomnated oluton referred to a the Pareto et of oluton. They repreent the bet degn and are characterzed by the defnton that no other oluton ext that uperor n all the objectve. In the applcaton of evolutonary algorthm to olve mult-objectve problem, the concept of Pareto optmalty often employed to fnd the non-domnated Pareto et (Foneca and Flemng 995; Tech and Schemann 000; Ztzler and Thele 999). In th paper, to addre the MORS problem, we ncorporate the concept of Pareto optmalty nto the rankng and electon cheme. We try to provde a non-domnated Pareto et of degn to the decon maker, rather than reducng the problem to a ngle-objectve model and provdng a ngle bet degn a n Butler, Morrce, and Mullarkey (00), Morrce, Butler, and Mullarkey (998), and Swher and Jacobon (00). The problem condered n th tudy now tated a follow. Suppoe that we have a et of n degn, where each evaluated n term of m ndependent objectve. We want to fnd the non-domnated (Pareto) et of degn by runnng mulaton. The problem to determne an optmal allocaton of the mulaton replcaton to the degn, o that the non-domnated et of degn can be found at the leat expene n term of mulaton replcaton. In th paper, we aume that the number of non-domnated degn (K) n the pace known n advance. The paper organzed a follow. Secton ntroduce a performance ndex to meaure how non-domnated a degn n the cae of mult-objectve problem. The poteror dtrbuton of the mean performance of a degn dcued n Secton 3. Secton 4 propoe a mulaton replcaton allocaton procedure for the MORS problem. Secton 5 preent ome computatonal reult, and fnally ome concluon and future reearch drecton are ummarzed n Secton 6. MEASUREMENT OF NON-DOMINATED DESIGNS IN MULTI-OBJECTIVE PROBLEMS To ncorporate the concept of Pareto optmalty nto the rankng and electon cheme, we frt need to fnd a way to meaure how non-domnated a degn. Wthout lo of generalty, we aume that mnmzaton of the objectve our goal throughout th paper. Alo, we aume that the random varable under tudy follow contnuou dtrbuton.. Comparng Uncertan Performance Meaure of Two Degn When conderng Pareto optmalty, we are tryng to fnd a complete et of thoe non-domnated degn. Suppoe we have two degn and j, each of whch evaluated n term of m performance meaure a llutrated below. µ : µ, µ,..., µ m µ : µ, µ,..., µ j j j jm In a noe-free tuaton, degn j domnate degn, denoted by µ j p µ, f the followng condton hold wth at leat one nequalty beng trct: µ jk µ k for k =,,..., m. However, f ftne value µ k and µ jk are random,.e., ubject to noe, then we have to conder the probablty that degn j domnate degn, a expreed n the followng condton wth at leat one nequalty beng trct: P( µ p µ ) = P( µ µ for k,,..., m). j jk k = Under the condton that the performance meaure are ndependent from one another and they follow contnuou dtrbuton, we have m P ( µ p µ ) = P( µ µ ). () j k = jk k

. A Performance Index to Meaure the Non-domnated Degn We now ntroduce a performance ndex to meaure how non-domnated a degn, when the performance meaure are ubject to noe. Gven n degn, we calculate the cumulatve probablty of degn beng domnated by other degn: n ψ = P ( µ j p µ ). () j=, j Performance ndex ψ meaure the um of the probablte that other degn are better than degn. Therefore, f ψ cloe to 0, then the probablty that other degn are better than degn low; and the probablty that degn non-domnated hgh, o that t hould be ncluded n the Pareto et. We etablh ome notaton. ψ * : A predefned requred performance ndex for degn n the Pareto et to be retaned at the end of the mulaton. S : The Pareto et contanng all non-domnated degn. p : The number of replcaton allocated to degn. K : The number of non-domnated degn known n advance. N max : The maxmum total number of mulaton replcaton avalable. The general dea to perform 0 replcaton for each degn ( =,,,n), etmate the performance ndex ψ for each degn, and then rank the degn n acendng order of ψ, wth ψ ( ) ψ () L ψ ( K ) L ψ ( n). Snce ψ the performance ndex meaurng the um of probablte that other degn are better than degn, the K degn wth the mallet ψ value are very lkely to be the nondomnated degn at the current mulaton tage; therefore, we put them nto the Pareto et, gven that K the known number of non-domnated degn. Then our problem become: determne the optmal allocaton of the replcaton to the degn o that each degn n the Pareto et ha performance ndex le than the requred performance ndex ( ψ * ), and the total number of mulaton replcaton mnmzed. The problem can be formulated a follow: ubject to : ψ mn ( k ) 0 n = ψ * for all k K for all =,,..., n (3) Alternatvely, we can mnmze the larget performance ndex for degn n the Pareto et, whle atfyng the contrant that the total number of replcaton wthn a predefned lmt, N max. The problem become: ubject to: mn ψ ( K ),,..., n n Nmax = (4) 0 for all =,,..., n In th tudy, though our fnal goal to olve problem (3), we adopt a equental approach, where at each tep, gven a total number of (<< N max ) replcaton to be allocated, problem (4) condered. 3 POSTERIOR DISTRIBUTIONS BASED ON BAYESIAN MODEL AND SIMULATION OUTPUT When we try to dentfy the bet degn among everal degn, we compare random varable repreentng the mean whoe poteror dtrbuton can be derved baed on the mulaton output. Therefore, before preentng a method to olve problem (3) or (4), we need to know how to get the poteror dtrbuton of the random varable repreentng the mean, and how each dtrbuton would change upon addtonal replcaton allocated to the correpondng degn. Suppoe that F% k the random varable repreentng the poteror mean performance for the kth objectve of degn, and F ˆk the random varable repreentng the poteror mean performance for the kth objectve of degn after addtonal replcaton are allocated to degn. Now we llutrate how to get the poteror dtrbuton for F% k and F ˆk baed on a Bayean model and mulaton output. Aume that we ue the followng addtonal notaton: f k : Smulaton ample for the kth objectve of degn. f k : The ample mean of the mulaton output for the kth objectve of degn. f k = f = k. µ k : The unknown mean performance meaure for the kth objectve of degn. σ k : The known varance of the kth objectve of degn. F% k : A random varable repreentng the poteror mean performance for the kth objectve of degn.

F ˆk : A random varable repreentng the poteror mean performance for the kth objectve of degn after addtonal replcaton have been allocated to degn. Suppoe that the mulaton output f k follow a normal dtrbuton wth unknown mean µ k, and known varance σ k, where the unknown mean µ k telf a random varable wth pror dtrbuton N( η k, v k ) ; then accordng to DeGroot (970), the poteror dtrbuton of µ k : F% ~ N k k k + v k f k, k v k σk + vk σk + vk σ η σ ( ). If the varance v k of the pror dtrbuton of µ k very large, then lttle pror knowledge avalable for the performance of the degn before conductng the mulaton. In that cae, the poteror dtrbuton of µ k can be approxmated a k σ F% k ~ N( fk, ). whch make ntutve ene. In our tudy, we aume that the known varance of the mulaton output ( σ k ) mply the ample varance (whch actually a random varable ued to etmate σ k ). Our etmate for σ k updated whenever addtonal mulaton replcaton are allocated to certan degn. If the varance σ k alo aumed to be unknown, the poteror d- trbuton of µ k wll follow a more complex dtrbuton. To examne how the probablty dtrbuton of F% k change after addtonal replcaton are allocated to degn, uppoe we conduct 0 replcaton on degn frt, after whch an addtonal replcaton are allocated to degn. Then the poteror dtrbuton for the kth objectve of degn F ~ N ˆk + fk, 0 = 0 σk ( ). 0 + + If mall, then a good approxmaton to the poteror dtrbuton fk, 0 = 0 σk F ˆk ~ N( ). (5) + 0 4 A REPLICATIONS ALLOCATION PROCEDURE FOR THE MORS PROBLEM From (5), we ee that addtonal replcaton allocated to a certan degn hould be mall enough o that the ample mean doe not change much after runnng the addtonal replcaton. Therefore, we adopt a equental approach to olve problem (3): we teratvely perform a number of tep, wth a mall number of replcaton allocated to the degn at each tep. Specfcally, at each tep, gven a total number of replcaton to allocate, we conder olvng problem (4), whch to fnd degn() that can gan the hghet decreae n the overall performance ndex of degn n the Pareto et. To olve problem (4), we examne the change on the performance ndex ψ upon addtonal replcaton allocated to degn d, ay ψ d. Then allocate addtonal replcaton to thoe degn that can gan the hghet ncreae n the total change of the performance ndex for degn elected nto the Pareto et. 4. Change on Performance Index upon Addtonal Replcaton Allocated to a Degn d We how how the performance ndex ψ of degn change upon addtonal replcaton ( d ) allocated to a certan degn d. Gven F% k and F ˆk a defned n Secton 3, from () and (), we have degn performance ndex n m % % ψ = PF ( F ). jk k j=, j k = Frt of all, uppoe d, o that addtonal replcaton are not allocated to degn. Snce only the dtrbuton for the m objectve of degn d wll change, we have m m ˆ d PFdk Fk PFdk Fk k= k= ψ = ( % % ) ( % ). (6) Otherwe, f d =, then the dtrbuton for degn objectve wll change, o n m n m ψ ( ) ( ˆ d = PF% jk F% k PF% jk Fk ). (7) j=, j k= j=, j k= From (6) and (7), we ee that the performance ndex ψ the um of n component. However, when calculatng the change n ψ ( ) upon addtonal replca- ψ d

ton allocated to degn d, we only need to calculate the dfference between two component for the n cae for whch d. Only for one cae ( d = ) do we need to calculate the dfference between all n component. Th help to mprove the computatonal effcency of the propoed algorthm. 4. Outlne of the Mult-Objectve Computng Budget Allocaton (MOCBA) Algorthm We propoe the followng procedure to allocate mulaton replcaton to degn for the MORS problem. We call the et of two procedure the MOCBA algorthm. Gven that p the number of degn elected to allocate addtonal replcaton, we have PROCEDURE I Step 0: Perform 0 replcaton for each degn ( =,,,n). Then total number of obervaton N = n 0. Step : Calculate performance ndex ψ for each degn ( =,,,n). Sort the degn =,,,n n acendng order of ψ a ψ ( ), ψ (),..., ψ ( n). Put the K degn wth the mallet value of ψ nto the Pareto et (S p ). Step : If ψ (K ) < ψ * or N > N max, go to Step 5. Step 3: Solve problem (4) by callng PROCEDURE II. Select p degn w, w,..., wp and the correpondng number of addtonal replcaton w ( =,,,p) from PROCEDURE II. Step 4: Perform ( =,,,p) replcaton from thoe w elected degn, et N = N +, and go to Step. Step 5: Output the K bet degn. PROCEDURE II Step : For each degn p = w S p and for each degn d =,,,n, calculate the change n performance ndex ψ d upon addtonal replcaton from degn d. Step : Calculate the total change n the performance ndex for degn n S p. ψd = ψd. Sp Step 3: Sort the degn n decendng order of ψ d a w, w,..., w n, wth the correpondng change n the performance ndex denoted a,,, ψ w ψ w ψ wn. Allocate addtonal replcaton w, =,,,p for the frt p degn ( w, w,..., wp ) a follow. wp w ψwp =, p ψ w = ψ w = * w =,,..., p p ψ wp 5 COMPUTATIONAL RESULTS To examne the performance of MOCBA, we make two comparon. One agant the theoretcal optmal allocaton (TOA) and the theoretcal unform allocaton (TUA); and the other wth the unform computng budget allocaton (UCBA) algorthm. In the followng computatonal experment, when callng PROCEDURE II to olve problem (4), the number of degn (p) elected to allocate addtonal replcaton ( ) et at, and the number of replcaton et at 5. 5. Comparon wth TOA and TUA Suppoe we know the true mean and varance of the performance meaure of the n degn. Then, gven a maxmum total number of replcaton (N max ) avalable, we can determne how to optmally allocate thee replcaton to the degn o that the performance ndexe ( ψ ) for degn n the Pareto et are mnmzed. Th what theoretcal optmal allocaton mean. Suppoe the number of replcaton allocated to degn, and ψ (), ψ (),, ψ (K ),, ψ (n) an ordered equence n acendng order of the value of ψ. Gven that K the number of non-domnated degn n the Pareto et, we want to fnd an optmal oluton to problem (4). To fnd the optmal allocaton of the mulaton replcaton, we conduct a brute force earch. Specfcally, we fx the maxmum number of replcaton N max = 400 and the number of degn n = 5. We conder the cae when degn are evaluated n term of 3 objectve. The mean and tandard devaton to generate the degn are lted n Table. From Table, we know that degn domnated by degn 0 and ; degn 3 domnated by degn ; degn 4 domnated by degn and 3; and degn 0 and are nondomnated degn. For the TOA, from the brute force earch, the number of replcaton allocated to each degn (R) and t percentage of total number of replcaton (θ ) are hown n Table. The Kth ( nd ) performance ndex reached for the bet allocaton 5.6 E-4.

Table : Mean and Standard Devaton to Generate the Degn Degn Mean Std Mean Std Mean 3 Std 3 0 6 9 44 9 56 9 7 9 40 9 64 9 8 9 45 9 65 9 3 9 9 4 9 66 9 4 0 9 43 9 67 9 Table : Number of Replcaton Needed for the TOA Degn 0 3 4 R 05 4 40 78 35 θ 6.3 35.5 0.0 9.5 8.8 For the TUA, mlarly, we uppoe the true mean and tandard devaton of the degn are known a n Table. We want to determne f we allocate the replcaton unformly to each degn how many replcaton hould be allocated to each degn, o that the Kth ( nd ) performance ndex wthn the requred performance ndex ψ * = 5.6 E-4. The number of replcaton needed for each degn (R) are llutrated n Table 3. In th cae, 0 replcaton hould be allocated to each degn, and a total of 550 replcaton are needed to attan the ame performance ndex a n TOA. Table 3: Number of Replcaton Needed for the TUA Degn 0 3 4 R 0 0 0 0 0 To compare the propoed heurtc (MOCBA) wth TOA and TUA, we generate 0 problem ntance baed on the mean and tandard devaton a llutrated n Table. We et the requred performance ndex ( ψ * ) to be 5.6 E-4. The replcaton needed for the 0 ntance from MOCBA are llutrated n Table 4. On average for the 0 ntance, the total number of replcaton needed for the 5 degn 97, whch le than 400 (N max ) for TOA and 550 for TUA. It eem that the heurtc (MOCBA) even take fewer replcaton than the theoretcal optmal one. The reaon may be due to the fact that the MOCBA equental; and at each tep, t can make ue of the amplng nformaton from the prevou tep to make decon regardng the allocaton of addtonal replcaton. Chen, He, and Yücean (003) alo preented mlar fndng. The followng Table 5 how the performance comparon among the three algorthm. Table 5 llutrate, n our computatonal experment, how replcaton are allocated to each degn for MOCBA, TOA and TUA. Th nclude the number of replcaton allocated to each degn (R) and the correpondng percentage of the total number of replcaton (θ ). From Table 5, we can ee that the dtrbuton of the replcaton among the degn mlar for MOCBA and TOA; ee column θ for MOCBA, TOA and TUA. The reult ndcate that MOCBA effectve. Table 4: Number of Replcaton Needed for the MOCBA Intance Degn No. 0 3 4 0 4 55 34 9 9 40 34 5 5 5 83 3 7 00 3 44 5 6 3 5 4 09 0 57 95 5 5 7 7 9 7 34 6 8 45 0 37 5 7 85 56 6 79 8 5 37 77 9 9 4 56 9 6 8 0 5 5 5 4 37 9 4 40 47 6 95 3 3 63 05 7 4 8 57 0 53 5 5 6 6 7 56 9 6 36 80 4 53 37 7 65 68 5 5 8 8 9 96 7 46 0 9 97 6 5 4 Table 5: Comparon of MOCBA wth TOA and TUA Degn MOCBA TOA TUA No. R θ R θ R θ 0 88 9.5 05 6.3 0 0 0 37. 4 35.5 0 0 7.5 40 0.0 0 0 3 48 6.0 78 9.5 0 0 4 9 9.9 35 8.8 0 0 Total 97 00 400 00 550 00 5. Comparon wth Unform Computng Budget Allocaton (UCBA) In th ecton we preent reult obtaned from comparng MOCBA wth UCBA. In UCBA, we teratvely allocate the ame number of replcaton to each degn, untl the requred performance meaure met. We conder the cae when degn are evaluated n term of 3 objectve. We generate 5 degn baed on gven mean and tandard devaton. The mean ued for each degn are hown n Table 6, and the tandard devaton 3 for all degn. From the mean of the degn, we know that degn 0,, 3, 4, 8 are non-domnated. To tet the robutne of the algorthm, we generate 0 problem ntance.

Table 6: Mean Ued to Generate the 5 Degn Degn No. Mean Mean Mean 3 0 5 44 56 6 4 64 9 43 63 3 4 58 4 8 38 6 5 3 43 64 6 8 45 60 7 9 39 63 8 0 40 60 9 4 6 0 4 44 64 6 46 66 8 48 68 3 30 50 70 4 3 5 7 5 34 54 74 6 36 56 76 7 38 58 78 8 40 60 80 9 4 6 8 0 4 60 80 36 6 8 4 56 68 3 3 6 66 4 34 60 8 In the followng comparon of MOCBA wth UCBA, we et the ntal number of run 0 = 5. Alo, we et the requred performance ndex ψ * = 0.00 a the toppng crteron. Fgure llutrate the number of replcaton needed for MOCBA and UCBA. From Fgure, we can ee that the average peedup of MOCBA over UCBA about tme. Number of Replcaton Needed 4000 3500 3000 500 000 500 000 500 Number of Replcaton for MOCBA and UCBA 0 0 3 4 5 6 7 8 9 0 3 4 5 6 7 8 9 MOCBA UCBA Intance Number Fgure : Comparon of Total Number of Replcaton for MOCBA and UCBA Fgure llutrate for the 0 problem ntance generated, the average number of replcaton that are allocated to each degn for both MOCBA and UCBA. From the MOCBA plot, we can clearly ee that the followng degn are allocated more replcaton: (a) thoe degn that hould be n the Pareto et, and (b) thoe degn whoe performance are very cloe to degn n (a). Th ndcate that our MOCBA algorthm effectve. Average Number of Replcaton 70 60 50 40 30 0 0 0 Average Number of Replcaton for Each Degn 0 3 4 5 6 7 8 9 03456789034 MOCBA UCBA Degn Number Fgure : Comparon of Average Number of Replcaton for Each Degn 6 CONCLUSIONS In th paper, we preent a framework for the rankng and electon problem when the degn are evaluated n term of more than one performance meaure: the mult-objectve rankng and electon (MORS) problem. We ncorporate the concept of Pareto optmalty nto the rankng and electon cheme, and try to fnd all the non-domnated degn n the Pareto et rather than a ngle bet degn. We preent a mple equental oluton method (MOCBA) to olve the problem. Computatonal reult how that the propoed algorthm effcent n term of the total number of replcaton needed to fnd the Pareto et, at leat n comparon wth the TOA (theoretcal optmal allocaton) and TUA (theoretcal unform allocaton) wth known true mean and varance, a well a wth the UCBA (unform computng budget allocaton). Compared to the TOA and TUA, MOCBA take fewer replcaton; whle compared to UCBA, the peedup of MOCBA over UCBA about tme. In the current tudy, we aume that the number of non-domnated degn n the Pareto et known. In future reearch, we may relax th aumpton and propoe method to fnd the correct non-domnated Pareto et. REFERENCES Bechhofer, R. E., T. J. Santner, and D. M. Goldman. 995. Degn and Analy of Experment for Stattcal Selecton, Screenng, and Multple Comparon. New York: Wley. Butler, J., D. J. Morrce, and P. W. Mullarkey. 00. A multple attrbute utlty theory approach to rankng and electon. Management Scence 47(6): 800 86. Chen, C. H., H. C. Chen, and L. Da. 996. A gradent approach for martly allocatng computng budget for

dcrete event mulaton. In Proceedng of the 996 Wnter Smulaton Conference (ed. J. M. Charne, D. J. Morrce, D. T. Brunner, and J. J. Swan), 398 405. Pcataway, New Jerey: Inttute of Electrcal and Electronc Engneer. Chen H. C., L. Da, C. H. Chen, and E. Yücean. 997. New development of optmal computng budget allocaton for dcrete event mulaton. In Proceedng of the 997 Wnter Smulaton Conference (ed. S. Andradóttr, K. J. Healy, D. H. Wther, and B. L. Nelon), 334 34. Pcataway, New Jerey: Inttute of Electrcal and Electronc Engneer. Chen, C. H., D. H. He, and E. Yücean. 003. Better-thanoptmal mulaton run allocaton? In Proceedng of the 003 Wnter Smulaton Conference (ed. S. Chck, P. J. Sánchez, D. Ferrn, and D. J. Morrce), 490 495. Pcataway, New Jerey: Inttute of Electrcal and Electronc Engneer. Chck, S. E. 997. Selectng the bet ytem: A decon theoretc approach. In Proceedng of the 997 Wnter Smulaton Conference (ed. S. Andradóttr, K. J. Healy, D. H. Wther, and B. L. Nelon), 36 333. Pcataway, New Jerey: Inttute of Electrcal and Electronc Engneer. DeGroot, M. H. 970. Optmal Stattcal Decon. McGraw-Hll, Inc. Dudewcz, E. J. and V. S. Taneja. 978. Multvarate rankng and electon wthout reducton to a unvarate problem. In Proceedng of the 978 Wnter Smulaton Conference (ed. H. J. Hghland, L. G. Hull, and N. R. Nelen), 07 0. Pcataway, New Jerey: Inttute of Electrcal and Electronc Engneer. Foneca, C. M. and P.J. Flemng. 995. An overvew of evolutonary algorthm n multobjectve optmzaton. Evolutonary Computaton 3(): 6. Goldman, D. and B. L. Nelon. 994. Rankng, electon and multple comparon n computer mulaton. In Proceedng of the 994 Wnter Smulaton Conference (ed. J. D. Tew, S. Manvannan, D. A. Sadowk, and A. F. Sela), 9 99. Pcataway, New Jerey: Inttute of Electrcal and Electronc Engneer. Goldman, D. and B. L. Nelon. 998. Stattcal creenng, electon, and multple comparon procedure n computer mulaton. In Proceedng of the 998 Wnter Smulaton Conference (ed. D. J. Medero, E. F. Waton, J. S. Caron, and M. S. Manvannan), 59 66. Pcataway, New Jerey: Inttute of Electrcal and Electronc Engneer. Morrce, D. J., J. Butler, and P. W. Mullarkey. 998. An approach to rankng and electon for multple performance meaure. In Proceedng of the 998 Wnter Smulaton Conference (ed. D. J. Medero, E. F. Waton, J. S. Caron, and M. S. Manvannan), 79 75. Pcataway, New Jerey: Inttute of Electrcal and Electronc Engneer. Rnott, Y. 978. On two-tage electon procedure and related probablty-nequalte. Communcaton n Stattc A7 (8): 799 8. Swher, J. R. and S. H. Jacobon. 00. Evaluatng the degn of a famly practce healthcare clnc ung dcrete-event mulaton. Health Care Management Scence 5 (): 75 88. Swher, J. R., S. H. Jacobon, and E. Yücean. 003. Dcrete-event mulaton optmzaton ung rankng, electon, and multple comparon procedure: A urvey. ACM Tranacton on Modelng and Computer Smulaton 3 (): 34 54. Tan, Y. P. 003. Mult-objectve genetc algorthm for robut flght chedulng. Mater The. Department of Indutral and Sytem Engneerng, Natonal Unverty of Sngapore. Tech, J. and R. Schemann. 000. Pareto-front exploraton wth uncertan objectve. Techncal report No. 3/00, Computer Engneerng Laboratory, Unverty of Paderborn. Ztzler, E. and L. Thele. 999. Multobjectve evolutonary algorthm: A comparatve cae tudy and the trength Pareto approach. IEEE Tranacton on Evolutonary Computaton 3(4): 57 7. AUTHOR BIOGRAPHIES LOO HAY LEE an Atant Profeor n the Department of Indutral and Sytem Engneerng, Natonal Unverty of Sngapore. He receved h B.S. (Electrcal Engneerng) degree from the Natonal Tawan Unverty n 99 and h S.M. and Ph.D. degree n 994 and 997 from Harvard Unverty. He currently a enor member of IEEE, vce predent of ORSS, and a member of INFORMS. H reearch nteret nclude mulatonbaed optmzaton, producton chedulng and equencng, logtc and upply chan plannng, and vehcle routng. H emal addre <eleelh@nu.edu.g> and h webte <www.e.nu.edu.g/taff/ leelh/>. EK PENG CHEW an Aocate Profeor and Deputy Head (Academc) n the Department of Indutral and Sytem Engneerng, Natonal Unverty of Sngapore. He receved h Ph.D. degree from the Georga Inttute of Technology. H reearch nteret nclude logtc and nventory management, ytem modelng and mulaton, and ytem optmzaton. H emal addre <ecep@nu.edu.g> and h web addre <www.e.nu.edu.g/taff/chewep/>. SUYAN TENG currently a Reearch Fellow n the Department of Indutral and Sytem Engneerng, Natonal Unverty of Sngapore. She receved her Ph.D. degree from Natonal Unverty of Sngapore n 004.

Her reearch nteret nclude mulaton-baed optmzaton and heurtc algorthm development for olvng vehcle routng and chedulng problem. Her e-mal addre <ety@nu.edu.g>. DAVID GOLDSMAN a Profeor n the School of Indutral and Sytem Engneerng at the Georga Inttute of Technology. H reearch nteret nclude mulaton output analy and rankng and electon theory. He ha done appled work n the health care and tranportaton ndutre. He an actve partcpant n the Wnter Smulaton Conference, havng been Program Char n 995. He repreent the Inttute of Indutral Engneer on the Board of the annual WSC. H emal addre <man@ye.gatech.edu> and h webte <www.ye.gatech.edu/~man>. Lee, Chew, Teng, and Goldman