Efficient Simulation Budget Allocation for Selecting an Optimal Subset 1

Transcription

1 Effcent Smulaton Budget Allocaton for Selectng an Optmal Subset Chun-Hung Chen and Dongha He Department of Systems Engneerng & Operatons Research George Mason Unversty 4400 Unversty Drve, MS 4A6 Farfax, VA 030 Mchael Fu Robert H. Smth School of Busness and Insttute for Systems Research Unversty of Maryland College Park, MD Abstract We consder a varaton of the subset selecton problem n rankng and selecton, where motvated by recently developed global optmzaton approaches appled to smulaton optmzaton, our obectve s to dentfy the top-m out of k desgns based on smulated output. Usng the optmal computng budget framework, we formulate the problem as that of maxmzng the probablty of correctly selectng all of the top-m desgns subect to a constrant on the total number of samples avalable. For an approxmaton of ths correct selecton probablty, we derve an asymptotcally optmal allocaton procedure that s easy to mplement. umercal experments ndcate that the resultng allocatons are superor to other methods n the lterature, and the relatve effcency ncreases for larger problems. Ths work has been supported n part by the atonal Scence Councl of the Republc of Chna under Grant SC 95-8-E , by SF under Grants IIS and DMI-0330, by ASA Ames Research Center under Grants AG--643 and A05CV6G, by FAA under Grant 00-G-06, and by AFOSR under Grant FA Correspondng author: Professor Chun-Hung Chen, Tel: ; Fax: ; Emal: cchen9@gmu.edu; Web: mason.gmu.edu/cchen9

2 . Introducton We consder the problem of selectng the top m out of k desgns, where the performance of each desgn s estmated wth nose (uncertanty). The prmary context s smulaton, where the goal s to determne the best allocaton of smulaton replcatons among the varous desgns n order to maxmze the probablty of selectng all top-m desgns. Ths problem settng falls under the well-establshed branch of statstcs known as rankng and selecton or multple comparson procedures (cf. Bechhofer, Santner, and Goldsman 995). In the context of smulaton, Goldsman and elson (998) provde an overvew of ths feld; see also Andradottr et al. (005). The prmary motvaton for the settng consdered n ths paper comes from some recent developments n global optmzaton that, when appled to the smulaton settng, requre the selecton of an elte subset of good canddate solutons n each teraton of the algorthm. Examples of these nclude genetc algorthms (Holland 975, Chambers 995), the cross entropy method (CE, see Rubnsten and Kroese 004), the model reference adaptve search method (MRAS, cf. Hu, Fu, and Marcus 006ab), and more generally, evolutonary populaton-based algorthms that requre the selecton of an elte populaton n the evolutonary process (see Fu, Hu, and Marcus 006). Instead of tryng to fnd a subset that contans the sngle best among a currently generated set of canddate solutons, the obectve s to fnd an optmal subset such that all members are among the best performers n that canddate set. The reason for ths requrement s that ths entre subset s used to update the subsequent populaton or samplng dstrbuton that drves the search for addtonal canddates. A subset wth poor performng solutons wll result n an update that leads the search n a possbly msleadng drecton. The overall effcency of these types of smulaton optmzaton algorthms depends on how effcently we smulate the canddates and correctly select the top-m desgns. The algorthm developed heren s generc enough that t can be ntegrated wth any such smulaton-based evolutonary optmzaton search algorthms. Most of the rankng-and-selecton research has focused on dentfyng the best desgn. Typcal of these are two-stage or sequental procedures that ultmately return a sngle choce as the estmated optmum, e.g., Dudewcz and Dalal (975) and Rnott (978). Even the tradtonal subset selecton procedures am at dentfyng a subset that contans the best desgn, datng back to Gupta (965), who presented a sngle-stage procedure for producng a subset (of random sze) contanng the best desgn wth a specfed probablty. Extensons of ths work relevant to the smulaton settng nclude Sullvan and Wlson (989), who derve a two-stage subset selecton procedure that determnes a subset of maxmum sze m that, wth a specfed probablty, contans desgns that are all wthn a pre-specfed amount of the optmum. Ths ndfference zone procedure approach also results n a subset of random sze, and the desgns are assumed to follow a normal dstrbuton, wth ndependence between desgns assumed and unknown and unequal moments. The prmary motvaton for such procedures s screenng, whereby the selected subset can be scrutnzed further to fnd the sngle optmum. Ths s n contrast to the motvaton for our settng, as alluded to earler. More recently, these procedures

3 have also been ncorporated nto smulaton optmzaton, but n a dfferent manner, where the rankng-and-selecton procedure s ncorporated n order to be able to make statstcally vald nferences rather than drvng the actual optmzaton process tself; see, e.g., Buchholz and Thümmler (005), Boesel, elson, and Km (003), and elson et al. (00), who also consder the settng of unknown and unequal varances; see the references theren for the cases of known or unknown but equal varances. Swsher, Jacobson, and Yücesan (003) ncludes a dscusson of subset selecton n the context of smulaton optmzaton along ths ven. ote that these approaches are stll focused on selectng a subset contanng the sngle best. As a result, the selected subset may also contan very poor solutons, whch can affect the convergence rate of procedures such as MRAS and the CE method when appled to the smulaton optmzaton settng, where the use of the selecton procedures are n the teratve updatng steps and not n the fnal determnaton of the optmum. To reterate, nstead of selectng the very best desgn from a gven set or fndng a subset that s hghly lkely to contan the best desgn, the obectve n ths papers s to fnd all top-m desgns. About the only substantve work we are aware of addressng ths problem s Koeng and Law (985), who along the lnes of the procedure n Dudewcz and Dalal (975), develop a two-stage procedure for selectng all the m best desgns (see also Law & Kelton 000 for an overvew of the procedure). The number of addtonal smulaton replcatons for the second stage s computed based on a least favorable confguraton, resultng n very conservatve allocatons, so that the requred computatonal cost s much hgher than actually needed. To mprove the effcency of allocatng smulaton replcatons among competng desgns, Chen et al. (997, 000), Chen and Kelton (000), Chck and Inoue (00ab), Hyden and Schruben (000), Lee and Chew (003), Tralovc and Pao (004), and Fu et al. (006) have approached the rankng-and-selecton problem from the perspectve of allocatng a fxed number of smulaton replcatons n order to maxmze the probablty of correct selecton, under a framework called optmal computng budget allocaton. Intutvely, to ensure a hgh probablty of correct selecton, a larger porton of the computng budget should be allocated to those desgns that are crtcal n the process of dentfyng the best desgn. In terms of tradtonal rankng and selecton, for example, ths results n the use of both the means and varances n the allocaton procedures (for normally dstrbuted desgn performances), rather than ust the varances, as n Dudewcz and Dalal (975) and Rnott (978). However, all of ths work has focused on selectng the sngle best, and there has been no research nvolvng subset selecton. Ths paper ams to fll ths gap by provdng an effcent allocaton procedure for selectng the m best desgns. ote that among the selected m desgns, there s no further rankng done wthn the set. Agan, ths s consstent wth the requrements of the CE method and MRAS approach, as well as other evolutonary populaton-based methods that requre an elte populaton of some type. The paper s organzed as follows. In the next secton, we formulate the optmal computng budget allocaton problem for selectng the top-m desgns. Secton 3 derves an allocaton scheme based on approxmatng the correcton selecton probablty and then carryng out an 3

4 asymptotc analyss. The performance of the technque s llustrated wth a seres of numercal examples n Secton 4. Secton 5 concludes the paper.. Problem Statement We ntroduce the followng notaton: T = total number of smulaton replcatons (budget), k = total number of desgns, m = number of top desgns to be selected n the optmal subset, S m = set of m (dstnct) ndces ndcatng desgns n selected subset, = number of smulaton replcatons allocated to desgn, X = -th smulaton replcaton for desgn, J = = X, sample mean for desgn, J = mean for desgn, = varance for desgn,, = J - J. The obectve s to fnd a smulaton budget allocaton that maxmzes the probablty of selectng the optmal subset, defned as the set of m (< k) best desgns, for m a fxed number. Our approach s developed based on Bayesan settng (e.g., Inoue and Chck 998). The mean of the smulaton output for each desgn, J, s assumed unknown and treated as a random varable, whose posteror dstrbuton s updated as smulaton proceeds. Wthout loss of generalty, we wll take as the m best desgns those desgns wth the m smallest means (but ths s unknown), so that n terms of our notaton, the correct selecton event s defned by S m contanng all of the m smallest mean desgns: I I CS m { J J ) } = { S m S m max J mn J }. () ( S S The optmal computng budget allocaton (OCBA) problem s gven by m m max, L, k P{CSm} s.t k = T. () 4

5 Here k denotes the total computatonal cost assumng the smulaton executon tmes for dfferent desgns are roughly the same. Ths formulaton mplctly assumes that the computatonal cost of each replcaton s constant across desgns. The smulaton budget allocaton problems gven n Chen et al. (000) s actually a specal case of () wth m =. For notatonal smplfcaton, we wll drop the m n P{CS m } n the remanng dscusson. ote that rank order wthn the subset s not part of the obectve. In ths paper, we wll take S m to be the m desgns wth the smallest sample means. Let J r be the r-th smallest (order statstc) of { J, J,..., J k },.e., J J... J k. Then, the selected subset s gven by S m {,,..., m }. We assume that the smulaton output samples {X } are normally dstrbuted and ndependent from replcaton to replcaton,.e., X, X,..., X, are..d. (J, ), as well as ndependent across desgns. The normalty assumpton s typcally satsfed n smulaton, because the output s obtaned from an average performance or batch means, so that Central Lmt Theorem effects usually hold. 3. Approxmate Asymptotcally Optmal Allocaton Scheme To solve the OCBA problem (), we estmate P{CS} usng the Bayesan model presented n Chen et al. (000) and He et al. (006). After the smulaton s performed, a posteror dstrbuton for the unknown mean J, p( J X, =,..., ), s constructed based on two peces of nformaton: () pror knowledge of the system s performance, and () current smulaton output. Thus, n the Bayesan framework, the probablty of correct selecton defned by () s gven by P{CS} = P{ J < J, S m and S m }, (3) where J, =,...,k, denotes the random varable whose probablty dstrbuton s the posteror dstrbuton of desgn. As n Chen et al. (000), we assume that the unknown mean J has a conugate normal pror dstrbuton and consder non-nformatve pror dstrbutons, whch mples that no pror knowledge s avalable about the performance of any desgn before conductng the smulatons, n whch case the posteror dstrbuton of J s (cf. DeGroot 970) J ( J, ). After the smulaton s performed, J can be calculated, can be approxmated by the sample varance, and the P{CS} gven by Equaton (3) can then be estmated usng Monte Carlo smulaton. However, snce estmatng P{CS} va Monte Carlo smulaton s tme-consumng and the purpose of budget allocaton s to mprove smulaton effcency, we adopt an approxmaton of P{CS} usng a lower bound. 5

6 3. Approxmatng the Probablty of Correct Selecton For a constant c, P{CS} = P{ J < P{ J < c and J, S m and S m } J > c, S m and S m } = P { J c} P { J c} APCSm, (4) S m S m where the last lne s due to ndependence across desgns. We refer to ths lower bound for P{CS}, whch can be computed easly and elmnates the need for extra Monte Carlo smulaton, as the Approxmate Probablty of Correct Selecton for m best (APCSm). Determnng an approprate value for c wll be deferred to later ths secton. Usng the approxmaton gven by Equaton (4), the OCBA problem () becomes max,, k S m P { J c} P { J c} S m s.t k = T. (5) ow we solve OCBA problem (5), assumng the varables { } are contnuous. 3. Asymptotcally Optmal Soluton For notaton smplfcaton, we defne the varable = J - c, =,,,k. For S m, P( J 0 < c)= π ( ) e ( x ) ( ) dx = ( ) π t e dt, and for S m, 6

7 P( J > c) = 0 π ( ) e ( x ) ( ) dx = ( ) π t e ow let F be the Lagrangan relaxaton of (5), wth Lagrange multpler λ: F = S m dt. P { J c} P J k { c} - λ( -T) S m = = S m ( ) π t e dt S m ( ) t e k dt - λ( -T). π = Furthermore, the Karush-Kuhn-Tucker (KKT) (Walker 999) condtons of ths problem can be stated as follows. For S m, F = P { J c} P { J c} S m S m ( ) e π - λ = 0. (6) For S m, F = P { J c} P { J c} S S m ( ) e m π - λ = 0. (7) Also, F k = 0 returns the budget constrant T =0. λ = To examne the relatonshp between and for, we consder three cases: () S m, and S m : Equatng the expressons n Equatons (6) and (7), P { J r c} r S m r S m r P { J r c} ( ) e π - λ 7

8 = P { J r c} P { J r c} S r S m Smplfyng, ( ) e r m π r - λ. P{ J c} Takng the log on both sdes, ( ) e = P{ J c} e ( ). log( P{ J c} ) +log( )- log( )=log( P{ J c} ) +log( )- log( ). (8) ow, we consder the asymptotc lmt T wth = α T, Equaton (8) becomes k = α =. Substtutng for, log( α P{ J c} ) T +log( )- log( α T ) α =log( P{ J c} ) T +log( )- log( α T ). Dvdng by T, log( P{ J c } ) α T + log( )- log( α T T ) T = log( P{ J c } ) T α + log( T )- log( α T ). T and then takng T yelds α = α. Therefore, we obtan the rato between α and α or between and as: = α α = for S m, and S m. (9) 8

9 () Both, S m and.: From Equaton (6), F F = =0 yelds r S m r P { J r c} P { J r c} r S m ( ) e π ( ) 3 - λ = P { J r c} P { J r c} r S r m r S m ( ) e π ( ) - λ. 3 Then, P{ J c} e ( ) = P{ J c} e ( ). Followng the analogous dervaton that led to Equaton (9) yelds the same result = α α = for, S m and. (0) (3), S m, and : Agan, followng the same dervaton procedures that led to Equatons (9) and (0) yelds = α α = for, S m and. () Thus, snce Equatons (9), (0), and () are dentcal, we wrte = α α =,, {,,..., k}, and. () In concluson, f a soluton satsfes Equaton (), then the KKT suffcent condtons must hold asymptotcally, so that the correspondng soluton s a locally optmal soluton to the Lagrangan relaxaton of the OCBA problem (5). We therefore have the followng result. 9

10 Theorem. The allocaton gven by () s asymptotcally (as T ) a locally optmal soluton for OCBA problem (5), where = J - c, for c a constant, and the varances,, K, k are fnte,.e., APCSm s asymptotcally maxmzed by the allocaton gven by (). 3.3 Determnaton of c Value The parameter c mpacts the qualty of the approxmaton APCSm to P{CS}. Snce APCSm s a lower bound of P{CS}, choosng c to make APCSm as large as possble s lkely to provde a better approxmaton of APCSm to P{CS}. Fgure s provded to help explan our choce of c, by gvng an example of probablty densty functons for, =,,..., k. J ote that APCSm s a product of P{ J < c} for S m and P{ J > c} for S m. Consder the case Var( J ) = Var( J ) =... = Var( J ). For any c, P{ J k < c} > P{ J < c} >... > P{ J m < c}, and P{ J m+ > c} < P{ J m+ > c} <... < P{ J k > c}. To prevent APCSm from beng small, we want to choose c to avod any of the product terms beng too small, partcularly for P{ J m < c} and P{ J m+ > c}, because one of these two terms s the smallest one n the product no matter what c s. A good choce of c s a number between J and J, because m () f c = c' < J, then P{ J m m < c'} < 0.5. The smaller (c' - J ), the smaller P{ J m m resultng n a negatve mpact on APCSm; () f c = c" > J m+, then P{ J m+ > c"} becomes small and so does APCSm. m+ < c}, Wth these consderatons, one would lke to maxmze both (c - J m ) and ( J m+ - c), or to maxmze both P{ J m < c} and P{ J m+ > c}. We choose c = ( J + J m m+ ) /, whch n theory maxmzes {(c - J ) + ( J m m+ - c) }, and n numercal testng results n good performance whle requrng neglgble computaton cost. 0

11 J J m J m+ J k J J J m m+ J k c' c c" Fgure. An example of probablty densty functons for J, =,,..., k, c' < J m < c < J m+ < c". 3.4 Sequental Allocaton Scheme The allocaton gven by () assumes known varances. In practce, a sequental algorthm s used to estmate these quanttes usng the updated sample varances. Furthermore, the constant c and sample means are also updated durng each teraton. Each desgn s ntally smulated wth n 0 replcatons n the frst stage, and addtonal replcatons are allocated ncrementally wth replcatons to be allocated n each teraton. In summary, we have the followng algorthm (assumng T-kn 0 s a multple of ). OCBA-m Allocaton Procedure IITIALIZE l 0; l l l Perform n 0 smulaton replcatons for all desgns; = = L = k = n0. k = l LOOP WHILE < T DO UPDATE Calculate sample means and sample varance usng the new smulaton output; Compute c = ( J + J m m+ ) /. ALLOCATE Increase the computng budget by and calculate the new budget allocaton, l+ l+ l+,,,, accordng to (). k l+ l SIMULATE Perform addtonal max( -,0) smulatons for desgn, =,,k; ED OF LOOP l l +.

12 4. umercal Testng and Comparson wth Other Allocaton Procedures In ths secton, we test the OCBA-m algorthm by comparng t on several numercal experments wth dfferent allocaton procedures: Equal Allocaton, whch smulates all desgn alternatves equally; the Koeng and Law (985) procedure denoted by ; Proportonal To Varance (PTV), whch s a modfcaton of that allocates replcatons proportonal to the estmated varances; and the OCBA allocaton algorthm for selectng only the best desgn (Chen et al. 000), denoted by OCBA-. For notatonal smplcty, we assume J [] < J [] < < J [k], so desgn [] s the best and correct selecton would be S m = {[], [],..., [m]} (but ths s unknown a pror). 4. Computng Budget Allocaton Procedures Equal Allocaton The smulaton budget s allocated equally to all desgns,.e., = T/k for each. The performance of equal allocaton wll serve as a benchmark for comparson. ( Koeng and Law 985) The two-stage procedure of Koeng and Law (985) selects a subset of specfed sze m, wth probablty at least P*, so that the selected subset s exactly the actual subset wth the best (smallest) expected values, provded that J [m+] - J [m] s no less than an ndfference zone, d. As n our settng, the orderng wthn the selected subset does not matter. In the frst stage, all desgns are smulated for n 0 samples. Based on the sample varance estmate ( S ) obtaned from the frst stage and gven the mnmum correct selecton probablty P*, the number of addtonal smulaton samples for each desgn n the second stage s determned by: = max(n 0 +, h 3 S ( n 0 ) / d ), for =,,, k, (3) where s the nteger round-up functon, and h 3 s a constant that depends on k, P*, and n 0. Proportonal To Varance (PTV) Ths s a sequental modfed verson of the procedure, based on the observaton that (3) mples that s proportonal to the estmated sample varances. Thus, the PTV procedure sequentally determnes {} based on the newly updated sample varances by replacng the ALLOCATE step n the OCBA-m algorthm by S l+ l = C S, for =,,, k,

13 where C l s determned based on the total smulaton samples equals to the gven computng budget at teraton l. ote that the ndfference-zone parameter has been removed n ths modfcaton n order to make t comparable to the other procedures. OCBA- (Chen et al. 000) The sequental OCBA procedure of Chen et al. (000) allocates the computng budget wth the obectve of selectng the best desgn,.e., m =, for whch extensve numercal testng has demonstrated ts effcency (e.g., Branke et al. 006). Whle t s not desgned for m >, we test ths procedure here for benchmarkng purposes, and denote t be OCBA-. 4. umercal Results To compare the performance of the procedures, we carred out numercal experments for several typcal selecton problems. In comparng the procedures, the measurement of effectveness used s the P{CS} estmated by the fracton of tmes the procedure successfully fnds all the true m- best desgns out of 00,000 ndependent experments. Because ths penalzes ncorrect selectons equally e.g., a subset contanng the top-, top-,..., and top-(m-) desgns and mssng only the top-m desgn s treated no dfferently than a subset contanng not a sngle of the top-m desgns n our numercal experments, we also nclude a second measure of selecton qualty, the so-called expected opportunty cost E[OC], where m OC ( J. = J [ ] ) Ths measure penalzes partcularly bad choces more than mldly bad choces. For example, when m = 3, a selecton of {top-, top-, top-4} s better than {top-, top-, top-5}, and both are better than {top-, top-3, top-5}. ote that OC returns a mnmum value of 0 when all the top-m desgns area correctly selected. The estmated E[OC] s the average of OC estmate over the 00,000 ndependent experments. Each of the procedures smulates each of the k desgns for n 0 = 0 replcatons ntally (followng recommendatons n Koeng and Law 985 and Law and Kelton 000). allocates addtonal replcatons n a second stage (so the total number s not fxed a pror), whereas the other procedures allocate replcatons ncrementally by =50 each tme untl the total budget, T, s consumed. For each level of computng budget, we estmate the acheved P{CS} and E[OC]. Snce s a two-stage ndfference-zone procedure, we must specfy the values for the desred probablty of correct selecton, P*, and the ndfference zone d to satsfy the condton that J [m+] - J [m] d, where a smaller d mples a hgher requred computaton cost based on Equaton (3). In practce, the value of J [m+] or J [m] s unknown beforehand, but for benchmarkng purposes, we set d = J [m+] - J [m], whch leads to the mnmum computatonal requrement (or maxmum effcency) for the procedure. As s done for the other procedures, the resultng P{CS} and E[OC] can be estmated over the 00,000 ndependent experments. Snce 3

14 the requred computaton cost also vares from one experment to another, we wll ndcate the average number of total replcatons based on the 00,000 ndependent experments. Example. Equal varance There are 0 alternatve desgns, wth dstrbuton (, 6 ) for desgn =,,, 0. The goal s to dentfy the top-3 desgns va smulaton samples,.e., m=3 n ths example. To characterze the performance of dfferent procedures as a functon of T, we vary T between 00 and 8000 for all of the procedures other than, and the estmated acheved P{CS} and E[OC] as a functon of T s shown n Fgure. For, we test two cases P* = 0.9 and P* = 0.95, and the correspondng estmated P{CS} and E[OC] vs. the average total smulaton replcatons are shown as two sngle ponts (the trangle and crcle) n Fgure. We see that all procedures obtan a hgher P{CS} and lower E[OC] as the avalable computng budget ncreases. However, OCBA-m acheves the hghest P{CS} and lowest E[OC] for the same amount of computng budget. It s nterestng to observe that OCBA-, whch performs sgnfcantly better than Equal Allocaton and PTV when the obectve s to fnd the sngle best desgn, fares worse n ths example than these two allocatons when the obectve s changed to fndng all the top-3 desgns. Equal allocaton performs almost dentcally to PTV, whch makes sense, snce the varance s constant across desgns. Specfcally, the computaton costs to attan P{CS} = 0.95 for OCBA-m, OCBA-, Equal, and PTV are 800, 300, 950, 000, respectvely. ot surprsngly, the performance of s along the performance curve of PTV, snce bascally allocate the computng budget based on desgns' varance. However, acheves a substantally hgher P{CS} than the desred level (e.g., exceedng 0.99 for the target mnmum of P* = 0.9) by spendng a much hgher computng budget than actually needed, consstent wth the fact that typcal two-stage ndfference-zone procedures are conservatve. 4

15 Equal PTV OCBA- OCBA-m P*=90% P*=95% P{CS} T Fgure a. P{CS} vs. T usng four sequental allocaton procedures and (trangle for P*=90% and crcle for P*=95%) for Example Equal PTV OCBA- OCBA-m E[OC] T P*=90% P*=95% Fgure b. E[OC] vs. T usng four sequental allocaton procedures and (trangle for P*=90% and crcle for P*=95%) for Example. 5

16 Example. Varance ncreasng n value of mean Ths s a varant of Example. All settngs are preserved except that the varance s ncreasng n the desgn ndex, so good desgns have smaller varances. Specfcally, the desgns are dstrbuted (, ) for desgn =,,, 0. Agan, m = 3. The test results shown n Fgure 3 are qualtatvely smlar to those n Example. OCBA-m acheves the hghest P{CS} for the same amount of computng budget. However, PTV (and ) performs poorly n ths example because good desgns receve relatvely less computng budget due to ther smaller varances, whch tend to slow down the process of dstngushng good desgns. Specfcally, the computaton costs to attan P{CS} = 0.95 for OCBA-m, OCBA-, Equal, and PTV are 350, 750, 700, 50, respectvely. Equal PTV OCBA- OCBA-m P*=90% P*=95% P{CS} T Fgure 3a. P{CS} vs. T usng four sequental allocaton procedures and (trangle for P*=90% and crcle for P*=95%) for Example. 6

17 Equal PTV OCBA- OCBA-m E[OC] T 3800 P*=90% P*=95% Fgure 3b. E[OC] vs. T usng four sequental allocaton procedures and (trangle for P*=90% and crcle for P*=95%) for example. Example 3. Varance decreasng n value of mean The thrd example s another varant of Examples and, but ths tme the varance s decreasng n the desgn ndex,.e., the dstrbuton s (, (-) ) for desgn =,,, 0. Under ths settng, good desgns have larger varance. Agan, m = 3. The test results shown n Fgure 4 are smlar to those n the prevous examples, wth agan OCBA-m performng the best. However, n contrast to Example, PTV (and ) performs relatvely well n ths example, because good desgns receve much more computng budget due to ther hgher varances. On the other hand, OCBA- performs poorly, because t spends an excess amount of the computng budget to dstngush between the very top desgns, snce ts obectve s to fnd the best. In ths example, the computaton costs to attan P{CS} = 0.95 for OCBA-m, OCBA-, Equal, and PTV are 400, 7900, 3050, 00, respectvely

18 Equal PTV OCBA- OCBA-m P*=90% P*=95% P{CS} T Fgure 4a. P{CS} vs. T usng four sequental allocaton procedures and (trangle for P*=90% and crcle for P*=95%) for Example Equal PTV OCBA- OCBA-m E[OC] P*=90% 5300 T P*=95% Fgure 4b. E[OC] vs. T usng four sequental allocaton procedures and (trangle for P*=90% and crcle for P*=95%) for Example 3. 8

19 Example 4. (s,s) nventory problem The fourth example s a typcal (s, S) nventory polcy problem based on the example that was ntroduced by Koeng and Law (985) and later analyzed by elson and Mateck (995). We extend the sze of that orgnal example from 5 desgns to 0 desgns. When random demand brngs the nventory of system on hand down to s unts, the nventory s reordered to level S., for =,,, 0. The 0 systems are defned by the parameters (s, s,, s 0 ) = (0, 0, 0, 40, 40, 40, 60, 60, 60, 80) and (S, S,..., S 0 ) = (30, 40, 50, 50, 60, 70, 70, 80, 90, 90), respectvely. The systems wth polcy (s 3, S 3 ), (s 6, S 6 ) and (s, S ) are the top-3 desgns (m = 3). The test results shown n Fgure 5 are smlar to those n prevous examples, n that OCBA-m s clearly the top performer agan; however, ths tme OCBA- s the runner up by a slght margn. The computaton costs to attan P{CS} = 0.95 for OCBA-m, OCBA-, Equal, and PTV are 500, 00, 650, 350, respectvely. Equal PTV OCBA- OCBA-m P*=90% P*=95% P{CS} T Fgure 5a. P{CS} vs. T usng four sequental allocaton procedures and (trangle for P*=90% and crcle for P*=95%) for Example 4. 9

20 Equal PTV OCBA- OCBA-m E[OC] T P*=90% P*=95% Fgure 5b. E[OC] vs. T usng four sequental allocaton procedures and (trangle for P*=90% and crcle for P*=95%) for Example 4. Example 5. Larger-scale problem Ths s a varant of Example (constant varance), wth the number of desgns ncreased to 50. The alternatves have dstrbuton (, 0 ) for desgn =,,, 50, and m = 5. Snce 's performance bascally follows that of PTV, but ts requred computng budget s far beyond the range we are consderng here, we exclude from the numercal testng. Fgure 6 depcts the smulaton results. As n earler examples, OCBA-m acheves the hghest P{CS} and the lowest E[OC] wth the same amount of computng budget; however, the performance gap between OCBA-m and other procedures s substantally greater. Ths s because a larger desgn space allows the OCBA-m algorthm more flexblty n allocatng the computng budget, resultng n even better performance. On the other hand, OCBA- performs poorly, because t spends a lot of computng budget on dstngushng the very top desgns, a tendency that s penalzed even more for larger m. Agan, snce the varance s constant across desgns, the performance of Equal and PTV are nearly ndstngushable. In ths example, the computaton costs to attan P{CS} = 0.95 for OCBA-m, OCBA-, Equal, and PTV are 4050, 3050, 7050, 700, respectvely. 0

21 Equal PTV OCBA- OCBA-m P{CS} T Fgure 6a. P{CS} vs. T usng four sequental allocaton procedures for Example Equal PTV OCBA- OCBA-m E[OC] T Fgure 6b. E[OC] vs. T usng four sequental allocaton procedures for Example 5.

22 5. Conclusons We present an effcent allocaton procedure for dentfyng the top-m desgns out of k (smulated) competng desgns. The obectve s to maxmze the smulaton effcency, expressed as the probablty of correct selecton wthn a gven computng budget. We propose a heurstc to approxmate the assocated correct selecton probablty, and then derve an asymptotcally optmal allocaton procedure for ths approxmate probablty. umercal testng ndcates that the allocaton procedure s sgnfcantly more effcent and robust than other methods n the lterature, wth the relatve effcency ncreasng n problem sze. Furthermore, although the procedure was derved based on an asymptotc dervaton, the numercal results ndcate that the procedure s effectve even when the computng budget s small. Fnally, the numercal results llustrate that the allocaton specfed by the orgnal OCBA algorthm (Chen et al. 000), desgned for selectng the sngle best desgn, does not perform well n selectng the top-m desgns, provdng another motvaton for the need of a new methodology when the obectve s extended beyond selectng ust the best desgn. References. Andradottr, S., D. Goldsman, B. W. Schmeser, L. W. Schruben, and E. Yücesan, Analyss Methodology: Are We Done? Proceedngs of the 005 Wnter Smulaton Conference, pp , December Bechhofer, R.E., T.J. Santner, and D.M. Goldsman, Desgn and Analyss of Experments for Statstcal Selecton, Screenng, and Multple Comparsons, John Wley & Sons, Boesel, J., B.L. elson, and S.H. Km, Usng Rankng and Selecton to Clean up After Smulaton Optmzaton, Operatons Research 5, 84-85, Branke, J., S. E. Chck, and C. Schmdt, Selectng a Selecton Procedure, submtted to Management Scence, Buchholz, P. and A. Thümmler, Enhancng Evolutonary Algorthms wth Statstcal Selecton Procedures for Smulaton Optmzaton, Proceedngs of the Wnter Smulaton Conference, 84-85, Chambers, L., Practcal Handbook of Genetc Algorthms, CRC Press, Chen, C. H., J. Ln, E. Yücesan, and S. E. Chck, Smulaton Budget Allocaton for Further Enhancng the Effcency of Ordnal Optmzaton, Journal of Dscrete Event Dynamc Systems: Theory and Applcatons, Vol. 0, pp. 5-70, Chen, E. J. and W. D. Kelton, An Enhanced Two-Stage Selecton Procedure, Proceedngs of the Wnter Smulaton Conference, pp , Chen, H. C., C. H. Chen, L. Da, and E. Yucesan, ew Development of Optmal Computng Budget Allocaton For Dscrete Event Smulaton, Proceedngs of the 997 Wnter Smulaton Conference, pp , December 997.

23 0. Chck, S. and K. Inoue, ew Two-Stage and Sequental Procedures for Selectng the Best Smulated System, Operatons Research, Vol. 49, pp , 00.. Chck, S. and K. Inoue, ew Procedures to Select the Best Smulated System Usng Common Random umbers, Management Scence, 47(8), pp , 00.. DeGroot, M. H., Optmal Statstcal Decsons. McGraw-Hll, Inc., Dudewcz, E. J. and S. R. Dalal, Allocaton of Observatons n Rankng and Selecton wth Unequal Varances, Sankhya, B37, pp. 8-78, Fu, M. C., J. Q. Hu, C. H. Chen, and X. Xong, Smulaton Allocaton for Determnng the Best Desgn n the Presence of Correlated Samplng, IFORMS Journal on Computng, accepted for publcaton, Fu, M. C., J. Hu, and S. I. Marcus, Model-Based Randomzed Methods for Global Optmzaton, Proceedngs of the 7th Internatonal Symposum on Mathematcal Theory of etworks and Systems, , Goldsman, D. and B. L. elson, Comparng Systems va Smulaton, J. Banks, ed. Handbook of Smulaton: Prncples, Methodology, Advances, Applcatons, and Practce, John Wley & Sons, ew York. pp , Gupta, S. S., On Some Multple Decson (Selecton and Rankng) Rules, Technometrcs 7: 5 45, He, D., S. E. Chck, C. H. Chen, The Opportunty Cost and OCBA Selecton Procedures n Ordnal Optmzaton, to appear n IEEE Transactons on Systems, Man, and Cybernetcs-- Part C, Holland, J. H., Adaptaton n atural and Artfcal Systems, The Unversty of Mchgan Press, Hu, J., M. C. Fu, and S. I. Marcus, A Model Reference Adaptve Search Algorthm for Global Optmzaton, Operatons Research, accepted for publcaton, 006a.. Hu, J., M. C. Fu, and S. I. Marcus. A Model Reference Adaptve Search Algorthm for Stochastc Global Optmzaton, workng paper, 006b.. Hyden, P. and L. Schruben, Improved Decson Processes Through Smultaneous Smulaton and Tme Dlaton, Proceedngs of the 000 Wnter Smulaton Conference, pp , Inoue, K., and S. E. Chck, "Comparson of Bayesan and Frequentst Assessments of Uncertanty for Selectng the Best System," Proceedngs of the 998 Wnter Smulaton Conference, pp December, Koeng, L. W. and A. M. Law, A Procedure for Selectng a Subset of Sze m Contanng the l Best of k Independent ormal Populatons, Communcaton n Statstcs - Smulaton and Communcaton, B4, pp , Km, S.-H. and elson, B.L Selectng the best system. Chapter 8 n Handbooks n Operatons Research and Management Scence: Smulaton, S.G. Henderson and B.L. elson, eds., Elsever. 6. Law, A. M. and W. D. Kelton, Smulaton Modelng & Analyss. McGraw-Hll, Inc.,

24 7. Lee, L. H. and E. P. Chew, A Smulaton Study on Samplng and Selectng under Fxed Computng Budget, Proceedngs of 003 Wnter Smulaton Conference, pp , December elson, B. L. and F. J. Mateck, Usng Common Random umbers for Indfference-Zone Selecton and Multple Comparsons n Smulaton, Management Scence, 4, pp , Rubnsten, R.Y. and D.P. Kroese, The Cross-Entropy Method: A Unfed Approach to Combnatoral Optmzaton, Monte-Carlo Smulaton, and Machne Learnng. Sprnger, Sullvan, D. W. and J. R. Wlson, Restrcted Subset Selecton Procedures for Smulaton, Operatons Research, 37:5 7, Swsher, J.R., S.H. Jacobson, and E. Yücesan, Dscrete-Event Smulaton Optmzaton Usng Rankng, Selecton, and Multple Comparson Procedures: A Survey, ACM Transactons on Modelng and Computer Smulaton 3, 34-54, Walker, R. C, Introducton to Mathematcal Programmng, Prentce Hall, Upper Saddle Rver, J, Tralovc, L. and L. Y. Pao, Computng Budget Allocaton for Effcent Rankng and Selecton of Varances wth Applcaton to Target Trackng Algorthms, to appear n IEEE Transactons on Automatc Control,