B. Maddah ENMG 6 Simulatio Output Aalysis (, Chapters 10 &11 Law) Comparig alterative system cofiguratio Sice the output of a simulatio is radom, the comparig differet systems via simulatio should be doe o the basis of a statistical aalysis. For example, whe comparig two systems, oe system ca be better o some replicatios ad worse o others. Tellig which system is better requires ca be oly doe approximately based o statistical aalysis. Example 1 Suppose a bak is cosiderig two possible ATM systems. o Zippy: M/M/1 queue with arrival rate 1 ad oe fast server with mea service time 0.9 miutes. o Kluky: M/M/ queue with arrival rate 1 ad two slow servers with mea service time 1.8 miutes each. 1
The performace measure of iterest is the mea delay of the first 100 customers, d(100) (system starts empty). The true measures (evaluated based o exact aalysis) are d Z (100) = 4.13 ad d K (100) = 3.70. So Kluky is better. Simulatig these two systems ad comparig the output of oe ru at a time i 100 replicatios, gives the followig. I 53 Out of 100 replicatios, d Z (100) < d K (100). That is, P{wrog aswer} = 0.53. This meas that a aalysis based o a sigle ru is likely to lead to wrog aswers. To improve the compariso oe could simulate each system for replicatio ad the base the decisio o comparig the averages of the replicatios.
Doig 100 comparisos this way (with a total of 100 replicatios) gave the followig results for differet s. A dot plot illustrates what s happeig. This example highlights the eed for methods to assess the ucertaity ad give statistical bouds or guaratees for coclusios ad decisios. 3
Cofidece Itervals for the Differece of Systems Cosider two alterative systems with performace measures i, i = 1, (the mea of somethig). Suppose that i replicatios are made for system i. Let X ij be the observatio from system i o replicatio j. The idea is to use the X ij s to build a cofidece iterval for If the iterval misses 0, we coclude there is a statistical differece betwee the two systems. We ca also have a idea of how sigificat is the differece betwee the two systems. There are two approaches for buildig cofidece itervals for Paired-t ad two-sample-t. Paired-t cofidece Iterval Assume replicatios of each system are performed. Let Z j = X 1j X j. The, the Z j s are IID. The paired-t approach works by buildig a cofidece iterval based o the sample mea ad variace for Z, Z ( Z Z ( )) j j1 j1 Z Z ( ), S ( ) j. 1 4
Assume that the Z j s are ormally distributed (which may be justified by the cetral limit theorem). The, the followig is a approximate 100(1 ) percet cofidece iterval for SZ ( ) Z ( ) t 1,1 /. Oe importat fact about the paired-t cofidece iterval is that X 1j ad X j eed ot to be idepedet. This could be useful whe utilizig a variace reductio techique. Example Cosider comparig two alterative orderig policies i a (s, S) system ((0, 40) ad (0, 80)). The measure of performace of iterest is the average mothly cost i a 10 moth horizo. The simulatio results of five replicatios of the two systems are as follows. 5
The, Z(5) 4.98 ad S Z (5) 1.19, ad usig a 10% sigificace level, t 4, 0.95 A approximate 90% t-paired cofidece iterval is 1.19 4.98.13 4.98 3.33 (1.65,8.31). 5 Sice the iterval does ot cotai 0, this implies that the two (s, S) policies are sesibly differet. I additio, the secod policy seems to be better (as it has lower cost). A two-sample cofidece iterval This method allows samples of uequal sizes (i.e. 1 ) from both systems. However, it requires that X 1j ad X j be idepedet. This ca be doe by usig differet radom umber streams for obtaiig X 1j ad X j. The method works by estimatig the mea ad variace for each sample separately. That is, oe first estimates 1 1 X ( X X ( )) 1 j 1 j 1 1 j1 j1 1 1 1 1 1 1 1 X ( ), S ( ), X ( X X ( )) j j j1 j1 1 X ( ), S ( ). 6
A approximate 100(1 ) % cofidece iterval for is S1 ( 1 ) S ( ) X1( 1 ) X ( ) t fˆ,1 /, 1 where S1 ( 1 ) / 1 S ( ) / fˆ. S1 ( 1 ) / 1 /( 1 1) S ( ) / /( 1) Example 3 Applyig the two-sample cofidece iterval approach for the ivetory system i Example, gives X (5) 15.57, X (5) 10.59, S (5) 4, S (5) 3.76. 1 1 The, f ˆ 7.99 8, ad t ˆ,0.95 1.86, ad approximate 90% cofidece iterval is f 4.98 1.86(1.46) 4.98.3 (.66, 7.3). This also idicates that the two policies are sesibly differet with the secod policy beig better. Paired-t versus two-sample cofidece iterval 7
Comparig steady-state measures of two system The approaches for buildig cofidece itervals require iid observatios X 11, X 1,, X 11, ad X1, X,, X each system. These ca be obtaied easily for termiatig simulatio. For o-termiatig steady-state simulatios such observatios ca be obtaied by the methods discussed before such as replicatio/deletio ad batch meas. Cofidece Itervals for the Differece of > Systems If more that two systems are to be compared, the, from cofidece itervals for the differece of the performace measure betwee differet pairs of the systems are required. A key issue here is the validity of idividual cofidece itervals for multiple comparisos. Specifically, if c cofidece itervals are to be developed with a overall cofidece level of 1 the each iterval must have a cofidece level 1 c. (This is called the Boferroi iequality.) This implies that makig multiple comparisos require a large umber of replicatios to achieve the high cofidece level 1 c. 8
Comparig with a stadard Here, oe of the systems (say system 1) is the stadard (e.g., the existig cofiguratio). We eed to compare each of the other k systems with the stadard, ad obtai cofidece itervals o k The, to obtai a overall cofidece of 1 each iterval should have a cofidece level 1 k 1). Example 4 The simulatio of five differet (s, S) policies i a ivetory system gave the followig results i 5 replicatios. The performace measure of iterest is the average mothly cost over a horizo o 10 moths. The first policy is cosidered the stadard system. 9
The desired overall cofidece level is 90%. So, itervals for k k =,3,4,5, are developed with a cofidece level (1 0.1/4)% = 97.5%. The cofidece itervals are as follows. Here we see that system 4 ad 5 are obviously worse tha the stadard, while system ad 3 are ot ecessarily better. All pair-wise compariso Here all systems are compared pair-wise. This would be doe if all alteratives receive equal cosideratio. Cofidece itervals for i 1 i, i 1 = 1,, k, i = 1,, k, i 1 i. The total umber of comparisos to be made is k(k 1)/. So to obtai a overall cofidece of 1 the cofidece level for every iterval should be 1 k(k 1)/]. 10
Variace Reductio Variace reductio techiques (VRT) are methods to reduce the variace (i.e. icrease precisio) of simulatio output without doig more rus. They are based o settig up the simulatio i smart ways that beefit from correlatios betwee differet radom variables to reduce variability. VRTs are ot free though. E.g., some additioal programmig effort may be eeded. The basic relatio used i a VRT is var( ax by) a var( X ) b var( Y) ab cov( X, Y). If the last term is egative tha the variace of ax + by could be reduced. Recall that the sample covariace based o observatios of X ad Y is cov( XY, ) i1 ( X X )( Y Y ) i 1 i 11
Variace reductio usig commo radom umbers (CRN) This is used whe comparig the simulatio outputs from two systems. The idea is to use the same stream of radom umbers to compare both systems. Ituitively, this is equivalet to comparig the two systems uder similar coditios. Suppose that iid observatios are available from the output of each system, X 11, X 1,, X 1, ad X 1, X,, X. Cosider the paired differece Z j = X 1j X j, j = 1,,. The, var( Z ) var( X ) var( X ) cov( X, X ) j 1 j j 1 j j This implies var(z j ) is reduced if X 1j ad X j are positively correlated (i.e. cov(x 1j, X 1j ) > 0). This ca be achieved by usig the same stream of radom umber to estimate X 1j ad X 1j. Oe critical poit is sychroizatio. That is, usig radom umbers for the same purpose i both simulatios. Oe way to achieve this is to use dedicated streams for each source of radomess (e.g. oe for arrival times ad oe for service times.) 1
I additio, usig a radom variates geeratio method which uses 1 U(0,1) to get 1 variate X ad which uses a mootoe trasformatio U X helps i sychroizatio. The iverse trasform method is highly desired here. CRN works well if the two systems uder compariso react i a similar way to chages i the uderlyig radom umber streams. If ot, we could get cov(x 1, X ) < 0 ad the method backfires. Example 5 CRN was applied to the comparig the two ATM cofiguratios i Example 1 based o 100 replicatios. CRN was applied i differet ways by sychroizig arrival times oly (A), service times oly (S), ad both arrival ad service times. The results were as follows. (I meas o sychroizatio e.g. usig oe radom umber stream for all purposes.) 13
Variace reductio usig atithetic variates (AV) This method is used for variace reductio of a sigle system (i order to get a more precise output). Like CRN, AV works by recyclig radom umbers. The idea is to ru replicatio j based o pairs of two replicatios. Oe replicatio, uses radom umbers U j1, U j,, U jm to estimate a measure of performace X (1) j, ad the other ru uses radom umbers 1 U j1, 1 U j,, 1 U jm, to estimate () a similar measure X j. (1) The, use X j = (X j + X () j )/ as the output of replicatio j. The use of U ad (1 U) is sought to iduce egative (1) () correlatio betwee X j ad X j. (1) The, the variace of X j is less tha the variace of X j ad () X j sice var(x j ) = [var(x (1) j ) + var(x () j ) + cov(x (1) j, X () j )] / 4. The ituitio behid AV is that couterbalacig a large observatio with a small oe leads somewhere close to the mea. Note that AV requires sychroizatio like CRN so that U ad 1 U for the same purpose. 14
Variace reductio usig cotrol variates (CV) This method works by relatig the radom variable of iterest, X, (for which we wish to estimate the mea) to aother radom variable with kow mea, Y. E.g, i a queueig simulatio, X could be the delay time ad Y the service time. The idea is to use our kowledge of E[Y] = v to cotrol X whe X ad Y are correlated. Specifically, defie X C = X a(y v), The, E[X C ] = E[X]. The choice of a ca be made i a way that miimizes var[x C ]. Note that var[x C ] = var[x] + a var[y] acov(x,y). This is a secod degree polyomial i a whose miimum is a* = cov(x, Y)/var[Y]. The issue ow is how to estimate cov(x, Y) or eve var[y]. This ca ofte be oly doe through sample data, which teds to bias the estimatio of E[X] through E[X C ]. 15