2WB05 Simulation Lecture 7: Output analysis

Size: px

Start display at page:

Download "2WB05 Simulation Lecture 7: Output analysis"

Dominick Johnston
5 years ago
Views:

1 2WB05 Simulation Lecture 7: Output analysis Marko Boon December 17, 2012

2 Outline 2/33 Output analysis of a simulation Confidence intervals Warm-up interval Common random numbers

3 Output analysis of a simulation 3/33 Confidence intervals Let X 1, X 2,..., X n be independent realizations of a random variable X with unknown mean µ and unknown variance σ 2. Sample mean X(n) = 1 n X i n Sample variance S 2 (n) = 1 n 1 Clearly X(n) is an estimator for the unknown mean µ. How can we construct a confidence interval for µ? i=1 n (X i X(n)) 2 i=1

4 Confidence intervals 4/33 Central limit theorem states that for large n ni=1 X i nµ σ n is approximately a standard normal random variable, and this remains valid if σ is replaced by S(n). Hence, let z β = 1 (β) (e.g., z = 1.96), then P( z 1 α/2 ni=1 X i nµ S(n) n z 1 α/2 ) 1 α or equivalently ( P X(n) z 1 α/2 S(n) n µ X(n) + z 1 α/2 S(n) n ) 1 α

5 Confidence intervals 5/33 Conclusion An approximate 100(1 α)% confidence interval for the unknown mean µ is given by X(n) ± z 1 α/2 S(n) n As a consequence, to obtain one extra digit of the parameter µ, the required simulation time increases with approximately a factor 100.

6 Confidence intervals 100 confidence intervals for the mean of uniform random variable on ( 1, 1); each interval is based on 100 observations /

7 Confidence intervals 7/33 Remark: If the observations X i are normally distributed, then ni=1 X i nµ S(n) n has for all n a Student s t distribution with n 1 degrees of freedom; so an exact confidence interval can be obtained by replacing z 1 α/2 by the corresponding quantile of the t distribution with n 1 degrees of freedom.

8 Confidence intervals Remark: Recursive computation of the sample mean and variance of the realizations X 1,..., X n of a random variable X: 8/33 and for n = 2, 3,..., where X(n) = n 1 n S 2 (n) = n 2 n 1 S2 (n 1) + 1 n X(n 1) + 1 n X n X(1) = X 1, S 2 (1) = 0. ( Xn X(n 1) ) 2

9 Warm-up interval 9/33 Problem of the initialization effect We are interested in the long-term behaviour of the system and maybe the choice of the initial state of the simulation will influence the quality of our estimate. One way of dealing with this problem is to choose N very large and to neglect this initialization effect. However, a better way is to throw away in each run the first k observations.

10 Warm-up interval Let W 1, W 2,..., W N be realizations of waiting times in a single run, and suppose we want to estimate the steady-state mean waiting time E(W ), defined as E(W ) = lim j E(W j ) 10/33 by the sample mean W N = 1 N N j=1 W j

11 Warm-up interval In this estimate there are two types of errors: Systematic error, or bias This means that E( W N ) = E(W ), due to the influence of the initial conditions, which may not be representative for steady-state behavior; Sampling (or random) error The estimator W N is of course a random variable. 11/33

12 Warm-up interval To reduce the systematic error, we delete the initial observations, say W 1,..., W k, and use the remaining observations W k+1,..., W N to estimate E(W ) by the truncated sample mean 12/33 W k,n = 1 N k N j=k+1 W j Then one expects that W k,n is less biased than W N, since the observations near the beginning of the simulation may not be representative for steady-state behavior; the parameter k is called the warm-up interval.

13 Warm-up interval 13/33 Choosing the warm-up interval We like to pick k such that E( W k,n ) E(W ). If k is too small, then E( W k,n ) may be significantly different from E(W ); If k is too large, then the variance of W k,n (the sampling error) may be too large (its variance is proportional to 1/(N k)).

14 Warm-up interval 14/33 Graphical procedure Our goal is determine a value k such that E(W j ) E(W ) for all j > k. The presence of variability of the process W 1, W 2,... makes it hard to determine k from a single run. Therefore, the idea is to make n independent replications (by using different random numbers) and employing the following steps:

15 Warm-up interval 15/33 1. Make n independent replications (or runs), each of length N; let W (i) j denote the j-th waiting time in run i. 2. Let W j = 1 n n i=1 The averaged process W 1, W 2,... has means and variances W (i) j E( W j ) = E(W j ), var( W j ) = var(w j )/n. So its mean behavior is the same as the original process, but it has a smaller (1/n-th) variance. 3. Plot W j and choose k such that beyond k the process W 1, W 2,... appears to have converged.

16 Warm-up interval Waiting times for the M/M/1 queue with λ = 0.5 and µ = 1; 16/

17 Warm-up interval Averaged waiting times for the M/M/1 queue with λ = 0.5 and µ = 1; number of replications is /

18 Warm-up interval Averaged waiting times for the M/M/1 queue with λ = 0.5 and µ = 1; number of replications is /

19 Warm-up interval Averaged waiting times for the M/M/1 queue with λ = 0.5 and µ = 1; number of replications is /

20 Warm-up interval To smooth high-frequency oscillations in W 1, W 2,... (but leave the trend) one may consider the moving average W j (w) (where w is the window size) defined as: 20/33 W j (w) = 1 2w + 1 w i= w W j+i for j = w + 1, w + 2,..., N w, and W j (w) = 1 2 j 1 j 1 i= ( j 1) W j+i for j = 1,..., w. The warm-up interval k can then be determined from the plot of W j (w) for j = 1,..., N w.

21 Warm-up interval Moving average of averaged waiting times for the M/M/1 queue with λ = 0.5 and µ = 1; number of replications is 10 and window size is /

22 Warm-up interval Moving average of averaged waiting times for the M/M/1 queue with λ = 0.5 and µ = 1; number of replications is 10 and window size is /

23 Warm-up interval Moving average of averaged waiting times for the M/M/1 queue with λ = 0.5 and µ = 1; number of replications is 100 and window size is /

24 Warm-up interval 24/33 Batch means Instead of doing n independent runs, we try to obtain n independent observations by making a single long run and, after deleting the first k observations, dividing this run into n subruns. The advantage is that we have to go through the warm-up period only once.

25 Batch means Let W 1, W 2,..., W nn be the output of a single run, where we have already deleted the first k observations and renumbered the remaining ones. Hence W 1, W 2,..., W nn will be representative for the steady-state. We divide the observations into n batches of length N. Thus, batch 1 consists of batch 2 of and so on. Let W (i) N W 1, W 2,..., W N ; W N+1, W N+2,..., W 2N, be the sample (or batch) mean of the N observations in batch i, so 25/33 W (i) N = 1 N i N j=(i 1)N +1 W j

26 Batch means The W (i) N s play the same role as the ones in the independent replication method. Unfortunately, the W (i) N s will now be dependent. But, under mild conditions, for large N the W (i) N s will be approximately independent, each with the same mean E(W ). Hence, for N large enough, it is reasonable to treat the W (i) N s as i.i.d. random variables with mean E(W ); thus W n,n ± z 1 δ/2 S n,n n 26/33 provides again a 100(1 δ)% confidence interval for E(W ), with W n,n and Sn,N 2 variance of the realizations W (1) N,..., W (n) N. again the sample mean and

27 Common random numbers If we compare the performance of two systems with random components it is in general better to evaluate both systems with the same realizations of the random components. If X and Y are estimators for the performance of the two systems, then var(x Y ) = var(x) + var(y ) 2 cov(x, Y ). In general, use of common random numbers leads to positively correlated X and Y : cov(x, Y ) > 0 Hence, the variance of X Y (and thus the corresponding confidence interval) will be smaller than in case X and Y are independent (when generated with independent random numbers). 27/33

28 Example: Job scheduling Suppose that N jobs have to be processed on M identical machines. The processing times are independent and exponentially distributed with mean 1. We want to compare the completion time of the last job, C max, under two different strategies: Longest Processing Time First (LPTF) Shortest Processing Time First (SPTF) 28/33 Remark: Let X 1,..., X N be independent exponentials with mean 1; denote the smallest one by X (1), the second smallest one by X (2), and so on. Then for i = 1,..., N X (i) d = YN + + Y N i+1 where Y 1,..., Y N are independent exponentials with E(Y i ) = 1/i (Y i is the minimum of i exponentials with mean 1).

29 Example: Job scheduling Results for M = 2, N = 10 without using common jobs for the SPTF and LPTF strategy (n is the number of experiments): and the results using common jobs: n Cmax S PT F Cmax L PT F mean st.dev. half width 95% CI n Cmax S PT F Cmax L PT F mean st.dev. half width 95% CI /33 Hence, using common jobs reduces the (sample) standard deviation of Cmax S PT F the confidence interval for its mean with a factor 5! Cmax L PT F, and thus the width of

30 Example: Job scheduling Results for M = 2, N = 10 without using common jobs for the SPTF and LPTF strategy (n is the number of experiments): and the results using common jobs: n Cmax S PT F Cmax L PT F mean st.dev. half width 95% CI Blah blah blah... Conclusion: common random numbers is better! n Cmax S PT F Cmax L PT F mean st.dev. half width 95% CI /33 Hence, using common jobs reduces the (sample) standard deviation of Cmax S PT F the confidence interval for its mean with a factor 5! Cmax L PT F, and thus the width of

31 Example: Production system We want to determine the reduction in the mean waiting time when we have an extra machine. To compare the two situations we want to use the same realizations of arrival times and processing times in the simulation. Assume that processing times of jobs are generated when they enter production. Then, the order in which arrival and processing times are generated depends on the number of machines: synchronization problem. 31/33 Solution: Use separate random number streams for different random variables (arrival and processing times); Design the simulation model such that it guarantees that exactly the same realizations of random variables are generated. In this problem, the second aproach can be realized by assigning to each job a processing time immediately upon arrival.

32 Example: Production system Results for λ = 4, µ = 1 and the number of machines M is 5, resp. 6. In each run N = 10 5 waiting times are generated; W (i) (i) i N M = 5 M = The standard deviation of the (i) is equal to If both systems use independent realizations of arrival and processing times the standard dev. of (i) is 0.029; so common random numbers yields a reduction of 25%. 32/33

33 Example: Production system Results for λ = 4, µ = 1 and the number of machines M is 5, resp. 6. In each run N = 10 5 waiting times are generated; W (i) (i) i N M = 5 M = Blah blah blah... Merry Christmas! The standard deviation of the (i) is equal to If both systems use independent realizations of arrival and processing times the standard dev. of (i) is 0.029; so common random numbers yields a reduction of 25%. 33/33

X i. X(n) = 1 n. (X i X(n)) 2. S(n) n

X i. X(n) = 1 n. (X i X(n)) 2. S(n) n Confidence intervals Let X 1, X 2,..., X n be independent realizations of a random variable X with unknown mean µ and unknown variance σ 2. Sample mean Sample variance X(n) = 1 n S 2 (n) = 1 n 1 n i=1