Online Supplement to Efficient Computation of Overlapping Variance Estimators for Simulation INFORMS Journal on Computing

Transcription

1 Online Supplement to Efficient Computation of Overlapping Variance Estimators for Simulation INFORMS Journal on Computing Christos Alexopoulos H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia 30332, USA, Nilay Tanık Argon Department of Industrial and Systems Engineering, University of Wisconsin Madison, Madison, Wisconsin , USA, David Goldsman H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia 30332, USA, Natalie M. Steiger Maine Business School, University of Maine, Orono, Maine , USA, Gamze Tokol Decision Analytics, Atlanta, Georgia 30306, USA, James R. Wilson Edward P. Fitts Department of Industrial and Systems Engineering, North Carolina State University, Campus Box 7906, Raleigh, North Carolina , USA, The purpose of this online supplement is to present the following: (i linear time algorithms to obtain overlapping area and Cramér von Mises estimators; (ii experimental results on the bias, variance, meansquared error, and efficiency of the NBM and overlapping variance estimators; and (iii analytical results on the bias and variance of the overlapping estimators when they are applied to i.i.d. normal data.. Algorithms We start with the algorithms for computing overlapping area and CvM estimators with several weighting functions in O(n time. Figures and 2 present the algorithms for obtaining the overlapping area estimator A O (f ; b, m for weighting functions f r (t r p0 c pt p and f cos,j (t 8πj cos(2πjt, respectively. Figure 3 presents an algorithm to obtain the overlapping CvM estimator C O (g; b, m with weighting function g(t r p0 c pt p.

2 Figure : Algorithm for the Overlapping Area Estimator with Polynomial Weighting Functions Step : Initialization V 0, a 0 0, a 0, S (p 0 for p 0,...,r, and k Step 2: Calculate S (p 0,m for p 0,...,r; a 0(f r ; a (f r ; and V 0 (f r S (p S (p + k p Y k for p 0,...,r; V V f r (k/ms (0 Until k m + V V + S (0 a /m a 0 a 0 + f r (k/m, k k + a a + kf r (k/m Step 3: Calculate W i (f r, V i (f r for i,...,n m; accumulate A O (f r ; b, m for all overlapping batches A O V 2 /m 3 and i Until i n m + Return A O A O/ (n m + Calculate S (p i,m backwards in p to save storage space p r S (p p ( px x0 ( p x S (x + m p Y i+m Y i I {p0}, p p Until p W r p0 c p S (p /m p, V V + (Y i+m Y i a /m W + Y i a 0 A O A O + V 2 /m 3, i i + 2. Efficiency of the Variance Estimators In Alexopoulos et al. (2006, we summarize the results of an experimental performance evaluation of the new overlapping variance estimators; and we find that they perform efficaciously compared with their nonoverlapping counterparts at least with respect to the measures of bias and variance. In particular, there is little difference between corresponding nonoverlapping and overlapping estimators when it comes to bias; yet overlapping variance estimators often achieve substantial savings compared with nonoverlapping variance estimators when it comes to variance. We have also shown that all of the overlapping estimators studied herein require computation on the same order of magnitude as that of NBM, the simplest estimator in terms of computation. A natural question to ask is: just how equal are the O(n computational efforts required by the various variance estimators? We might reasonably expect that among these O(n estimators, NBM requires less work than any of the batched STS estimators, which in turn require less work than the corresponding overlapping STS estimators. The good news is that, for all practical purposes, the differences in computational effort 2

3 Figure 2: Algorithm for the Overlapping Area Estimator with Trigonometric Weighting Functions Step : Initialization j user-selected positive integer, V 0, a 0 0, a 0, Q 0, Q 0, S (0 0, and k Step 2: Calculate S (0 0,m, Q m, Q m, a 0(f cos,j, a (f cos,j, and V 0 (f cos,j S (0 S (0 + Y k, V V f cos,j (k/ms (0 Q Q + cos(α j ky k, Q Q + sin(α j ky k a 0 a 0 + f cos,j (k/m, a a + kf cos,j (k/m, k k + Until k m + V V + S (0 a /m Step 3: Calculate Q i, Q i, W i(f cos,j, V i (f cos,j for i,...,n m; accumulate A O (f cos,j ; b, m for all overlapping batches A O V 2 /m 3 and i Q Q + cos(α j (i + my i+m cos(α j iy i Q Q + sin(α j (i + my i+m sin(α j iy i W 8πj (cos(α j iq + sin(α j iq V V + (Y i+m Y i a /m W + Y i a 0 A O A O + V 2 /m 3, i i + Until i n m + Return A O A O /(n m + are meaningless, especially when one regards the observations as expensive compared to the subsequent computational costs once the observations are in hand. Nevertheless, we undertook a great deal of Monte Carlo work to evaluate the actual effort necessary to compute the various variance estimators. In one of our simplest experiments, we generated n 60,000 observations of the waiting time process for the M/M/ queue with 80% traffic intensity; and we organized the resulting time series into batches of size m 4,000, 8,000, and 6,000 so that we could obtain precise estimates of the total computing time C, bias B, variance V, and mean-squared error M of the corresponding variance estimators N (b, m, A O (f 0 ; b, m, A O (f 2 ; b, m, C O (g 0 ; b, m, and C O (g2 ; b, m for b 40, 20, and 0. Table reports the results based on 6,000 i.i.d. replications of each of the variance estimators under study. Each total computing-time entry C in Table is the sum of two components:. On an IBM T43p laptop computer with an Intel Pentium M processor having a speed of 2.3 GHz and 3

4 Figure 3: Algorithm for the Overlapping CvM Estimator with Polynomial Weighting Functions Step : Initialization S 0, M 0, J 0, a p 0 for p {0,, 2}, S (p 0 for p 0,...,r + Z (p 0 for p 0,...,r, l, and k Step 2: Calculate S (p 0,m for p 0,...,r +, U l for l 2,...,m+, M(, and a p (g for p {0,, 2}; initialize the second term of (4 from Alexopoulos et al. (2006 S (p S (p + k p Y k for p 0,...,r +, J J + g(k/mks (0 /m if l U 2 g(/my else Calculate Z (p l,,l backwards in p to save storage space p r Z (p p ( px x0 Z (x + Y l, p p Until p U l+ U l + Y l a 0 r p0 c p Z (p /m p + g(l/ms (0 endif a p a p + k p g(k/m for p {0,, 2}, M M + g(k/ms (0, l l +, k l Until l m + Step 3: Calculate W i (g r+ for i,...,n m, U l for l m + 2,...n, M(l for l 2,...,n m + ; accumulate the third term of (4 from Alexopoulos et al. (2006 and C O (g; b, m K M, G a 0, R Y M, C O a 2 (S (0 /m 2 2S (0 J, i Calculate S (p i,m backwards in p to save storage space p r + S (p p ( px x0 ( p x S (x + m p Y i+m Y i I {p0}, p p Until p W r p0 c p S (p+ /m p+, J J + W Y i a /m K K + r p0 c p S (p /m p a 0 Y l m, M M + K U l m+, R R + Y l m+ M if l n m + Calculate Z (p l,,m backwards in p to save storage space p r Z (p p ( px x0 Z (x + Y l (m + p Y l I {l m+2}, p p Until p U l+ U l + Y l a 0 r p0 c p Z (p /m p else G G g((l n + m 2/m, S S + Y l, p r Calculate Z (p l,l (n m,m backwards in p to save storage space Z (p p ( px x0 Z (x + Y l I {ln m+2} (m + p Y l (m+, p p Until p U l+ U l + Y l G r p0 c p Z (p /m p g((l n + m/ms endif C O C O + a 2 (S (0 /m 2 2S (0 J, l l +, i i + Until l n + l n m + 2 Step 4: Calculate M(l for l n m + 2,...,n; accumulate the third term of (4 and C O (g; b, m M M U l, R R + Y l M, l l + Until l n + Return C O (C O + R/(m 2 (n m + 4

5 Performance Table : Efficiency Analysis of the Variance Estimators Variance Estimator Measure N (b, m A O (f 0 ; b, m A O (f 2 ; b, m C O (g 0 ; b, m C O (g2 ; b, m Results for m 4,000 and b 40 C v.est (sec C (sec B V 430, ,27 332, , ,708 M 43, , ,62 25,74 326,96 Q Q Results for m 8,000 and b 20 C v.est (sec C (sec B V 637, , ,47 33,398 47,670 M 637,87 38,64 426,48 37,036 47,77 Q Q Results for m 6,000 and b 0 C v.est (sec C (sec B V,084, ,573 68,96 430, ,620 M,084, ,092 68,936 43, ,762 Q Q

6 .00 GB of RAM, we computed the average execution time C sim 4.22 seconds per run for an Arena simulation model (Kelton, Sadowski, and Sturrock 2004 of the M/M/ queue in which 60,000 customer waiting times are merely computed and tabulated in an Arena Record module before the associated entity is disposed. 2. On the same laptop computer, we computed the average execution time C v.est (in seconds to compute the variance estimators N (b, m, A O (f 0 ; b, m, A O (f 2 ; b, m, C O (g 0 ; b, m, and C O (g2 ; b, m from a set of n 60,000 queue waiting times for the M/M/ queue that were already resident in computer memory. To compute C v.est for, say, the NBM variance estimator N (b, m, we used the PORTLIB function DTIME of Fortran PowerStation (Microsoft Corp. 995 to obtain the total CPU time required to compute 6,000 replications of N (b, m; and then a similar experiment was performed for the overlapping variance estimators A O (f 0 ; b, m, A O (f 2 ; b, m, C O (g 0 ; b, m, and C O (g2 ; b, m separately. Thus for each variance estimator in Table, we report C sim, C v.est, and C C sim + C v.est. We take the NBM variance estimator N (b, m as the baseline with performance measures C 0, B 0, V 0, and M 0 ; and for any other variance estimator with corresponding performance measures C, B, V, and M, we see that the relative efficiency statistics are given by Q C V /( C0 V 0 and Q C M /( C0 M 0. ( On any reasonably sized practical applications, the estimator computation times are insignificant compared to the times required to obtain the observations, thus rendering NBM noncompetitive in terms of the relative efficiency measures Q and Q. 3. Exact Results for i.i.d. Normal Case The purpose here is simply to see whether the new estimators perform as advertised on an example for which we are able to calculate the expected value and variance of the variance estimators exactly. Suppose that we sample from an i.i.d. standard normal process, {Y i : i,...,n}. After some tedious algebra that is detailed below, we obtain the results displayed in Table 2 describing the bias and variance properties of some of the proposed overlapping variance estimators. In Table 2 all the bias results are exact; on the other hand, in the tabulated expressions for the variance of the overlapping estimators, we omitted terms of the form 6

7 O(/m (for fixed b. We see that the entries in Table 2 indeed match up nicely with those in Table of Alexopoulos et al. (2006 with γ 0. Of course, in this very special case, the natural estimator for σ 2 would have been the usual sample variance S 2, which is unbiased and has variance 2/(n. What follows are the derivations of the entries appearing in Table 2. Table 2: Bias and Variance for Variance Estimators in I.i.d. Normal Case Estimator Bias Variance A O (f 0 ; b, m m 2 24b 3 35(b 2 A O (f 2 ; b, m m 2 2m 4 m 6 354b (b 2 C O (g 0 ; b, m m 2 88b 5 20(b 2 C O (g 2 ; b, m 4 m 2 5 m 4 O(b, m b (b 2 4b 3 b 2 +4b+6 3(b 4 Overlapping Area Estimator We first obtain exact results for the overlapping area estimator. Using the arguments found in, e.g., 4 of Foley and Goldsman (999, one can apply a little algebraic elbow grease to obtain A O i (f ; m m ] 2 h(f ; j, my m 3 i+j, i,...,n m +, where h(f ; j,m m m l m f(l m f( l m, l lj and where we notice that the h(f ; j,m s need only be calculated once beforehand. We now calculate the mean and variance of the overlapping area estimator for the current example. First, 7

8 we give a general expression for the mean. Since the Y i s are i.i.d. with unit variance, we have EA O (f ; b, m] EA O (f ; m] ( m E m 2 ] h(f ; j, my 3 j ( m Var m h(f ; j, my 3 j m h 2 (f ; j, m. (2 m 3 Before moving ahead to the variance of the overlapping area estimator, we state a useful lemma. Lemma (Patel and Read 996 If (Z,Z 2 is bivariate normal with marginal means equal to zero, then Cov(Z 2,Z2 2 2Cov2 (Z,Z 2. In addition, define for k 0,, 2,..., R A k Cov(A O (f ; m, AO +k (f ; m Cov ( m 2 h(f ; i, my m 3 i, ( m 2 h(f ; j, my m 3 k+j 2 ( m m 6 Cov2 h(f ; i, my i, 2 m 6 m 2 m 6 m ik+ m h(f ; j, my k+j m h(f ; i, mh(f ; j,mcov(y i,y k+j ] 2 (by Lemma h(f ; i, mh(f ; i k, m] 2, (3 since the Y i s are i.i.d. Then the definition of A O (f ; b, m and stationarity imply that (n m + Var(A O (f ; b, m n m + k n m R0 A + 2 ( k n m + k R0 A + 2 ( k n m + again since the underlying observations are i.i.d. Cov(A O i (f ; m, AO j (f ; m R A k R A k, (4 8

9 Now we see what happens to the area and variance of the overlapping area estimator when we apply specific weighting functions to the i.i.d. standard normal process. Example For the overlapping area estimator with constant weighting function f 0 (t 2, it is easy to show that h(f 0 ; j,m ( 2 j m +. 2 Plugging this into Equations (2 (4 leads to EA O (f 0 ; b, m] m 2, which matches up with the appropriate entry in Table of Alexopoulos et al. (2006 (taking into account that γ 0, and Var(A O (f 0 ; b, m (m2 m 5 (24b m m 3 b + o(m 3 b ] 35m 5 m(b + ] 2. 24b 3 35(b 2, for large m and b, which is in synch with the appropriate variance entry from Table of Alexopoulos et al. (2006. Example 2 Consider the first-order unbiased overlapping area estimator with quadratic weights f 2 (t 840(3t 2 3t + /2. After some algebra, we have h(f 2 ; i, m 20 4i 3 6i 2 (m + + 2i(m 2 + 3m + m(m + ] 2m 2. Again using (2 (4, we eventually obtain and EA O (f 2 ; b, m] 2m6 + 7m m m 6, Var(A O (f 2 ; b, m 354m + O(/m 4290(bm m + 26m2 + O(. 354b (bm m (b, 2 for large m and b, in agreement with the results found in Table of Alexopoulos et al. (2006. Remark Some additional algebra not reported here allows us to verify the overlapping area estimator results for the AR( process analyzed via Monte Carlo in 4. of Alexopoulos et al. (

10 Overlapping CvM Estimator We next look at the overlapping CvM estimator for an i.i.d. standard normal process. As for the overlapping area estimator, we begin with a general expression for the expected value of the variance estimator. Since the Y i s are i.i.d. with unit variance, we have EC O (g; b, m] EC O (g; m] m g( k m 2 m k2 E(Ȳ,m O Ȳ,k O 2 ] k m g( k m 2 m k2 Var(Ȳ,m Ȳ,k k m g( k m 2 m k2 Var(Ȳ,m + Var(Ȳ,k 2Cov(Ȳ,m, Ȳ,k ] k m ( g( k m 2 m k2 m + k 2 m k m ( g( k m 2 m k2 k. (5 m k Before undertaking our calculation of the overlapping CvM estimator s variance, we start off with a preliminary calculation: For i, j,..., m and k 0,, 2,..., Cov ( Ȳ,m O Ȳ,i O, Ȳ +k,m O Ȳ +k,j O m m Cov(Y m 2 α,y k+β jm im α β i α β m Cov(Y α,y k+β + ij m α β i α β j Cov(Y α,y k+β j Cov(Y α,y k+β max(m k, 0 min(j, max(m k, 0 max(i k, 0 + min(j, max(i k, 0 m2 jm im ij (since the Y i s are i.i.d. with unit variance ( + k m( i m, if k<iand j i k k + ( ( k m 2 i j m k ( i ( m j, if k<iand m k<j m, if k<iand i k<j m k k, if i k<mand j m k m 2 ( ( k m j, if i k<mand m k<j m 0, if m k. (6 0

11 For k 0,, 2,..., let Rk C Cov(C O (g; m, CO +k (g; m Cov m g( i m 2 m i2( Ȳ,m O Ȳ,i O 2, m g( j m 2 m j 2( Ȳ+k,m O Ȳ +k,j O 2 2 g( i m 4 m g( j m i2 j 2 Cov (Ȳ 2,m O Ȳ,i O, Ȳ +k,m O Ȳ +k,j O, the last step following from Lemma. Evocation of Equation (6 then reveals that Rk C 2 k 2 k m k g( i m 4 m g( j m i2 j 2 + ( k 2 k ( g( i m m g( j m i2 j m m 4 + ( + k m + 2 ik+ ji k+ +k 2 ik+ m k ik+ jm k+ i k g( i m g( j m j ( 2 i m jm k+ 2 g( i m g( j m i2 j 2 ( k m 2 + ( k i g( i m g( j m ( i m ( 2 j m 2 ( j m 2, (7 0 for k m. Finally, as in the derivation of Equation (4, we have for k 0,,...,m, and Rk C (n m + Var(C O (g; b, m R0 C + 2 ( k 2 k Rk C n m +. (8 With Equations (5, (7, and (8 in hand, we can calculate the mean and variance of the overlapping CvM estimator when specific weighting functions are applied to the i.i.d. normal process. Example 3 For the overlapping CvM estimator with constant weighting function g 0 (t 6, we obtain EC O (g 0 ; b, m] m 2, which is consistent with Table of Alexopoulos et al. (2006 (with γ 0, and Var(C O (g 0 ; b, m (88b 5m7 + 88m 6 + m 5 (588b O(m 4 b 20m 5 (m(b b 5 20(b 2, for large m and b, which is again consistent with Table of Alexopoulos et al. (2006. Example 4 For the overlapping CvM estimator with quadratic weighting function g2 (t t 50t 2,wehave EC O (g 2 ; b, m] + 4 m 2 5 m 4,

12 which agrees with Table of Alexopoulos et al. (2006 (with γ 0; and Var(C O (g2 ; b, m m0 (bm m + O(m 9 b m 9 (m(b b (b, 2 for large m and b, which agrees with Table of Alexopoulos et al. (2006. Overlapping Batch Means Estimator Finally, we examine the OBM estimator for an i.i.d. standard normal process. The expected value is given by (n m + (n m nm EO(b, m] E(Ȳ O i,m Ȳ n 2 ] Var(Ȳ O i,m Ȳ n ] Var(Ȳi,m O 2Cov(Ȳ i,m O, Ȳ n + Var(Ȳ n (n m + m 2 n + ], n so that EO(b, m], in agreement with Table of Alexopoulos et al. (2006 (with γ 0. As for the variance of the OBM estimator, we have (n m + 2 (n m 2 Var(O(b, m 2n 2 m 2 ( Cov (Ȳi,m O 2 Ȳ n 2,(Ȳj,m O Ȳ n 2 Cov 2 (Ȳ O i,m Ȳ n, Ȳ O j,m Ȳ n (by Lemma Cov(Ȳi,m O, Ȳ j,m O Cov(Ȳ i,m O, Ȳ n Cov(Ȳj,m O, Ȳ n + Var(Ȳ n Cov(Ȳi,m O, Ȳ j,m O n (n m + Var(Ȳ,m n] O ] 2 (n m + m ] 2 m j + 2 (n m + j n m 2 ] 2 (n m + j Cov(Ȳ,m O, Ȳ +j,m O n 2 n ] n 2 jm ] 2 (n m + j,

13 so that, after carrying out some algebra, Var(O(b, m 4b3 b 2 + 4b + 6 2b 2 + 3(b 4 3m(b 2b(b4 3b 3 + 5b 2 0b (n m + (b 4 + b2 (b 2 6b + 8 3(n m + 2 (b 4 4b3 b 2 + 4b + 6 3(b 4 + O(/m, which is in agreement with Table of Alexopoulos et al. (2006. References Alexopoulos, C., N. T. Argon, D. Goldsman, N. M. Steiger, G. Tokol, and J. R. Wilson Efficient computation of overlapping variance estimators for simulation. Technical report. School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, GA. Foley, R. D., and D. Goldsman Confidence intervals using orthonormally weighted standardized times series. ACM Transactions on Modeling and Computer Simulation Kelton, W. D., R. P. Sadowski, and D. T. Sturrock Simulation with Arena, 3rd ed. McGraw-Hill, New York. Microsoft Corp Fortran PowerStation Reference. Microsoft Corp., Redmond, WA. Patel, J. K., and C. B. Read Handbook of the Normal Distribution, 2nd ed. Marcel Dekker, New York. 3