Variance Parameter Estimation Methods with Data Re-use Christos Alexopoulos, David Goldsman, Melike Meterelliyoz Claudia Antonini James R. Wilson Georgia Institute of Technology, Atlanta, GA, USA Universidad Simón Bolívar, Sartenejas, 1080, Venezuela North Carolina State University, Raleigh, NC, USA October 12, 2008
Outline 1 Introduction 2 Background Basics and Assumptions Standardized Time Series Estimators Estimators Based on Batches Estimators Based on Overlapping Batches 3 Estimators with Data Re-use Folded Estimators Folded Overlapping Area Estimators Reflected Estimators 4 Conclusions and Future Work
Introduction Introduction Objective Computation of point and confidence interval (CI) estimators for the mean of a stationary discrete time process {X j : j 1}. The point estimate for the mean µ is the sample mean X n. To evaluate the performance of Xn, we need an estimate of its variance Var( X n ). For i.i.d. simulation output, S 2 (n)/n is the obvious estimator. For correlated data, S 2 (n)/n can be severely biased. Valid CIs are obtained based on estimators of σ 2 n = nvar( X n ) or σ 2 = lim n σ 2 n.
Introduction Literature Review 1 Techniques for estimating σ 2 Nonoverlapping Batch Means (NBM). Overlapping Batch Means (OBM) [Song and Schmeiser, 1993]. Standardized Time Series (STS) [Schruben, 1983]. 2 Related STS Methods Area estimators. Cramér von Mises (CvM) estimators. Permuted STS estimators [Calvin and Nakayama, 2006].
Introduction Motivation Re-use of data: Obtain different sample paths from the same data. Different estimators of σ 2 from each sample path. Linear combination of estimators: Smaller variance. Same effort, more information.
Background Basics and Assumptions Assumptions A A.1 The process {X i : i 1} is stationary. A.2 The process {X i : i = 1, 2,...} satisfies the following functional central limit theorem. For each positive integer n, let Y n (t) nt ( X nt µ) σ for t [0, 1], (1) n where is the greatest integer function. Then there exist constants σ > 0 and µ such that Y n ( ) W( ), where W( ) is a standard Brownian motion process on [0,1] and denotes convergence in distribution. A.3 k=1 k2 R k <, where R k = Cov(X 1, X 1+k ), and σ 2 j= R j (0, ).
Background Basics and Assumptions Assumptions F F.1 The [ function f( ) is normalized so that 1 ] Var f(t)b(t) dt = 1, where B( ) is a standard 0 Brownian bridge process on [0, 1]. F.2 d2 dt 2 f(t) is continuous at every t [0, 1]. F.3 f(t) = f(1 t) for every t [0, 1].
Background Basics and Assumptions Definitions γ 1 2 l=1 lr l. For t [0, 1]: F (t) t 0 f(s) ds and F F (1). F (t) t 0 F (s) ds and F F (1). Useful Weight Functions For t [0, 1]: f 0 (t) 12. f 2 (t) 840(3t 2 3t + 1/2).
Background Standardized Time Series Estimators Standardized Time Series T n (t) nt ( X n X nt ) σ, for t [0, 1]. n Under Assumptions A, [ n( Xn µ), σt n ( ) ] [σw(1), σb( )].
Background Standardized Time Series Estimators Weighted Area Estimator and Limiting Functional A(f; n) [ 1 n n j=1 f ( j ( j ) n) ] 2 [ 1 2 σtn n A(f) f(t)σb(t) dt]. 0 A(f) D = σ 2 χ 2 1. Under Assumptions A and F, E[A(f; n)] = σ 2 [(F F ) 2 + F 2 ]γ 1 2n + o(1/n). If {A 2 (f; n) : n = 1, 2,...} is uniformly integrable, then lim Var[A(f; n)] = Var[A(f)] = n 2σ4.
Background Estimators Based on Batches Nonoverlapping Batches Form b nonoverlapping batches each consisting of m observations (assuming n = bm). Batch i consists of observations {X (i 1)m+j : j = 1,..., m}. Standardized Time Series from Batch i T i,m (t) mt ( X i,m X i, mt ) σ, m for t [0, 1] and i = 1,..., b, where X i,j 1 j j X (i 1)m+l, for j = 1,..., m. l=1
Background Estimators Based on Batches Batched Area Estimator A(f; b, m) 1 bm 2 b m f ( j ) m σti,m ( j m ) i=1 j=1 2, A(f; b, m) σ 2 χ 2 b /b. Under Assumptions A and F, E[A(f; b, m)] = σ 2 [(F F ) 2 + F 2 ]γ 1 2m + o(1/m). (2) If {A 2 (f; b, m) : m = 1, 2,...} is uniformly integrable, then lim bvar[a(f; b, m)] = Var[A(f)] = m 2σ4.
Background Estimators Based on Batches Nonoverlapping Batch Means Estimator N (b, m) m b 1 N (b, m) σ 2 χ 2 b 1 /(b 1). b ( X i,m X n ) 2. i=1 E[N (b, m)] = σ 2 (b+1)γ 1 bm + o(1/m). lim (b 1)Var[N (b, m)] = m 2σ4.
Background Estimators Based on Overlapping Batches Overlapping Batches Form n m + 1 overlapping batches each consisting of m observations. Batch i consists of observations {X i+j : j = 0,..., m 1}. Define b n/m.
Background Estimators Based on Overlapping Batches Overlapping Area Estimators A o (f; b, m) 1 m 2 (n m + 1) n m+1 i=1 m j=1 where, for t [0, 1] and j = 1, 2,..., n m + 1, f ( j ) ( m σt o j i,m m ) 2, T o i,m(t) mt ( X o i,m X o i, mt ) σ m and j 1 Xo i,j 1 X i+l. j l=0 Under Assumptions A and F, for t [0, 1] and s [0, b 1], A o (f; b, m) A o (f; b) 1 b 1 b 1 0 [ σ 1 0 f(u)b W,s (u) du] 2 ds, where B W,s (t) t[w(s + 1) W(s)] [W(s + t) W(s)].
Background Estimators Based on Overlapping Batches Overlapping Area Estimators If Assumptions A and F hold, then E[A o (f; b, m)] = σ 2 [(F F ) 2 + F 2 ]γ 1 2m + o(1/m). (3) If {[A o (f; b, m)] 2 : m = 1, 2,...} is uniformly integrable, Var[A o (f; b, m)] Var[A o (f; b)] m = 4σ4 (b 1) 2 for fixed b, where, for y [0, 1], 1 0 (b 1 y)p 2 (y) dy, (4) p(y) F (1)[F (1 y) F (1 y) y F (1)] + F (y) F (1) 1 y 0 f(u) F (y + u) du.
Background Estimators Based on Overlapping Batches Overlapping Batch Means Estimator O(b, m) Under Assumptions A, n m+1 nm ( (n m + 1)(n m) X i,m o X n ) 2. i=1 E[O(b, m)] = σ 2 (b2 + 1)γ 1 mb(b 1) + o(1/m). lim m Var[O(b, m)] = (4b3 11b 2 + 4b + 6)σ 4 3(b 1) 4 4σ4 3b.
Estimators with Data Re-use Outline 1 Introduction 2 Background Basics and Assumptions Standardized Time Series Estimators Estimators Based on Batches Estimators Based on Overlapping Batches 3 Estimators with Data Re-use Folded Estimators Folded Overlapping Area Estimators Reflected Estimators 4 Conclusions and Future Work
Estimators with Data Re-use Folded Estimators Folding Operation 1.5 1 Brownian bridge Reflected portion 2 1.5 0.5 1 0 0.5 0.5 1 0 1.5 0 0.2 0.4 0.6 0.8 1 0.5 0 0.2 0.4 0.6 0.8 1 Take a Brownian Bridge. Reflect the second half portion through the first half. Take the difference between the two portions. Stretch it over the [0,1] interval.
Estimators with Data Re-use Folded Estimators Batched Folded Area Estimator A (k) (f; b, m) 1 bm 2 b m f ( j ( j ) m) σt(k),i,m m i=1 j=1 where, for t [0, 1], k = 1, 2,... and i = 1,..., b, T (k),i,m (t) T (k 1),i,m ( t 2 ) T (k 1),i,m(1 t 2 ), Theorem Under Assumptions A and F, A (k) (f; b, m) σ 2 χ 2 b /b. If m is even, E[A (1) (f; b, m)] = σ 2 F 2 γ 1 m + o(1/m). (5) 2,
Estimators with Data Re-use Folded Estimators Batched Folded Area Estimator If {A 2 (k)(f; b, m) : m = 1, 2,...} is uniformly integrable, then lim Var[A (k)(f; b, m)] = 2σ4 m b. For weight functions under consideration, Cov[A (0) (f; b, m), A (k) (f; b, m)] = 0. Linear combination estimator Ā (0,1) (f; b, m) A (0)(f; b, m) + A (1) (f; b, m). 2 lim m Var[Ā(0,1)(f; b, m)] = σ4 b.
Estimators with Data Re-use Folded Estimators Approximate Asymptotic Bias and Variance for Different Estimators Area (m/γ 1 )Bias (b/σ 4 )Var A(f; b, m) Eq. (2) 2 A(f 0 ; b, m) 3 2 A(f 2 ; b, m) o(1) 2 A (1) (f; b, m) Eq. (5) 2 A (1) (f 0 ; b, m) 3 2 A (1) (f 2 ; b, m) o(1) 2 Ā (0,1) (f 0 ; b, m) 3 1 Ā (0,1) (f 2 ; b, m) o(1) 1 N (b, m) 1 2 O(b, m) 1 1.333
Estimators with Data Re-use Folded Estimators Examples 1 First-order autoregressive (AR(1)) process: X i = φx i 1 + ɛ i for i = 1, 2,..., where 1 < φ < 1, X 0 is a standard normal random variable, and the ɛ i s are i.i.d. N(0, 1 φ 2 ) random variables that are independent of X 0. R k = φ k for all k = 0, ±1, ±2,... σ 2 = (1 + φ)/(1 φ) and γ = 2φ/(1 φ) 2. Consider φ = 0.9, hence σ 2 = 19. 2 We run 10,000 replications using b = 32.
Estimators with Data Re-use Folded Estimators Expected Values of Folded Estimators Based on the AR(1) Process 19 18.5 Expected Values 18 17.5 17 16.5 Level 0 f0 Level 0 f2 Level 1 f0 Level 1 f2 NBM 1000 2000 3000 4000 5000 6000 m: batch size
Estimators with Data Re-use Folded Estimators Estimated Standard Deviations of Variance Estimators for the AR(1) Process for b = 32 Level-0 Estimators m A (0) (f 0 ; b, m) A (0) (f 2 ; b, m) 512 4.54 4.74 1024 4.63 4.75 2048 4.71 4.76 4096 4.74 4.74 Level-1 Folded Estimators m A (1) (f 0 ; b, m) A (1) (f 2 ; b, m) 512 4.49 4.53 1024 4.58 4.64 2048 4.69 4.77 4096 4.67 4.67 Linearly Combined Estimators m Ā (0,1) (f 0 ; b, m) Ā (0,1) (f 2 ; b, m) 512 3.19 3.27 1024 3.25 3.31 2048 3.33 3.38 4096 3.32 3.33
Estimators with Data Re-use Folded Overlapping Area Estimators Folded Overlapping Area (FOA) Estimators Idea: Combination of folding and overlapping operations. Estimators based on overlapping batches are always less variable than estimators based on nonoverlapping batches. Folded overlapping estimators can have smaller variance than unfolded overlapping analogues. Let s put these tricks together.
Estimators with Data Re-use Folded Overlapping Area Estimators Estimated Standard Deviations of Variance Estimators for the AR(1) Process for b = 32 Level-0 Folded Area Estimators m A (0) (f 0 ; b, m) A o (0)(f 0 ; b, m) A (0) (f 2 ; b, m) A o (0)(f 2 ; b, m) 512 4.54 2.79 4.74 3.03 1024 4.63 2.84 4.75 3.09 2048 4.71 2.81 4.76 3.07 4096 4.74 2.82 4.74 3.09 4.75 2.78 4.75 3.04 Level-1 Folded Area Estimators m A (1) (f 0; b, m) A o (1)(f 0; b, m) A (1) (f 2; b, m) A o (1)(f 2; b, m) 512 4.49 2.72 4.53 2.45 1024 4.58 2.76 4.64 2.49 2048 4.69 2.77 4.77 2.51 4096 4.67 2.74 4.67 2.48 4.75 2.72 4.75 2.47
Estimators with Data Re-use Reflected Estimators Reflected Estimators Reflection Prniciple If W(t) is a Brownian motion on t [0, 1], then { Wc W(t) if t < c (t) = 2W(c) W(t) if t c is also a Brownian motion process, where c [0, 1] is any reflection point. 1 original reflected 0.5 0 0.5 0 0.2 0.4 0.6 0.8 1
Estimators with Data Re-use Reflected Estimators Reflected STS with Reflection Point c: Tn,c(t) nt ( X nt X n) σ. n (a indicates the reflected analogue of the original quantity) Reflected Brownian Bridge: Bc (t) = Wc (t) twc (1) { W(t) t[2w(c) W(1)] if 0 t c = W(t) + 2(1 t)w(c) + tw(1) if c < t 1.
Estimators with Data Re-use Reflected Estimators Under Assumptions A, T n,c(t) B c (t). Reflected weighted area estimator and limiting functional: A c(f; n) [ 1 n n j=1 f ( ] j ) 2 n σt n,c ( j n ) [ 1 2 A c(f) f(t)σbc (t) dt]. Expected values depend on reflection point c. 0
Estimators with Data Re-use Reflected Estimators Linear Combination of Reflected Area Estimators k To obtain the estimator α i A c i (f) with minimum variance i=1 among k reflected area estimators, we solve min α,c subject to [ k ] Var α i A c i (f) i=1 k α i = 1 i=1 0 c 1 c 2 c k 1, α j IR, 0 c j 1; j = 1,..., k, where α i and c i for i = 1,..., k are weights and reflection points respectively.
Estimators with Data Re-use Reflected Estimators Variance of Linear Combination Estimator for f 0 ( σ 4 ) 2 1.8 1.6 Variance 1.4 1.2 1 0.8 1 2 3 4 5 6 7 # of Estimators
Estimators with Data Re-use Reflected Estimators Estimated Expected Values and Variances of Linearly Combined Area Estimators for the AR(1) Process with n = 10, 000 f 0 f 2 Number of Expected Variance Expected Variance Estimators Value Value 1 18.90 710.35 18.97 721.09 2 18.88 354.45 18.95 358.77 3 18.84 289.38 18.90 291.93 4 18.85 264.99 18.93 270.55 5 18.82 253.66 18.91 259.12 6 18.79 247.68 18.92 253.93 7 18.83 245.67 18.92 250.17
Conclusions and Future Work Outline 1 Introduction 2 Background Basics and Assumptions Standardized Time Series Estimators Estimators Based on Batches Estimators Based on Overlapping Batches 3 Estimators with Data Re-use Folded Estimators Folded Overlapping Area Estimators Reflected Estimators 4 Conclusions and Future Work
Conclusions and Future Work Summary I Obtained estimators using data re-use techniques: Folding, overlapping, and reflection. Folding increases bias, but as m increases, bias levels decrease to those of unfolded counterparts. Linear combination of folded estimators reduces variance significantly. While having roughly the same bias, the FOA estimators have smaller variance for all weight functions. Variance reduction is 33% for the weight function f 2 compared to unfolded overlapping area estimators. All folded estimators can be computed in O(n) time. Linear combination of reflected STS estimators reduces variance significantly. Bias may increase due to reflection.
Conclusions and Future Work Future Work Higher levels of folding. Linear combinations of area and CvM estimators at different folding levels. Folded overlapping CvM estimators. Sequential procedures that deliver required batch and sample sizes. Algorithms that give better and quicker solutions to optimization problem for linear combination of reflected estimators. Combination of batching with reflection. Comprehensive comparison of variance parameter estimators in literature.
Conclusions and Future Work References I Alexopoulos, C., C. F. Antonini, D. Goldsman, M. Meterelliyoz. Performance of folded variance estimators for simulation. Submitted to ACM TOMACS. Alexopoulos, C., N. T. Argon, D. Goldsman, G. Tokol, J. R. Wilson. Overlapping variance estimators for simulations. Operations Research 55 1090 1103, 2007. Antonini C. F. Folded variance estimators for stationary time series. PhD thesis. H. Milton Stewart School of ISyE, Georgia Tech, Atlanta, GA, 2005. Calvin, J. M., and M. K. Nakayama. Permuted standardized time series for steady-state simulations. Mathematics of Operations Research 31 351 368, 2006. Glynn, P. W., and W. Whitt. Estimating the asymptotic variance with batch means. Operations Research Letters 10 431 435, 1991.
Conclusions and Future Work References II Meterelliyoz, M., C. Alexopoulos, D. Goldsman. Folded overlapping area estimators for simulation. Technical Report. H. Milton Stewart School of ISyE, Georgia Tech, Atlanta, GA, 2008. Schruben, L. W. Confidence interval estimation using standardized time series. Operations Research 31 1090 1108, 1983. Song, W. M. T., and B. W. Schmeiser. Variance of the sample mean: Properties and graphs of quadratic-form estimators. Operations Research 41 501 517, 1993.
Conclusions and Future Work THANK YOU! QUESTIONS AND COMMENTS?