Proceedings of the 2015 Winter Siulation Conference L. Yilaz, W. K. V. Chan, I. Moon, T. M. K. Roeder, C. Macal, and M. D. Rossetti, eds. JACKKNIFED VARIANCE ESTIMATORS FOR SIMULATION OUTPUT ANALYSIS Keal Dinçer Dingeç Christos Alexopoulos David Goldsan H. Milton Stewart School of Industrial and Systes Engineering Georgia Institute of Technology Atlanta, GA 30332-0205, USA Wenchi Chiu Fenxi LLC Belont, CA 94002, USA Jaes R. Wilson Edward P. Fitts Departent of Industrial and Systes Engineering North Carolina State University Raleigh, NC 27695-7906, USA Tûba Aktaran-Kalaycı Advanced Analytics, Big Data AT&T Atlanta, GA 30319, USA ABSTRACT We develop new point estiators for the variance paraeter of a steady-state siulation process. The estiators are based on jackknifed versions of nonoverlapping batch eans, overlapping batch eans, and standardized tie series variance estiators. The new estiators have reduced bias and can be anipulated to reduce their variance and ean-squared error copared with their predecessors, facts which we deonstrate analytically and epirically. 1 INTRODUCTION A fundaental goal in siulation output analysis is the estiation of the unknown ean µ of a steady-state siulation-generated output process, {Y j : j= 1,2,...,n}. The obvious point estiator for this task is the saple ean, Y n n 1 n i=1 Y i. It also proves useful to get a handle on the saple ean s variability, so a long-standing area of research has involved estiating the easure σ 2 n nvar(y n ) or (alost equivalently) the variance paraeter, σ 2 li n σ 2 n = j= R j, where the covariance function R j Cov(Y 1,Y 1+ j ), for j = 0,1,... Knowledge of σ 2 helps us to ake precision and confidence stateents about Y n as an estiator for µ. Over the years, a significant literature has developed with the proble of estiating σ 2 in ind, for exaple, the ethods of nonoverlapping batch eans (NBM) (Scheiser 1982), overlapping batch eans (OBM) (Meketon and Scheiser 1984), and standardized tie series (STS) (Schruben 1983). These ethods typically divide the tie series {Y j : j = 1,2,...,n} into possibly overlapping batches of size, and calculate estiators for σ 2 that have been proven to be consistent as and b n/ both go to infinity that is, the ean squared errors (MSEs) of these estiators go to zero as and b. Broadly speaking, the batch size governs the bias coponent of the estiator s MSE, while the quantity b ost directly affects the variance coponent of MSE (Goldsan and Meketon 1986, Song and Scheiser 1995, and Aktaran-Kalaycı et al. 2011). When the budget n is fixed, one faces the classical bias-variance trade-off when selecting and b. The goal of this paper is to use siple jackknifing technology to facilitate large reductions in bias at the price of only odest increases in variance the result of which will be iproved MSE. 978-1-4673-9743-8/15/$31.00 2015 IEEE 459
This article is organized as follows. We present background aterial in Section 2 to introduce the NBM, OBM, and STS variance estiators that will be used in the sequel. The ain jackknifing results for these estiators are given in Section 3. A coparison of the various estiators is undertaken in Section 4. Conclusions and ongoing work are detailed in Section 5. 2 BACKGROUND In this section, we define the NBM, OBM, and STS estiators for σ 2. 2.1 Nonoverlapping Batch Means Estiator Here we divide the steady-state output {Y j : j = 1,2,...,n} into b contiguous, nonoverlapping batches of observations, each of length, where we assue that n = b. Thus, the ith nonoverlapping batch consists of observations {Y (i 1)+k : k=1,2,...,} for i=1,2,...,b. We define the nonoverlapping batch eans by Y i, 1 k=1 Y (i 1)+k, for i = 1,2,...,b. It is well known that under ild oent and ixing conditions, these batch eans can be regarded as approxiately independent and identically distributed (i.i.d.) noral rando variables as the batch size increases. This iediately allows us to use the scaled saple variance of the batch eans as the NBM estiator for σ 2 (Glynn and Whitt 1991, Steiger and Wilson 2001), N (b,) b b 1 (Y i, Y n ) 2 σ 2 χb 1 2 as, b 1 i=1 where the sybol denotes convergence in distribution and χ 2 ν is a χ 2 rando variable with ν degrees of freedo. Under ild conditions, several papers (e.g., Chien, Goldsan, and Melaed 1997, Goldsan and Meketon 1986, Song and Scheiser 1995) find the expected value of the NBM estiator. For instance, consider the following standing assuption. Assuption A: The process{y j } is stationary with ean µ and exponentially decaying covariance function R j =O(δ j ) for soe δ (0,1), so that j l R j = O( l δ ) and j= j l R j = γ l j=1 2 + O(l δ ) for l=0,1,2,..., where the Big-Oh notation g()=o(h()) eans that for soe finite constants C and 0, we have g() C h() for all 0, and where γ l 2 j=1 jl R j, l=0,1,2,... Under Assuption A, Aktaran-Kalaycı et al. (2007) show that E[N (b,)] = σ 2 γ 1(b+1) + O(δ ). (1) b In addition, the NBM estiator s variance is given by 2.2 Overlapping Batch Means Estiator li (b 1)Var[N (b,)] = 2σ 4 for fixed b. Now we for n +1 overlapping batches, each of size. In particular, the ith overlapping batch is coposed of the observations {Y i+k : k=0,1,..., 1}, for i=1,2,...,n +1; and the ith overlapping batch ean is Y O i, 1 k=0 1 Y i+k, for i=1,2,...,n +1. Finally, the OBM estiator for σ 2 is the appropriately scaled saple variance of the overlapping batch eans (Meketon and Scheiser 1984), O(b,) n +1 n (n +1)(n ) i=1 (Y O i, Y n ) 2. 460
Under Assuption A with b=n/ 2, Aktaran-Kalaycı et al. (2007) show that E[O(b,)] = σ 2 γ 1(b 2 + 1) n(b 1) + γ 1 + γ 2 (n )(n +1) + O(δ ) (2) (also see Goldsan and Meketon 1986 and Song and Scheiser 1995, aong others). Further, Daerdji (1995) finds that for large batch size and fixed saple-to-batch-size ratio b, li Var[O(b,)] = (4b3 11b 2 + 4b+6)σ 4 3(b 1) 4 4σ 4 3b for large b. So the OBM estiator has about the sae bias as, but only 2/3 the variance of the NBM estiator. 2.3 Standardized Tie Series Nonoverlapping Area Estiator For purposes of this subsection, we divide the steady-state siulation output {Y j : j = 1,2,...,n} into b = n/ nonoverlapping batches, as in Section 2.1. Schruben (1983) defined the standardized tie series fro nonoverlapping batch i by T i, (t) t (Y i, Y i, t ) σ for t [0,1] and i=1,2,...,b, where is the floor function and Y i, j j 1 j k=1 Y (i 1)+k is the jth cuulative saple ean fro batch i, for i=1,2,...,b and j= 1,2,...,. The STS nonoverlapping area estiator for σ 2, based on b batches and weight function f( ), is defined as where A i ( f ;) [ 1 and where f( ) satisfies the conditions 1 1 0 0 A( f ;b,) 1 b k=1 b i=1 A i ( f ;), f(k/)σt i, (k/)] 2 for i=1,2,...,b, f(s) f(t) ( in(s,t) st ) dsdt = 1 and d 2 dt 2 f(t) is continuous at every t [0,1]. (3) Under a ild functional central liit theore assuption (cf. Goldsan, Meketon, and Schruben 1990), it turns out that A( f ;b,) σ 2 χ 2 b/ b as. Moreover, where E[A( f ;b,)] = σ 2 [(F F)2 + F 2 ]γ 1 2 + O(1/ 2 ), (4) s u F(s) f(t)dt for s [0,1], F F(1), F(u) F(s)ds for u [0,1], and F F(1). 0 0 Further, under ild conditions and as long as the weight function f( ) satisfies Conditions (3), we have for fixed b, li bvar[a( f ;b,)] = 2σ 4. 461
Schruben s original area estiator uses the constant weight f 0 (t) 12 for all t [0,1], for which under Assuption A, Aktaran-Kalaycı et al. (2007) derive the fine-tuned result E[A( f 0 ;b,)] = σ 2 3γ 1 σ 2 2 + γ 1+ 2γ 3 3 + O(δ ), (5) indicating that A( f 0 ;b,) is soewhat biased in. The good news is that it is easy to choose a weight such as f 2 (t) 840(3t 2 3t+ 1/2) that yields an estiator having F = F = 0 in Equation (4), i.e., which is first-order unbiased for σ 2. In fact, under Assuption A, Aktaran-Kalaycı et al. (2007) obtain the fine-tuned result 3 MAIN RESULTS E[A( f 2 ;b,)] = σ 2 + 7(σ 2 6γ 2 ) 2 2 + 35(γ 1+ 2γ 3 ) 2 3 + O(1/ 4 ). (6) In this section, we deliver the ain results of the article. Section 3.1 describes a siple jackknife calculation that reduces estiator bias. Then Sections 3.2 3.4 show how to use the jackknife on the NBM, OBM, and STS nonoverlapping area estiators. Section 4 copares the perforances of the various estiators. 3.1 A Rough and Ready Tool One of the easiest ways to reduce estiator bias is via the use of jackknifing (Quenouille 1949, Quenouille 1956, Tukey 1958, Efron 1982). In the ensuing discussion, we will work with siple block jackknife versions of our original NBM, OBM, and STS estiators. In order to provide otivation, suppose that V (n) is a generic estiator for σ 2 based on n observations. Further suppose that E[V (n)]=σ 2 +c/n+o(1/n 2 ) for soe appropriate c. If we define a jackknife version of V (n) by V J (n) V (n) rv (rn), 0<r<1, (7) 1 r then an eleentary calculation reveals that E[V J (n)]=σ 2 +O(1/n 2 ), thereby yielding a first-order unbiased estiator for σ 2. While this bias reduction is greatly satisfying, the party is spoiled a bit by a variance increase, Var[V J (n)] = 1 ( Var[V (n)]+r 2 (1 r) 2 Var[V (rn)] 2rCov[V (n), V (rn)] ) 1 ( (1+r 2 (1 r) 2 )Var[V (n)] 2rCov[V (n), V (rn)] ), where the approxiation is due to the fact that Var[V (n)] and Var[V (rn)] (for fixed r) typically converge to the sae constant for large n. We will use this trick or an easy variant in the upcoing sections. 3.2 Jackknifing the NBM Estiator We apply a slight variant of Equation (7) to obtain the jackknifed NBM estiator, N J (b,,r) β N (b,r)n (b,)+[1 β N (b,r)]n (b/r,r), (8) where we assue for convenience that b/r and r are integers and we take β N (b,r) b+r b(1 r). 462
Equations (1) and (8) iediately reveal that E[N J (b,,r)]=σ 2 +O(δ ), i.e., N J (b,,r) has exponentially decaying bias. After carrying out additional algebra, the details of which are given in Dingeç et al. (2015), one can calculate the variance of the jackknifed NBM estiator, li Var[N J(b,,r)] = 2σ 4 b 1 [( 1+r r 2 ) ] b 2 +(r+ r 2 )b+r 2 (1 r)b(b r) 2σ 4 b 1 W(b,r), where W(b,r) represents a variance inflation factor over the original NBM estiator N (b,), with 1+r r2 li W(b,r) = b 1 r > 1 for 0<r<1. For exaple, for r= 1/8 and 1/2, the above liiting inflation factors are 71/56 1.3 and 5/2, respectively. 3.3 Jackknifing the OBM Estiator We can reove the OBM estiator s first-order bias ter displayed in Equation (2) by jackknifing, O J (b,,r) β O (b,r)o(b,)+(1 β O (b,r))o(b/r,r), where we again assue for convenience that b/r and r are integers and we take β O (b,r) (b 1) ( b 2 + r 2) (1 r)b[b 2 b(r+ 1) r]. After soe algebra, we find that the expected value of the jackknifed OBM estiator is [ (b E[O J (b,,r)] = σ 2 3 br+ r 2 r)+b 2 r ] (γ 1 + γ 2 ) + b [ (b 1)+1 ][ (b r)+1 ][ b 2 (1+r)b r ]+ O(δ ). (9) Thus, O J (b,,r) is first-order unbiased for σ 2, which is an iproveent over the analogous expected value result for O(b,) fro Equation (2). Moreover, Dingeç et al. (2015) find that li b E[O J (b,,r)] = σ 2 + O(δ ), suggesting that the bias will exhibit exponential decay for large b. Dingeç et al. (2015) also obtain the variance of the jackknifed OBM estiator, Var[O J (b,,r)] 4σ 4 3b (1+2r)+O(1/b2 ) for large b. For r = 1/2 and large b, the jackknifed OBM estiator has approxiately 2 ties the variance of the regular OBM estiator; this penalty goes up to a factor of at ost 3 for r 1. 3.4 Jackknifing the STS Nonoverlapping Area Estiator For notational convenience, we teporarily work with area estiators consisting of b = 1 batch of observations; odifications for b > 1 batches of observations will be discussed starting in Section 3.4.3. Therefore, consider Section 2.3 s area estiator fro the first batch of observations, A( f ;) A 1 ( f ;). We will exaine the effects of jackknifing on this area estiator with weights f 0 (t) and f 2 (t). 3.4.1 Area Estiator with Weight f 0 (t) Recall that the expected value of the area estiator with constant weight f 0 (t)= 12 is given by Equation (5), where it is revealed that A( f 0 ;) is first-order biased as an estiator of σ 2. The good news is that Equation (7) gives us a recipe to eliinate this first-order bias via the jackknifed estiator A J1 ( f 0 ;,r) A( f 0;) ra( f 0 ;r). 1 r 463
After soe algebra, our hopes are realized, since E[A J1 ( f 0 ;,r)] = σ 2 + σ 2 r 2 (1+r)(γ 1+ 2γ 3 ) r 2 3 + O(δ ). However, Dingeç et al. (2015) show that there is a steep price to be paid in ters of variance for this first-order unbiasedness. Naely, Var[A J1 ( f 0 ;,r)] Var[A( f 0 ;)] 1+r2 2r 4 (1 r) 2 ; (10) for instance, if r=1/2, the variance inflates by a factor of 4.5, so that Var[A J1 ( f 0 ;,r)] 9σ 4. Without becoing discouraged by this variance inflation, let us apply our jackknifing technology again to reove other bias ters. In fact, if we are interested in eliinating a generic area estiator s O(1/ l ) bias ter for soe positive integer l, Dingeç et al. (2015) show that the estiator A Jl ( f ;,r) A( f ;) rl A( f ;r) 1 r l, r (0,1), (11) does the trick. Siilarly, if we want to siultaneously eliinate two bias ters, say of orders O(1/ l 1) and O(1/ l 2) for positive integers l 1 and l 2 with l 2 >l 1, then it is easy to show that the following estiator is right for the job, A Jl1,l 2 ( f ;,r) A( f ;) (rl 1+ r l 2)A( f ;r)+r l 1+l 2 A( f ;r 2 ) (1 r l, r (0,1). 1 )(1 r l 2). For exaple, suppose that we would like to siultaneously eliinate the first- and third-order bias ters for the area estiator with weight f 0. Then we find after a bit of algebra that the expected value of the estiator A J1,3 ( f 0 ;,r) is ) E[A J1,3 ( f 0 ;,r)] = σ (1+ 2 1 r r(1 r 3 ) 2 + O(δ ), where the first- and third-order bias ters have indeed been eliinated, while the second-order ter still reains. Fortuitously, we can avoid a third jackknife and eliinate the O(1/ 2 ) bias ter by applying a siple anipulation. Let r(1 r 3 ) 2 ζ(,r) 1 r+ r(1 r 3 ) 2, so that the estiator has exponential convergence of its expected value to σ 2, A J 1,3 ( f 0 ;,r) ζ(,r)a J1,3 ( f 0 ;,r) (12) E [ A J 1,3 ( f 0 ;,r) ] = σ 2 + O(δ ). As with Equation (10), Dingeç et al. (2015) find that as, the asyptotic variance inflation caused by the double jackknife is Var[A J 1,3 ( f 0 ;,r)] Var[A( f 0 ;)] 1+r2 + r 3 + r 5 (1 r) 3 (1+r+ r 2 ), r (0,1); and for r=0.5, this ratio is 45/7 6.4, so that Var[A J 1,3 ( f 0 ;,r)] 12.857σ 4. 464
3.4.2 Area Estiator with Weight f 2 (t) We apply the technology of Equation (11) with l=2 to Equation (6) with b=1 to reove the quadratic bias ter, resulting in E[A J2 ( f 2 ;,r)] = E[A( f 2;)] r 2 E[A( f 2 ;r)] 1 r 2 = σ 2 35(γ 1+ 2γ 3 ) 2r(1+r) 3 + O(1/4 ), r (0,1), which is second-order unbiased, as proised. Siilar to Equation (10), we calculate the asyptotic variance inflation caused by the jackknife as, Var[A J2 ( f 2 ;,r)] Var[A( f 2 ;)] 1+r4 1 2 r5 [7+3r( 7+4r)] 2 (1 r 2 ) 2, r (0,1), so that for r=0.5, the inflation factor is about 1.88194, i.e., Var[A J2 ( f 2 ;,r)] 3.764σ 4. Notice that this inflation factor is significantly saller than the inflation factors for A J1 ( f 0 ;,r) and A J 1,3 ( f 0 ;,r). We can repeat the jackknifing exercise to reove higher-order bias ters, but instead refer the reader to Dingeç et al. (2015) for the detailed results. 3.4.3 Batching of Jackknifed Area Estiators The single-batch variance estiators A Jl ( f ;,r) and A J 1,3 ( f 0 ;,r) defined by Equations (11) and (12) are easily generalized for b>1 batches. Siply let A Jl ( f ;b,,r) 1 b b i=1 A Jl,i( f ;,r) and A J 1,3 ( f 0 ;b,,r) 1 b b i=1 A J 1,3,i( f 0 ;,r), where the i subscript indicates that the coponent estiator is fro the ith nonoverlapping batch, i = 1,2,...,b. Assuing that the A Jl,i( f ;,r) estiators fro different nonoverlapping batches are i.i.d., we see that E[A Jl ( f ;b,,r)]=e[a Jl ( f ;,r)] and Var [ A Jl ( f ;b,,r) ] = Var [ A Jl ( f ;,r) ] /b; and siilarly for AJ 1,3 ( f 0 ;b,,r). 4 COMPARISON OF ESTIMATORS In this section, we copare the perforances of different jackknifed estiators presented in Section 3 based on their biases, variances, and MSEs. 4.1 Bias, Variance, and Mean Squared Error Table 1 suarizes the ain results fro Section 3 for r = 1/2, along with asyptotically optial MSE results. The MSE of an estiator for the variance paraeter σ 2 balances bias and variance. To this end, consider the generic variance estiator V (n) for σ 2. Suppose that the bias of V (n) is of the for Bias[V (n)] = c/ k for soe constant c, batch size, and k > 0, where we ignore saller-order ters. Further suppose that the variance of V (n) is of the for Var[V (n)]=v/b for soe constant v and saple-to-batch-size ratio b = n/. In such cases, the MSE of V (n) as an estiator of σ 2 is MSE[V (n)] = Bias 2 [V (n)]+var[v (n)] c2 2k + v b, where the approxiation is the direct result of ignoring sall-order ters. As described in Goldsan and Meketon (1986), Song and Scheiser (1995), and Aktaran-Kalaycı et al. (2011), the iniu value of 465
Table 1: Approxiate bias, variance, and optial MSE forulas for large b and and r = 1/2. (Bias results for O J (b,,1/2) are for the special case in which b.) Estiator Bias b Variance/σ 4 MSE N (b,) γ 1 ( γ1 σ 4 2 3 n N J (b,,1/2) O(δ ) 5 O(ln(n)/n) ) 2/3 O(b,) γ 1 ( γ1 σ 4 4/3 2.289 n ) 2/3 O J (b,,1/2) O(δ ) 8/3 O(ln(n)/n) A( f 0 ;b,) 3γ 1 ( γ1 σ 4 2 6.240 n ) 2/3 A J1 ( f 0 ;b,,1/2) 2σ 2 ( σ 5 2 9 12.622 n A J 1,3 ( f 0 ;b,,1/2) O(δ ) 90/7 O(ln(n)/n) ) 4/5 A( f 2 ;b,) A J2 ( f 2 ;b,,1/2) 7(σ 2 [ 6γ 2 ) (σ 2 6γ 2 )σ 8 2 2 2 4.740 n 2 [ 70(γ 1 + 2γ 3 ) (γ1 + 2γ 3 )σ 12 3 3 3.764 11.545 n 3 ] 2/5 ] 2/7 this quantity (at least asyptotically for large values of the run length n and hence for large and b) is [ ( v ) ] 2 k MSE 1+2k [V (n)] = (1+2k) c. 2nk For the variance estiators N J (b,,r), O J (b,,r), and A J 1,3 ( f 0 ;b,,r) for which the bias is of the for O(δ ), a ore-delicate analysis is required to show that the iniu MSE is of order O(ln(n)/n); see Dingeç et al. (2015). 4.2 Exact Bias Exaple We present exact closed-for bias results for a particular stochastic process a stationary autoregressive process of order 1 [AR(1)]. The AR(1) is defined by Y i = φy i 1 + ε i for i=1,2,..., where 1<φ < 1, Y 0 N(0,1), and the ε i s are i.i.d. N(0,1 φ 2 ) rando variables, independent of Y 0. The covariance function of the process is R j = φ j, for j= 0,±1,±2,... As shown by Aktaran-Kalaycı et al. (2007), we have σ 2 =(1+φ)/(1 φ),γ 1 = 2φ/(1 φ) 2,γ 2 = 2φ(1+φ)/(1 φ) 3, and γ 3 = 2φ(1+4φ+φ 2 )/(1 φ) 4. Aktaran-Kalaycı et al. (2007) give closed-for forulas for the expected values of N (b, ), O(b, ), and A( f 0 ;b,) for the AR(1) process. The expected value of the vanilla NBM estiator is E[N (b,)] = σ 2 γ 1 b (b+1 b2 φ φ b b 1 ) ; (13) 466
and, for large b, we have Dingeç, Alexopoulos, Goldsan, Wilson, Chiu, and Aktaran-Kalaycı li E[N (b,)] = σ 2 γ 1(1 φ ). b Equations (8) and (13) give the expected value of the jackknifed NBM estiator, E[N J (b,,r)] = σ 2 γ [( 1 b 2 1 ) φ r +(r 2 b 2 )φ +(1 r 2 )φ b]. (b 1)(1 r)(b r) For large b, we obtain The expected value of the regular OBM estiator is E[O(b,)] = σ 2 li E[N J(b,,r)] = σ 2 γ 1(φ r φ ). (14) b (1 r) ( b 2 + 1 ) γ 1 b(b 1) + γ 1 + γ 2 (n )(n +1) + γ 1φ n [ b+ φ n b 2( 1+φ n 2+1 φ n +1) ]. (1 φ)(n +1) We will not give the tedious expression here for E[O J (b,,r)], but as per Equation (9), the jackknifed estiator O J (b,,r) is first-order unbiased for σ 2. What is uch ore interesting is that for the special case in which we let b, soe algebra reveals that the expected value of the jackknifed OBM estiator is the sae as the right-hand side of Equation (14). So, asyptotically, the jackknifed OBM estiator has the sae bias as the jackknifed NBM estiator. But we also know that the jackknifed OBM estiator has an asyptotic variance that is saller than that of the jackknifed NBM estiator. Thus, perhaps it will be the case that, asyptotically, O J (b,,r) will have a saller MSE than N J (b,,r) though this is not quite borne out by the respective MSE entries of Table 1, which only indicate that the MSEs are of the sae order O(ln(n)/n). Continuing, the expected value of the original area estiator A( f 0 ;b,) is E[A( f 0 ;b,)] = σ 2 3γ 1 σ 2 2 + γ 1+ 2γ 3 3 3γ 1 while that of the jackknifed version turns out to be (1+ σ 2 ) 2 φ, E [ AJ 1,3 ( f 0 ;b,,r) ] = σ 2 3r 2 γ 1 φ r2 (1 r) 2 (1+r+ r 2 ) + o(φ r2 /), where the little-oh notation g() = o(h()) eans that g()/h() 0 as. For this AR(1) exaple, note that the bias of the jackknifed area estiator is O(φ r2 /), whereas the biases of the jackknifed NBM and OBM estiators are of the saller order O(φ r /). Table 2 presents exact bias results for the regular N (b,), O(b,), and A( f 0 ;b,) estiators and the jackknifed N J (b,,r), O J (b,,r), and AJ 1,3 ( f 0 ;b,,r) estiators with various batches sizes, batch count b=n/=10, and r = 1/2 for an AR(1) process with φ = 0.9 (in which case σ 2 = 19). We see that jackknifing draatically reduces estiator bias, as anticipated by the underlying theory. Note that the N J (b,,r) and AJ 1,3 ( f 0 ;b,,r) estiators have exponentially decaying bias as the batch size increases, while O J (b,,r) sees to decay a bit ore slowly (since b=10 is sall ). 4.3 Asyptotically Optial Mean Squared Error Exaple With these exact expected bias results as well asyptotic variance results fro Dingeç et al. (2015) for the general jackknife paraeter r in hand, we optiize the MSEs with respect to the batch count b, the batch 467
Table 2: Exact biases of the N (b,), O(b,), and A( f 0 ;b,) estiators and the jackknifed N J (b,,r), O J (b,,r), and A J 1,3 ( f 0 ;b,,r) with b=10 and r=1/2 for an AR(1) process with φ = 0.9 (σ 2 = 19). N (b,) N J (b,,r) O(b,) O J (b,,r) A( f 0 ;b,) AJ 1,3 ( f 0 ;b,,r) 64 3.09 0.22 3.14 0.21 7.72 2.21 128 1.55 0.004 1.58 0.001 4.13 0.19 256 0.77 2E 06 0.79 0.0008 2.10 0.002 512 0.39 2E 12 0.39 0.0002 1.05 1E 06 1024 0.19 2E 24 0.20 5E 05 0.53 7E 13 2048 0.10 3E 48 0.10 1E 05 0.26 6E 25 size, and r, for an AR(1) process with φ = 0.9. The optial values are found by nuerical iniization of the asyptotic MSE, which is given by the su of the exact squared bias and the asyptotic variance. These results are displayed in Table 3 for a selection of n-values. We see that the jackknifed estiators quickly outperfor their non-jackknifed counterparts in ters of optial MSE as n increases. Specifically, N J (b,,r) perfors quite well, though for large-enough n (and hence large-enough b and ), the jackknifed OBM estiator O J (b,,r) eventually overtakes it. 5 CONCLUSIONS AND ONGOING WORK We have shown that the use of jackknifing is an effective way to draatically reduce the bias and ean squared error of estiators of a steady-state siulation s variance paraeter σ 2. This is particularly noteworthy in light of the fact that jackknifing typically increases estiator variance. We presented results for nonoverlapping batch eans, overlapping batch eans, and certain standardized tie series area estiators. In Dingeç et al. (2015), we generalize our work in the following ways: We consider other STS area estiator weighting functions, specifically, the general class given in Foley and Goldsan (1999). We consider other classes of STS estiators, for exaple, estiators based on Craér von Mises (Goldsan et al. 1999), Durbin Watson (Batur et al. 2009), folded (Alexopoulos et al. 2010), and reflected (Meterelliyoz et al. 2015) functionals of Brownian bridges, as well as overlapping versions thereof (see, e.g., Alexopoulos et al. 2007). We derive approxiate distributions of the various variance estiators not just the first two central oents. We work with jackknife estiators that take advantage of batching in a ore-efficient way. When carrying out siultaneous jackknifing to eliinate ultiple orders of bias, we work with ultiple r-values not just a single value as in the current paper. We are certainly encouraged by the fact that jackknifing alost always decreases bias significantly for this bulleted list of estiators both in theory and on practical experiental stochastic processes. ACKNOWLEDGMENTS We thank the referees for their eaningful coents and suggestions. Keal Dingeç was supported by the TÜBİTAK (The Scientific and Technological Research Council of Turkey) 2219 International Postdoctoral Research Scholarship Progra. Christos Alexopoulos, David Goldsan, and Jaes R. Wilson were partially supported by National Science Foundation grants CMMI-1233141/1232998. REFERENCES Aktaran-Kalaycı, T., C. Alexopoulos, N. T. Argon, D. Goldsan, and J. R. Wilson. 2007. Exact Expected Values of Variance Estiators in Steady-State Siulation. Naval Research Logistics 54 (4): 397 410. 468
Table 3: Asyptotically optial b,, r, and MSE for the N (b,), O(b,), and A( f 0 ;b,) estiators and the jackknifed N J (b,,r), O J (b,,r), and A J 1,3 ( f 0 ;b,,r) estiators for an AR(1) process with φ = 0.9 (σ 2 = 19). (The optial b and are rounded to the nearest integer.) N (b,) N J (b,,r) n b MSE b r MSE 256 9 28 120.267 12 22 0.07 130.620 512 14 36 75.764 19 27 0.13 78.367 1024 23 45 47.728 31 33 0.18 46.034 2048 36 57 30.067 53 38 0.21 26.665 4096 57 72 18.941 93 44 0.24 15.266 8192 91 90 11.932 166 49 0.26 8.647 16384 144 114 7.517 299 55 0.28 4.850 32768 229 143 4.735 545 60 0.30 2.697 65536 363 181 2.983 999 66 0.31 1.488 O(b,) O J (b,,r) n b MSE b r MSE 256 8 33 91.781 11 22 0.17 103.248 512 12 41 57.819 19 26 0.27 58.784 1024 20 52 36.423 34 30 0.34 33.110 2048 31 65 22.945 61 34 0.41 18.526 4096 50 82 14.455 110 37 0.46 10.300 8192 79 103 9.106 202 41 0.50 5.689 16384 126 130 5.736 373 44 0.54 3.121 32768 200 164 3.614 693 47 0.57 1.701 65536 317 207 2.276 1295 51 0.60 0.922 A( f 0 ;b,) AJ 1,3 ( f 0 ;b,,r) n b MSE b r MSE 256 4 59 250.166 5 47 0.03 233.765 512 7 75 157.595 8 65 0.03 151.602 1024 11 94 99.279 13 81 0.06 95.683 2048 17 118 62.542 21 96 0.10 58.631 4096 27 149 39.399 37 111 0.13 35.026 8192 44 188 24.820 64 127 0.15 20.516 16384 69 237 15.635 114 143 0.17 11.832 32768 110 298 9.850 205 160 0.18 6.739 65536 175 375 6.205 370 177 0.19 3.799 469
Aktaran-Kalaycı, T., C. Alexopoulos, D. Goldsan, and J. R. Wilson. 2009. Optial Linear Cobinations of Overlapping Variance Estiators for Steady-State Siulation. In Advancing the Frontiers of Siulation: A Festschrift in Honor of George Sauel Fishan, edited by C. Alexopoulos, D. Goldsan, and J. R. Wilson, 291 328. New York: Springer Science+Business Media. Aktaran-Kalaycı, T., C. Alexopoulos, D. Goldsan, and J. R. Wilson. 2011. On the Mean-Squared Error of Variance Estiators for Coputer Siulations. In Proceedings of the 2011 Winter Siulation Conference, edited by S. Jain, R. R. Creasey, J. Hielspach, K. P. White, and M. Fu, 549 555. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers, Inc. Alexopoulos, C., C. F. Antonini, D. Goldsan, and M. Meterelliyoz. 2010. Perforance of Folded Variance Estiators for Siulation. ACM Transactions on Modeling and Coputer Siulation, 20(3): article 11. Alexopoulos, C., N. T. Argon, D. Goldsan, G. Tokol, and J. R. Wilson. 2007. Overlapping Variance Estiators for Siulation. Operations Research 55 (6): 1090 1103. Batur, D., D. Goldsan, and S.-H. Ki. 2009. An Iproved Standardized Tie Series Durbin Watson Variance Estiator for Steady-State Siulation. Operations Research Letters 37:285 289. Chien, C.-H., D. Goldsan, and B. Melaed. 1997. Large-Saple Results for Batch Means. Manageent Science 43:1288 1295. Daerdji, H. 1995. Mean-Square Consistency of the Variance Estiator in Steady-State Siulation Output Analysis. Operations Research 43 (2): 282 291. Dingeç, K., T. Aktaran-Kalaycı, C. Alexopoulos, W. Chiu, D. Goldsan, and J. R. Wilson 2015. Jackknifed Variance Estiators for Siulation Output Analysis. Technical Report, H. Milton Stewart School of Industrial and Systes Engineering, Georgia Institute of Technology, Atlanta, Georgia. Efron, B. 1982. The Jackknife, the Bootstrap, and Other Resapling Plans. Philadelphia, Pennsylvania: Society for Industrial and Applied Matheatics. Foley, R. D., and D. Goldsan. 1999. Confidence Intervals Using Orthonorally Weighted Standardized Tie Series. ACM Transactions on Modeling and Siulation 9:297 325. Glynn, P. W., and W. Whitt. 1991. Estiating the Asyptotic Variance with Batch Means. Operations Research Letters 10:431 435. Goldsan, D., K. Kang, and A. F. Seila. 1999. Craér von Mises Variance Estiators for Siulations. Operations Research 47:299 309. Goldsan, D., and M. S. Meketon. 1986. A Coparison of Several Variance Estiators. Technical Report, School of Industrial and Systes Engineering, Georgia Institute of Technology, Atlanta, Georgia. Goldsan, D., M. S. Meketon, and L. W. Schruben. 1990. Properties of Standardized Tie Series Weighted Area Variance Estiators. Manageent Science 36:602 612. Meketon, M. S., and B. W. Scheiser. 1984. Overlapping Batch Means: Soething For Nothing? In Proceedings of the 1984 Winter Siulation Conference, edited by S. Sheppard, U. W. Pooch, and C. D. Pegden, 227 230. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers, Inc. Meterelliyoz, M., C. Alexopoulos, D. Goldsan, and T. Aktaran-Kalaycı. 2015. Reflected Variance Estiators for Siulation. To appear in IIE Transactions, accessed at www.tandfonline.co/doi/full/10.1080/0740817x.2015.1005776 Quenouille, M. H. 1949. Probles in Plane Sapling, Annals of Matheatical Statistics 20 (3): 355-375. Quenouille, M. H. 1956. Notes on Bias in Estiation. Bioetrika 43 (3-4): 353 360. Scheiser, B. W. 1982. Batch Size Effects in the Analysis of Siulation Output. Operations Research 30:556 568. Schruben, L. W. 1983. Confidence Interval Estiation Using Standardized Tie Series. Operations Research 31:1090 1108. Sheran, M. 1995. On Batch Means in the Siulation and Statistical Counities. In Proceedings of the 1995 Winter Siulation Conference, edited by C. Alexopoulos, K. Kang, W. R. Lilegdon, and D. Goldsan, 297 302. Piscataway, New Jersey: Institute of Electrical and Electronics Engineers, Inc. 470
Song, W.-M., and B. W. Scheiser. 1995. Optial Mean-Squared-Error Batch Sizes. Manageent Science 41:110 123. Song, W.-M. T. 1996. On the Estiation of Optial Batch Sizes in the Analysis of Siulation Output Analysis. European Journal of Operational Research 88:304 309. Steiger, N. M., and J. R. Wilson. 2001. Convergence Properties of the Batch-Means Method for Siulation Output Analysis. INFORMS Journal on Coputing 13 (4): 277 293. Tukey, J. W. 1958. Bias and Confidence in Not Quite Large Saples (abstract). Annals of Matheatical Statistics 29: 614. AUTHOR BIOGRAPHIES TÛBA AKTARAN-KALAYCI received her B.S. degree in Industrial Engineering fro Bilkent University in Ankara, Turkey, and her Ph.D. fro the H. Milton Stewart School of Industrial and Systes Engineering at the Georgia Institute of Technology. She is currently Director of Advanced Analytics, Big Data at AT&T in Atlanta, Georgia. Her research interests lie in siulation output analysis, financial applications, and risk anageent. Her e-ail address is taktaran@gail.co. CHRISTOS ALEXOPOULOS is a professor in the H. Milton Stewart School of Industrial and Systes Engineering at the Georgia Institute of Technology. His research interests are in the areas of siulation, statistics, and optiization of stochastic systes. He is a eber of INFORMS and an active participant in the Winter Siulation Conference, having been Proceedings Co-Editor in 1995, Associate Progra Chair in 2006, and a eber of the Board of Directors since 2008. He is also an Area Editor of the ACM Transactions on Modeling and Coputer Siulation. His e-ail address is christos@isye.gatech.edu, and his Web page is www.isye.gatech.edu/ christos. WENCHI CHIU is a data analyst at Fenxi LLC, where her projects focus on entertainent, edia, and technology arket research. She received her Ph.D. degree in Industrial Engineering fro National Tsing Hua University, Taiwan, and later did postdoc research at the H. Milton Stewart School of Industrial and Systes Engineering at the Georgia Institute of Technology. Her research at Georgia Tech concentrated on siulation and statistical applications in healthcare. Her e-ail address is wenchichiu@gail.co. KEMAL DİNÇER DİNGEÇ is a post-doctoral researcher in the H. Milton Stewart School of Industrial and Systes Engineering at the Georgia Institute of Technology. He was previously a post-doc at Boğaziçi University s Industrial Engineering Departent. His research interests include stochastic odels in anufacturing and finance, Monte Carlo siulation, and variance reduction ethods for financial siulations. His eail address is keal.dingec@boun.edu.tr. DAVID GOLDSMAN is a professor in the H. Milton Stewart School of Industrial and Systes Engineering at the Georgia Institute of Technology. His research interests include siulation output analysis, ranking and selection, and healthcare siulation. He was Progra Chair of the Winter Siulation Conference in 1995 and a eber of the WSC Board of Directors between 2001 2009. He is currently a trustee of the WSC Foundation. His e-ail address is san@gatech.edu, and his Web page is www.isye.gatech.edu/ san. JAMES R. WILSON is a professor in the Edward P. Fitts Departent of Industrial and Systes Engineering at North Carolina State University. His current research interests are focused on probabilistic and statistical issues in the design and analysis of siulation experients, with special ephasis on applications in healthcare and production. As a WSC participant, he served as Proceedings Editor (1986), Associate Progra Chair (1991), and Progra Chair (1992). During the period 1997 2004, he was a eber of the WSC Board of Directors. He is a eber of ACM, ASA, and ASEE; and he is a Fellow of IIE and INFORMS. His e-ail address is jwilson@ncsu.edu, and his Web page is www.ise.ncsu.edu/jwilson. 471