Recursive Estimation of Time-Average Variance Constants Through Prewhitening

Transcription

1 Recursive Estimation of Time-Average Variance Constants Through Prewhitening Wei Zheng a,, Yong Jin b, Guoyi Zhang c a Department of Mathematical Sciences, Indiana University-Purdue University, Indianapolis, IN 460, United States b Department of Finance, Insurance and Real Estate, University of Florida, Gainesville, FL 3605, United States c Department of Mathematics and Statistics, University of New Mexico, Albuquerque, NM 87131, United States Abstract The time-average variance constant (TAVC) has been an important component in the study of time series. Many real problems request a fast and recursive way in estimating TAVC. In this paper we apply AR(1) prewhitenning filter to the recursive algorithm by Wu (009b), so that the memory complexity of order O(1) is maintained and the accuracy of the estimate is improved. This is justified by both theoretical results and simulation studies. Keywords: Central limit theorem, consistency, linear process, Markov chains, martingale, Monte Carlo, nonlinear time series, prewhitening, recursive estimation, spectral density. 010 MSC: 60F05 (Primary), 60F17 (Secondary) 1. Introduction 5 10 The time-average variance constant (TAVC) plays an important role in many time series inference such as unit root testing and statistical inference of the mean. Throughout the paper, we focus on a stationary time series {X i } i Z with mean µ = E(X i ) and finite variance. The TAVC typically has the representation of σ = k Z γ(k), where γ(k) = cov(x 0,X k ) is the covariance function at lag k. The TAVC is proportional to the spectral density function evaluated at the origin by a constant and the estimation of the latter has been extensively studied in literature. See for instance Stock (1994), Song and Schmeiser (1995), Phillips and Xiao (1998), Politis, Romano and Wolf (1999), Bühlmann (00), Lahiri (003), Alexopoulos and Goldsman (004), Haldrup and Jansson (005) and Jones et al. (006). In many applications, one has to sequentially update the estimate of σ while the observations are being accumulated one by one. For example, in MCMC experiments, while the observations are sequentially Corresponding address: Department of Mathematical Sciences, Indiana University-Purdue University Indianapolis 40 N. Blackford, LD 70, Indianapolis, IN, 460, United States. Tel.: ; fax: URL: weizheng@iupui.edu (Wei Zheng), yong.jin@warrington.ufl.edu (Yong Jin), gzhang13@gmail.com (Guoyi Zhang) Preprint submitted to Journal of LATEX Templates November 7, 014

2 generated, one would like to construct the confidence interval of µ by X n ± z α/ˆσ n / n to decide if the sample size n is sufficient to make the interval small enough. Here z α/ is the (1 α/)th percentile of the standard normal distribution and X n and ˆσ n are the sequentially updated estimates of µ and σ respectively, based on the first n observations. For the traditional methods of estimating σ, the computation time and the memory required will increase for each update of ˆσ n as n increases. This is prohibitive especially when multiple MCMC chains are run simultaneously. To address this issue, Wu (009b) modified the batch mean approach into a recursive version without sacrificing the convergence rate of O(n 1/3 ). Meanwhile, the memory complexity of each update is reduced from the traditional O(n) to O(1). In this paper, we incorporate the prewhitening idea into Wu (009b) s approach in order to achieve reduced mean squared error (MSE) for the estimation of σ. The idea of applying prewhitening (Press and Tukey, 1956) to the time series for TAVC (or the spectral density function) estimation has a long history. See for instance Andrews and Monahan (199), Lee and Phillips (1994), Phillips and Sul (003) and Sul, Phillips and Choi (005). The benefit of prewhitening is that the resulting residuals are less dependent than the original data and one can obtain more accurate estimate of the TAVC for the residuals due to larger effective degrees of freedom. From the latter, one can easily retrieve the estimate of σ. To be specific, suppose the residuals {e i } i Z are generated from {X i } i Z by the mechanism of a(l)x i = b(l)e i, where L is the lag operate and a( ) and b( ) are polynomial functions so that their roots do not fall in the unit circle. It is well known that σ = a(1) b(1) σe, where σe is the TAVC of {e i } i Z. If {X i } i Z follows an ARMA model, {e i } i Z becomes the white noise sequence under the proper choice of a( ) and b( ). In the latter case, the estimation of σe could be a lot more accurate than that of the original sequence. Of course, a( ) and b( ) are unknown even under the ARMA model, not to mention the original sequence may possibly follows a nonlinear model. However, by plugging in data-driven estimate of a( ) and b( ) according to some model fitting criteria, the transformed data {e i } i Z typically is less dependent than the original data and hence the advantage of prewhitening still remains. For the sake of simplicity of implementing the algorithm, we shall adopt the AR(1) prefilter in constructing recursive estimate of the TAVC. By this small modification, we would achieve substantial improvements estimation efficiency under some circumstances. Theoretical conditions are given for deciding if the prewhitening is needed or not. The rest of the paper is organized as follows. Section provides the algorithms of recursive estimation of TAVC. In Section 3, we derive the asymptotic properties of the proposed estimators. In Section 4, we discuss efficiency comparisons between the proposed method and Wu s method in terms of both theoretical results and simulation studies. Section 5 concludes the paper. All the proofs are included in Appendix.

3 Proposed algorithms Throughout the paper, we assume µ = 0 without loss of generality. We first propose an algorithm for estimating σ through prewhitening when µ is known. Then we generalize it to the unknown case. Here are some frequently used notations throughout the paper: a k = ck p ; t i = k N a ki ak i<a k+1 ; ρ = γ(1)/γ(0); ˆγ n (k) = n 1 n i=k+1 X ix i k ; γ n (k) = n 1 n i=k+1 (X i X n )(X i k X n ); ˆρ n = ˆγ n (1)/ˆγ n (0); ρ n = γ n (1)/ γ n (0); and v n = n l i, where l i = i t i When µ is known 55 To estimate σ, Wu (009b) proposed the statistic v 1 n n ( i X j ), which allows the recursive algorithm with the memory complexity of O(1). One way of improving the accuracy of the estimation is to prewhiten the original sequence so that the transformed sequence has weaker dependence and the corresponding estimation of TAVC would be more accurate. Here, we adopt the AR(1) model as the prefilter to produce a new sequence, i.e. e i = X i ρx i 1 with ρ = γ(1)/γ(0). Due to the relationship σ = σ e/(1 ρ), it is reasonable to estimate σ by V n /(v n (1 ρ) ) when ρ is known, where V n = Wi and W i = (X j ρx j 1 ). When ρ is unknown, one only need to plug in the estimate of ρ and hence σ can be estimated by ˆσ = ˆV n /(v n (1 ˆρ n ) ), (1) where ˆV n = n Ŵ i,n and Ŵi,n = i (X j ˆρ n X j 1 ). It can be shown that ˆV n = X j + ˆρ n X j 1 ˆρ n n X j X j To simplify notations, let S n,0 = n X i, S n,1 = n i= X ix i 1, W i,0 = i X j, W i,1 = i X j 1, V n,0 = n W i,0, V n,1 = n W i,1, and V n, = n W i,0w i,1. As a result, ˆρ n = S n,1 /S n,0 and ˆV n = V n,0 + ˆρ nv n,1 ˆρ n V n,. Now ˆσ n can be calculated recursively by the following Algorithm. Algorithm 1. At stage n, we store (k n, v n, X n, S n,0, S n,1, ˆρ n, W n,0, W n,1, V n,0, V n,1, V n, ). Note that t n = a kn. When n = 1, the vector is (1, 1, X 1, X 1, 0, 0, X 1, 0, X 1, 0, 0). At stage n + 1, we update the vector by If n + 1 = a 1+kn, let W n+1,0 = X n+1, W n+1,1 = X n and k n+1 = 1 + k n ; Otherwise, let W n+1,0 = W n,0 + X n+1, W n+1,1 = W n,1 + X n and k n+1 = k n.. Let S n+1,0 = S n,0 +X n+1, S n+1,1 = S n,1 +X n+1 X n, V n+1,0 = V n,0 +W n+1,0, V n+1,1 = V n,1 +W n+1,1, V n+1, = V n, + W n+1,0 W n+1,1, and v n+1 = v n + (n + a kn+1 ). 3

4 3. Let ˆρ n+1 = S n+1,1 /S n+1,0. 4. Calculate ˆV n+1 = V n+1,0 + ˆρ n+1v n+1,1 ˆρ n+1 V n+1, 70 Output: ˆσ n+1 = ˆV n+1 /(v n+1 (1 r) )... When µ is unknown In this case, we have to centralize the data by the sample average and then apply Algorithm 1 to the centralized data. Hence one can estimate σ by σ = Ṽn/(v n (1 ρ n ) ), () where Ṽ n = W i,n, Wi,n = (X j X n ρ n (X j 1 X n )). (3) Equation (3) can be rewritten as Ṽ n = Ṽ n + (1 ρ n ) X n d n (1 ρ n ) X n U n, (4) 75 where ρ n = n i= (X i X n )(X i 1 X n )/ n (X i X n ), Ṽn = n ( i (X j ρ n X j 1 )), d n = n l i and U n = n l i i (X j ρ n X j 1 ). Note that Ṽ n is similar with ˆV n except that the ˆρ n therein is replaced by ρ n. Also note that X n+1 = (n X n + X n+1 )/(n + 1) and ρ n = n i= X ix i 1 + X n (X 1 + X n ) (n + 1) X n n X i n X. n We can update σ n recursively by the following Algorithm Algorithm. At stage n, we store (k n, v n, d n, X 1, X n, X n, S n,0, S n,1, ρ n, W n,0, W n,1, V n,0, V n,1, V n,, U n ). Note that t n = a kn. When n = 1, the vector is (1, 1, 1, X 1, X 1, X 1, X1, 0, 0, X 1, 0, X1, 0, 0, X 1 ). At stage n + 1, we update the vector by 1. If n + 1 = a 1+kn, let W n+1,0 = X n+1, W n+1,1 = X n and k n+1 = 1 + k n ; Otherwise, let W n+1,0 = W n,0 + X n+1, W n+1,1 = W n,1 + X n and k n+1 = k n.. Let S n+1,0 = S n,0 + Xn+1, S n+1,1 = S n,1 + X n+1 X n, and X n+1 = (n X n + X n+1 )/(n + 1). 3. Let ρ n+1 = (S n+1,1 + X n+1 (X 1 + X n+1 ) (n + ) X n+1)/(s n+1,0 (n + 1) X n+1). 4. Let V n+1,0 = V n,0 + Wn+1,0, V n+1,1 = V n,1 + Wn+1,1, V n+1, = V n, + W n+1,0 W n+1,1, and v n+1 = v n + (n + a kn+1 ), d n+1 = d n + (n + a kn+1 ). 5. Let U n+1 = U n + (n + a kn+1 )(W n+1,0 ρ n+1 W n+1,1 ) and calculate Ṽ n+1 = V n+1,0 + ρ n+1v n+1,1 ρ n+1 V n+1,. 6. Calculate Ṽn+1 = Ṽ n+1 + (1 ρ n+1 ) X n+1 d n+1 (1 ρ n+1 ) X n+1 U n+1 90 Output: σ n+1 = Ṽn+1/(v n+1 (1 ρ n+1 ) ). 4

5 3. Asymptotical properties of the estimators In this section, we shall study the asymptotic properties of ˆσ n and σ as defined by (1) and () respectively. Theorem 1 derives the asymptotic properties of V n, which leads to Theorem and Corollary 1 regarding the asymptotic properties of ˆσ n and σ n. Define a b = min(a, b). A random variable ξ is said to be in L p (p > 0) if ξ p := [E( ξ p )] 1/p <, where ξ = ξ. For two sequences {a n } and {b n }, write a n b n, if lim n a n /b n = 1; write a n = o(b n ) if lim n a n /b n = 0; write a n = O(b n ) if lim sup n a n /b n < ; write a n = o a.s (b n ) if lim n a n /b n = 0 almost surely; write a n b n or a n = O(b n ) if there exists a constant c > 0, such that 1/c a n /b n c for all large n. Throughout the paper, we assume that {X i } is a stationary causal process of the form X i = g(f i ), F i = (ε 1,..., ε i 1, ε i ), and {ε i } are iid. Let X i = g(f i ), where F i = (F 1, ε 0, ε 1,..., ε i ) and ε 0 is an iid copy of ε i s. The quantity δ α (i) = X i X i α is called the physical dependence measure by Wu (005). Recall e i = X i ρx i 1 and define e i = X i ρx i 1. Let θ α(i) := e i e i α δ α (i), it can be shown that the following two conditions are equivalent δ α (i) <, (5) i=0 θ α (i) <. (6) i=0 105 in view of X i = k 0 ρk e i k. Particularly, for the AR(1) model we have θ α (i) = ε 0 ε 0 α 1 i=0. Since e i is adapted to F i and E(e i ) = 0, under mild condition of P 0 e i <, where P i = E( F i ) E( F i 1 ), (7) i=0 we have the functional central limit theorem 1 e i, 0 t 1 n i nt d {σ e IB(t), 0 t 1}, 110 where IB is the standard Brownian motion and σ e = i=0 P 0e i = k Z E(e 0e k ) = (1 ρ) σ. Note that (7) is weaker than (5). Now that σ e is the TAVC for the time series {e i }. Wu (009b) proposed V n /v n to estimate σ e. The properties of V n are given in the following theorem, and its proof follows the same way as in Wu (009b). 5

6 Theorem 1. Let a k = ck p, k 1, where c > 0 and p > 1 are constants. (i) Assume that E(X i ) = 0, X i L α and (6) holds for some α (, 4], we have V n E(V n ) α/ = O(n τ ), max V n E(V n ) n N = O(N τ logn), α/ where τ = /α + 3/ 3/(p). (ii) Assume that E(X i ) = 0, X i L α and (6) holds for some α > 4, we have V n E(V n ) α/ = O(n 3/(p) ), max V n E(V n ) n N = O(N 3/(p) ). α/ (iii) Assume that E(X i ) = 0, and for α > 4, we have P 0 e i α <, (8) i=0 V n E(V n ) lim = σ ep c 3/(p), n n 3/(p) 1p 9 (iv) Assume that EX i = 0, and for some q (0, 1] either (a) or (b) k=0 P 0X k < and are satisfied, we have k q P 0 e k < (9) k=0 k q P 0 (e k ρe k+1 ) <, (10) k=0 E(V n ) v n (1 ρ) σ = O(n 1+(1 q)(1 1/p) ). Lemma 1 below indicates that the magnitude of ˆV n V n and Ṽn V n are both negligible compared to that of V n v n (1 ρ) σ. Hence the almost sure bounds of ˆσ σ and σ σ could be resorted to Theorem Lemma 1. Assume p > 1, E(X i ) = 0, X i L α, and (5) holds with α 4. (i) For 1 < β < α, let β = β, then we have max ˆV k V k k n = O(n 3/ 1/p n (1 1/p)(/β 1) ) β/4 (ii) If 4/3 β < α and the conditions of Theorem 1 (iv) hold, we have where φ(p) = (p) 1 q(1 1/p). ˆV n /v n (1 ρ) σ β/4 = O(n φ(p) ), (11) 6

7 130 As will be shown later, (11) plays the critical role in determining the deviation of ˆσ and σ from σ. It can be shown easily that φ(p) reaches its maximum of q/(q + 1) at p = 1 + (q) 1. By imposing the latter condition, we derive the following resutls regarding asymptotic properties of ˆσ n and σ n. Theorem. Suppose E(X i ) = 0, (9) holds with q (0, 1], and p = 1 + (q) 1. (i) Assume X i L α and (5) holds with α > 4, then we have ˆσ n σ = o a.s. (n q/(q+1) logn). (1) (ii) Assume X i L 4 and (5) holds with α = 4, then we have ˆσ n σ = o a.s. (n q/(q+1) log +ɛ n) (13) for any small ɛ > Corollary 1. Suppose E(X i ) = 0, (9) holds with q (0, 1], and p = 1 + (q) 1. (i) Assume X i L α and (5) holds with α > 4, then we have σ n σ = o a.s. (n q/(q+1) logn). (14) (ii) Assume X i L 4 and (5) holds with α = 4, then we have σ n σ = o a.s. (n q/(q+1) log +ɛ n) (15) for any small ɛ > 0. (iii) Assume X i L α and (5) holds with α 4, then we have Ṽn/v n (1 ρ) σ β/4 = O(n q/(q+1) ) (16) 140 when 4/3 β < α. 4. The comparison of efficiencies Assume that (8) holds with α > 8 and (9) holds with q = 1. Theorem suggests optimal p = 3/ in the sequence a k = ck p. Let S n = n e i, η = k=1 kγ(k), and η e = k=1 ke(e 0e k ). As l, we have lσ e E(S l ) = η e + o(1). 145 Following the proof of Lemma 1 and the arguments in Section 4 of Wu (009b), we have ( Ṽn/v n σe V n /v n σe σe 4 16c /3 + η 56 ) n / c 4/3 7

8 In minimizing the MSE, we choose c = 4 η e /(3σ e) so that By direct calculations we have Ṽn/v n σe 14/3 η/3 35/3 e σe 8/3 n /3. (17) σ σ = n (1 ρ n ), where n = Ṽn/v n σ e + ( ρ ρ n )( ρ n ρ)σ. By Proposition and (17) we have Let ˇσ be the Wu s estimator of σ. n 14/3 η/3 35/3 e σe 8/3 n /3. (18) Theorem 3. Suppose E(X i ) = 0, (9) holds with q (0, 1], and p = 1 + 1/q. (i) Assume X i L α, and (5) holds with α > 40. for 40 β < α. (ii) Assume X i L α, and (5) holds with α > 40. We have σ σ β/0 n β/0 (1 ρ) = o(n 1/3 ) (19) σ σ ˇσ σ ( 1 γ(1) ) /3 (1 ρ) := λ. (0) η To estimate c, we can refer to Algorithm 3 in Wu (009b) except that ˆγ(k) therein should be replaced by an estimate of E(e 0 e k ) = (1 + ρ )γ(k) ρ(γ(k + 1) + γ(k 1)). Therefore, we could substitute ρ and γ(k) by their estimates based on a small portion of the initial data. Since λ < 1 if and only if γ(1)/η > 0, the proposed algorithm here would outperform Wu s algorithm whenever γ(1) 0 and kγ(k)/γ(1) > 1. k 4.1. ARMA model Consider ARMA( p, q) model a(l)x i = b(l)ε i, (1) 160 where a( ) and b( ) are polynomials of orders p and q respectively. Suppose a(z) = p 0 (1 r iz) mi with p0 m i = p and r i < 1. In the sequel, we will discuss the calculation of λ for the case of p > q. No extra difficulty is needed to deal with the case of p q. From Brockwell and Davis (1991), the autocovariance function has the representation of p 0 γ(k) = m i 1 j=0 d ij k j r k i, k 0, () 8

9 MSE Ratio of MSE AR(1) Direct Mean Variance Figure 1: λ for ARMA Model with a(z) = (1 0.8z)( z)(1 0.0z) and b(z) = z. 165 where d ij could be found by solving the system of linear equations () for k = 0, 1,..., p 1. See Brockwell and Davis (1991) or Karansos (1998) for details. Define Ξ i (r) = k=1 ki r k, then Ξ 0 (r) = r/(1 r) and Ξ i (r), i 1 could be derived recursively by the relation Ξ i+1 (r) = rξ i (r), i 0. For example, we have Ξ 1 (r) = r/(1 r) and Ξ (r) = (r + r)/(1 r) 3. For model (1) we have p0 γ(1) (1 ρ) η = mi 1 p0 j=0 d ij r i /(1 ρ) mi 1 j=0 d ij Ξ j+1 (r i ) and ρ = p0 mi 1 j=0 d ij r i. p0 d i0 If all the r i s are distinct, these two equations reduce to p γ(1) (1 ρ) η = d ir i /(1 ρ) p p d and ρ = d ir i ir i /(1 r i ) p d. (3) i with the convention of d i = d i0. Simulation studies are carried out for three different examples of Model (1) and the results are displayed in Figures 1 3. In all these three examples, we adopted σε = 1, and the value of λ is calculated by (3). The black line therein is associated with Wu s estimator and the red line is associated with the proposed estimator. Figures 1 and verify that when λ < 1 the proposed method outperforms Wu s. The main reason of the smaller MSE is due to the reduced bias by introducing the prewhittening. One the other hand, Figure 3 shows that when λ > 1 the prewhittening is not recommended. 9

10 MSE Ratio of MSE AR(1) Direct Mean Variance Figure : λ 0.0 for ARMA Model with a(z) = (1+0.9z)(1+0.8z)(1+0.5z)(1 0.z) and b(z) = 1+0.9z +0.8z +0.8z 3. MSE Ratio of MSE AR(1) Direct Mean Variance Figure 3: λ.9947 for ARMA Model with a(z) = (1 0.8z)( z)(1 0.z) and b(z) = z + 0.8z. 10

11 4.. AR(1) model In an AR(1) model, the autoregressive coefficient equals to ρ = γ(1)/γ(0). The AR(1) model could be written as X i ρx i 1 = ε i, where ε i W N(0, σε). (4) Since e i = ε i and σ e = (1 ρ) σ, we have E(V n ) v n (1 ρ) σ = 0. Hence (11) holds when φ(p) = (p) 1, which implies that p should be chosen as small as possible. Note that we have X i = j=0 ρj ε i j from the model, the condition ε i L α, i Z implies that X i L α and (5) holds. We will now investigate the case of p = 1 for Model (4). Proposition 1. Suppose the stationary time series {X i } are from Model (4). Let p = 1 and c N in the sequence a k = ck. (i) Assume ε 0 L α with α 4, we have V n E(V n ) α/ = O( n) (ii) Assume ε 0 L 4, and let κ = ε 0 4 /σ 4 ε. We have V n E(V n ) lim n n = φ c,κ σ 4 ε, where φ c,κ = (c + 1)(c + 1)(κ 1)/6 + c(c 1)/3. (iii) Assume that ε 0 L α for α > 4. For 4 β < α, we have 190 lim sup n where S n = n X i and R n = n ε i. Ṽn V n β/4 n σ ρ G β/ 1 c (iv) Assume ε 0 L α for α, then for 0 < β < α we have c S i β R i β, lim n ( ρ ρn )( ρ n ρ) β/4 = ( ρ) σ ρ G β/4. n By Proposition 1 and the fact v n n(c + 1)/, we have n α = O( n) for α and lim sup n n n φc,κ c + 1 σ ε + 31/4 σ ρ c(c + 1) c S i 8 R i 8 + (1 ρ)σ ρ σ. Now let C p = 18p 3/ (p 1) 1/. By Burkholder s inequality and stationarity, we have R i 8 C 8 i ε0 8, where C By Theorem 1 of Wu (007), we have S i 8 C 8 i(1 ρ ) 1 ε 0 8. By Hölder s inequality, we have R i 8 i ε 0 8 and S i 8 i(1 ρ ) 1 ε 0 8. Incorporating these bounds we have c S i 8 R i 8 (1 ρ ) 1 ε 0 c(c + 1) 8B c, 11

12 195 where B c = c i min(c 8,i) c(c+1). Note that B c reaches its minimum 1/ at two points, 1 and. Since the latter is not applicable, we have B c B 1 = 1/ for c N. Note that φ c,k /(c+1) is an increasing function of c, it is reasonable to adopt c = 1, i.e. a k = k, when implementing Algorithm, for which we have lim sup n n κ 1σε + 3 1/4 σ ρ (1 ρ ) 1 ε σ ρ (1 ρ) 1 σε. n By the same argument as for (19), when ε 0 L α with α > 40, we have σ σ = O(n 1/ ). 00 This bound is better than O(n 1/3 ) resulted from the sequence a k = ck 3/. See Lee and Phillips (1994) for a similar result in the context of non-recursive estimation. To implement the algorithm in Wu (009b), we adopt a k = ck 3/. c is computed as c = 8 ρ 3(1 ρ ), ε 0 ρ since η = (1 ρ) (1 ρ ), and σ = ε 0. Hence if AR(1) is the true model and we adopt c = p = 1 (1 ρ) for the prewhittening method, we have σ σ ˇσ σ The simulation study is carried out for three different AR(1) models, namely ρ = 0.9, 0.1 and 0.9. As shown by Figures 4-6, our method (red) performs significantly better than Wu s method (black) for AR(1) model Nonlinar models We start with a GARCH model: X t = σ t ε t, where σ t = α 0 + p α ix t i + q j=1 β iσ t j. Since {X i} are uncorrelated and there is no need to estimate the TAVC by batch method. Meanwhile, the squared terms {Xi } is of interest in some studies. See Bollerslev (1986), Ding and Granger (1996), and Karanasos 10 (1999). Let Z t = X t α 0 1 E(ε 0 ) p α i q j=1 β, t Z. j It is well known that {Z t } follows an ARMA model as follows. max(p,q) Z t ((Eε 0)α i + β i )Z t i = µ t p β i µ t j, t Z, (5) where {u t } is the white noise process. Here, α i = 0 for i > p and β j = 0 for j > p. By (5), the recursive estimation of the TAVC of the squared terms reduces to that of the regular ARMA model in Section 4.1. j=1 1

13 MSE Ratio of MSE AR(1) Direct Mean Variance Figure 4: Comparison of the two methods under the model X i 0.9X i 1 = ε i. MSE Ratio of MSE AR(1) Direct Mean Variance Figure 5: Comparison of the two methods under the model X i + 0.1X i 1 = ε i. 13

14 MSE Ratio of MSE AR(1) Direct Mean Variance Figure 6: Comparison of the two methods under the model X i + 0.9X i 1 = ε i. 15 Further, a simulation study is carried out for an ARMA GARCH model as follows. X t = 0.9X t 1 + Z t + 0.Z t 1 (6) Z t = σ t ɛ t σ t = Z t 1 0.Z t + 0.3σ t σ t Figure 7 shows that our method outperforms Wu s method. 5. Conclusions 0 5 This paper studies a prewhitened version of Wu s algorithm for estimating the TAVC in a recursive way. Asymptotic properties of the estimate is derived for a general causal stationary process and simulation studies are carried out for both linear and nonlinear time series models. prefilter, i.e. Here we adopt the simplest the AR(1) model, which achieves significant improvements on the estimation efficiency whenever γ(1) 0 and k kγ(k)/γ(1) > 1. In practice, one may keep two lines of updating algorithms simultaneously in the initial stage and then choose one from them according to the latter condition. The simple form of AR(1) facilitates the construction and implementation of the recursive estimation of TAVC with the memory complexity of O(1). 14

15 MSE AR(1) Direct mseratio[100:l] Index Mean Variance Figure 7: Comparison of the two methods under Model (6) Appendix Proof of Theorem 1 Observe that (i), (iii) and (iv)(a) are direct results from Corollary 3, Theorem (ii), and Theorem (iii) of Wu (009b) by replacing X i therein by e i. Also (ii) could be proved by the same argument as (i) by using the inequality P i k Wi α/ n C α Pi k Wi α/ 30 instead of P i k Wi α/ α/ n C α Pi k Wi α/ α/ as in Wu (009b) since α > 4. Now we prove (iii)(b). Since E( n X i) = k <n (n k )γ(k) and E( n X i)( n X i 1) = k <n (n k )γ(k + 1), we have EWi = E (X j ρx j 1 ) = [ (l i k ) (1 + ρ )γ(k) ργ(k + 1) ], k <l i 15

16 where l i = i t i + 1. For q (0, 1] we have l i (1 ρ) σ EWi = min( k, l i ) [ (1 + ρ )γ(k) ργ(k + 1) ] k Z = min(k, l i ) [ (1 + ρ )γ(k) ργ(k + 1) ργ(k 1) ] Since = k=1 min(k, l i )E [X 0 (e k ρe k+1 )] (7) k=1 E[X 0 (e k ρe k+1 )] = i Z E(P i X 0 )(P i (e k ρe k+1 )) i Z P i X 0 P i (e k ρe k+1 ), 35 we have li (1 ρ) σ EWi l 1 q i = O(l 1 q i ). k q P i X 0 P i (e k ρe k+1 ) k=1 i Z Hence EV n v n (1 ρ) σ = l i (1 ρ) σ EWi O(l 1 q i ) = O(nb 1 q m ) = O(n 1+(1 q)(1 1/p) ) Proof of Lemma1 By Cauchy s inequality, n i= X ix i 1 ( n i= X i )1/ ( n 1 X i )1/ n X i, hence we have ˆρ n 1 for any n N. By Theorem 1 (iii) of Wu (007) and Proposition (iii) we have Ŵ i,n Wi β/4 Ŵi,n W i β/3 Ŵi,n + W i β = (ˆρ n ρ)x j 1 X j (ˆρ n + ρ) j=t β/3 i 4 ˆρ n ρ β/ X j 1 β ) = O (n 1/ l /β i X j 1 β (8) 16

17 Since max k n ˆV k V k n Ŵ i W i 40, we have max ˆV k V k k n = O(n 1/ v n max β/4 i n l/β 1 i ) = O(n 3/ 1/p n (1 1/p)(/β 1) ). The application of (i) with β 4/3 gives max k n ˆV k V k = O(n 3/(p) ), hence (ii) follows from β/4 Theorem 1 in view of v n n 1/p. Proof of Theorem We prove (ii) first and then prove (i). By Lemma 1 and Theorem 1(i), we have max ˆV n EV n = O( d( 3/(p)) d), n d β/4 45 for any 4 > β >. By choosing 4 > β > 4/(1 + ɛ) we have max n d ˆV n EV n β/4 β/4 <, (9) βd( 3/(p))/4 d (+ɛ)β/4 d=1 which by the Borel-Cantelli lemma implies that ˆV n EV n = o a.s. (n ( 3/(p)) log +ɛ n) (30) Since v n n 1/p and p = 1 + 1/q, by (30) and Theorem 1(iv) we have ˆV n /v n (1 ρ) σ = o a.s. (n q/(q+1) (logn) +ɛ ). (31) Now we have ( ) ( ) 1 ρ 1 ρ ˆσ n σ = ˆσ n σ + σ σ 1 ˆρ n 1 ˆρ n = ˆV n /v n (1 ρ) σ (1 ˆρ n ) + ( ρ ˆρ n)(ˆρ n ρ) (1 ˆρ n ) σ = o a.s. (n q/(q+1) log +ɛ n) (3) 50 in view of (31) and Proposition 3. Now (i) could be proved similarly by choosing β (4, α) and setting ɛ = 1 in (9). Proof of Corollary 1 We will prove the corollary based on the expression (4). By the same argument for (30) we have Ṽn EV n = o a.s. (n ( 3/(p)) logn) (33) 17

18 under (i) and Ṽ n EV n = o a.s. (n ( 3/(p)) log +ɛ n) (34) 55 under (ii). Let U n = n l i i X j. By the same argument as in Corollary 1 of Wu (009b) we have for α (, 4] and integers r, m N that U r m U r (m 1) α = O( r(5/ /p) m /p ). Then by Proposition 1(i) of Wu (007) we have max d d r U 1 i d i U r m U r (m 1) α α α r=0 m=1 d = r(5/ /p) (d r)(1/α+ /p) r=0 = O( d(5/ /p) ) 1/α 60 Since max1 i d U i α α αd(5/ /p) n α/ < By Borel-Cantelli Lemma we have U n = o a.s. (n 5/ /p log 1/ n). Since Theorem 1 of Wu (007) implies n X i α = O( n), by the same argument as in Proposition 3 we have X n = o a.s. (n 1/ log n). Thus X n Un = o a.s. (n /p log 3 n) and X nd n = o a.s. (n /p log 4 n) in view of d n n 3 /p. Since ρ n 1, we have Ṽ n Ṽ n = o a.s. (n /p log 4 n). (35) 65 By the same argument for (3), (i) and (ii) follows in view of (33), (34), and (35). By Remark 4 of Wu (009b), we have Ṽn Ṽ n α/ = O(n /p ). Also by the same argument as Theorem (iii), we have Ṽ n /v n (1 ρ) σ β/4 = O(n q/(q+1) ) for 4/3 β < α. Hence (iii) follows. Proof of Theorem 3 By Hölder s inequality, we have n (1 ρ n ) n β/0 (1 ρ) n β/0 (1 ρ n ) n (1 ρ) β/0 = n ( ρ ρ n )( ρ n ρ) (1 ρ n ) (1 ρ) β/0 4 n β/4 ρ n ρ (1 ρ) (1 ρ n ) β/16 = ρ n ρ (1 ρ n ) O(n 1/3 ) (36) β/16 18

19 By Lemma 3 of Kan and Wang (010), we have ρ n cos(π/(n + 1)). Hence (1 ρ n ) = O(n 4 ). By Schwartz s inequality, we have ρ n 1. Choose 0 < c < 1 ρ and β < β 0 < α, we have ρ n ρ (1 ρ n ) ρ n ρ (1 ρ n ) 1 [ ρ n ρ>c] + ρ n ρ β/16 (1 ρ n ) 1 [ ρ n ρ c] β/16 = P ( ρ n ρ > c) 16/β O(n 4 ) + ρ n ρ β/16 O(1) = [ n( ρ n ρ) β0/ β 0/ n β0/4 ] 16/β O(n 4 ) + O(n 1/ ) = O(n 4(1 β0/β) ) + O(n 1/ ) β/16 = o(1) (37) 70 Then (i) follows from (36) and (37). By direct calculation, we have η e = (1 ρ) η γ(1). Then a direct application of (18), (19) with β = 40 here and (3) of Wu (009b) gives σ σ ˇσ σ = ηe /3 σe 8/3 (1 ρ) 4 η /3 σ 8/3 ( η e (1 ρ) η ) /3 = λ (38) 75 Proof of Proposition 1 Let m = n/c in the proof. For (i), let H k = kc i=(k 1)c+1 (W i EW i ), then V mc EV mc = m k=1 H k and H k L α/. By Burkholder s inequality, we have V mc EV mc α/ C p m H k α/, where C p = 18(p) 3/ (p 1) 1/. Then (i) follows in view of V n EV n (V mc EV mc ) α/ = O(1). For (ii) we have Ri E R i = = = k=1 (n + 1 i)(ε i Eε i ) + (n + 1 j)ε i ε j 1 i<j n (n + 1 i) ε 0 Eε (n + 1 j) σε 4 n(n + 1)(n + 1) 6 = nφ n,κ σ 4 ε. 1 i<j n ε 0 Eε 0 + n (n 1) σε 4 3 Since W i d = Rli, by independence we have V n EV n mcφ c,κ σ 4 ε nφ c,κ σ 4 ε, 19

20 To prove (iii), we adopt the notation in (4) and have Ṽn V n = ( ρ n ρ) X j 1 e j ( ρ n ρ) X j 1 (39) 80 We deal with the two parts in the RHS of (39) separately. ( ρ n ρ) X j 1 e j ρ n ρ β/ β/4 n ρ n ρ β/ Hence we have lim sup n n 1/ ( ρ n ρ) X j 1 e j X j 1 e j β/ X j 1 β n 1/ n( ρ n ρ) β/ (m + 1) 1 σ ρ G β/ c β/4 e j β c S i β R i β c S i β R i β. (40) Further we have ( ρ n ρ) n X j 1 β/4 ( ρn ρ) ( β/α) 1 β/ β/α ρ n ρ β/α β/ X j 1 X j 1 β/α n 1/ δ n( ρ n ρ) β/α β/ (m + 1) α α/ c S i α, where δ = 1 (1 β/α) > 0. Hence we have ( ρ n ρ) n X j 1 = o( n). (41) β/4 85 Then (iii) follows from (39), (40), and (41). For (iv), we have ( ρ ρ n )( ρ n ρ) = ( ρ)( ρ n ρ) ( ρ n ρ) Since ( ρ n ρ) β/4 = O(n 1 ) and lim n n ρn ρ β/4 = σ ρ G β/4, (iv) follows. 0

21 Proposition. (i) Assume E(X i ) = 0, X i L α and (5) holds with α >. For 0 < β < α, the following sequences are all uniformly integrable: {[n 1/ (ˆγ n (k) γ(k))] β/, n 1} {[n 1/ (ˆρ n (k) ρ(k))] β/, n 1} {[n 1/ ( γ n (k) γ(k))] β/, n 1} {[n 1/ ( ρ n (k) ρ(k))] β/, n 1} 90 (ii) Assume E(X i ) = 0, X i L 4 and (5) holds with α = 4, we have n(ˆρn ρ) n( ρn ρ) d d N(0, σ ρ), N(0, σ ρ), where σ ρ = i=0 P 0X i ẽ i /γ(0) and ẽ i = X i 1 ρx i. (iii) Assume E(X i ) = 0, X i L α and (5) holds with α 4. Then for 0 < β < α n(ˆρ n ρ) β/ σ ρ G β/, n( ρ n ρ) β/ σ ρ G β/, where σ ρ is as defined in (ii) and G stands for a standard normal distribution. Proof: First of all, we have P 0 X i X i+k α/ X i X i+1 X ix i+k α/ X i α x i+k x i+k α + X i+k α x i X i α = O(δ α (i)) (4) 95 Since X i X i+k γ(k) = l Z P lx i X i+k, we have by Burkholder s inequality that n (X i X i+k γ(k)) = P l X i X i+k α/ l Z α/ C α P l X i X i+k l Z α/ = ( n ) 1/ O(δ α (i l)) l Z ( n ) 1/ = O(δ α (i l)) l Z = O( n) (43) 1/ 1

22 Hence n 1/ (ˆγ n (k) γ(k)) α/ = n 1/ (X i X i+k γ(k)) + n 1/ X i X i+k α/ i=n k+1 α/ = O(1) (44) Hence {[n 1/ (ˆγ n (k) γ(k))] β/, n 1} is uniformly integrable. Note that n γ n (k) nˆγ n (k) α/ = X k n X i + X n n X i (n + k) X n i=n k+1 X k n X i α/ + (n + k) X n α/ k X n α X k α + (n + k) X n α α/ = O(1) (45) in view of X n α = O(n 1/ ) by Theorem 1 of Wu (007). Then (44) and (45) implies n 1/ γ n (k) γ(k) α/ = O(1), i.e. {[n 1/ ( γ n (k) γ(k))] β/, n 1} is uniformly integrable. Define the (m+1) dependent sequence X i,m = E(X i F i i m ), i Z, where F i i m = (ε i m, ε i m+1,..., ε i ). Also define γ m (k) = EX 0,m X k,m, which could be estimated by ˆγ n,m (k) = n 1 n i=k+1 X i,mx i+k,m. By the same argument as in (4), (43), (44), and (45), we could prove that {[n 1/ (ˆγ n,m (k) γ m (k))] β/, n 1} and {[n 1/ ( γ n,m (k) γ m (k))] β/, n 1} are both uniformly integral for any m N in view of X i,m α X i α < and i=0 X i,m X i,m α = m i=0 X i,m X i,m α (m + 1) X i,m α <. Then by the same argument as in Theorem 1 of Lee (1996), we could prove that {[n 1/ (ˆρ n (k) ρ(k))] β/, n 1} and {[n 1/ ( ρ n (k) ρ(k))] β/, n 1} are both uniformly integrable. For (ii), (4) follows by applying the delta method to Theorem 1 of Wu (009a) and (4) follows similarly in view of (45). (iii) follows from (i) and (ii). 310 Proposition 3. If E(X i ) = 0, X i L 4 and (5) holds with α = 4, we have ˆγ n (k) γ(k) = o a.s. (n 1/ log n), ˆρ n (k) ρ(k) = o a.s. (n 1/ log n), γ n (k) γ(k) = o a.s. (n 1/ log n), (46) ρ n (k) ρ(k) = o a.s. (n 1/ log n). (47)

23 Proof: By Theorem 1 of Wu (009a), we have n 1/ T n,k N(0, σ k) where T n,k = n (X ix i k γ(k)) and σ k = i=0 P 0X i X i k. In the following, we will write T n,k as T n. Let 1 < q <, (43) implies that {[n 1/ T n,k ] q, n N} is uniformly integral. Hence we have T n q = O( n) (48) By Proposition 1 (i) of Wu (007) and (48), we have max T n n d q d [ d r T r q 1/q q] r=0 = d/ Hence, max n d T n q q qd /d <, 315 which implies T n = o(n 1/ log n) almost surely. Then we have ( ˆγ n (k) γ(k) = 1 n T n ) k X i X i k = o a.s. (n 1/ log n), and hence ˆρ n (k) ρ(k) = ˆγ n(k) γ(k) ˆγ n (0) = o a.s. (n 1/ log n) By similar arguments, we could prove (46) and (47) in view of (45). γ(k)(ˆγ n(0) γ(0)) γ(0)ˆγ n (0) References 30 Alexopoulos, C. and Goldsman, D. (004). To batch or not to batch? ACM Transactions on Modeling and Computer Simulation Andrews, D. W. K. and Monahan, J. C. (199). An improved heteroskedasticity and autocorrelation consistent covariance matrix estimator. Econometrica

24 Bühlmann, P. (00). Bootstraps for time series. Statistical Science Bollerslev, T. (1986). Generalized autoregressive conditional heteroskedasticity. Journal of Econometrics Brockwell, P. J., and Davis, R. A. (1991). Time Series: Theory and Methods, nd ed. Springer, New York. Ding, Z., Granger, C.W.J. (1996). Modeling volatility persistence of speculative returns: a new approach. Journal of Econometrics Engle, R.F. (198). Autoregressive conditional heteroscedasticity with estimates of the variance of U.K. inflation. Econometrica Haldrup, N. and Jansson, M. (005). Improving size and power in unit root testing. Tech. rep., Department of Economics, University of Aarhus. Jones, G. L., Haran, M., Caffo, B. S. and Neath, R. (006). Fixed-width output analysis for Markov chain Monte Carlo. Journal of the American Statistical Association Kan, R. and Wang, X. (010). On the Distribution of the Sample Autocorrelation Coefficients. Journal of Econometrics Karanasos, M. (1998). A New Method for Obtaining the Autocovariance of an AMRA Model: An Exact Form Solution. Econometric Theory Karanasos, M. (1999). The second moment and the autocovariance function of the squared errors of the GARCH model, Journal of Econometrics Lahiri, S. N. (003). Resampling methods for dependent data. Springer, New York. S. Lee. (1996). Sequential estimation for the autocorrelations for linear processes. Annals of Statistics Lee, C. C. and Phillips, P. C. B. (1994). An ARMA prewhitened long-run variance estimator. Manuscript, Yale University Phillips, P. C. B. and Sul, D. (003). Dynamic panel estimation and homogeneity testing under cross section dependence, The Econometrics Journal Phillips, P. C. B. and Xiao, Z. (1998). A primer on unit root testing. Journal of Economic Surveys Politis, D. N., Romano, J. P. and Wolf, M. (1999). Subsampling. Springer, New York. Press, H. and Tukey, J.W. (1956). Power Spectral Methods of Analysis and Their Application to Problems in Airplane Dynamic. Flight Test Manual, NATO, Advisory Group for Aeronautical Research and Development Vol. IV-C Reprinted as Bell System Monograph No

25 Song, W.M. and Schmeiser, B.W. (1995). Optimal mean-squared-error batch sizes. Management Science Stock, J. H. (1994). Unit roots, structural breaks and trends. Handbook of econometrics, Vol. IV. North-Holland, Amsterdam Sul, D., Phillips, P. C. B. and Choi C. Y. (005). Prewhitening Bias in HAC Estimation. Oxford Bulletin Of Economics And Statistics Wu, W. B. (005). Nonlinear system theory: Another look at dependence. Proceedings of the National Academy of Sciences USA Wu, W. B. (007). Strong invariance principles for dependent random variables. Annals of Probability Wu, W. B. (009a). An asymptotic theory for sample covariances of Bernoulli shifts. Stochastic Processes and their Applications Wu, W. B. (009b). Recursive Estimation of Time-Average Variance Constants. Annals of Applied Probability