STAT 720 TIME SERIES ANALYSIS. Lecture Notes. Dewei Wang. Department of Statistics. University of South Carolina

Transcription

1 STAT 720 TIME SERIES ANALYSIS Spring 2015 Lecture Notes Dewei Wang Department of Statistics University of South Carolina 1

2 Contents 1 Introduction Some examples Time Series Statistical Models Stationary Processes Measure of Dependence Examples Identify Non-Stationary Time Series Linear Processes AR(1) and AR(p) Processes AR(1) process AR(p) process MA(1) and MA(q) Processes ARMA(1,1) Processes Properties of X n, γ X (h) and ρ X (h) For X n For γ X (h) and ρ X (h) Autoregressive Moving Average (ARMA) Processes Definition Causality and Invertibility Computing the ACVF of an ARMA(p, q) Process First Method Second Method Third Method The Spectral Representation of a Stationary Process Complex-Valued Stationary Time Series The Spectral Distribution of a Linear Combination of Sinusoids Spectral Densities and ARMA Processes Causality, Invertibility and the Spectral Density of ARMA(p, q) Prediction of Stationary Processes Predict X n+h by X n Predict X n+h by {X n,..., X 1, 1} General Case The Partial Autocorrelation Fucntion (PACF) Recursive Methods for Computing Best Linear Predictors Recursive Prediction Using the Durbin-Levinson Algorithm Recursive Prediction Using the Innovations Algorithm

3 5.5.3 Recursive Calculation of the h-step Predictors, h Recursive Prediction of an ARMA(p, q) Process h-step prediction of an ARMA(p, q) process Miscellanea Estimation for ARMA Models The Yule-Walker Equations and Parameter Estimation for Autoregressive Processes Preliminary Estimation for Autoregressive Processes Using the Durbin-Levinson Algorithm Preliminary Estimation for Moving Average Processes Using the Innovations Algorithm Preliminary Estimation for ARMA(p, q) Processes Recursive Calculation of the Likelihood of an Arbitrary Zero-Mean Gaussian Process Maximum Likelihood Estimation for ARMA Processes Asymptotic properties of the MLE Diagnostic checking Nonstationary process Introduction of ARIMA process Over-differencing? Seasonal ARIMA Models Seasonal ARMA models Seasonal ARIMA Models Regression with stationary errors Multivariate Time Series Second order properties of multivariate time series Multivariate ARMA processes State-Space Models State-Space Models

4 1 Introduction 1.1 Some examples Question: What is a time series? Answer: It is a random sequence {X t } recorded in a time ordered fashion. Question: What are its applications? Answer: Everywhere when data are observed in a time ordered fashion. For example: Economics: daily stock market quotations or monthly unemployment rates. Social sciences: population series, such as birthrates or school enrollments. Epidemiology: the number of influenza cases observed over some time period. Medicine: blood pressure measurements traced over time for evaluating drugs. Global warming? Example 1.1. (Johnson & Johnson Quarterly Earnings) Figure 1.1 shows quarterly earnings per share for the U.S. company Johnson & Johnson. 84 quarters (21 years) measured from the 1st quarter of 1960 to the last quarter of require(astsa) par(mar=c(4,4,2,.5)) plot(jj, type="o", ylab="quarterly Earnings per Share",col="blue") Quarterly Earnings per Share Time Figure 1.1: Johnson & Johnson quarterly earnings per share, 84 quarters, 1960-I to 1980-IV 1

5 Example 1.2. (Global Warming) Figure 1.2 shows the global mean land-ocean temperature index from 1880 to 2009 with the base period require(astsa) par(mar=c(4,4,2,.5)) plot(gtemp, type="o", ylab="global Temperature Deviations",col="blue") Global Temperature Deviations Time Figure 1.2: Yearly average global temperature deviations ( ) in degrees centigrade. 2

6 Example 1.3. (Speech Data) Figure 1.3 shows a small.1 second (1000 point) sample of recorded speech for the phrase aaa hhh. require(astsa) par(mar=c(4,4,2,.5)) plot(speech,col="blue") speech Time Figure 1.3: Speech recording of the syllable aaa hhh sampled at 10,000 points per second with n = 1020 points Computer recognition of speech: use spectral analysis to produce a signature of this phrase and then compare it with signatures of various library syllables to look for a match. 3

7 1.2 Time Series Statistical Models A time series model specifies the joint distribution of the sequence {X t } of random variables; e.g., P (X 1 x 1,..., X t x t ) for all t and x 1,..., x t. where {X 1, X 2,... } is a stochastic process, and {x 1, x 2,... } is a single realization. Through this course, we will mostly restrict our attention to the first- and second-order properties only: E(X t ), Cov(X t1, X t2 ) Typically, a time series model can be described as X t = m t + s t + Y t, (1.1) where m t : trend component; s t : seasonal component; Y t : Zero-mean error. The following are some zero-mean models: Example 1.4. (iid noise) The simplest time series model is the one with no trend or seasonal component, and the observations X t s are simply independent and identically distribution random variables with zero mean. Such a sequence of random variable {X t } is referred to as iid noise. Mathematically, for any t and x 1,..., x t, P (X 1 x 1,..., X t x t ) = t P (X t x t ) = t F (x t ), where F ( ) is the cdf of each X t. Further E(X t ) = 0 for all t. We denote such sequence as X t IID(0, σ 2 ). IID noise is not interesting for forecasting since X t X 1,..., X t 1 = X t. Example 1.5. (A binary {discrete} process, see Figure 1.4) As an example of iid noise, a binary process {X t }is a sequence of iid random variables X t s with P (X t = 1) = 0.5, P (X t = 1) = 0.5. Example 1.6. (A continues process: Gaussian noise, see Figure 1.4) {X t } is a sequence of iid normal random variables with zero mean and σ 2 variance; i.e., X t N(0, σ 2 ) iid 4

8 Example 1.7. (Random walk) The random walt {S t, t = 0, 1, 2,... } (starting at zero, S 0 = 0) is obtained by cumulatively summing (or integrating ) random variables; i.e., S 0 = 0 and S t = X X t, for t = 1, 2,..., where {X t } is iid noise (see Figure 1.4) with zero mean and σ 2 variance. Note that by differencing, we can recover X t ; i.e., S t = S t S t 1 = X t. Further, we have ( ) E(S t ) = E X t = t t E(X t ) = t ( ) 0 = 0; Var(S t ) = Var X t = t t Var(X t ) = tσ 2. set.seed(100); par(mfrow=c(2,2)); par(mar=c(4,4,2,.5)) t=seq(1,60,by=1); Xt1=rbinom(length(t),1,.5)*2-1 plot(t,xt1,type="o",col="blue",xlab="t",ylab=expression(x[t])) t=seq(1,60,by=1); Xt2=rnorm(length(t),0,1) plot(t,xt2,type="o",col="blue",xlab="t",ylab=expression(x[t])) plot(c(0,t),c(0,cumsum(xt1)),type="o",col="blue",xlab="t",ylab=expression(s[t])) plot(c(0,t),c(0,cumsum(xt2)),type="o",col="blue",xlab="t",ylab=expression(s[t])) X t X t t t S t S t t t Figure 1.4: Top: One realization of a binary process (left) and a Gaussian noise (right). Bottom: the corresponding random walk 5

9 Example 1.8. (white noise) We say {X t } is a white noise; i.e., X t WN(0, σ 2 ), if {X t } is uncorrelated, i.e., Cov(X t1, X t2 ) = 0 for any t 1 and t 2, with EX t = 0, VarX t = σ 2. Note that every IID(0, σ 2 ) sequence is WN(0, σ 2 ) but not conversely. Example 1.9. (An example of white noise but not IID noise) Define X t = Z t when t is odd, X t = 3Zt 1 2 2/ 3 when t is even, where {Z t, t = 1, 3,... } is an iid sequence from distribution with pmt f Z ( 1) = 1/3, f Z (0) = 1/3, f Z (1) = 1/3. It can be seen that E(X t ) = 0, Var(X t ) = 2/3 for all t, Cov(X t1, X t2 ) = 0 for all t 1 and t 2, since Cov(Z t, 3Zt 1 2 2/ 3) = 3Cov(Z t, Zt 2 ) = 0. However, {X t } is not an iid sequence. Since when Z 2k is determined fully by Z 2k 1. Z 2k 1 = 0 Z 2k = 2/ 3, Z 2k 1 = ±1 Z 2k = 3 2/ 3. A realization of this white noise can be seen from Figure 1.5. set.seed(100); par(mfrow=c(1,2)); par(mar=c(4,4,2,.5)) t=seq(1,100,by=1); res=c(-1,0,1) Zt=sample(res,length(t)/2,replace=TRUE); Xt=c() for(i in 1:length(Zt)){ Xt=c(Xt,c(Zt[i], sqrt(3)*zt[i]^2-2/sqrt(3)))} plot(t,xt,type="o",col="blue",xlab="t",ylab=expression(x[t])) plot(c(0,t),c(0,cumsum(xt)),type="o",col="blue",xlab="t",ylab=expression(s[t])) X t S t t t Figure 1.5: One realization of Example 1.9 If the stochastic behavior of all time series could be explained in terms of the white noise model, classical statistical methods would suffice. Two ways of introducing serial correlation and more smoothness into time series models are given in Examples 1.10 and

10 Example (Moving Averages Smoother) This is an essentially nonparametric method for trend estimation. It takes averages of observations around t; i.e., it smooths the series. For example, let X t = 1 3 (W t 1 + W t + W t+1 ), (1.2) which is a three-point moving average of the white noise series W t. See Figure 1.9 for a realization. Inspecting the series shows a smoother version of the first series, reflecting the fact that the slower oscillations are more apparent and some of the faster oscillations are taken out. set.seed(100); w = rnorm(500,0,1) # 500 N(0,1) variates v = filter(w, sides=2, rep(1/3,3)) # moving average par(mfrow=c(2,1)); par(mar=c(4,4,2,.5)) plot.ts(w, main="white noise",col="blue") plot.ts(v, main="moving average",col="blue") white noise w Time moving average v Time Figure 1.6: Gaussian white noise series (top) and three-point moving average of the Gaussian white noise series (bottom). 7

11 Example AR(1) model (Autoregression of order 1): Let X t = 0.6X t 1 + W t (1.3) where W t is a white noise series. It represents a regression or prediction of the current value X t of a time series as a function of the past two values of the series. set.seed(100); par(mar=c(4,4,2,.5)) w = rnorm(550,0,1) # 50 extra to avoid startup problems x = filter(w, filter=c(.6), method="recursive")[-(1:50)] plot.ts(x, main="autoregression",col="blue",ylab=expression(x[t])) autoregression X t Time Figure 1.7: A realization of autoregression model (1.3) 8

12 Example (Random Walk with Drift) Let X t = δ + X t 1 + W t (1.4) for t = 1, 2,... with X 0 = 0, where W t is WN(0, σ 2 ). The constant δ is called the drift, and when δ = 0, we have X t being simply a random walk (see Example 1.7, and see Figure 1.8 for a realization). X t can also be rewritten as t X t = δt + W j. set.seed(150); w = rnorm(200,0,1); x = cumsum(w); wd = w +.2; xd = cumsum(wd); par(mar=c(4,4,2,.5)) plot.ts(xd, ylim=c(-5,45), main="random walk",col="blue") lines(x); lines(.2*(1:200), lty="dashed",col="blue") j=1 random walk X t Time Figure 1.8: Random walk, σ = 1, with drift δ = 0.2 (upper jagged line), without drift, δ = 0 (lower jagged line), and a straight line with slope.2 (dashed line). 9

13 Example (Signal in Noise) Consider the model X t = 2 cos(2πt/ π) + W t (1.5) for t = 1, 2,..., where the first term is regarded as the signal, and W t WN(0, σ 2 ). Many realistic models for generating time series assume an underlying signal with some consistent periodic variation, contaminated by adding a random noise. Note that, for any sinusoidal waveform, A cos(2πωt + φ) (1.6) where A is the amplitude, ω is the frequency of oscillation, and φ is a phase shift. set.seed(100); cs = 2*cos(2*pi*1:500/50 +.6*pi); w = rnorm(500,0,1) par(mfrow=c(3,1), mar=c(3,2,2,1), cex.main=1.5) plot.ts(cs, main=expression(2*cos(2*pi*t/50+.6*pi)),col="blue") plot.ts(cs+w, main=expression(2*cos(2*pi*t/50+.6*pi) + N(0,1)),col="blue") plot.ts(cs+5*w, main=expression(2*cos(2*pi*t/50+.6*pi) + N(0,25)),col="blue") 2cos(2πt π) cos(2πt π) + N(0, 1) cos(2πt π) + N(0, 25) Figure 1.9: Cosine wave with period 50 points (top panel) compared with the cosine wave contaminated with additive white Gaussian noise, σ = 1 (middle panel) and σ = 5 (bottom panel). 10

14 2 Stationary Processes 2.1 Measure of Dependence Denote the mean function of {X t } as µ X (t) = E(X t ), provided it exists. And the autocovariance function of {X t } is γ X (s, t) = Cov(X s, X t ) = E[{X s µ X (s)}{x t µ X (t)}] Preliminary results of covariance and correlation: for any random variables X, Y and Z, Cov(X, Y ) = E(XY ) E(X)E(Y ) and Corr(X, Y ) = ρ XY = Cov(X, Y ) Var(X)Var(Y ) ρ XY 1 for any X and Y 2. Cov(X, X) = Var(X) 3. Cov(X, Y ) = Cov(Y, X) 4. Cov(aX, Y ) = acov(x, Y ) 5. Cov(a + X, Y ) = Cov(X, Y ) 6. If X and Y are independent, Cov(X, Y ) = 0 7. Cov(X, Y ) = 0 does not imply X and Y are independent 8. Cov(X + Y, Z) = Cov(X, Z) + Cov(Y, Z) 9. Cov( n i=1 a ix i, m j=1 b jy j ) = n i=1 m j=1 a ib j Cov(X i, Y j ) Verify 1 9 as a HW problem. The time series {X t } is (weakly) stationary if 1. µ X (t) is independent of t; 2. γ X (t + h, h) is independent of t for each h. We say {X t } is strictly (or strongly) stationary if (X t1,..., X tk ) and (X t1 +h,..., X tk +h) have the same joint distributions for all k = 1, 2,..., h = 0, ±1, ±2,..., and time points t 1,..., t k. This is a very strong condition. 11

15 Theorem 2.1. Basic properties of a strictly stationary time series {X t }: 1. X t s are from the same distribution. 2. (X t, X t+h ) = d (X 1, X 1+h ) for all integers t and h. 3. {X t } is weakly stationary if E(Xt 2 ) < for all t. 4. Weak stationary does not imply strict stationary. 5. An iid sequence is strictly stationary. Proof. The proof is quite straightforward and thus left as a HW problem. Example 2.1. (q dependent strictly stationary time series:) One of the simplest ways to construct a time series {X t } that is strictly stationary is to filter an iid sequence. Let {Z t } IID(0, σ 2 ), define X t = g(z t, Z t 1,..., Z t q ) for some real-valued function g. Then {X t } is strictly stationary and also q dependent; i.e., X s and X t are independent whenever t s > q. A process, {X t } is said to be a Gaussian process if the n dimensional vector X = (X t1,..., X tn ), for every collection of time points t 1,..., t n, and every positive integer n, have a multivariate normal distribution. Lemma 2.1. For Gaussian processes, weakly stationary is equivalent to strictly stationary. Proof. It suffices to show that every weakly stationary Gaussian process {X t } is strictly stationary. Suppose it is not, then there must exists (t 1, t 2 ) T and (t 1 + h, t 2 + h) T such that (X t1, X t2 ) T and (X t1 +h, X t2 +h) T have different distributions, which contradicts the assumption of weakly stationary. In this following, unless indicated specifically, stationary always refers to weakly stationary. Note, when {X t } is stationary, r X (t + h, h) can be written as γ X (h) for simplicity since γ X (t + h, h) does not depend on t for any given h. Let {X t } be a stationary time series. Its mean is µ X = µ X (t). Its autocovariance function (ACVF) of {X t } at lag h is γ X (h) = Cov(X t+h, X t ). Its autocorrelation function (ACF) of {X t } at lag h is ρ X (h) = γ X(h) γ X (0) = Corr(X t+h, X t ) 12

16 Theorem 2.2. Basic properties of γ X ( ): 1. γ X (0) 0; 2. γ X (h) γ(0) for all h; 3. γ X (h) = γ X ( h) for all h; 4. γ X is nonnegative definite; i.e., a real valued function K defined on the integers is nonnegative definite if and only if n a i K(i j)a j 0 i,j=1 for all positive integers n and real vectors a = (a 1,..., a n ) T R n. Proof. The first one is trivial since γ X (0) = Cov(X t, X t ) = Var(X t ) 0 for all t. The second is based on the Cauchy-Schwarz inequality: γ X (h) = Cov(X t+h, X t ) Var(X t+h ) Var(X t ) = γ X (0). The third one is established by observing that γ X (h) = Cov(X t+h, X t ) = Cov(X t, X t+h ) = γ X ( h). The last statement can be verified by n 0 Var(a T X n ) = a T Γ n a = a i γ X (i j)a j i,j=1 where X n = (X n,..., X 1 ) T and Γ n = Var(X n ) = = Cov(X n, X n ) Cov(X n, X n 1 ) Cov(X n, X 2 ) Cov(X n, X 1 ) Cov(X n 1, X n ) Cov(X n 1, X n 1 ) Cov(X n 1, X 2 ) Cov(X n 1, X 1 ). Cov(X 2, X n ) Cov(X 2, X n 1 ) Cov(X 2, X 2 ) Cov(X 2, X 1 ) Cov(X 1, X n ) Cov(X 1, X n 1 ) Cov(X 1, X 2 ) Cov(X 1, X 1 ) γ X (0) γ X (1) γ X (n 2) γ X (n 1) γ X (1) γ X (0) γ X (n 3) γ X (n 2). γ X (n 2) γ X (n 3) γ X (0) γ X (1) γ X (n 1) γ X (n 2) γ X (1) γ X (0) Remark 2.1. An autocorrelation function ρ( ) has all the properties of an autocovariance function and satisfies the additional condition ρ(0) = 1. 13

17 Theorem 2.3. A real-valued function defined on the integers is the autocovariance function of a stationary time series if and only if it is even and non-negative definite. Proof. We only need prove that for any even and non-negative definite K( ), we can find a stationary process {X t } such that γ X (h) = K(h) for any integer h. It is quite trivial to choose {X t } to be a Gaussian process such that Cov(X i, X j ) = K(i j) for any i and j Examples Example 2.2. Consider {X t = A cos(θt) + B sin(θt)} where A and B are two uncorrelated random variables with zero means and unit variances with θ [ π, π]. Then µ X (t) = 0 γ X (t + h, t) = E(X t+h X t ) = E[{A cos(θt + θh) + B sin(θt + θh)}{a cos(θt) + B sin(θt)}] = cos(θt + θh) cos(θt) + sin(θt + θh) sin(θt) = cos(θt + θh θt) = cos(θh) which is free of h. Thus {X t } is a stationary process. Further ρ X (h) = cos(θh) 14

18 Example 2.3. For white noise {W t } WN(0, σ 2 ), we have { { σ 2 if h = 0; µ W = 0, γ W (h) =, ρ W (h) = 0 otherwise, 1 if h = 0; 0 otherwise, rho=function(h,theta){i(h==0)*1} h=seq(-5,5,1); s=1:length(h); y=rho(h,.6) plot(h,y,xlab="h",ylab=expression(rho[x](h))) segments(h[s],y[s],h[s],0,col="blue") ρ X (h) h 15

19 Example 2.4. (Mean Function of a three-point Moving Average Smoother). See Example 1.10, we have X t = 3 1 (W t 1 + W t + W t+1 ), where {W t } WN(0, σ 2 ). Then µ X (t) =E(X t ) = 1 3 [E(W t 1) + E(W t ) + E(W t+1 )] = 0, γ X (t + h, t) = 3 9 σ2 I(h = 0) σ2 I( h = 1) σ2 I( h = 2) does not depend on t for any h. Thus, {X t } is stationary. Further ρ X (h) = I(h = 0) I( h = 1) + 1 I( h = 2). 3 rho=function(h,theta){i(h==0)+2/3*i(abs(h)==1)+1/3*i(abs(h)==2)}; h=seq(-5,5,1); s=1:length(h); y=rho(h,.6); plot(h,y,xlab="h",ylab=expression(rho[x](h))); segments(h[s],y[s],h[s],0,col="blue") ρ X (h) h 16

20 Example 2.5. MA(1) process (First-order moving average): X t = W t + θw t 1, t = 0, ±1, ±2,..., where {W t } WN(0, σ 2 ) and θ is a constant. Then µ X (t) =0 γ X (h) =σ 2 (1 + θ 2 )I(h = 0) + θσ 2 I( h = 1) ρ X (h) =I(h = 0) + θ I( h = 1). 1 + θ2 rho=function(h,theta){i(h==0)+theta/(1+theta^2)*i(abs(h)==1)} h=seq(-5,5,1); s=1:length(h); y=rho(h,.6) plot(h,y,xlab="h",ylab=expression(rho[x](h))); segments(h[s],y[s],h[s],0,col="blue") ρ X (h) h 17

21 Example 2.6. AR(1) model (Autoregression of order 1). Consider the following model: X t = φx t 1 + W t, t = 0, ±1, ±2,..., where {W t } WN(0, σ 2 ) and W t is uncorrelated with X s for s < t. Assume that {X t } is stationary and 0 < φ < 1, we have µ X = φµ X µ X = 0 Further for h > 0 γ X (h) =E(X t X t h ) = E(φX t 1 X t h + W t X t h ) =φe(x t 1 X t h ) + 0 = φcov(x t 1 X t h ) =φγ X (h 1) = = φ h γ X (0). And γ X (0) = Cov(φX t 1 + W t, φx t 1 + W t ) = φ 2 γ X (0) + σ 2 γ X (0) = Further, we have γ X (h) = γ X ( h), and σ2 1 φ 2. ρ X (h) = φ h. rho=function(h,phi){phi^(abs(h))} h=seq(-5,5,1); s=1:length(h); y=rho(h,.6) plot(h,y,xlab="h",ylab=expression(rho[x](h))); segments(h[s],y[s],h[s],0,col="blue") ρ X (h) h 18

22 Example 2.7. (Mean Function of a Random Walk with Drift). See Example 1.12, we have t X t = δt + W j, t = 1, 2,..., j=1 where {W t } WN(0, σ 2 )Then µ X (t) = E(X t ) = δt. Obviously, when δ is not zero, {X t } is not stationary, since its mean is not a constant. Further, if δ = 0, t+h γ X (t + h, t) =Cov W j, j=1 = min{t + h, t}σ 2 is, again, not free of t. Thus {X t } is not stationary for any δ. Example 2.8. The MA(q) Process: {X t } is a moving-average process of order q if t j=1 W j X t = W t + θ 1 W t θ q W t q, where {W t } WN(0, σ 2 ) and θ 1,..., θ q are constants. We have µ X (t) =0 q h γ X (h) =σ 2 j=0 θ j θ j+ h I( h q). Proposition 2.1. If {X t } is a stationary q correlated time series (i.e., Cov(X s, X t ) = 0 whenever s t > q) with mean 0, then it can be represented as an MA(q) process. Proof. See Proposition on page 89 of Brockwell and Davis (2009, Time Series Theory and Methods). 19

23 2.1.2 Identify Non-Stationary Time Series After learning all the above stationary time series, one question would naturally arise is that, what kind of time series is not stationary? Plotting the data would always be helpful to identify the stationarity of your time series data. Any time series with non-constant trend is not stationary. For example, if X t = m t + Y t with trend m t and zero-mean error Y t. Then µ X (t) = m t is not a constant. For example, the following figure plots a realization of X t = t + Y t, where {Y t } N(0, 1) iid. set.seed(100); par(mar=c(4,4,2,.5)) t=seq(1,100,1); Tt=1+.05*t; Xt=Tt+rnorm(length(t),0,2) plot(t,xt,xlab="t",ylab=expression(x[t])); lines(t,tt,col="blue") X t t 20

24 Any time series with seasonal trend is not stationary. For example, if X t = s t +Y t with seasonal trend s t and zero-mean error Y t. Then µ X (t) = s t is not a constant. For example, the following figure plots a realization of X t = t + 2 cos(πt/5) + 3 sin(πt/3) + W t, where {Y t } N(0, 1) iid. set.seed(100); par(mar=c(4,4,2,.5)); t=seq(1,100,1); Tt=1+.05*t; St=2*cos(pi*t/5)+3*sin(pi*t/3); Xt=Tt+St+rnorm(length(t),0,2) plot(t,xt,xlab="t",ylab=expression(x[t])); lines(t,tt+st,col="blue") X t t Any time series with non-constant variance is not stationary. {S t = t j=1 X t} with X t being iid N(0, 1). For example, random walk set.seed(150); par(mar=c(4,4,2,.5)); t=seq(1,200,by=1); Xt1=rnorm(length(t),0,1) plot(c(0,t),c(0,cumsum(xt1)),type="o",col="blue",xlab="t",ylab=expression(s[t])) S t t 21

25 Another way you may have already figured out by yourself of identifying stationarity is based on the shape of the autocorrelation function (ACF). However, in applications, you can never know the true ACF. Thus, a sample version of it could be useful. In the following, we produce the estimators of µ X, ACVF, and ACF. Later, we will introduce the asymptotic properties of these estimators. For observations x 1,..., x n of a time series, the sample mean is The sample auto covariance function is γ X (h) = 1 n n h x = 1 n n x t. t=1 (x t+ h x)(x t x), for n < h < n. t=1 This is like the sample covariance of (x 1, x h+1 ),..., (x n h, x n ), except that we normalize it by n instead of n h, we subtract the full sample mean. This setting ensures that the sample covariance matrix Γ n = [ γ X (i j)] n i,j=1 is nonnegative definite. The sample autocorrelation function (sample ACF) is ρ X (h) = γ X(h), for n < h < n. γ X (0) The sample ACF can help us recognize many non-white (even non-stationary) time series. 22

26 Some guidelines: Time series: Sample ACF White noise Zero for h > 0 Trend Slow decay Periodic Periodic MA(q) Zero for h > q AR(1) Decays to zero exponentially set.seed(100); par(mfrow=c(5,2)) par(mar=c(4,4,2,.5)) #White Noise WN=rnorm(100,0,1); plot(1:n,wn,type="o",col="blue",main="white Noise",ylab=expression(X[t]),xlab="t"); acf(wn) #Trend t=seq(1,100,1); Tt=1+.1*t; Xt=Tt+rnorm(length(t),0,4) plot(t,xt,xlab="t",ylab=expression(x[t]),main="trend") lines(t,tt,col="blue") acf(xt) #Periodic t=seq(1,150,1) St=2*cos(pi*t/5)+3*sin(pi*t/3) Xt=St+rnorm(length(t),0,2) plot(t,xt,xlab="t",ylab=expression(x[t]), main="periodic") lines(t,st,col="blue") acf(xt) #MA(1) w = rnorm(550,0,1) v = filter(w, sides=1, c(1,.6))[-(1:50)] plot.ts(v, main="ma(1)",col="blue",ylab=expression(x[t]),xlab="t") acf(v) #AR(1) w = rnorm(550,0,1) x = filter(w, filter=c(.6), method="recursive")[-(1:50)] plot.ts(x, main="ar(1)",col="blue",ylab=expression(x[t]),xlab="t") acf(x) 23

27 White Noise Series WN X t ACF t Trend Series Lag Xt X t ACF t Periodic Series Lag Xt X t ACF t MA(1) Series Lag v X t ACF t AR(1) Series Lag x X t ACF t Lag 24

28 The typical procedure of time series modeling can be described as 1. Plot the time series (look for trends, seasonal components, step changes, outliers). 2. Transform data so that residuals are stationary. (a) Estimate and subtract m t, s t. (b) Differencing (c) Nonlinear transformations (log, ) 3. Fit model to residuals. Now, we introduce the difference operator and the backshift operator B. Define the lag-1 difference operator: (think first derivative ) X t = X t X t 1 t (t 1) = X t X t 1 = (1 B)X t where B is the backshift operator, BX t = X t 1. Define the lag-s difference operator, s X t = X t X t s = (1 B s )X t, where B s is the backshift operator applied s times, B s X t = B(B s 1 X t ) and B 1 X t = BX t. Note that If X t = β 0 + β 1 t + Y t, then if X t = k i=0 β it i + Y t, then X t = β 1 + Y t. k X t = k!β k + k Y t, where k X t = ( k 1 X t ) and 1 X t = X t. if X t = m t + s t + Y t and s t has period s (i.e., s t = s t s for all t), then s X t = m t m t s + s Y t. 25

29 2.2 Linear Processes Every second-order stationary process is either a linear process or can be transformed to a linear process by subtracting a deterministic component, which will be discussed later. The time series {X t } is a linear process if it has the representation X t = µ + j= ψ j W t j, for all t, where {W t } WN(0, σ 2 ), µ is a constant, and {ψ j } is a sequence of constants with j= ψ j <. if we define ψ(b) = j= ψ jb j, then the linear process X t = µ + ψ(b)w t. Note that the condition j= ψ j < ensures that X t is meaningful; i.e., X t < almost surely. Since E W t σ for all t and P ( X t α) 1 α E X t 1 µ + ψ j E W t j α j= 1 µ + σ ψ j 0 as α 0. α j= Before proceeding, we provide a brief introduction of several types of convergence in statistics. We say a sequence of random variables X n converges in mean square to a random variable X (denoted by X n L 2 X) if E(X n X) 2 0 as n. More generally, we have convergence in r-th mean, denoted by X n L r X, if E( X n X r ) 0 as n. Also, we say that X n converges in probability to X, denoted by X n p X, if P { X n X > a} 0, for all a > 0, as n. X n converges in distribution to X, denoted by X n p X, if F n (X) F (X) as n, at the continuity points of F ( ). The last one is convergence almost surely denoted by X n a.s. X (which will not be used in this course). The relationship between these convergences is, for r > 2, L r L2 p d. 26

30 This course mainly focuses on convergence in mean square. One easy way to prove this convergence is through the use of the following theorem: Theorem 2.4. (Riesz-Fisher Theorem, Cauchy criterion.) X n converges in mean square if and only if lim E(X m X n ) 2 = 0. m,n Example 2.9. Linear process X t = j= ψ jw t j, then if j= ψ j <, we have j= ψ jw t j converges in mean square. Proof. Defining S n = n j= n ψ jw t j, we have E(S m S n ) 2 =E = m j n m j n ψ j W t j ψj 2 σ 2 σ 2 2 m j n ψ j 2 0 as m, n. Lemma 2.2. Linear process {X t } defined above is stationary with µ X =µ (2.1) γ X (h) =σ 2 Proof. Equation (2.1) is trivial. For (2.2), we have γ X (h) =E j= = = j= k= ψ j W t+h j j= j= ψ j+h ψ j. (2.2) ψ j W t j ψ j ψ k E(W t+h j W t k ) = ψ j ψ k I(k = j h)σ 2 = σ 2 j= k= ψ j ψ k γ W (h j + k) ψ j ψ j h = σ 2 j= k= j= j= ψ j+h ψ j. Proposition 2.2. Let {Y t } be a stationary time series with mean 0 and covariance function γ Y. If j= ψ j <, then the time series X t = ψ j Y t j = ψ(b)y t j= 27

31 is stationary with mean 0 and autocovariance function γ X (h) = j= k= ψ j ψ k γ Y (h j + k) It can be easily seen that white noise, MA(1), AR(1), MA(q) and MA( ) are all special examples of linear processes. White noise: choose µ, and ψ j = I(j = 0), we have X t = µ + j= ψ jw t j = µ + W t WN(µ, σ 2 ). MA(1): choose µ = 0, ψ j W t + θw t 1. = I(j = 0) + θi(j = 1), we have X t = µ + j= ψ jw t j = AR(1): choose µ = 0, ψ j = φ j I(j 0), we have X t = µ + j= ψ jw t j = j=0 φj W t j = W t + φ j=0 φj W t 1 j = W t + φx t 1 MA(q): choose µ = 0, ψ j = I(j = 0) + q k=1 θ ki(j = k), we have X t = µ + j= ψ jw t j = W t + θ 1 W t θ q W t q. MA( ): choose µ = 0, ψ j = k=0 θ ki(j = k), we have X t = µ + j= ψ jw t j = j=0 ψ jw t j. 2.3 AR(1) and AR(p) Processes AR(1) process In this section, we provide closer investigation on the AR(1) process which has been briefly introduced in Example 2.6. An AR(1) process was defined in Example 2.6 as a stationary solution {X t } of the equations X t φx t 1 = W t, for all t, (2.3) where {W t } WN(0, σ 2 ), and Z t is uncorrelated with X s for s < t. When φ = 0, it is so trivial that X t = W t. When 0 < φ < 1, we now show that such a solution exists and is the unique stationary solution of (2.3). In the above, we have already shown that X t = φ j W t j, (2.4) j=0 28

32 This can be found easily through the aid of φ(b) = 1 φb and π(b) = j=0 φj B j. We have The last step is due to π(b)φ(b) = (1 φb) X t φx t 1 = W t π(b)(x t φx t 1 ) = π(b)w t π(b)φ(b)x t = π(b)w t X t = π(b)w t = φ j W t j. φ j B j = j=0 j=0 φ j B j φ j B j = 1 j=0 which is similarly to the summation of geometric series: φ j B j = j=0 (φb) j = j=0 1 1 φb. It can be easily seen that it is stationary with mean 0 and ACVF γ X (h) = σ 2 φ h /(1 φ 2 ), which are the same as in Example 2.6. Further, we show this solution is unique. Suppose {Y t } is another stationary solution, then by iterating, we have j=1 Y t =φy t 1 + W t =W t + φw t 1 + φ 2 Y t 2 = =W t + φw t φ k W t k + φ k+1 Y t k 1 Then E Y t k φ j W t j j=0 2 = φ 2k+2 E(Yt k 1 2 ) 0 as k. This implies that Y t is equal to the mean square limit j=0 φj W t j and hence the uniqueness is proved. When φ > 1, the series defined in (2.4) does not converge. However, we can rewrite (2.3) in the form By iterating, we have X t = φ 1 W t+1 + φ 1 X t+1. X t = φ 1 W t+1 φ k 1 W t+k+1 + φ k 1 X t+k+1, 29

33 which shows that X t = φ j W t+j j=1 is the unique stationary solution of (2.3). However, this is very hard to interpret, since X t is defined to be correlated with future values of Z s. Another way to look at this case is to define a new sequence Wt =X t 1 φ X t 1 = (φ φ 1 )X t 1 + W t = (φ φ 1 ) φ j W t 1+j + W t = 1 φ 2 W t (1 φ 2 ) φ j W t+j j=1 Standard arguments yields that (left as a HW problem) j=1 E(W t ) =0 γ W (h) = σ2 I(h = 0); φ2 i.e., {W t } is a new white noise sequence with mean 0 and variance σ 2 /φ 2, then we have a new AR(1) model X t = φ X t 1 + W t with φ = 1/ φ < 1. Thus, we can rewrite the unique stationary solution as X t = φ j Wt j j=0 which now does not depend on future values. assumes that φ < 1. Thus, for an AR(1) model, people typically When φ = 1. If there is a stationary solution to (2.3), check Cov(X t 1, W t ) = Cov(X t 1, X t φx t 1 ) = γ X (1) φγ X (0) = 0 This holds if and only if X t = φx t 1 + b for some constant b. Then {W t = b} is a constant process. Since {W t } is a white noise, then b has to be zero. Now we have X t = φx t 1. When φ = 1, X t has to be all zeros. When φ = 1, then X t are all constants. So if we quire σ > 0, then there is no stationary solution; if more broadly, we allow σ = 0, i.e., {W t = 0}, then when φ = 1, there is a stationary solution which is X t = 0; when φ = 1, there is also a stationary solution that X t = µ X. In the following, we require σ > 0. 30

34 Remark 2.2. This example introduced a very important terminology: causality. We say that {X t } is a causal function of {W t }, or more concisely that {X t } is a causal autoregressive process, if X t has a representation in terms of {W s, s t}; i.e., the current status only relates to the past events, not the future. A linear process {X t } is causal (strictly, a causal function of {W t }), if there is a ψ(b) = ψ 0 + ψ 1 B + ψ 2 B 2 + with j=0 ψ j < such that X t = ψ(b)w t = ψ j W t j. j=0 When φ < 1, AR(1) process {X t } is a causal function of {W t }. When φ > 1, AR (1) process is not causal. Proposition 2.3. AR(1) process φ(b)x t = W t with φ(b) = 1 φb is causal if and only if φ < 1 or the root z 1 of the polynomial φ(z) = 1 φz satisfies z 1 > AR(p) process An AR(p) process {X t } is a stationary process that satisfies X t φ 1 X t 1 φ p X t p = W t where {W t } WN(0, σ 2 ). Equivalently, φ(b)x t = W t where φ(b) = 1 φ 1 B φ p B p. Recall that, for p = 1, φ(b) = 1 φ 1 B, and for this AR(1) model, X t is stationary only if φ 1 1. This is equivalent to that for any z R such that φ(z) = 1 φz satisfis z 1, or for any z C such that φ(z) = 1 φz satisfis z 1. Now for the AR(p) model, similarly, we should have for any z C such that φ(z) = 1 φ 1 z φ p z p satisfis z 1. Theorem 2.5. A (unique) stationary solution to φ(b)x t = W t exists if and only if φ(z) = 1 φ 1 z φ p z p = 0 z 1 Further, this AR(p) process is causal if and only if φ(z) = 1 φ 1 z φ p z p = 0 z > 1 (2.5) 31

35 When (2.5) is satisfied, based on causality, we can write X t = ψ(b)w t where ψ(b) = ψ 0 + ψ 1 B + ψ 2 B 2 + for some ψ j s satisfying j=0 ψ j <. The question is then, how to calculate ψ j s? One way is to matching the coefficients. φ(b)x t = W t and X t = ψ(b)w t 1 = ψ(b)φ(b) 1 = (ψ 0 + ψ 1 B + ψ 2 B 2 + )(1 φ 1 B φ p B p ) 1 = ψ 0, 0 = ψ 1 φ 1 ψ 0, 0 = ψ 2 φ 1 ψ 1 φ 2 ψ 0. 1 = ψ 0, 0 = ψ j (j < 0), 0 = φ(b)ψ j (j > 0). 2.4 MA(1) and MA(q) Processes Now, we look at the MA(1) process defined in Example 2.5. An MA(1) process {X t } is defined as X t = W t + θw t 1 where {W t } WN(0, σ 2 ). Obviously, {X t } is a causal function of {W t }. But more importantly is about another terminology: invertibility. Just as causality means that X t is expressible in terms of {W s, s t}, invertibility means that W t is expressible in terms of {X s, s t}. A linear process {X t } is invertible (strictly, a invertible function of {W t }), if there is a π(b) = π 0 + π 1 B + π 2 B 2 + with j=0 π j < such that W t = π(b)x t = π j X t j. j=0 Obviously, AR(1) process is invertible. Back to the MA(1) process: When θ < 1, we have (1 + θb) 1 = ( θ) j B j j=0 32

36 Thus X t = W t + θw t 1 = (1 + θb)w t (1 + θb) 1 X t = W t W t = ( θ) j X t j. j=0 We have {X t } as a invertible function of {W t }. when θ > 1, the sum j=0 ( θ)j X t j diverges, but we can write Now, MA(1) is not invertible. W t = θ 1 W t+1 + θ 1 X t+1 = θ 1 X t+1 θ 2 X t+2 + θ 2 W t+2 = = ( θ) j X t+j. When θ = 1, we have X t = W t + W t 1. If we have W t = j=0 π jx t j, then X t = π j X t j + π j X t 1 j = j=0 j=0 j=0 j=1 π j X t j + π j 1 X t j = π 0 X t + (π j + π j 1 )X t j. Thus, we have have π j + π j 1 = 0 and π 0 = 1, which means π j = ( 1) j. Then j=1 π j < j=0 is not possible. So MA(1) is not invertible when θ = 1; similarly when θ = 1. One may notice that, similarly as the case of φ = 1 in the AR(1) model, if we allow σ = 0, then we have X t = 0 and W t = 0 = X t so invertible. But this is a nonsense case. So in the following, we require σ > 0. Sec 4.4 in Brockwell and Davis (2009, Time Series Theory and Methods) defines invertibility in a more general way that is if we can express W t as W t = j=0 π jx t j. It does not require that j=0 π j <. With this more general meaning invertibility, we have X t is invertible when θ = 1. In the remaining context, we will keep our more realistic restriction of j=0 π j <. Proposition 2.4. MA(1) process X t = θ(b)w t where θ(b) = 1 + θb is not invertible if and only if θ 1 or the root z 1 of the polynomial θ(z) = 1 + θz satisfies z 1 1. j=1 33

37 Theorem 2.6. The MA(q) process X t = π(b)w t where π(b) = 1 + θ 1 B + + θ q B q is not invertible if and only if π(z) = 1 + θ 1 z + + θ q z q = 0 z 1. Based on invertibility, we can write W t = π(b)x t where π(b) = π 0 + π 1 B + π 2 B 2 + for some π j s satisfying j=0 π j <. The question is then, how to calculate π j s? One way is to matching the coefficients. X t = θ(b)w t and W t = π(b)x t 1 = π(b)θ(b) 1 = (π 0 + π 1 B + π 2 B 2 + )(1 + θ 1 B + + θ p B p ) 1 = π 0, 0 = π 1 + θ 1 π 0, 0 = π 2 + θ 1 π 1 + θ 2 π 0. 1 = π 0, 0 = π j (j < 0), 0 = θ(b)π j (j > 0). 34

38 2.5 ARMA(1,1) Processes In this subsection we introduce, through an example, some of the key properties of an important class of linear processes known as ARMA (autoregressive moving average) processes. This example is the ARMA(1,1) processes. Higher-order ARMA processes will be discussed later. The time series {X t } is an ARMA(1,1) process if it is stationary and satisfies (for every t) where {W t } WN(0, σ 2 ), σ > 0, and φ + θ 0. Let us find the expression of {X t } in terms of {W t }: When φ = 0, we have the trivial solution X t = W t + θw t 1. X t φx t 1 = W t + θw t 1, (2.6) When 0 < φ < 1, we have meaning full definition of j=0 φj B j. Then applying it to both sides of (2.6) provides that φ j B j (1 φb)x t = X t = φ j B j (1 + θb)w t j=0 j=0 = φ j B j + θ φ j B j+1 W t j=0 =W t + (φ + θ) j=0 φ j 1 W t j. (2.7) This is one MA( ) process, and of course stationary. For the uniqueness, suppose we have another stationary solution Y t, then we have j=1 Y t = W t + θw t 1 + φy t 1 = W t + (θ + φ)w t 1 + θφw t 2 + φ 2 Y t 2 = = W t + (θ + φ)w t 1 + (θ + φ)φw t (θ + φ)φ k 1 W t k + φ k+1 Y t k 1 Then E Y t W t (φ + θ) k φ j 1 W t j j=1 2 = φ 2k+2 E(Yt k 1 2 ) 0 as k. Hence, solution (2.7) is the unique stationary solution of (2.6) providing that φ < 1. 35

39 When φ > 1, we have X t = θφ 1 W t φ 1 W t+1 + φ 1 X t+1 = θφ 1 W t (θ + φ)φ 2 W t+1 φ 2 W t+2 + φ 2 X t+2 k = = θφ 1 W t (θ + φ) φ j 1 W t+j φ k 1 W t+k+1 + φ k 1 X t+k+1 j=1 Then E X t θφ 1 W t (θ + φ) k φ j 1 W t+j j=1 =φ 2k 2 E(W t+k+1 + X t+k+1 ) 2 0 as k. 2 Thus, we have a unique stationary solution of (2.6) when φ > 1 as X t = θφ 1 W t (θ + φ) φ j 1 W t+j. (2.8) Again, this solution depends on future values of W t. j=1 When φ = 1, there is no stationary solution of (2.6) (left as a HW problem). stationary ARMA(1,1) process when φ = 1. Thus, no Summary: A stationary solution of the ARMA(1,1) equations exists if and only if φ 1. If φ < 1, then the unique stationary solution is given by (2.7). In this case, we say that {X t } is causal or a causal function of {W t }, since X t can be expressed in terms of the current and past values {W s, s t}. If φ > 1, then the unique stationary solution is given by (2.8). In this case, we say that {X t } is noncausal since X t is then a function of current and future values {W s, s t}. For invertibility, we have, by switching the roles of X t and W t, and the roles of φ and θ, If θ < 1, then ARMA(1,1) process is invertible as W t = X t (φ + θ) ( θ) j 1 X t j. If θ > 1, then ARMA(1,1) process is noninvertible as W t = φθ 1 X t + (θ + φ) ( θ) j 1 W t+j. j=1 j=1 36

40 If θ = 1, the ARMA(1,1) process is invertible under the more general definition of invertibility same as in the MA(1) process. Without this more general setting, we say the ARMA(1,1) process is noninvertible when θ = 1. Like the argument in last subsection of AR(1) model, if the ARMA(1,1) process {X t } is noncausal and noninvertible; i.e., φ > 1 and θ > 1, then we define φ(b) = 1 φ 1 B and θ(b) = 1 + θ 1 B and let W t = θ 1 (B) φ(b)x t Once verifying that {Wt } WN(0, σ ) 2 and φ(b)xt = θ(b)w t, (2.9) we have {X t } being a causal and invertible ARMA(1,1) process relative to the white noise sequence {Wt }. Threre, from a second-order point of view, nothing is lost by restricting attention to causal and invertible ARMA(1,1) models. This statement is also true for higher-ordered ARMA models. Now, we show (2.9). It is easy to see that θ(b)w t = θ(b) θ 1 (B) φ(b)x t = φ(b)x t. It suffices to show {Wt } is a white noise. This is left as a HW problem. 37

41 2.6 Properties of X n, γ X (h) and ρ X (h) For X n Recall that, for observations x 1,..., x n of a time series, the sample mean is The sample auto covariance function is γ X (h) = 1 n n h x = 1 n n x t. t=1 (x t+ h x)(x t x), for n < h < n. The sample autocorrelation function (sample ACF) is t=1 ρ X (h) = γ X(h) γ X (0). Estimation of µ X : The moment estimator of the mean µ X of a stationary process {X t } is the sample mean X n = n 1 Obviously, it is unbiased; i.e., E(X n ) = µ X. Its mean squared error is Var(X n ) =E(X n µ X ) 2 =n 2 n i=1 j=1 =n 2 n i j= n n X t. (2.10) t=1 n Cov(X i, X j ) = n 2 n i=1 j=1 n γ X (i j) (n i j )γ X (i j) = n 1 n γ X (0) = + 2 n 1 }{{ n } n h=1 is Var(X n) when {X t} are iid ( 1 h n h= n ( 1 h ) γ X (h) n ) γ X (h). Depending on the nature of the correlation structure, the standard error of X n may be smaller or larger than the white noise case. Consider X t = µ + W t 0.8W t 1, where {W t } WN(0, σ 2 ), then Var(X n ) = γ X(0) n + 2 n 1 ( 1 h ) γ X (h) = 1.64σ2 1.6(n 1)σ2 n n n n 2 < 1.64σ2. n h=1 38

42 And if X t = µ + W t + 0.8W t 1, where {W t } WN(0, σ 2 ), then Var(X n ) = γ X(0) n If γ X (h) 0 as h, we have + 2 n 1 ( 1 h ) γ X (h) = 1.64σ2 1.6(n 1)σ2 + n n n n 2 > 1.64σ2. n h=1 Var(X n ) γ X(0) n Thus, X n converges in mean square to µ. If h= γ X(h) <, then nvar(x n ) = n h= n γ X (0) + 2 n h=1 + 2 γ X(h) 0 as n. n ( 1 h ) n h=1 γ X (h) = γ X (0) + 2 (n h)γ n 1 h X(h) h=1 i=1 = γ X (0) + 2 γ X(i) n n n γ X (i) = i=1 h= γ X (h) = γ X (0) h= ρ X (h). One interpretation could be that, instead of Var(X n ) γ X (0)/n, we have Var(X n ) γ X (0)/(n/τ) with τ = h= ρ X(h). The effect of the correlation is a reduction of sample size from n to n/τ. Example For linear processes, i.e., if X t = µ + j= ψ jw t j with j= ψ j <, then h= γ X (h) = h= h= σ 2 = σ 2 j= = σ 2 σ 2 j= j= ψ j j= ψ j ψ j+h ψ j ψ j+h h= 2 ψ j ψ j+h < To make inference about µ X (e.g., is µ X = 0?), using the sample mean X n, it is necessary to know the asymptotic distribution of X n : If {X t } is Gaussian stationary time series, then, for any n, n(xn µ X ) N ( 0, n h= n ( 1 h ) ) γ X (h). n 39

43 Then one can obtain exact confidence intervals of estimating µ X, or approximated confidence intervals if it is necessary to estimate γ X ( ). For the linear process, X t = µ + j= ψ jw t j with {W t } IID(0, σ 2 ), j= ψ j < and j= ψ j 0, then n(xn µ X ) AN(0, ν), (2.11) where ν = h= γ X(h) = σ 2 ( j= ψ j) 2. The proof of (2.11) can be found in Page 238 of Brockwell and Davis (2009, Time Series Theory and Methods). Very roughly, recall then γ X (h) = σ 2 lim nvar(x n) = lim n n n h= n = lim n σ2 = σ 2 j= ψ j ψ j+h, ( 1 h ) γ X (h) n n ( ψ j 1 h ) ψ j+h n h= n j= j= ψ j 2 The above results for the linear process, also hold for ARMA models. Naturally, (X n 1.96 ν/n, X n ν/n). is an approximated 95% confidence interval for µ X. as Since ν is typically unknown, naturally, we have an approximated 95% confidence interval of µ X (X n 1.96 ˆν/n, X n ˆν/n), once we can obtain an estimator ˆν of ν = h= γ X(h). One intuitive way is to use ˆν = h= γ X(h). However, based on finite sample {X 1,..., X n }, it is impossible to obtain a reasonable estimator of γ X (h) for h n. Then, why not use ˆν = n 1 h= (n 1) γ X(h). Vary sadly and interestingly, this ˆν is always zero. A compromising estimator ˆν is then [ n] ( ˆν = 1 h ) [ γ X (h) n] h= [ n] If we known the model of the time series, i.e., we have explicit formula of γ X (h). For example, 40

44 say we have an AR(1) {X t } with mean µ X satisfies X t µ X = φ(x t 1 µ X ) + W t, we have γ X (h) = φ h σ 2 /(1 φ 2 ) and consequently, ν = σ 2 /(1 φ) 2. Then we have ν = σ 2 (1 φ) 2 41

45 2.6.2 For γ X (h) and ρ X (h) Estimators of γ X (h) and ρ X (h) is defined by n h γ X (h) =n 1 (X t+ h X n )(X t X n ), (2.12) t=1 ρ X (h) = γ X(h) γ X (0). (2.13) First let us check that ˆν = n 1 h= (n 1) γ X(h) is always zero. ˆν = n 1 h= (n 1) n h n 1 t=1 (X t+ h X n )(X t X n ) n n 1 n h =n 1 (X t X n ) 2 + 2n 1 (X t+h X n )(X t X n ) t=1 h=1 t=1 n =n 1 (Xt 2 2X t X n + X 2 n 1 n h n) + 2n 1 (X t+h X t X t X n X t+h X n + X 2 n) t=1 h=1 t=1 n =n 1 Xt 2 X 2 n 1 n h n + 2n 1 (X t+h X t X t X n X t+h X n + X 2 n) t=1 h=1 t=1 n =n 1 Xt 2 nx 2 n 1 n h n + 2n 1 X t+h X t = 0. t=1 h=1 t=1 To check the bias of γ X (h), let us look at the case when h = 0. We have γ X (0) = n 1 n (X t X n ) 2. t=1 Even in iid case, this is an biased estimator (sample variance is biased which has (n 1) 1 instead of n 1 ). Expression for E{ γ X (h)} is messy (try your best to derive it as a HW problem). Let s consider instead It can be seen that γ X (h) = 1 n n h (X t+ h µ X )(X t µ X ). t=1 E{γ X (h)} = n h γ X (h) γ X (h); n i.e., biased. Rather than using γ X (h), you might seem more natural to consider γ X (h) = n h 1 (X n h t+ h µ X )(X t µ X ), t=1 42

46 since now we have E{ γ X (h)} = γ X (h); an unbiased estimator. Now replacing µ X with X n, we have two estimators: γ X (h) and n n h γ X(h) respectively, called biased and unbiased ACVF estimators (even though latter one is actually biased in general!). Generally speaking, both γ X (h) and ρ X (h) are biased even if the factor n 1 is replaced by (n h) 1. Nevertheless, under general assumptions they are nearly unbiased for large sample size (conduct a simulation study to see the bias of both estimators as a HW problem). Now, let us talk about the reason of why we like γ X (h), and why I think this is very brilliant. Lemma 2.3. For any sequence x 1,..., x n, the sample ACVF γ X satisfies: 1. γ X (h) = γ X ( h) 2. γ X is nonnegaitve definite, and hence 3. γ X (0) 0 and γ X (h) γ X (0) Proof. The first one is trivial. It suffices to prove the second property which is equivalent to show that for each k 1 the k-dimensional sample covariance matrix γ X (0) γ X (1) γ X (k 1) γ Γ k = X (1) γ X (0) γ X (k 2).... γ X (k 1) γ X (k 2) γ X (0) is nonnegative definite. To see that, we have, for k n, Γ k = n 1 MM T, where Y 1 Y 2 Y k 0 0 Y M = 1 Y 2 Y k Y 1 Y 2 Y k 0 0 is a k 2k matrix with Y i = X i X n for i=1,..., n and Y i = 0 for i = n + 1,..., k. Note that, if Γ m is nonnegative definite, then all Γ k s are nonnegative definite for all k < m. The nonnegative definite property is not always true if n 1 is replaced by (n h) 1. Further, when h n or for h slightly smaller thann, there is no way to reliably estimate γ X (h) and ρ X (h) since the information around there are too little. Box and Jenkins (1976) suggest that useful estimates of correlation ρ X (h) can only be made if n is roughly 50 or more and h n/4. 43

47 It will be important to be able to recognize when sample autocorrelations are significantly different from zero so that we can select the correct model to fit our data. In order to draw such statistical inference, we need the following asymptotic joint distribution. Theorem 2.7. For an IID process {W t }, if E(W 4 t ) <, we have ρ W (h) = where I h is a h h identity matrix. ρ W (1). ρ W (h) AN(0, n 1 I h ). (2.14) Remark 2.3. For {W t } IID(0, σ 2 ), then ρ W (l) = 0 for l 0. From Theorem 2.7, we have, for large n, ρ W (1),..., ρ W (h) is approximately independent and identically distributed normal random variables form N(0, n 1 ). If we plot the sample autocorrelation function ρ W (k) as a function of k, approximately 0.95 of the sample autocorrelations should lie between the bounds ±1.96 n. This can be used as a check that the observations truly are from an IID process. In Figure 2.3, we have plotted the sample autocorrelation ρ W (k) for k = 1,..., 40 for a sample of 200 iid N(0, 1). set.seed(150); Wt=rnorm(200);par(mar=c(4,4,1.5,.5));acf(Wt,lag.max=40) ACF Lag It can be seen that all but one of the autocorrelations lie between the bounds ±1.96 n, and Remark 2.4. This theorem yields several procedures of testing H 0 : iid vs H a : not iid. Method 1: Based on the values of sample ACF: If for one h, ρ X (h) ± z α/2 / n does not contain zero, reject H 0. 44

48 Method 2: The portmanteau test I: Instead of checking ρ X (h) for each h, it is also possible to consider the single statistics Q = n h ρ 2 X(j). j=1 Under H 0, Q χ 2 h. Thus, rejection region is Q > χ2 h (1 α). Method 3: The portmanteau test II (Ljung and Box, 1978). Q LB = n(n + 2) h ρ 2 X(j)/(n j) which is better approximated by χ 2 h, thus the same rejection region. Method 4: The portmanteau test III: if wanted to test residuals {R t } rather than a time series {X t }, then Q LB = n(n + 2) j=1 h ρ 2 R(j)/(n j) j=1 which is better approximated by χ 2 h p instead, where p is the number of parameters estimated in forming {R t }. Method 5: Turning point test Method 6: Difference-Sign test Method 7: Rank test Design a simulation study to compare these testing procedures (as a HW problem). 1. Learn them by yourself. At least you should know when is okay to use which test, how to calculate the test statistic and when to reject the null. 2. Set a reasonable sample size and generate an iid sequence. Apply each of these method to test for IID. If rejected, count by 1, if not, count it by zero. 3. Repeat step 2 for 1000 times. Record how many of them lead to the conclusion of rejection (it should be around the value of α) 4. Then, start making your model be more and more Non-IID, for example, you can general X t φx t = W t, in the beginning set φ = 0 then you have IID. Then set φ = seq(0.02, 1, by = 0.02). Each time, you do 1000 replications to obtain a rejection rate (as the power). 5. Plot the power curve to see which methods is the most powerful. 6. Summarize your simulation result and turn in with your homework. 45