Figure 29: AR model fit into speech sample ah (top), the residual, and the random sample of the model (bottom).

Transcription

1 Original ACF Residual ACF Generated ACF Figure 29: AR model fit into speech sample ah (top), the residual, and the random sample of the model (bottom). Theorem 5.6. If (X i ) is stationary AR(p) with (W i ) WN(σ 2 ), and ˆφ (n) is the YW estimates based on X 1,...,X n, then n(ˆφ(n) φ) n N(0,σ 2 Γ 1 ), in distribution. Moreover, ˆσ 2 σ 2 in probability. For proof, see Brocwell and Davis, Theorem Eample 5.7. The mobile phone (GSM) standard involves lossy speech compression based on linear prediction coefficients, which is engineering terminology AR coefficients. Figure 29 shows a the beginning of a 0.5 second speech sample of ah at 8000 samples per second, the corresponding residual of AR(100) fit to the sample, and a random sample of AR(100). (The compression includes both the AR coefficients and an additional residual compression, which involves a lot of fine tuning...) 42 In the AR model, we have (X i µ) p φ j (X i j µ) = W i,

2 and we could thin also the following estimator Definition 5.8 (Conditional sum of squares). The conditional sum of squares, or conditional least squares estimator of φ = φ 1,...,φ n and µ is (ˆφ,ˆµ) := argmin S c( 1,..., n ;φ,µ), (φ,µ) n ( p 2. S c ( 1,..., n ;φ,µ) := i µ φ j ( i j µ)) (9) i=p+1 If we differentiate wrt. µ and set to zero, we get ˆµ = n i=p+1 ( i p φ j i j ) (n p)(1 p φ j) = 1 ( n p ) i +R n,p, n p i=p+1 if n is large, because R n,p is a residual term of order O(p 2 ). If we substitute µ = in (9) and differentiate wrt. φ and set to zero, we get that the estimates ˆφ j should satisfy 1 n n ( i )( i ) i=p+1 which is, for large n, close to p ( 1 ˆφ j n n i=p+1 ) ( i )( i j ) = 0, ˆγ = p ˆφ jˆγ j, which corresponds to the Yule-Waler estimate. This shows that for large n, the Yule-Waler estimates are going to be similar to the CSS estimates. Eample 5.9. Assume net an AR(1) with Gaussian white noise (W n ) i.i.d. N(0,σ 2 ). We can write the lielihood now as ( l( 1,..., n ) = N 1 µ;0, ) σ 2 n N ( 1 φ 2 i µ;φ 1 ( i 1 µ),σ 2), 1 where N(,m,σ 2 ) stands for the Gaussian p.d.f. Denoting ˆ i = i µ, we can epand logl = 1 φ2 1 2σ 2 ˆ σ 2 i=2 n (ˆ i φ 1ˆ i 1 ) 2 n 2 logσ log(1 φ2 1)+c i=2 = 1 2σ 2S c( 1,..., n ;φ 1,µ)+R 1 ( 1,σ 2,φ 2 1,µ). 43

3 As the number of samples increases, the conditional sum of squares term will dominate, meaning that the maimum lielihood estimator will be similar to the CSS, and hence the YW estimators. Remar Finding both the CSS and the ML estimates requires, in general, iterative numerical optimisation methods. The story about the maimum lielihood estimates of a general AR(p) is similar: The first p variables follow a multivariate Gaussian distribution, and the conditional sum of squares term will be the dominating one, so the maimum lielihood estimates will coincide with the YW and CSS estimates asymptotically. Keep in mind, however, that with any finite n, the Yule-Waler, the CSS and the ML estimates will all be generally different, and may be preferable in specific applications. The R arima used ML by default. Question. Can you give some reasons why each of the three variants could be useful in certain applications? 5.3 General ARMA We saw earlier that the Yule-Waler method of estimation for the AR models was straightforward and efficient, and coincides asymptotically with the CSS and the ML estimators. There are not as simple and well-behaved methods for the estimation of parameters in the general ARMA model, and one has to resort to numerical optimisation. If the time series is not too long, and if the fitted model does not have too many parameters, it is common to use maimum lielihood estimation in the ARIMA contet, assuming Gaussian white noise. This is also what the R standard ARIMA fitting tool arima does. The ML estimation requires iterative numerical optimisation methods, and weshallnottaeacloserlooatthemethodsnow.instead,wequotethefollowing asymptotic normality result. Theorem (not eaminable) Suppose β = (φ,θ) are parameters of an ARMA(p,q) process (X i ) with Gaussian (W i ) WN(σ 2 ), which satisfies Condition Let ˆβ (n) = (ˆφ (n),ˆθ (n) ) stand for the ML estimator calculated from (X 1,...,X n ), then, n(ˆβ(n) β) n N ( 0,V(β) ), in distribution, where the covariance matri V(β) can be epressed as [ ] EUU V(β) = σ 2 T EUV T 1 EV U T EV V T, U = (U p,...,u 1 ) V = (V q,...,v 1 ), 44

4 where (U i ) and (V i ) are AR(p) and AR(q) processes satisfying φ(b)u i = W i and θ(b)v i = W i. In case of p = 0 or q = 0, the corresponding matrices vanish. See Brocwell and Davis, Section 8.8. We note that the confidence intervals of the estimated parameters may be (and are) calculated from the Hessian, ] ) 1 1 ([ n V(β) 2 l(ˆβ), β i β j where l is the log-lielihood of β. i,j 6 Model selection tools Perhaps the most challenging question in time series analysis is to find a satisfactory model class. In the case of real data, there rarely is a correct model, and what is satisfacotry depends on what the model is used for. We will net loo some tools which may indicate which models might be appropriate. The first thing to do is always to inspect the data. Plot the time series, or if it is long, loo for shorter segments of the data at the time. Loo for obvious trends or other behaviour suggesting non-stationary. Inspect also cyclic or nearcyclic behaviour. Autocorrelation plots and the periodogram can be helpful, they can also suggest cyclic components. Remember to try also transformations, such as log-transform or Bo-Co transforms. It can be instructive to loo at the data agains lagged versions of itself, that is, the scatterplots of (X i,x i+ ). Eample 6.1. Lag plots of the speech data in Eample 5.7. lag.plot(, layout=c(2,3), set.lags=c(1:3, 32, 72, 104), pch=".") 6.1 Non-stationarity If the data shows signs of non-stationarity, recall that you may inspect the differences. Note, however, that if removal of a trend maes the data seem stationary (data is trend stationary), it is possible to consider the original data with an eternal regressor handling the trend. Remar 6.2. Unit root tests were mentioned in Section 4.5. We shall not discuss unit root tests further, but only note that there are some implemented in R library tseries: the Augmented Dicey-Fuller test adf.test and the Phillips- Perron test pp.test. 45

5 lag 1 lag 2 lag 3 lag 32 lag lag Figure 30: Lag plots of the speech data in Eample Autocorrelation Theorem 1.17 stated asymptotic normality of the sample ACF in case of white noise. The following generalises that for a large class of linear processes. Theorem 6.3. (Not eaminable) Suppose (X i ) is a stationary process given by X i = µ+ j= c j W j, (W i ) WN(σ 2 ), with coefficients satisfying j c j < and j j c2 j <, and let (ρ ) be its autocorrelation. Then, for any h N, the sample autocorrelations ˆρ (n) calculated from X 1,...,X n satisfy (n) n (ˆρ 1 ρ 1,..., ˆρ (n) ) n h ρ h N(0,W h ) in distribution, where the limiting covariance matri is given as [W h ] ij = ( )( ) ρ+i ρ i 2ρ i ρ ρ+j ρ j 2ρ j ρ =1 For a proof, see Brocwell and Davis Theorem Remar 6.4. Theorem 6.3 justifies us to say that (ˆρ 1,..., ˆρ h ) is approimately N ( (ρ 1,...,ρ h ),W h /n ) forlargen.notethattheorem6.3holdsforanystationary 46

6 ARMA(p,q), because we now that it has representation with c j = 0 for j < 0 and c j decay eponentially in j. Corollary 6.5. Suppose (X i ) is a MA(q). Then, for all i > q, n(ˆρi ρ i ) N(0,σ 2 i), where Proof. Left aso an eercise. q σi 2 = 1+2 ρ 2 j. The result of Corollary 6.5 is used by R acf, when called with ci.type="ma". It displays confidence bounds based on 1 ˆσ 2 = 1+2 ˆρ 2 j. Question. Can you eplain what is the rationale of this? What about the possible caveats? Eample 6.6. Comparison of ACF with standard and MA confidence bounds. = arima.sim(model=list(ma=c(3/4, -1/5, -1/5)), 100) par(mfrow=c(2, 1)) acf(); acf(, ci.type="ma") 6.3 Partial autocorrelation function Definition 6.7. The partial autocorrelation function (PACF) (α ) 0 of a zeromean finite variance stationary process (X i ) is defined as ρ 0 = 1 = α 0 and ρ 1 = α 1, and for 2 through α = Corr(X +1 ˆX +1 2:, X 1 ˆX 1 2: ), where ˆX +1 2: and ˆX 1 2: are the best linear predictors (in the mean square sense) 12, of X +1 and X 1 given X 2,...,X, respectively. Theorem 6.8. The partial autocorrelation of AR(p) process α = 0 for all > p. 12. Recall that the best linear predictor ˆX of a random variable X given Y 1,...,Y n in the mean square sense is ˆX = n c jy j, where the constants (c 1,...,c n ) are chosen to minimise E[( ˆX X) 2 ]. That is, the partial autocorrelation is correlation of the residuals of X +1 and X 1 after regressing with X 2,...,X. 47

7 Proof. Let > p, then we have [ p ] E[X +1 X 2,...,X ] = E φ j X +1 j +W +1 X 2,...,X = p φ j X +1 j, because W +1 is independent of X 2,...,X and j in the sum. Because conditional epectation minimises the mean square error, we deduce that (in this specific case!) ˆX +1 2: = p φ jx +1 j. We then conclude, α = Corr(W +1, X 1 ˆX 1 2: ) = 0. Theorem 6.9. Let ρ be the autocorrelation of a stationary process (X i ), and assume β () = (β () 1,...,β () ) is the solution of R () β () = ρ (), where ρ () = (ρ 1,...,ρ ) and [R () ] ij = ρ i j for 1 i,j. Then, the partial autocorrelation α = β () for 2. The proof can be found, for eample, from Brocwell and Davis, Corollary () Definition The sample PACF ˆα = ˆβ, where ˆβ () satisfies ˆR ()ˆβ() = ˆρ (), with ˆR () and ˆρ () standing for the R () and ρ () with the autocorrelations replaced with their corresponding sample quantities. 48

8 Remar The partial autocorrelations can be calculated iteratively with the Levinson-Durbin algorithm, as discussed in Section 5.2. Note that each ˆα p corresponds to the last estimated autocorrelation coefficient ˆφ p of the AR(p) Yule- Waler estimates. Eample (Not eaminable) The PACF of MA(1) is (eercise) α = ( θ 1) (1 θ1) 2. 1 θ 2(+1) 1 The general story is similar (and very much eaminable!): For MA(q), the PACF will not vanish but will tail off, just lie the ACF for AR(p). Theorem Assume (X i ) is AR(p), then the partial autocorrelation coefficient α (n) calculated from X 1,...,X n satisfies (n) n nα N(0,1), for > p. This follows from from asymptotic of Yule-Waler estimates stated in Theorem5.6(cf.BrocwellandDavisE.8.15).ItisthebasisofthePACFconfidence intervals in R. Eample Figure 6.14 shows ACF and PACF of simulated AR(1) with φ 1 = 3/4, MA(1) with θ 1 = 3/4 and random wal. n = 300.ar <- arima.sim(model=list(ar=c(3/4)), n).ma <- arima.sim(model=list(ma=c(3/4)), n).rw <- cumsum(rnorm(n)) par(mfrow=c(3,3)) ts.plot(.ar); acf(.ar); pacf(.ar) ts.plot(.ma); acf(.ma); pacf(.ma) ts.plot(.rw); acf(.rw); pacf(.rw) 6.4 Information criteria Autocorrelation and partial autocorrelation plots can suggest certain low-order AR and MA models, respectively. For a mied ARMA(p,q) with p > 1 and q > 1, neither ACF or PACF vanish, providing little help. Further, comparing AR, ARMA and MA models can be very difficult, because the model families are not ordered (how to compare AR(2) and ARMA(1, 1), say?). It is possible (and quite popular) to base model choice based on generic information criteria such as Aaie s AIC, AIC with correction AICc, and the 49

9 Figure 31: Data (left), ACF (middle) and PACF (right) of AR(1) (top), MA(1) (middle) and random wal (bottom) of Eample Bayesian BIC: AIC := 2lnl ML +2 2n AICc := 2lnl ML + n 1 BIC := 2lnl ML +lnn, = AIC 2( +1) n 1 where n is the length of the data, l ML is the lielihood of the ML estimate and = p+q +2 is the number of parameters (including mean and variance). Remar Note that AICc converges to AIC as n tends to infinity, but the penalty of BIC will be higher, leading to favour models with less parameters. fit <- arima(, order=c(p,0,q)) n <- length(); <- q+p+2 fit.aic <- AIC(fit) fit.aicc <- AIC(fit, =2*n/(n--1)) fit.bic <- AIC(fit, =log(n)) ThereisevenanautomaticARIMAmodelfittingtoolinR,whichsearches for the best (low order) ARIMA in terms of some of the information criteria. 50