MAT3379 (Winter 2016)

Transcription

1 MAT3379 (Winter 2016) Assignment 4 - SOLUTIONS The following questions will be marked: 1a), 2, 4, 6, 7a Total number of points for Assignment 4: 20 Q1. (Theoretical Question, 2 points). Yule-Walker estimation for AR(p) models. Assume that Z t are i.i.d random variables with mean 0 and variance σ 2 Z. (a) Consider AR(1) model X t = ϕx t 1 + Z t. Use Theorem 4.2 in the Lecture Notes to derive confidence intervals for ϕ. (b) Consider AR(2) model X t = ϕ 1 X t 1 + ϕ 2 X t 2 + Z t. Use Theorem 4.2 in the Lecture Notes to derive confidence intervals for ϕ 1 and ϕ 2. Solution to Q1: (a) For the first part, Theorem 4.2 for p = 1 tells us that ( ˆϕ N ϕ, 1 ) n σ2 ZΓ 1 1. We have Γ 1 1 = 1/γ X (0). The parameter γ X (0), the variance of the sequence X t, can be estimated by the sample variance ˆγ X (0). We know that ˆϕ = ˆγ X(1) ˆγ X (0) = ˆρ X(1), i.e. ˆϕ is the sample correlation at lag 1, while σ 2 Z can be estimated by ˆσ 2 Z = ˆγ X (0) ˆϕ ˆγ X (0), see eq. (21) in the Lecture Notes. In summary, the CI for ϕ is given by ˆϕ ± z(1 α/2) 1 ˆσ Z 2 n ˆγ X (0), where z(1 α/2) is the appropriate quantile of the standard normal random variable. (b) For the second part, Theorem 4.2 for p = 2 tells us that ( ( ˆϕ 1, ˆϕ 2 ) T N (ϕ 1, ϕ 2 ), 1 ) n σ2 ZΓ 1 2. Note that σzγ is a 2 2 matrix, called limiting variance-covariance matrix. The diagonal entries correspond to asymptotic variance of corresponding estimators, for example, the first diagonal entry is the asymptotic variance of Var( ˆϕ 1 ) that is used to construct confidence interval. The limiting variance-covariance matrix is given in Example 4.4 (Lecture Notes): σ 2 ZΓ 1 2 = σ 2 Z [ 1 ϕ 2 2 ϕ 1 (1 + ϕ 2 ) ϕ 1 (1 + ϕ 2 ) 1 ϕ 2 2 Hence, the theoretical confidence interval for ϕ 1 and ϕ 2 is, respectively, σ Z ˆϕ 1 ± z α/2 1 ϕ 2 n 2, σ Z ˆϕ2 ± z α/2 1 ϕ 2 n 2. (1) These confidence intervals are not practical since ϕ 1, ϕ 2 are unknown. Practical confidence intervals are 1 ˆϕ 1 ± z α/2 n 1 ˆϕ 2 2, 1 ˆϕ2 ± z α/2 n 1 ˆϕ

2 2 Now, we have to give formulas for ˆϕ 1 and ˆϕ 2. Use Yule-Walker equations to obtain ( (ϕ 1, ϕ 2 ) T = Γ 1 2 (γ X(1), γ X (2)) T 1 γx (0)γ = X (1) γ X (1)γ X (2) γx 2 (1) γ2 X (0) γ X (1)γ X (1) + γ X (0)γ X (2) Replacing γ X (k) with their estimated values we obtain e.g. Marking scheme for Q1: Part (a) - 2 points; part (b) will not be marked. ˆϕ 1 = ˆγ X(0)ˆγ X (1) ˆγ X (1)ˆγ X (2) ˆγ 2 X (1) ˆγ2 X (0). (2) ). Q2. (Theoretical Question, 4 points). Maximum Likelihood Estimation for AR(p) models. Consider AR(1) model X t = ϕx t 1 + Z t, where Z t are i.i.d. normal random variables with mean zero and variance σ 2 Z. Derive MLE for ϕ and σ 2 Z. (Hint: You should get formulas as in Lecture Notes, but I need to see calculations). Solution to Q2: The likelihood function is given by ( 1 L(σ Z, ϕ) = ( exp 2π) n σz n 1 2σZ 2 Hence, the log-likelihood is Taking derivative w.r.t. ϕ yields and ) n (X i ϕx i 1 ) 2. (3) i=1 l(σ Z, ϕ) = log L(σ Z, ϕ) = n log σ Z 1 2σ 2 Z n X i 1 (X i ϕx i 1 ) = 0 i=1 n (X i ϕx i 1 ) 2. i=1 n i=1 ˆϕ MLE = X i 1X i n. (4) i=1 X2 i 1 Taking derivative w.r.t. σ Z yields and Replacing ϕ with the MLE yields n σ Z + 1 σ 3 Z σ 2 Z = 1 n ˆσ 2 MLE = 1 n n (X i ϕx i 1 ) 2 = 0 i=1 n (X i ϕx i 1 ) 2. i=1 n (X i ˆϕ MLE X i 1 ) 2. (5) i=1 Marking scheme for Q2: 1 point for some computation leading to (4), 1 point for the correct formula (4); 1 point for some computation leading to (5), 1 point for the correct formula (5). Total: 4 points. Q3. (Practical Question).

3 3 (a) We have already fitted AR(4) to US unemployment data. We estimated parameters using the Yule-Walker procedure. (b) Predict the next observation (remember about the mean!). (c) Predict the past observations and verify quality of the prediction by plotting the original values and the predicted values on the same graph. Solution to Q3: Note: You do not need to perform such extensive analysis as I did below. I marked the most important parts, required for this question, in blue. Plot of data US.month.ts<-ts(US.month,start=c(1996,1),freq=12) ; par(mfrow=c(1,1)); plot.ts(us.month.ts); US.month.ts Time Figure 1. US unemployment data Data do not look stationary. Do decomposition. par(mfrow=c(1,1)); plot(decompose(us.month.ts)); Note: you may notice that observed (that is the original data set) is longer than trend and hence random. The latter sequence is obtained by computing observed - trend - seasonal. This is due to the fact that the function decompose uses moving average smoothing to remove the trend. There would be no problem if one uses the exponential smoothing. Recover the stationary part from this decomposition: Stationary<-decompose(US.month.ts)$random; Stationary=Stationary[7:120; mean=mean(stationary); MyTimeSeries=Stationary-mean; Note: Stationary has six NAs at the beginning and six NAs at the end. This is due to the smoothing effect described above. I needed to get rid of those NAs. (a) We analyse the stationary part. n=length(mytimeseries); fit.ar.yw<-ar(mytimeseries,method="yule-walker"); fit.ar.yw

4 4 Decomposition of additive time series random seasonal trend observed Time Figure 2. Decomposition of US unemployment data You can see that the selected order is AR(4). The estimated parameters are Coefficients: Order selected 4 sigma^2 estimated as I store the coefficients as phi=c(fit.ar.yw$ar[1,fit.ar.yw$ar[2, fit.ar.yw$ar[3,fit.ar.yw$ar[4) We analyse residuals to check if the model is well-fitted. Since the model fitted is AR(4), we will have four NAs in the residuals. We need to remove them. par(mfrow=c(1,3)) length(fit.ar.yw$resid) # you will get 114. plot.ts(fit.ar.yw$resid[5:114); acf(fit.ar.yw$resid[5:114); pacf(fit.ar.yw$resid[5:114); The output is on Figure 3. Note that there is almost no dependence left, except of lag 10. Which could mean that there is some seasonal effect every ten months (since this is the time resolution for our time series). I will ignore it but one should not do prediction for a longer period than 10 months. (b) Now, I will do prediction for the past of the stationary part. Remember about the mean! length(mytimeseries) # you get 114 past.prediction=phi[1*mytimeseries[4:113+phi[2*mytimeseries[3:112 +phi[3*mytimeseries[2:111+phi[4*mytimeseries[1:110+mean*(1-sum(phi)); past.prediction=c(stationary[1:4,past.prediction) A comment here: since the fitted model is AR(4), we can predict the data in the stationary part numbered 5, 6,..., 114. Therefore, I added to the prediction vector the first four observations. par(mfrow=c(1,1)) plot.ts(stationary); points(past.prediction, type="p", col="blue") You can see on Figure 4 that the prediction follows the stationary part, except of few places where the stationary part has large absolute values. (c) Now, I will predict the next observations for the stationary part. We use the prediction formula for AR(4). Remember about the mean! future.prediction=phi[1*mytimeseries[n+phi[2*mytimeseries[n-1

5 5 Series fit.ar.yw$resid[5:114 Series fit.ar.yw$resid[5:114 fit.ar.yw$resid[5: ACF Partial ACF Time Figure 3. Residuals for AR(4) fit Stationary Time Figure 4. Past prediction for the stationary part +phi[3*mytimeseries[n-2+phi[4*mytimeseries[n-3+mean*(1-sum(phi)); future.prediction; Marking scheme for Q3: This question will not be marked Q4. (Practical/Theoretical Question, 4 points) (a) Type

6 6 My.TS<-LakeHuron; help(lakehuron); mean=mean(my.ts); My.Centered.TS<-My.TS-mean(My.TS); The first command loads data set LakeHuron which is in-built in R. The second command shows description of the data set. The third command centers your data set. (b) Fit AR(2) model using the Yule-Walker estimator. Obtain ˆϕ 1, ˆϕ 2, ˆσ 2 Z. fit.ar<-ar(my.centered.ts,method="yule-walker"); We did this in class! (c) Verify that the command ar leads to the correct Yule-Walker estimator. Type ACF<-acf(LakeHuron) and read ˆρ X (1) and ˆρ X (2). Type var(lakehuron) to get ˆγ X (0). Using these information, compute ˆγ X (1), ˆγ X (2). Create a vector (ˆγ X (1), ˆγ X (2)) and call it gamma.vector. Create a matrix ˆΓ 2 and call it Gamma.matrix. Type solve(gamma.matrix)%*%gamma.vector; and compare the obtained values with part (b). Solution to Q4: By typing acf(my.centered.ts); pacf(my.centered.ts) we obtain Series My.Centered.TS Series My.Centered.TS ACF Partial ACF PACF suggest that AR(2) is a good fit, with ϕ 1 > 0 and ϕ 2 < 0. By typing fit.ar<-ar(my.centered.ts,method="yule-walker"); fit.ar we obtain Call: ar(x = My.Centered.TS, method = "yule-walker") Coefficients:

7 7 Order selected 2 sigma^2 estimated as A side note: the same will be obtained by typing ar(my.ts,method= yule-walker ) since the software centers the data anyway. Thus, ˆϕ 1 = , ˆϕ2 = , ˆσ 2 Z = (6) The graphs show ACF and PACF of residuals. They are obtained by typing: length(fit.ar$resid) # obtain 98, you have to remove two NAs. par(mfrow=c(1,2)) acf(fit.ar$resid[3:98); pacf(fit.ar$resid[3:98) Series Noise Series Noise ACF Partial ACF They suggest that the residuals form a white noise - AR(2) fir is appropriate. By typing ACF<-acf(LakeHuron); ACF; var(lakehuron); we obtain Autocorrelations of series LakeHuron, by lag and We can read that rho1=0.832; rho2=0.610; We fill in the matrix and the vector gamma0=var(lakehuron); gamma1=rho1*gamma0; gamma2=rho2*gamma0; Gamma.matrix=matrix(c(gamma0,gamma1,gamma1,gamma0),byrow=T,2,2); gamma.vector=c(gamma1,gamma2); solve(gamma.matrix)%*%gamma.vector; The result is [,1 [1, [2, (***)

8 8 Series X Series X ACF Partial ACF Figure 5. ACF and PACF which agrees with (6) (there is some difference due to rounding). Marking scheme for Q4: 2 points for getting ˆϕ 1, ˆϕ 2, ˆσ 2 Z. 2 points for getting the same values in (6) and (***). Total 4 points Q5. (Practical Question) The following exercise shows that it is hard to identify AR model with p 2. Download BadData.txt. Denote by X your data set. (a) Based on ACF and PACF argue that an AR(3) model can be chosen. (b) Type fit.ar<-ar(x,method="mle"); fit.ar; What order has been selected? Denote this order by p. (c) Use p from (b) and type fit.arima<-arima(x,order=c(3,0,0)); fit.arima; fit.arima1<-arima(x,order=c(p,0,0)); fit.arima1; Why did MLE select p, not 3? Solution to Q5: (a) ACF and PACF are messy. Ignoring significant values for large lags of PACF, we can argue that an AR(3) model can be chosen. (b) Type fit.ar<-ar(x,method="mle"); fit.ar; The output is Call: ar(x = X, method = "mle")

9 9 Coefficients: Order selected 5 sigma^2 estimated as The order selected is 5. p=5; (c) Use p from (b) and type fit.arima<-arima(x,order=c(3,0,0)); fit.arima; fit.arima1<-arima(x,order=c(p,0,0)) fit.arima1; MLE selects p = 5 since the absolute value of AIC is smaller. What happens here is that the covariances and partial covariances are messy, we ignored them in (a), but they influence MLE. Marking scheme for Q5: This question will not be marked. Q6. (Theoretical/Practical Question, 7 points). In this question we develop Yule-Walker estimator in AR(1, 1) and ARMA(1, 1) model and study its numerical performance. (a) Numerical experiment for AR(1): Recall from lectures that in AR(1) model X t = ϕx t 1 + Z t the Yule-Walker estimator is ˆϕ = ˆγ X(1) ˆγ X (0) = ˆρ X(1), ˆσ 2 Z = ˆγ X (0) ˆϕˆγ X (1) = ˆγ X (0) ˆρ 2 X(1)ˆγ X (0). Load into R the file Data-AR.txt. (Just type Data=scan() and then copy and paste). This is data set generated from AR(1) model with ϕ = 0.8. Type var(data) to obtain ˆγ X (0). Type ACF<-acf(Data). Then type ACF. You will get ˆρ X (h), the estimators of ρ X (h) for different lags. The second entry will be ˆρ X (1). Via the formula above this is also ˆϕ. Write the final values for ˆϕ and ˆσ 2 Z. Compare your estimated ˆϕ with the true ϕ. (b) Consider ARMA(1, 1) model X t = ϕx t 1 + Z t + θz t 1, ϕ < 1, so that the sequence X t is causal. Apply the Yule-Walker procedure to get the estimators for ϕ, θ and σ 2 Z = Var(Z t ). HINT: You should get ϕ = γ X(2) γ X (1), γ X(1) = ϕγ X (0) + θσz 2, γ X (0) = σz [1 2 + (θ + ϕ)2 1 ϕ 2. (c) Numerical experiment for ARMA(1, 1): Load into R the file Data-ARMA.txt. (Just type Data=scan() and then copy and paste). This is data set generated from ARMA(1, 1) model with ϕ = 0.8 and θ = 1. Write the final values for ˆϕ, ˆθ and ˆσ 2 Z. Compare your estimated ˆϕ with the true ϕ. Which estimate is more accurate, for ARMA(1, 1) or for AR(1)? Solution to Q6: (a) After typing var(data) we obtain ˆγ X (0) = After typing ACF<-acf(Data) and ACF we get

10 10 The second entry is ˆρ X (1) = Thus, ˆϕ = Recall that the true parameter was ϕ = 0.8. Furthermore, ˆσ Z 2 = = (b) For ARMA(1,1) we have γ X (h) = ϕ h 1 γ X (1), h 2; γ X (1) = σz [(θ 2 ϕ(θ + ϕ)2 + ϕ) + 1 ϕ 2 γ X (0) = σz [1 2 (θ + ϕ)2 + 1 ϕ 2. Clearly, γ X (2)/γ X (1) = ϕ that gives the first equation. The third equation is for free. Also, we immediately the second equation by applying the above formulas for γ X (1) and γ X (0). (c) Now, numerical experiment for ARMA(1, 1). After typing ACF<-acf(Data) and ACF we get From this we read: ˆρ X (1) = 0.509; ˆρ X (2) = Now, from the formula above ˆϕ = ˆγ X(2) ˆγ X (1) = ˆρ X(2) = 0.204/0.509 = 0.4. ˆρ X (1) Type var(data) to get ˆγ X (0) = From this we get ˆγ X (1) = ˆρ X (1)ˆγ X (0) = = We take the system of two equations obtained from Yule-Walker procedure and we replace values with their estimators We obtain and Also ˆγ X (1) = ˆϕˆγ X (0) + ˆθˆσ 2 Z, ˆγ X (0) = ˆσ 2 Z [ 1 + (ˆθ + ˆϕ) 2 1 ˆϕ = ˆγ X (1) = ˆϕˆγ X (0) + ˆθˆσ 2 Z = ˆθˆσ 2 Z = ˆγ X (0) = ˆσ 2 Z [ ˆσ 2 Z = /ˆθ. 1 + (ˆθ + ˆϕ) 2 1 ˆϕ 2 = ˆθ [ 1 + (ˆθ + 0.4) The second equation has two solutions, take the one which is smaller than 1: ˆϕ = 0.4; ˆθ = 0.14, ˆσ 2 Z = Note that the true parameters were ϕ = 0.8; θ = 1; σ Z = 1. Hence, the fact that MA part is present messes up estimation of the autoregressive coefficient ϕ. Marking scheme for Q6: Part (a) - 1 point for each correct value of ˆϕ and ˆσ 2 Z. Total 2 points for part (a); Part (c): 1 point each for each correct value of ˆϕ, ˆθ and ˆσ 2 Z. 1 point for a comment on accuracy of estimation of the AR coefficient. Total 4 points for part c). Part b) - 1 point for making a link between formulas for covariances and formulas that appear in the HINT. Total: 7 points... Q7. (Theoretical-Practical Question, 3 points). (a) One hundred observations from AR(1) yield the following sample statistics: x = 0, ˆγ X (0) = 1.1, ˆρ X (1) = Find the Yule-Walker estimators of ϕ and σ 2 Z. Write the confidence interval for ϕ. If X 100 = 1.5, what is the predicted value of X 101?

11 11 (b) Two hundred observations from AR(2) yield the following sample statistics: Solution to Q7: x = 3.82, ˆγ X (0) = 1.15, ˆρ X (1) = 0.427, ˆρ 2 = Find the Yule-Walker estimators of ϕ 1, ϕ 2 and σ 2 Z. Is the estimated model causal?. If X 100 = 3.84 and X 99 = 3.26, what is the predicted value of X 101? (a) We have the following formulas for Yule-Walker estimators in AR(1) case: ˆϕ = ˆρ X (1) = ˆρ X (1) = 0.42, ˆσ 2 Z = ˆγ X (0) ˆϕˆγ X (1) = ˆγ X (0) ˆϕ 2ˆγ X (0) = = The confidence interval for ϕ is 1 1 ˆϕ ± z α/2 n ˆσ ˆγX (0). Choose α = 0.05; then z = Thus ± = 0.42 ± The prediction for the 101st is ˆϕ X 100 = The squared error of the prediction is ˆσ 2 Z(1 + 1/n) = (1 + 1/100). (b) From the data we compute ˆγ X (1) = ˆρ X (1) ˆγ X (0) = 0.49, ˆγ X (2) = ˆρ 2 ˆγ X (0) = We have to solve ( ) ( ) ϕ1 = Γ 1 ˆγX (1) ϕ 2 2 ˆγ X (2) where We obtain The predicted value is Γ 2 = [ ˆγX (0) γ X (1) ˆγ X (1) γ X (0) ( ) ( ˆϕ = ˆϕ ˆX = ˆϕ 1 (X ) + ˆϕ 2 (X ) = 0.27 ( ) ( ) = Thus The autoregressive polynomial is ) ˆX 101 = = ϕ(z) = z 0.36z 2. It has two solutions: z 1 = 1.33 and z 2 = Both have absolut values bigger than one, thus the model is causal and stationary. Marking scheme for Q7: 3 points for part (a), one for each subpart. Part (b) will not be marked. Q8. (Practical Question) Find a data set. Fit an ARMA model. Estimate parameters. Do relevant prediction You have to provide relevant graphs and output that support your claims. Marking scheme for Q8:

12 12 This question will not be marked, however, I will provide some comments to your solutions.