ARIMA Models. Jamie Monogan. January 25, University of Georgia. Jamie Monogan (UGA) ARIMA Models January 25, / 38

Transcription

1 ARIMA Models Jamie Monogan University of Georgia January 25, 2012 Jamie Monogan (UGA) ARIMA Models January 25, / 38

2 Objectives By the end of this meeting, participants should be able to: Describe the logic of the Box-Jenkins modeling strategy. Define stationarity and describe the processes of a stationary series. Solve for algebraic properties of difference equations of ARIMA processes. Explain why Maximum Likelihood Estimation is essential for ARIMA models and how MLE is tailored to time series needs. Identify, estimate, and diagnose ARIMA models. Perform a static time series regression with ARIMA errors. Jamie Monogan (UGA) ARIMA Models January 25, / 38

3 Structure, Error, and Procedure Modeling Form Regression y t = {β 0 + β 1 x 1 + β 2 X 2 + } + u t = {structure} + error The two components are structure and error: Y = structure + error Box-Jenkins y t = [transfer function] + ARIMA Model y t =f(x) + N t So we start out working on ARIMA models of error aggregation, the N t, and then later develop transfer functions as tests of theories we care about. (Opposite what we usually do.) The causal flow of the transfer function cannot be observed until we successfully model the error processes. Jamie Monogan (UGA) ARIMA Models January 25, / 38

4 Structure, Error, and Procedure How a Time Series is Produced We assume the data generating process is: a t Linear filter z t A white noise input is systematically filtered into the observed time series. Getting Back to White Noise a t Linear filter z t implies z t Inverse of Linear filter a t So, if we can solve for z= f(a), then we can invert f and produce a (which is white noise). Jamie Monogan (UGA) ARIMA Models January 25, / 38

5 Structure, Error, and Procedure The Box-Jenkins Procedure Identification What class of models probably produced z t? Estimation What are the model parameters? Diagnosis Are the residuals, a t, from the estimated model white noise? Empirically Identifying the Error Process We can infer the data generating process because knowing the mathematics, we know the empirical signature. We will develop the signatures of a family of error aggregation models that are autoregressive (AR), integrated (I), moving average (MA) and all combinations ARIMA(P,D,Q) Jamie Monogan (UGA) ARIMA Models January 25, / 38

6 Structure, Error, and Procedure AR(1) Example AR(1): A very important special case Notation: AR(1) means autoregressive, 1st order Only the first lag of z appears in the equation z t = θ 0 + φ 1 z t 1 + a t What is its signature? To answer that question it is useful to transform the equation into shock form, where z is a function of all previous a s. z t = θ 0 + a t + φ 1 a t 1 + φ 2 a t 2 + φ 3 a t 3 + φ t 1 a 1 Jamie Monogan (UGA) ARIMA Models January 25, / 38

7 Structure, Error, and Procedure AR(1) Example AR(1) That s an ugly equation that has T terms, but it has useful information about the expected association of each of the a s with z t. At lag 1: φ, that is φ 1 2: φ 2 3: φ 3 4: φ 4 k: φ k etc. Jamie Monogan (UGA) ARIMA Models January 25, / 38

8 AR(1) Structure, Error, and Procedure AR(1) Example Since φ is constrained to be <1.0, that means that each exponential power of φ is a progressively smaller number, looking like: Jamie Monogan (UGA) ARIMA Models January 25, / 38

9 Structure, Error, and Procedure AR(1) Example Next Steps If we observe an empirical series that shows this (very common) pattern of autocorrelation (and a couple other details), we judge it to be AR(1) This is the IDENTIFICATION stage: we re using empirical evidence such as correlograms to determine the error process. Once we ve tentatively judged the class of model, we estimate the parameters of such a model using MLE (ARIMA ESTIMATION, more later). After we estimate the model, calculate the residuals, a t. Now the really neat part: If our judgment was correct, a t, the estimated residuals must be white noise and we know how to test for that property! (DIAGNOSIS) If it is white noise, then we can use this filtered series for our analysis. Jamie Monogan (UGA) ARIMA Models January 25, / 38

10 Stationarity and Integration Stationarity and Integration A stationary series is one that tends to return to some equilibrium level after being disburbed. A nonstationary series, or an integrated series, has no equilibrium. The most common integrated series is the random walk (i.e., DJIA). Box-Jenkins models are defined only for stationary time series. The good news: Integrated series can be made stationary by differencing them. (Which you know how to do.) How to Know Stationarity The ACF of a stationary series tends to approach zero after just a few lags And stay there. Integrated series show systematic behavior over very long lag lengths. In the regression tradition, we will develop the Dickey-Fuller test for unit roots. Jamie Monogan (UGA) ARIMA Models January 25, / 38

11 Stationarity and Integration Macropartisanship: A Non-stationary Series Jamie Monogan (UGA) ARIMA Models January 25, / 38

12 Notation AR(1) Notation: Box-Jenkins and Econometrics Enders: y t = α 0 + α 1 y t 1 + ɛ t Box-Jenkins: z t = θ 0 + φ 1 z t 1 + a t For MA processes, α becomes β and φ becomes θ Jamie Monogan (UGA) ARIMA Models January 25, / 38

13 Notation Notation for the Random Walk Process (an I(1) series) z t = z t 1 + a t z t z t 1 = a t (subtracting z t 1 ) B 0 z t Bz t = a t (by definition of B) (B 0 B)z t = a t (factoring by z t ) (1 B)z t = a t (the result) (1 B)z = a Jamie Monogan (UGA) ARIMA Models January 25, / 38

14 Notation The General ARMA(P,Q) Model p q z t = θ 0 + φ i z t i + θ i a t i + a t i=1 i=1 In Enders (Regression) Notation: p q y t = α 0 + α i y t i + β i ɛ t i + ɛ t i=1 i=1 Jamie Monogan (UGA) ARIMA Models January 25, / 38

15 Notation The Shock Form of AR and I Models The AR(1) Model Functional: z t = φz t 1 + a t where -1 < φ < 1 Backshift: (1 - φb)z t = a t Shock form: z t = a t + φa t 1 + φ 2 a t 2 + φ 3 a t φ T a 0 Jamie Monogan (UGA) ARIMA Models January 25, / 38

16 Notation The Shock Form of AR and I Models The MA(1) Model Functional: z t = θ 0 + a t - θ 1 a t 1 Backshift: z t = θ 0 + (1 - θ 1 B)a t The shock form is MA, so there is no expansion Jamie Monogan (UGA) ARIMA Models January 25, / 38

17 Notation The Shock Form of AR and I Models AR(1) inverted to shock form z t = φz t 1 + a t z t 1 = φz t 2 + a t 1 (from stationarity) z t = φ(φz t 2 + a t 1 ) + a t substituting z t = φa t 1 + φ 2 z t 2 + a t z t = φa t 1 + φ 2 a t 2 + φ 3 z t 3 + a t... z t = a t + φa t 1 + φ 2 a t 2 + φ 3 a t φ T a t T thus the ACF we expect to see is exponential decay: 1+ φ + φ 2 + φ Jamie Monogan (UGA) ARIMA Models January 25, / 38

18 I(1) in shock form Notation The Shock Form of AR and I Models z t = a t + a t 1 + a t 2 + a t a 0 Note difference from AR(1): There are no decay parameters on the old shocks. Their influence persists forever. Jamie Monogan (UGA) ARIMA Models January 25, / 38

19 Notation The Shock Form of AR and I Models MA(1) is in shock form in normal notation z t = θ 0 + a t - θ 1 a t 1 ACF: E(ρ 1 ) = -θ 1 /(1 + θ 2 1 ) Crude empirical rule of thumb: ACF(1)= -θ 1 /2. Which implies ACF(1) negative and less than.5 in absolute value. Jamie Monogan (UGA) ARIMA Models January 25, / 38

20 Notation The Shock Form of AR and I Models The Partial Autocorrelation Function The PACF shows the autocorrelation at lag k controlling for all previous lags. Thus it shows the effects at lag k which could not have been predicted from lower lags. In effect then, it shows the independent effects of processes at lag k. PACF(1) = ACF(1) Jamie Monogan (UGA) ARIMA Models January 25, / 38

21 Identification Issues Some Low Order Models White noise z = a Random walk z t - z t 1 = a t (1 - B)z = a z = a [i.e., cumulated white noise] AR(1) z t = φz t 1 + a t (1 - φb)z = a Note that (1 - φb) = a for φ=1.0 is then exactly a random walk. φ = 1.0 is called a unit root. Jamie Monogan (UGA) ARIMA Models January 25, / 38

22 Identification Issues Low Order with MA Components MA(1) z t = a t - θa t 1 z = (1 - θb)a IMA(1,1) (1 - B)z = (1 - θb)a Thus an IMA(1,1) is simply a MA(1) operating on first differences. ARMA(1,1) (1 - φb)z = (1 - θb)a Jamie Monogan (UGA) ARIMA Models January 25, / 38

23 Identification Issues Cookbook Rules for Identification For More Detail: Enders 2010, 68 AR(P) Exponential decay in the ACF, P significant spikes in the PACF I(1) Slow decay in the ACF, 1 significant spike in the PACF MA(Q) Q significant spikes in the ACF, exponential decay in the PACF Jamie Monogan (UGA) ARIMA Models January 25, / 38

24 Least Squares and MLE Estimation Some ARIMA models, e.g., AR(1), are essentially linear and could be estimated by least squares. For example z t = φ 1 z t 1 + a t can be estimated by least squares regression if you just drop the first case. R: z <- ts(data$z1) l.z <- lag(z, -1) data2 <- ts.union(z, l.z) reg.1 <- lm(z l.z, data=data2) Stata: tsset month, then reg z l.z The coefficient on the lagged dependent variable is a LS estimate of φ 1 Jamie Monogan (UGA) ARIMA Models January 25, / 38

25 Estimation LS and MLE, cont. In practice ARIMA software uses a generalized maximum likelihood algorithm for all ARIMA models. The φ s estimated by LS and ML are not identical, but the difference is nearly always trivial. This is not a case like OLS, where LS and ML solutions are proven identical when OLS assumptions hold. Jamie Monogan (UGA) ARIMA Models January 25, / 38

26 Estimation Maximum Likelihood Unmasked Maximum Likelihood Estimation is really nothing more than efficient trial and error. It has three components: 1 A function to be maximized, the log of likelihood for the equation Why log instead of likelihood itself? 2 An algorithm for generating efficient guesses of parameter values 3 Starting values for the parameters. Jamie Monogan (UGA) ARIMA Models January 25, / 38

27 Estimation Maximum Likelihood Estimation for the Nonlinear MA(1) Case MA(1) Model: z t = θ 0 - θ 1 a t 1 + a t a t is Normal(0,σ 2 ) Assume: a 0 = 0 (Conditional maximum likelihood) Problem: Find θ 1 such that L(θ 1 z, a 0 = 0) is a maximum Jamie Monogan (UGA) ARIMA Models January 25, / 38

28 Estimation Log of Likelihood for ARIMA Estimation LL(θ) = T 2 log(2π) T 2 log(σ2 ) This applies to any ARIMA(P,D,Q) model. T t=1 a 2 t 2σ 2 When the likelihood is known, as here, the problem reduces to finding out how to estimate θ and a t. Jamie Monogan (UGA) ARIMA Models January 25, / 38

29 Estimation MA(1) illustration z t = -θ 1 a t 1 + a t + θ 0 Drop θ 0 for simplicity Note inherent nonlinearity of -θ 1 a t 1 Both θ and a t 1 are unobserved quantities to be estimated Step by step Presume for the moment that we somehow know θ How do we estimate a t? Except for the first case; just solve one case at a time: z t is given Jamie Monogan (UGA) ARIMA Models January 25, / 38

30 Estimation Solve the MA(1) Equation for a t Just Algebra 1 z t = -θa t 1 + a t 2 z t + θa t 1 = + a t (adding θa t 1 to both sides) 3 a t = z t + θa t 1 (reversing) So, beginning at time zero, if we know a 0, we can solve for a 1, if we know a 1, we can solve for a 2, if we know a 2, we can solve for a 3,recursive all the way to a T So assuming or computing a value for a 0 is the key to everything. Jamie Monogan (UGA) ARIMA Models January 25, / 38

31 Estimation Conditional Maximum Likelihood Conditional Maximum Likelihood E(a t )=0.0; therefore Assume a 0 = 0.0 Then maximize L conditional on that assumption The assumption will be false, but its effect is transient That is guaranteed by the stationarity condition Jamie Monogan (UGA) ARIMA Models January 25, / 38

32 Estimation Unconditional Maximum Likelihood with Backforecasting Unconditional Maximum Likelihood Backforecast a 0 Because the backforecast is a product of known z and maximum likelihood estimates of θ and a, it will be optimum. Hence full ML is preferred to Conditional ML Why is this optimal? Because backforecasting makes an assumption about unobservables at the end of a series, And because the error in that assumption is transient, The backward forecast at the origin of the series will be unaffected by the transient error. Jamie Monogan (UGA) ARIMA Models January 25, / 38

33 Estimation Unconditional Maximum Likelihood with Backforecasting Backforecasting for MA(1) Given: z t = -θ 1 a t 1 + a t Or: z t - a t = -θ 1 a t 1 Or: (z t - a t )/-θ 1 = a t 1 Or: a t 1 = (z t - a t )/-θ 1 Then if we let a T = 0, we can solve recursively for each previous a t 1, including a 0 Jamie Monogan (UGA) ARIMA Models January 25, / 38

34 Extending ARIMA to Static Regressions Extension: The Multivariate Case We can now consider a case where we have a static time series regression with ARIMA errors. Y = β 0 + β 1 x 1 + β 2 x β k x k + N t This is static because the causal flow from x to y is not a function of time, just β. Jamie Monogan (UGA) ARIMA Models January 25, / 38

35 Extending ARIMA to Static Regressions Software The arima commands in R and Stata can handle any number of right-hand-side regressors, and ARIMA errors. R: mod.1 <- arima(data$y, order=c(1,0,0), xreg=cbind(data$x1,data$x2)) Stata: arima y x1 x2,ar(1) Both of these are for the AR(1) case and will produce a ML regression with ML estimated φ 1 simultaneously. This is still a very limited extension, because static models are not often appropriate specifications for longitudinal causality. If they were, the course could terminate now. But this is the step Hibbs (1974) took. Jamie Monogan (UGA) ARIMA Models January 25, / 38

36 Extending ARIMA to Static Regressions Percent Identifying as Liberal Over Time A Real Example of a Static ARIMA Regression Citation: OLS MLE for AR(1) Estimate S.E. Estimate S.E. Great Society intervention Party control duration Post-intervention trend Intercept ˆφ Radj N=70 Ellis, Christopher & James A. Stimson On Symbolic Conservatism in America. Presented at the APSA Annual Meeting, Chicago, September Jamie Monogan (UGA) ARIMA Models January 25, / 38

37 Extending ARIMA to Static Regressions Two Issues in Time Series Regressions 1 Correlated errors in the residuals violate OLS assumptions, producing inefficient β and biased σ 2, t, and p. That s the one we ve solved. 2 Dynamics: y is likely to be caused by previous values of x and y. This is the big one, producing biased and inconsistent β. This is a violation of the Gauss-Markov assumption of proper functional form. That is the focus of much of the rest of the course. Jamie Monogan (UGA) ARIMA Models January 25, / 38

38 Extending ARIMA to Static Regressions For Next Time Bring me a copy of the article you want to replicate. Write down the shock form for an AR(1) process. If you replace every φ with a θ, what kind of MA process is it? Why might this be worth knowing? Data exercise: Download the simulated dataset posted at For series z1-z6, identify, estimate, and diagnose the ARIMA process. Write a sentence or two justifing your identification decision. Present the results and your sentences in some concise manner. Reading: Enders sections Gujarati & Porter sections Jamie Monogan (UGA) ARIMA Models January 25, / 38