Chapter 12: An introduction to Time Series Analysis
Introduction In this chapter, we will discuss forecasting with single-series (univariate) Box-Jenkins models. The common name of the models is Auto-Regressive Integrated Moving Average (ARIMA) models. Such models that can handle seasonal patterns are called SARIMA models. Time domain time series analysis (ARIMA models) is based on the autocorrelation function. A function called the spectral density can account for the variation in a time series by cyclic components at different frequencies may be the basis for time series analysis, in which case we talk about frequency domain time series analysis. In this short treatment of the analysis of time series we will describe only the time domain approach to inferences about time series. ARIMA models attempt to forecasts future time series values for a particular series based on past values of the variable being forecast.
Introduction Time series data consists of pairs of observations (t, z t ), t = 1,..., n taken over equally spaced, discrete time intervals. Since the variable observed is measured only at discrete time intervals the data is said to be discrete data even though the variable observed may be continuous. A mechanical pen recording continuously the temperature for every instant of time on a roll of paper would be an example of continuous data. Examples of discrete time series are: (1) Rainfall on successive days; (2) Total sales figures each week for a number of successive weeks; (3) Yearly population values for a country or town; (4) Numbers of women unemployed each week in a city; (5) Daily stock prices; (6) Number of sunspots each year over a number of years;
Introduction (7) Dow Jones indexes for various goods; (8) Number of airline passengers per year; (9) Average monthly air temperature in a given local over a number of months; (10) Bank interest as recorded on each day; (11) Coal production per year; (12) Number of housing permits issued in a certain local per year; (13) Amount of real-estate loans per month issued in a given local; (14) Weekly profit margin for a particular firm; (15) College enrolment in a province per year; (16) Total yearly exports.
Introduction In regression analysis and many other types of statistical analysis we assume individual observations are independent, however in time series data we assume that individual observations are dependent, that is correlated. Often we assume that the current observation depends strongly on the previous observation in the time series, so that pair of observations separated by one time unit are highly correlated, that is have strong first order autocorrelation. It is common to see a time series model which relates the current z value to the two most recent observations. Thus ARIMA models are often used to make short-term forecasts. To build a time series model and make forecasts we usually need at least 50 observations, because we need a large sample sizes to estimate the autocorrelation accurately.
Stationary Time Series The ARIMA methods are useful for stationary time series. A time series is said to be stationary if there is no systematic change in the mean and the variance, and if strictly periodic variations have been removed. That is a stationary time series has a mean, variance, and autocorrelation function that are essentially constant through time. Note that there is a proper probability definition of stationarity in most books written on time series, and the mathematical student should consult one such book. Given n time series observations the usual formulas may be used to estimate the mean and variance, z = 1 n n i=1 z i and s 2 z = 1 n n (z i z) 2. i=1 Basically an ABIMA model is a simple or perhaps complicated equation relating a current z value to its past values.
Stationary Time Series Example 12.1 The following equation (sometimes called an autoregressive process of the first order) is an example of an ARIMA model, z t = C + φ 1 z t 1 + α t. In this equation C is a constant term, α t is the current random shock to the system, φ 1 is a parameter (like a regression coefficient) that relates the current z t value to the immediate past z t 1 value. This is an auto-regressive model of order one, denoted as ARIMA(1,0,0), which is a sub-case of the general ARIMA model. To fit this model to a time series data set in R package we use arima(data, order = c(1, 0, 0))
Stationary Time Series Example 12.2 Consider the following: z t = C θ 1 α t 1 + α t. Note that, C again is a constant term, and this model uses the parameter θ 1 to relate the immediate past shock to the current value of z t, so that the current value of z t now depends on an immediate past shock α t 1 and a current shock α t. This model is called a moving average model of order one, denoted as ARIMA(0,0,1), which is also a sub-case of the general ARIMA model. To fit this model to a time series data set in R package we use arima(data, order = c(0, 0, 1))
Stationary Time Series Example 12.3 Consider the following z t = C + φ 1 z t 1 θ 1 α t 1 + α t. This model is a mixed model, it contains both AR (autoregressive) and MA (moving average) terms. It is an ARIMA(1,0,1) process because the AR order is one and the MA order is one. To fit this model to a time series data set in R package we use arima(data, order = c(1, 0, 1))
Paradigm for the Box-Jenkins Modelling Procedure To fit the Box-Jenkins models we follow a four step procedure. (1) Identify the appropriate model. There are many (an infinite number of models) and the first step is to select a possible appropriate one. We will discuss, shortly, two major tools for selecting an appropriate model. (2) Fit the model chosen in (1) by using available time series data. The estimation procedure is usually based on maximum likelihood procedures or least squares. Under certain normality assumptions (random shocks normally distributed) least squares estimators are equivalent to maximum likelihood estimators. (3) Check the fitted model obtained in (2) for adequacy, for example, check the residuals. If the model is not adequate, then go back to Step (1). (4) If the model fitted in (2) is proven to be adequate in (3), then we may use the model to make forecasts.
Paradigm for the Box-Jenkins Modelling Procedure There are two functions which we calculate that help us to identify what model we should fit: (1) The ACF, the autocorrelation function and (2) The PACF, the partial autocorrelation function. We may use the R package to calculate both these functions, corresponding R commands are: (1) acf(data) and pacf(data). These commands will automatically make ACF and PACF graphs. Figure : ACF and PACF Plots
How do these graphs helps? Associated with each ARIMA model we would have a theoretical ACF and a theoretical PACF. The procedure is to estimate from the time series data the theoretical ACF and PACF. The estimates are called the empirical ACF and empirical PACF. We then compare the empirical functions to the theoretical functions. We will pick the model which have the theoretical ACF and PACF that agrees as closely as we can decide to the empirical ACF and PACF. This tentative model is then fitted and used for forecasting if it is adequate, otherwise we would try again to find a better model.
The Definition of the Empirical ACF and Empirical PACF For a stationary processes, the empirical ACF and empirical PACF are defined as following: (1) The Empirical ACF: Given discrete time series (t, z t ), t = 1,..., n, the autocorrelation coefficient at lag 1 is defined as r 1 = n 1 t=1 (z t z)(z t+1 z) n t=1 (z t z) 2. Similarly, we can define the autocorrelation coefficient at lag 2, lag 3 and so on. The autocorrelation coefficient at lag k is defined as r k = n k t=1 (z t z)(z t+k z) n t=1 (z t z) 2 A plot of r k against k is called the empirical correlogram.
The Definition of the Empirical ACF and Empirical PACF (2) The Empirical PACF: The PACF is an attempt to measure the correlation between points in the time series lagged (separated) by say k in such a way that the effects of the intervening time series z values between z t, and z t+k are accounted for. The symbol for the theoretical PACF is φ kk and the symbol for the empirical (estimated) PACF is ˆφ kk. Note that ρ k = φ kk when we have k = 1 since with lag one there are no intervening points. The calculation of the PACF is difficult but one way in which an estimate can be made is in terms of the autocorrelation matrix. Note that for a series of length n we may write a matrix of all the possible autocorrelations. For example for a series of length two (note we could not estimate correlation from a series of length 2, in fact, it is recommended that a series be of length 50 before we try and estimate correlations) we could have one autocorrelation ρ 1 and A matrix ( ) 1 ρ1 ρ 1 1
The Definition of the Empirical ACF and Empirical PACF For a series of size three we would have 1 ρ 1 ρ 2 ρ 1 1 ρ 1 ρ 2 ρ 1 1 Clearly for a series of length n we would have an n n matrix containing n 1 different autocorrelations. Of course one or more could be zero. Now the first PACF is define by the second by φ 11 = ρ 1, 1 ρ 1 ρ 1 ρ 2 φ 22 = 1 ρ 1 ρ 1 1 The same pattern is repeated for third and any higher order PACF. We can estimate these PACF s by replacing the autocorrelations by the estimated value r k. That is the matrix in the numerator if formed
The Definition of the Empirical ACF and Empirical PACF from the autocorrelation matrix in the denominator by replacing just the last column by the vector of possible correlations (ρ 1, ρ 2,..., ρ n ). Note that, (1) Stationary autoregressive (AR) processes have theoretical ACF s that decay or damp out toward zero (that is the r k s decrease in magnitude as k increases, r k k). But they have theoretical PACF s that cut off sharply to zero after a few spikes. The lag length of the last PACF spike equals the AR order (p) of the process. For example the AR(1) processes has one only PACF spike, all other φ kk are zeros. (2) Moving average (MA) processes have theoretical ACF s that cut off to zero after a certain number of spikes. The lag of the last ACF spike equals the MA order of the process. Their theoretical PACF s decay or die out toward zero (that is the φ kk s decrease in magnitude as k increases, φ kk k). Properties of a good time series model: (1) It uses the smallest number of coefficients needed to explain the data.
Further Notations (2) It is stationary ( that is it has AR coefficients which satisfy certain mathematical inequalities). (3) It is invertible (has MA coefficients which satisfy certain mathematical inequalities. (4) It had coefficients that are significant. (5) It has uncorrelated residuals. (6) It makes forecasts that are at least sensible. We need some further notation to represent complicated models and because most books and software packages use this additional notation. To reference most works on time series we must understand the standard notation. The Box-Jenkins approach uses two operators that permits compact representation of the models. The two operators are (1) The backshift operator, B, defined by B(z t ) = z (t 1).
Further Notations (2) The difference operator,, defined by (z t ) = z t z (t 1). Note that we have the relationship = 1 B. Thai is (z t ) = z t B(z t ). Furthermore, d = (1 B) d. Now, we take a look on some of the following examples: B 2 (z t ) = x (t 2), and B k (z t ) = x (t k). (1 B)z t = z t B(z t ) = z t z (t 1). Using the backshift operator, we can write the AR(2) time series model z t = C + φ 1 t (t 1) + φ 2 t (t 2) + α t as (z t φ 1 t (t 1) φ 2 t (t 2) ) = C + α t,
Further Notations which is equivalent to (1 φ 1 B φ 2 B 2 )z t = C + α t. Letting Φ 2 (B) = (1 φ 1 B φ 2 B 2 ), we can further simplify the above notation by Φ 2 (B)z t = C + α t. Similarly, we can represent a MA(2) time series model z t = C + α t θ 1 α(t 1) θ 2 α(t 2) by z t = C + (1 θ 1 B θ 2 B 2 )α t = C + Θ 2 (B)α t, where, Θ 2 (B) = (1 θ 1 B θ 2 B 2 ). Now we can write a ARIMA(2 0 2) model as Φ 2 (B)z t = C + Θ 2 α t.
Stationarity and Invertibility Stationarity In general, we can write a general ARIMA (p 0 q) model as where, Φ p (B)z t = C + Θ q α t, Φ p (B) = (1 φ 1 B φ 2 B 2... φ p B p ) and Θ q (B) = (1 θ 1 B θ 2 B 2... θ q B q ). We will discuss the general ARIMA(p d q) model later. In this section, we introduce two important characteristics of a time series. Stationary implies that the AR coefficients must satisfy certain conditions. Usually after estimating a model you check that the AR coefficients satisfy the following stationarity conditions. (1) Stationarity condition for AR(1): The requirement is the the coefficient in absolute value be less than one, that is φ 1 < 1.
Stationarity and Invertibility Why do we need stationary? In fact this is the requirement for all ARIMA(1,0,q) models. In practice we do not know φ 1 so we check the estimate ˆφ 1. (2) Stationarity condition for AR(2) or ARIMA(2,0,q) models: The requirement is that φ 2 < 1 φ 2 + φ 1 < 1 φ 2 φ 1 < 1. Note that all three conditions must hold for the ARIMA(2, 0, q) models to be stationary. We apply the conditions to the estimated parameters ˆφ 1 and ˆφ 2. Note that o check AR processes of order p > 2 is difficult, however a necessary condition for stationarity is that φ 1 + φ 2 +... + φ p < 1. Stationarity is necessary because a realization of a time series represents only one sample observation at each time point. If the mean
Stationarity and Invertibility How to checking for stationarity? changes with time then we would have only one observation at time point t to estimate the mean at that time point. We cannot start history again to get two realization of the series at each point in time. If we had two observations at a point in time then we could estimate the mean at that point in time by averaging the two observations. However we have only one observation at any point in time consequently we require stationarity, that is, that the mean value is not changing over time. If the mean is not changing over time then we can estimate the mean by averaging observations over time. Same thing applies to estimation of the variance and of the covariance. Consequently we require that neither the mean, nor the variance, nor the covariance change over time, i.e. that the time series be stationarity in terms of these parameters. This is usually called secondorder stationarity or weakly stationary. For normal processes this second-order stationarity implies strict stationarity since the multivariate normal distribution is completely characterized by its first
Stationarity and Invertibility Invertibility and second moments. To check the stationarity of a time series in practice, we follow the following steps: (1) Check the time series plot to see if either the mean or the variance appears to be changing over time dramatically. (2) Check the ACF, it should be decreasing rapidly to zero and reach zero at roughly lag 5 or 6. This means that the t-value for r 5 or r 6 should indicate that the autocorrelation is not significantly different from zero. (3) Check if the AR coefficients satisfy the inequality restrictions discussed above. Note that the above stationarity inequalities apply only to AR coefficients in general ARIMA(p, d, q) model. Invertibility implies that the MA coefficients must satisfy certain conditions. Usually after estimating a model you check that the MA coefficients satisfy the following invertibility conditions. (1) Invertibility for MA(1) processes: For an MA(1) or ARIMA(p,0,1) process, invertibility requires that the absolute value of θ 1 be less
Non-stationarity in the mean than one, i.e., θ 1 < 1. (2) Invertibility for MA(2) processes: For an MA(2) or ARMA(p,0,2) process to be invertible we must satisfy that θ 2 < 1 θ 2 + θ 1 < 1 θ 2 θ 1 < 1. Again for higher order MA processes (q > 2) it is complicated to check invertibility but a necessary condition for MA models is that θ 1 + θ 2 +... + θ p < 1. Note that the invertibility inequalities apply only to MA coefficients in general ARIMA(p, d, q) model. Our common sense tells us that in general more recent events (shocks) should have larger coefficients and more past events should have re-
Differencing duced weights. Invertibility guarantees that the weights on past observations will decline as we move further into the past. The general formula for an ARIMA process is denoted by AR1MA(p, d, q). In this model we now understand the meaning of AR and the p which indicates the order of the AR process, we understand MA and the q which indicates the order of the MA process. We have left to explain the meaning of the letter I and the the letter d. We will tackle this in the next lecture. The models discussed in previous require that the series be stationary. It may be that the series is stationary except with respect to the mean which may be, for example, increasing or decreasing with time. We now look at how we can handle a series that is non-stationary in the mean. If a different segments of a series behave much like the rest of the series after we allow for changes in level and/or slope then the series may be transformed into a stationary series simply by differencing. Differencing is a type of filtering, which can be used to remove trend.
Differencing Basically we difference a given series until it becomes stationary. Suppose we have a non-seasonal series and we denote the original observations by {y t, t = 1, 2,..., n} then a new differenced series, say, {z t, t = 1, 2,..., n} can be formed by z t = y t = y t y t 1 A series z t, created in this way by making first differences is called a first differences of y t. We hope that this series of first differences will have a constant mean. It is surprising how often simple first differences will produce a series with constant mean. Recall again that = (1 B). Now if first differences do not work then we try second differences. We can obtain second differences by first differences of the the first differences series z t. Let us call the series of second differences w t, then we have w t = z t z t 1 = (y t y t 1 ) (y t 1 y t 2 ) = y t 2y t 1 + y t 2
Differencing thus w t is the original series differenced twice. Note that this can be done using the difference operator or the backshift operator: or w t = (1 B) 2 y t = (1 2B + B 2 )y t = y t 2y t 1 + y t 2 w t = 2 y t = ( y t ) = (y t y t 1 ) Recall that, and = y t y t 1 = (y t y t 1 ) (y t 1 y t 2 ) = y t 2y t 1 + y t 2. Φ(B) = (1 φ 1 B φ 2 B 2 φ p B p ), Θ(B) = (1 θ 1 B θ 2 B 2 θ q B q ), We are now in a position to write the general nonseasonal ARIMA model, we can indicate the ARIMA (p, d, q) by Φ(B) d y t = C + Θ(B)α t.
Differencing For example the ARIMA (1, 1, 1) model is y t y t 1 = C + φ 1 (y t 1 y t 2 ) θ 1 α t 1 + α t. In the compact notation we can write this model as φ(b) 1 y t = C + θ(b)α t, with φ(b) = 1 θ 1 B and θ(b) = (1 θ 1 B). We can see that d stands for the number of times a realization of a series must be differenced to achieve a stationary mean. Usually not more than two times, i.e. usually d 2. Remarks on d: (1) If a series is nonstationary its estimated acf will decay very slowly and we should try differencing the series to produce a series which is stationary. (2) If the estimated AR coefficients do not satisfy the AR stationarity conditions we should try differencing.
The I in the term ARIMA (3) If we look at a time series plot and see that segments of the series differ only in level of the time series, then we should try d = 1. (4) If we look at a time series plot and see that both the level and the slope of segments of the series appear to be changing through time then we should try d = 2. Fitting an ARIMA(p,d,q) model to an original series y t consists of first differencing the original series y t d times resulting in, say, a differenced series z t to which we then fit an ARMA(p,q) (ARIMA(p,0,q)) model. Therefore since we had to difference the original series d times to get back to the original series we have to sum (often called integrate) the differenced series d times. That is while zs are differences of the ys, the ys are sums of the zs. The I in the term ARIMA refers to this summing or integrating. For example consider a differenced series z t, z t = (1 B)y t
Estimation then in terms of y t we have y t = (1 B) 1 z t = (1 + B + B 2 + )z t = z t + z t 1 + z t 2 + which shows that the original series is a sum of the differenced series. The coefficients in the ARIMA model must be obtained via some criteria. The most common criteria is chose values of C, φ and θ that maximizes the likelihood or chose estimates that result in the smallest sum of squared residuals (least squares). We will discuss the LS procedure briefly. Consider the AR(1) model (1 φ 1 B)z t = C + α t, or z t = C + φ 1 z t 1 + α t If we take expectations on both sides, then because of stationarity of the time series, we have µ = C + φ 1 µ + 0 C = µ(1 φ 1 )
Root-Mean-Squared Erroe and we can rewrite the model as z t = µ(1 φ 1 ) + φ 1 z t 1 + α t Now if we had a good estimate of ˆφ1 for φ 1 and ˆµ for µ, we could estimate the residual by ˆα t = z t ẑ t, where ẑ t = ˆµ(1 ˆφ 1 ) + ˆφ 1 z t 1. Thus we select parameter values ˆµ and ˆφ 1 such that the sum of squared residuals is minimum, i.e. SSR = ˆα t 2 = [ ] 2 z t ˆµ(1 ˆφ 1 ) + ˆφ 1 z t 1. t t Thus if ˆµ and ˆφ 1 are such that SSR is a minimum we have the least squares estimates. The following quantity is called the root-mean-squared error: SSR ˆσ α =
Root-Mean-Squared Erroe where n is all available time series observations and m is the number of estimated parameters. If we are considering two models we select the model with the smaller ˆσ α, which is called the root-mean-squared error or the estimated standard deviance of the random shocks.
The Ljung and Box χ 2 Test Dtatistic A ARIMA model assumes that the random shocks are independent. A successful model should have residual random shocks that are uncorrelated, however, if the model is not adequate to explain the data then the residuals from that model will contain effects not accounted for by the AR and MA terms in the model. Thus the residuals shocks will be associated and will not have zero autocorrelations. We may therefore construct a test statistics for the goodness of fit of the model by testing if a set of autocorrelations of lags, say, 1 to k are all zero. The null hypothesis is that for the shocks αs with test statistic H 0 : ρ 1 (α) = ρ 2 (α) = = ρ k (α) = 0 Q = n(n + 2) k i=1 (n 1) 1 r 2 i (ˆα) where n is the number of observations used to estimate the model.
Forecasting This statistic has approximately a χ 2 -distribution with (k m) degrees of freedom where m is the number of parameter estimated. Basically we reject for large Q, if we do not reject the null then this is evidence that the shocks estimated from the model have the required pattern that they are uncorrelated. Note: Some programs use the Box-Pierce statistics which performs the same test. For large sample sizes they are equivalent for small sample sizes the Ljung-Box statistics follows more closely the χ 2 - distribution use to calculate the p-value of the test statistics. Suppose we have fitted the ARIMA (1,0,1): z t = ˆµ(1 ˆφ 1 ) + ˆφ 1 z t 1 ˆθ 1 α t 1 + α t and now we want to forecast ahead say l time points i.e. we want to forecast (estimate) z t+1. Now the theoretical model ARIMA (1,0,1) states that z t = ˆµ(1 ˆφ 1 ) + ˆφ 1 z t 1 ˆθ 1 α t 1 + α t
Forecasting Now a best estimate of z t+1 (which is a random variable) is its mathematical expectation given our knowledge of past values as specified by the model, in this case the ARIMA(1,0,1) model. Let I t be the set of past values used in the model, then E(z t+1 I t ) = µ(1 φ 1 ) + φ 1 z t θ 1 α t where we assume E(α t+1 ) = 0. We do not know the parameter values so we replace them by the estimates given in the ARlMA(1,0,1) model fitted above. Thus we estimate z t+1 by ẑ t (1), where ẑ t (1) = ˆµ(1 ˆφ 1 ) + ˆφ 1 z t ˆθ 1 α t. Then we estimate z t+2 by ẑ t (2), where ẑ t (2) = ˆµ(1 ˆφ 1 ) + ˆφ 1 ẑ t (1) ˆθ 1 α t+1. where since we do not know α t+1 we replace it by its expected value of zero. Hence the above estimate of ẑ t (2) becomes ẑ t (2) = ˆµ(1 ˆφ 1 ) + ˆφ 1 ẑ t (1).
Models for Seasonal Data Similarly, we can build up a forecast for any future value. Of course the further we try to forecast into the future the larger will be the standard error of the forecast. Note that for stationary models the forecasts converge towards the mean of the series. Computer programs have been written to produce these forecasts and their standard errors. In particular R contains a time series packages for time series analysis, one for Box-Jenkins approach and one for the spectral analysis approach. Sometimes a time series will have a pattern that will repeat say every s time periods. When this happens we term the data to be periodic data. An example of periodic data is data with seasonal variation. Generally to remove the seasonal variation we must difference the observations by length s. That is we must difference observations separated by s time units i.e. lags of s, 2s, 3s, etc. The basic idea is that observations s time periods apart {z t, z t s, z t 2s,...} are similar. Observations that are similar every s time points are correlated, so we expect nonzero correlations between observations
Φ p (B) d z t = C + Θ q (B)α t Chapter 12: An introduction to Time Series Analysis Models for Seasonal Data separated by s time points or by 2s time points and so on. Thus if we construct an acf or a pacf we expect to find nonzero correlations at lags s, 2s, and so on. If the series lacks correlations at any lags except those that are multiples of s we have a purely seasonal process. An autoregressive model for this data can be written using one autoregressive parameter φ: z t = C + φ t s z t s + α t or (1 φ t s B s )z t = α t We may have to perform first-degree or second degree or Dth degree differencing of length s to obtain a stationary series, that is For example if D = 1 D s z t = (1 B s ) D z t. s z t = (1 B s )z t = z t z t s. Recall that the general model for the ARIMA(p, d, q) process is
Models for Seasonal Data The general model for the seasonal model is Φ P (B s ) D s z t = C + Θ Q (B s )α t. Now one way to combine the seasonal model with the local ARIMA model is to multiply them, then the general ARIMA(p, d, q)(p, D, Q), model is Φ p (B)Φ P (B s ) d D s z t = C + Θ Q (B s )Θ q (B)α t. For example ARIMA(0,0,1)(0,1,1) is a model which assumes that the process to be modelled has periodicity 4, since s = 4, and because D = 1, z t is differenced once by length four. Further since d = 0, there is no nonseasonal differencing for the local effect. Further since Q = l we have one seasonal MA term at lag 4. Again since
References q = 1 we have one nonseasonal MA term. Thus the model is thus z t z t 4 = C + (1 Θ 4 (B 4 ))(1 θ 1 B)α t = C + (1 Θ 4 (B 4 ))(α t θ 1 α t 1 ) = C + (α t θ 1 α t 1 ) Θ 4 (B 4 )(α t θ 1 α t 1 ) = C + α t θ 1 α t 1 Θ 4 α t 4 + Θ 4 θ 1 α t 5 z t = C + z t 4 θ 1 α t 1 Θ 4 α t 4 + Θ 4 θ 1 α t 5 + α t. All these model are easily fitted using the R, however finding the right model takes time and lots of experience. Most of this chapter are based on the following two books. References (1) Time Series Analysis and Forecasting, by O.D. Anderson (1) Forecasting with Univariate Box-Jenkins Models, by A. Pankratz