ARMA Estimation Recipes

Transcription

1 Econ. 1B D. McFadden, Fall Preliminaries ARMA Estimation Recipes hese notes summarize procedures for estimating the lag coefficients in the stationary ARMA(p,q) model (1) y t = µ +a 1 (y t-1 -µ) + + a p (y t-p -µ) + g t + b 1 g t b q g t-q, where y t is observed for t = 1,, and the g t are unobserved i.i.d. disturbances with mean zero and finite variance. he mean µ, the lag coefficients a 1,,a p and b 1,,b q, and are the parameters of the model. By assumption, a p ú 0 and b q ú 0. Define the polynomials A(z) = a 1 z + + a p z p and B(z) = b 1 z + + b q z q, and the lag operator L that for any series x t is defined as Lx t = x t-1. hen the model can be written () (I - A(L))(y t - µ) = (I + B(L))g t. he model is stationary if and only if the polynomial 1 - A(z) is stable; i.e., all its roots lie outside the unit circle. If the model is stationary, then the lag polynomial I - A(L) is invertible, and there is a MA() representation of the model, written formally as I % B(L) (3) y t - µ = "g t. I & A(L) Let z 1,,z p denote the roots of 1 - A(z), some of which may be repeated. hen this polynomial can be written 1 - A(z) = (1 - z/z 1 )""(1 - z/z p ). hen (I - A(L)) -1 = p k k'1 z ks "L s. Alternately, write (I - A(L)) -1 (I + B(L)) = sl s / (L). he identity min(p,s) i'1 () I + B(L} = (I - A(L))" sl s = 0I + ( 1 -a 1 0 )"L + + ( s - a i s-i )"L s + min(p,s) i'1 implies 0 = 1, 1 = a 1 + b 1, and s = a i s-i + b s, where b s = 0 for s > q. Another derivation of these conditions starts by noting that the covariance of y t and g t-m is m < 0. Multiplying (1) by g t-m and taking expectations then gives m for m $ 0, and zero for 1

2 (5) m = a 1 m a p m-p + b m, with m-s = 0 for s > m, b 0 = 1, and we define b m = 0 for m > q. hese are called the Yule-Walker equations. hey can be used recursively to obtain the coefficients s in the MA() representation. An implication of the MA() representation is (6) 0 / Var(y t ) = " s and m / cov(y t,y t-m ) = " s s+m for m > 0. Since cov(y t+m,y t ) = cov(y t,y t-m ) for m > 0, one has -m = m. It is sometimes convenient to summarize the autocovariances of a stationary series x in terms of an autocovariance generating function (ACGF) s'& (7) g x (z) = cov(x t,x t- s )@z s. he ACGF has a useful convolution property: If a linear transformation C(L) = s'& c s L s is applied to a stationary series x t, then y t = C(L)x t has g y (z) = C(z)C(1/z)g x (z). o verify this, note that cov(y t,y t-m ) = i'& s'& '& c i t-i,x t-m- ), and cov(x t-i,x t-m- ) = cov(x t,x t-m+i- ). hen m'& i'& '& g y (z) / z m cov(y t,y t-m ) = z i t-i,x t-m- ) m'& i'& '& i'& '& = c i z c t-i,x t-m- )z m+i- = c i z c x (z), with the last equality holding since summing over m for each fixed i and gives g x (z). Let t = g t + b 1 g t b q g t-q. hen E t = (b b q ), E t t-m = (b m b b q b q-m ) for 1 < m # q, and zero for m > q, and E t y t-m = (b m b q q-m ) for 0 # m # q, zero for m > q. he i.i.d. series g t has the constant ACGF g g (z) = since all its autocovariances are zero. hen, applying the ACGF convolution formula to t = (I + B(L))g t yields (8) g (z) = {1+B(z))(1+B(1/z)) = q s'&q (b*s*b b q b q-*s*)z s. For example, q = 1 yields g (z) = ((1+b 1 ) + b 1 z + b 1 z -1 ), and q = yields g (z) = ((1+b 1 +b ) + b 1 (1+b )z + b 1 (1+b )z -1 + b z +b z -1 ). Applying the convolution formula to y t = (I - A(L)) -1 t, it has the ACGF

3 1 (9) g y (z) = "g (z) = = 1 1%B(z) 1&A(1/z) 1%B(1/z) 1&A(1/z) (z)" (1/z). If all the roots of the polynomial 1 + B(z) lie outside the unit circle, then I + B(L) is invertible, and there is an AR() representation of (1), written formally as I & A(L) (10) "y t = g t. I % B(L) It is not a condition for stationarity that I + B(L) be invertible. However, it is always possible to re-scale the g s and redefine B(L) so that it has the same ACGF, but all the roots of 1 + B(z) lie outside or on the unit circle. For example, t = (I + L/z 1 )g t has ACGF ((1+z 1 - ) + z/z 1 + 1/zz 1 ), while t = (I + z 1 L)(g t /z 1 ) has ACGF ( /z 1 )((1+z 1 ) + zz 1 + z 1 /z), which is the same. hen one can factor B(L) into an invertible term and a non-invertible term that has unit roots. For some purposes, it is convenient to rewrite the ARMA model (1) by defining the (p+q) 1 vectors hn = (1,0,,0) with a one in the first component, rn = (1,0,,0,1,0,,0) with ones in the first and p+1 components and zeros elsewhere, and t N = (y t -µ,,y t-p+1 -µ,g t,,g t-q+1 ). hen (11) t+1 = F t + rg t+1 and y t+1 -µ = hn t+1, where an bn (1) F = I p&1,p 0 p&1,q 0 1,p 0 1,q 0 q&1,p I q&1,q with an = (a 1,,a p ), bn =(b 1,,b q ), I rs an r s matrix with ones down the diagonal and zeros elsewhere, and 0 rs an r s matrix of zeros. his is called a state space representation of the ARMA(p,q) model, with t called the state vector, t+1 = F t + rg t+1 called the state equation, and y t+1 -µ = hn t+1 called the observation equation. State space representations are not unique, and the one given here is easy to interpret but not minimal in terms of dimensionality; see Harvey, p , for a more compact and commonly used state space representation for the ARMA model. 3

4 . Prediction Let G t denote all of history up through t, including the realizations of y s and g s for s # t. hen, (13) E(y t+1 *G t ) = a 1 E(y t *G t ) + + a p E(y t-p+1 *G t ) + E(g t-1 *G t ) + b 1 E(g t- *G t ) + + b q E(g t-q-1 *G t ) = a 1 y t + + a p y t-p+1 + b 1 g t + + b q g t-q+1. As a shorthand, let y t+1*t = E(y t+1 *G t ); this is the forecast of y t+1 given G t that minimizes mean square error. An implication of the formula for y t+1*t is that g t = y t - y t*t-1. Similarly, the minimum-mse m-period ahead forecast y t+m*t = E(y t+m *G t ) is obtained using the recursion (1) y t+m*t = a 1 "y t-1+m*t + + a p "y t-p+m*t + b 1 g t+m-1*t + + b q g t+m-q*t, where y t-i+m*t = y t-i+m and g t+i-m*t = g t+i-m if I $ m, and g t+i-m*t = 0 if I < m. he conditional variance of the forecast error t+m*t = y t+m - y t+m*t is (15) v t+m*t = E( t+m*t*g t ). A convenient way of summarizing the forecasting formulas is in terms of the state space representation above, where the minimum MSE forecast is (16) t+m*t = F t+m-1*t = F m t. he forecast error is t+m*t = t+m - t+m*t = m&1 he conditional covariance matrix of the forecast errors is (17) V t+m*t = FV t+m-1*tfn + U = m&1 F s UF Ns, F s rg t+m-s = rg t-m + F t-m-1*t, implying g t+1 = 1,t+1-1,t+1*t. where U is an array that has a one in the northwest corner and zeros elsewhere. he updating formulas t+m*t = F t+m-1*t and V t+m*t = FV t+m-1*tfn + U are versions of what is called the Kalman filter. his formulation has broader application in state space models for time series analysis, a topic that will be discussed elsewhere. For some purposes, it is useful to predict backwards to period t from information after t. Multiplying the equations (I-A(L))y t = t and t = (I+B(L))g t by L -p and L -q respectively gives (18) E(y t y t+1,, t+1,) = (y t+p - a 1 y t+p a p -1y t+1 - t+p )/a p, E( t g t+1,, t+1,) = ( t+q - g t+q - b 1 g t+q b q-1 g t+1 )/b q.

5 3. Method of Moments Estimation he model (1) can be written as a regression model (19) y t = c + a 1 y t a p y t-p + t where c = (1 - a a p )µ and the MA(q) disturbances t have a covariance matrix given by the formulas following (6). Since t is correlated with y t-1,,y t-q but uncorrelated with earlier y's, (19) can be estimated using SLS with 1,y t-q-1,,y t-q-p as instruments, and observations t = p+q+1,,. his method then loses the first p+q observations in order to get the instruments. o estimate the MA coefficients, first retrieve the residuals t from the SLS estimation of (19), and form the empirical ACGF, q s'&q (0) g (z) = sz s, t'p%q%*s*%1 1 where s = t t-*s*, is an empirical estimate of E t t-*s*. From (8), the theoretical ACGF for is a bilinear function of the MA coefficients. hen, estimating the MA coefficients so that the theoretical and empirical ACGF coincide is relatively practical. As noted earlier, the solution is not in general unique, and it is preferable to pick estimates that give a stable root rather than an unstable one. his can be achieved by conducting the search over the q possible roots of 1 + B(z) subect to the constraint that these roots lie outside or on the unit circle. For the example of a MA(1) component in the ARMA model, matching the ACGF terms yields 0 = (1 + b 1 ) and 1 = b 1, which yields the quadratic 1-0b 1 + 1b 1 = 0. his has the solution b 1 = 0 / 1 ± ( 0-1 ) ½ / 1. If 0 > 1, there are two real roots, and 0 / 1 - ( 0-1 ) ½ / 1 gives the stable root. If 0 # 1, there is no real root, and the empirical ACGF cannot be matched exactly by a MA(1) model. In this case, b 1 = sign( 1 ) yields the MA(1) model with a root on or outside the unit circle that is closest to the empirical ACGF. For the example of a MA() component, writing 1 + B(z) in terms of its roots and matching the ACGF terms yields 0 = (1 + 1/z 1 + 1/z ), 1 = (1/z 1 )(1 + 1/z 1 z + 1/z ), and = /z 1 z. he estimator would then be obtained by a search in, z 1, and z subect to the restriction that z 1 and z are in the complex plane, on or outside the unit circle, and either are both real, or are convex congugates. he estimators described above are not the most efficient available among those employing only moment conditions. However, it will be convenient to develop the alternatives as estimators for the case of normal disturbances, and then note that they will be consistent even without normality. 5

6 . Maximum Likelihood Estimation Suppose the disturbances g t in (1) are normal. hen, vector (y 1,,y ) is then multivariate normal with mean µ and a covariance matrix that is a band matrix G, with all coefficients s places off the diagonal equal to s : 0 1 & & &3 & 1 0 & &3 (1) =. : : : " : : & &3 & 0 1 &1 & &3 1 0 he autocovariances s in are functions of the deep parameters of the model, and the coefficients of A(L) and B(L); these are given by the coefficients of the ACGF (9). A brute force approach to estimating the model, which is efficient under the assumptions of normality, is then to maximize the log likelihood function () L = -(/)log( ) - (½)log(det( )) - (½)(y 1 -µ,,y -µ) -1 (y 1 -µ,,y -µ)n in the parameters µ,, a 1,, a p, b 1,,b q. he likelihood function is minimized in µ at the sample mean µ. Using the formulas for differentiation of determinants and inverses, the first-order-condition for a parameter is (3) ML/M = -½ -1 (I - uun -1 )"M /M, where un = (y 1 -µ,..,y -µ ) is the vector of deviations from sample mean. he practical problem with the brute force approach is that the parameters of the ARMA process appear in L non-linearly, deep within the matrix. herefore, considerable effort in time-series analysis goes to reformulating the maximum likelihood problem in ways that are more tractable computationally. he model (1) can be rewritten as () g t = y t - a 1 y t a p y t-p - b 1 g t b q g t-q. aking expectations conditioned on the information G t-1, this equation implies 0 = y t*t-1 - a 1 y t a p y t-p - b 1 g t b q g t-q, and hence 6

7 (5) g t = y t - y t*t-1. he mapping from g s to y s is linear, with a transformation matrix that is triangular (i.e., y t depends only on g s at or before t) with ones on the diagonal. hen, the Jacobean of the transformation from (y 1,,y ) to (g 1,,g ) is one, and the log density of (g 1,,g ) conditioned on earlier g s is t'1 (6) L = -(/)log( ) - (½)log( ) - ½ g t /, with the g t defined (recursively) as functions of the lag parameters from (). his is called the predictive error decomposition of the likelihood. One approach to estimation is to condition on (y 1,,y p ) and (g p-q,,g p-1 ), so that () gives the g's for t = p+1,,, and then to maximize the conditional log likelihood, which is equivalent to minimizing the conditional sum of squares, t'p%1 (7) CSS = g t, in the lag parameters. his gives a pre-estimator (because of dependence on g p-1,..) that is equivalent to the solution at convergence obtained by iteratively applying least squares to the equation below, with the lagged g's computed using () and the lag coefficient estimates from the previous round: (8) y t = a 1 y t a p y t-p + b 1 g t b q g t-q + t for t = p+1,,. he final step is to get rid of the dependence of the estimator on the initial g's. his is often done in the computationally expedient way of replacing them by their unconditional expectations, which are zero. A superior procedure, only moderately more difficult, is described later. For AR(p) models, a useful simplification of () comes from noting that the density of (y 1,,y ) can be written as the product of the p-dimensional multivariate normal density of (y 1,,y p ) and the 1-dimensional conditional densities of y t given y t-1,,y t-p for t = p+1,,. In this formulation, the log likelihood is -1 (9) L = -(/)log( ) - ½log(det( p )) - ½(y 1 -µ,,y p -µ) p (y 1 -µ,,y p -µ) - ½log( ) - (1/ t'p%1 ) (y t -a 1 y t-1 --a p y t-p ), where p is the covariance matrix of the first p observations. If, further, one conditions on (y 1,,y p ), the resulting log likelihood is maximized when the quadratic form t'p%1 (y t -a 1 y t-1 --a p y t-p ) is 7

8 minimized; this is exactly the same as the regression (19) in the case q = 0. Conditioning on the first p observations, maximization of this likelihood function is equivalent to (non-iterative) least squares applied to the model (8), or to minimizing CSS in (7). Now consider the full stationary ARMA(p,q) model (1), and its representation (11) in state space form. Let z t denote the conditional expectation of t given y t,y t-1,,y 1. his is the optimal predictor of t given this information. Given normality, this is a linear function of the y s. Let P t denote the conditional MSE of the deviation t - z t ; i.e., P t = E{( t - z t )( t - z t )N*y t,y t-1,,y 1 }. Similarly, define z t t-1 = E( t y t-1,,y 1 ) and P t t-1 = E{( t - z t )( t - z t )N*y t-1,,y 1 }. Given the state equation t = F t-1 + rg t and the observation equation y t = hn t, and taking conditional expectations, one obtains the formulas (30) z t*t-1 = Fz t-1, P t*t-1 = FP t-1 FN + rrn, y t*t-1 = hnz t*t-1, and from these the updating relationships for proections, (31) t / y t - y t*t-1 = hn( t - z t*t-1), f t = hnp t t-1 h z t = z t*t-1 + P t*t-1h(y t - hnz t*t-1)/f t P t = P t*t-1 - P t*t-1hhnp t*t-1/f t. hese formulas (with a different notation) are derived and discussed in Harvey, p , for a more general model that includes stationary ARMA as a special case. he formulas in (31) can be employed, with an initialization for z 0 and P 0, to calculate the exact oint normal density function of (y 1,,y ): t'1 t'1 (3) L = -(/)log(b) - ½ log(f t ) - ½ < t /f t his is a predictive error decomposition form of the log likelihood. he maximum likelihood estimates can also be given an interpretation of minimizing a CSS; see Harvey, p. 90. Harvey, p. 88, also describes the construction of starting values. For the stationary ARMA model, they are z 0 = (I - F) -1 c, where c is a vector with µ in the first p components and 0 in the remaining q components, and vec(p 0 ) = F (I - FqF) -1 vec(rrn). Absent normality, the predictors above remain best linear predictors, and the estimators continue to have an interpretation of minimum CSS estimators that use all of the sample information on the first two moments, with a Bayesian interpretation of the starting values. 8