Estimating Time-Series Models

Transcription

1 Estimating ime-series Models he Box-Jenkins methodology for tting a model to a scalar time series fx t g consists of ve stes:. Decide on the order of di erencing d that is needed to roduce a stationary series y t = ( L) d x t that can be aroximated by an ARMA(,q) model (with intercet if the mean of y t is not zero).. By insecting the samle autocorrelations and artial autocorrelations of fy t g, determine tentative values for and q. 3. Estimate the lag coe cients by aroximate maximum likelihood assuming normally distributed innovations. 4. Comute aroximate standard errors and con dence intervals for the unknown coe cients. 5. Using various diagnostics, check if the tentative model was indeed aroriate. Box and Jenkins suggest informal ways to erform the rst two stes. A formal treatment of ste will aear in the second half of the course. A formal aroach to ste can be develoed using the theory of nonnested model selection; see, for examle, the book ime Series: heory and Methods by Brockwell and Davis. Here we shall discuss stes 3-5. Finally, in Section 6 we extend the discussion to ARMAX models. Some Alternative Estimators Consider the stationary ARMA(,q) model A (L)y t = B q (L)" t where the " t are white noise with zero mean and variance : he lag olynomials A (L) = L L and B q (L) = + L + + q L q are assumed to be invertible and have no common factors. Let be the column vector of the m = + q unknown lag coe cients. Suose we want to estimate the unknown arameters and using the vector of observations y = (y ; :::; y ) 0 : If the " t s are jointly normal, so are the y t s. Let () be a matrix de ned by Eyy 0 = (). Excet for an additive constant, the Gaussian log likelihood function is L(; ) = ln j()j ln y0 () y: Note that, for given ; L is maximized when = y 0 () y: hus the concentrated log likelihood function is L () = max L(; ) = ln j()j ln y0 () y hree alternative methods are commonly used in ractice for estimating :. Quasi Maximum Likelihood maximizes L in the arameter sace of : : (). Unconditional Least Squares uses the fact that j()j is negligible in large samles and minimizes y 0 () y: his method is described in more detail in Section 4 below.

2 3. Conditional Least Squares: If one conditions on y ; :::y and sets " ; " ; ; " q+ to zero, the likelihood for the remaining y s has the same form as () but j()j becomes a constant and y 0 () y becomes e 0 e; where e = (" + ; ; " ): As discussed in Section below, mimimizing e 0 e is a roblem in nonlinear least squares and can be accomlished using the Gauss-Newton algorithm. Because it is comutationally the simlest, conditional least squares is commonly used in ractice. It can be shown that all three estimators have the same limiting distribution even if the errors are not normal. We have the following basic result: heorem Assume i. the arameter sace for is such that the roots of A (z) = 0 and B q (z) = 0 are outside the unit circle and there are no common roots, ii. the true value 0 is in the interior of the arameter sace, iii. the " t are i.i.d. with mean zero and variance : hen all three estimators of 0 and converge in robability to the true value. Furthermore, for all three estimators, (^ 0 ) converges in distribution to a normal random variable with mean zero and covariance matrix V @ 0. If the " t are normally distributed, the estimators are asymtotically e cient. Note that, excet for the e ciency result, we do not need to assume the innovations are in fact normally distributed. Although the asymtotic variance exression does require that the innovations be conditionally homoskedastic, the i.i.d. assumtion can be weakened. For examle, Assumtion iii can be relaced by the following without a ecting the result: iii. both " t and " t are martingale di erence sequences (with resect to the ast history of " t ); that is, E(" t j " t ; " t ; :::) = 0 and E(" t j" t ; " t ; :::) = 0: In addition, E" + t < for some > 0: For a roof of the theorem, see W. Fuller, Introduction to Statistical ime Series, nd edition, Conditional Least Squares Maximum likelihood estimation is comutationally demanding when the samle size is large since the calculation involves inverting the matrix and comuting its determinant. Later we shall show how a comutational algorithm known as the Kalman lter can be emloyed to obtain exact maximum likelihood estimates by doing these calculations recursively. In ractice, most emirical economists maximize an aroximation to the likelihood based on a slight change in the initial conditions. For examle, in the AR() model, if we condition on the rst values y ; :::; y and examine the rocess starting at t = +, maximum likelihood is equivalent to a least

3 squares regression of y t on its lagged values. We now show that a similar modi cation of initial conditions in the general ARMA case leads to a nonlinear least squares regression. In every invertible ARMA(,q) model, the observed y s have a moving average reresentation exressing them linearly to current and lagged " s. Hence the likelihood function (the joint density of the observed y s) can be derived from the density of the " s using familiar change-of-variable techniques. Unfortunately, the maing y to " is tyically not one-to-one, so the calculation is nontrivial. However, conditioning on the observed values y ; :::; y and on " = " = ::: = " q+ = 0 necessarily leads to a one-to-one maing. Let e be the vector of white noise innovations (" + ; :::; " ) 0 and let y be the vector of observations (y + ; :::; y ) 0 : hen, by successive substitution, we can write e = Dy + d where D is a triangular matrix with ones on the diagonal; the vector d deends on y ; :::; y and is nonrandom because of our conditioning. hus we can write e = e(y; ) where the Jacobian j@e=@yj does not deend on. If e= is standard normal, the change-of-variable rule gives the density for y having the form f(y) = ( ) ( )= exf e(y; )0 e(y; )= g Maximum likelihood estimates of can be obtained by minimizing e 0 e: Although e is linear in y; it is nonlinear in as long as q (the number of moving average coe cients to be estimated) is greater than zero. his nonlinear least squares roblem can be solved using the Gauss-Newton algorithm. Let m = + q be the number of unknown arameters in and let n = be the number of observations after conditioning. (If there is an intercet, then m = + q + :) hen, de ning the n m matrix Z(; 0 and starting with some initial guess 0, we use the recursion r+ = r (Z 0 rz r ) Z 0 re r where Z r = Z( r ; y) and e r = e( r ; y): When Z 0 r r ' 0, one stos. For examle, suose y t = y t +" t +" t and we treat y as xed and assume " = 0. hen, using the equation " t = y t y t " t with " = 0, we nd " = y y ; " 3 = y 3 ( + )y + y ; " 4 = y 4 ( + )y 3 + ( + )y y ; etc. Clearly, the Jacobian is triangular with determinant equal to one. However, we do not have to exlicitly solve for the " 0 s in terms of the y s. Under normality, the MLE which minimizes the sum of squared innovations can be comuted using the following G-N algorithm:. For some initial 0 = ( 0 ; 0 ) 0 and starting with the initial condition " =@ =@ = 0, build u the vector e 0 and the ( ) matrix Z 0 = using the recursions " t = y t 0 y t 0 " t =@ = y t t t =@ = " t t =@. Regress e 0 on Z 0 and comute = 0 (Z 0 0 Z 0) Z 0 0 e0 : 3

4 3. Redo stes one and two using in lace of 0 so = (Z 0 Z ) Z 0 e : 4. Continue to convergence. Of course, this algorithm fails if, at iteration r, Z r has rank less than. So, for examle, one should not start with the initial choice 0 = (0; 0) 0 : Note that exogenous regressors can be easily handled by the Gauss-Newton algorithm. For examle, if the model is y t = y t + x t + " t + " t then Z is the ( ) 3 and the recursions are " t = y t 0 y t 0 x t 0 " t =@ = y t t t =@ = " t t t =@ = x t t =@ Of course, to insure that the algorithm converges to a global (and not just a local) minimum, alternative starting values should be used. Or at least the starting values should be chosen as some consistent method-of-moments estimator that has high robability of being near the true arameter value. If the model has been estimated by the Gauss-Newton algorithm, then a natural estimate of the asymtotic covariance matrix V is s (Z 0 Z= ), where Z = de=d and s = e 0 e=(n m); both e and Z are evaluated at the estimate ^ from the nal iteration. hat is, we behave as though ^ is normal with mean 0 and variance matrix s (Z 0 Z) : he usual tests and con dence regions based on the t and F-distributions are asymtotically valid. he G-N algorithm maximizes an aroximation to the likelihood since " is not really 0 and y is not really constant. However, the G-N estimate is usually close to the actual MLE as long as the arameters are far away from the noninvertibility boundaries jj = and jj =. 3 Score ests for ARMA Parameters Score test statistics are tyically much easer to comute than Wald test statistics but are asymtotically equivalent. If the null hyothesis uts r indeendent constraints on, we need only nd the constrained MLE e which involves a lower dimensional nonlinear estimation roblem. Using the outut of a Gauss-Newton iteration, we would evaluate e and the full n m matrix Z at e so the estimated score is e Z 0 e=es and the estimated information matrix is e Z 0 e Z=es. If e 0 e Z( e Z 0 e Z) e Z 0 e es is large comared to a chi-square(r) critical value, one rejects the hyothesis. If is comosed of two subvectors ( ; ) and the null hyothesis is = a; the score statistic takes a simler form. Partitioning the matrix Z as (Z ; Z ); we note that Z e 0 e = 0: hus the score statistic becomes e 0 e Z 0 ( e Z 0 e Z e Z 0 e Z ( e Z 0 e Z ) e Z 0 e Z ) e Z 0 e es. Score statistics are articularly useful for diagnostic checking. After tting an ARMA(,q) model, one might want to see whether an ARMA(+,q) or an ARMA(,q+) rovides a signi - cantly better t. Since Z e and e have already been comuted, one need only comute the additional 4

5 vector e Z : here is no need to comute another Gauss-Newton set of iterations. Note that a score test of the hyothesis that an ARMA(,q) is aroriate (in a maintained ARMA(+,q+) model) would fail since e Z would have rank. his re ects the fact that, under the null hyothesis, the AR and the MA olynomials have a common factor. An alternative diagnostic test is also commonly used. If an ARMA(,q) model is adequate, then the estimated innovations should behave like white noise. he low order samle autocorrelations of a white-noise series are asymtotically indeendent N(0; ) variables. So times the sum of the rst k squared samle autocorrelations would be aroximately distributed as chi-square-k. If the autocorrelations are comuted from the e; the estimated innovations after tting an ARMA(,q) model, the asymtotic theory is a ected by the estimation. For large k and, the Box-Pierce ortmanteau statistic (e + e + e k) is aroximately distributed as chi square with k q degrees of freedom if the model is correctly seci ed. 4 Unconditional Least Squares and Backcasting Although no longer used much in ractice, unconditional least squares is another way to avoid the comutational di culties involved in maximizing the exact Gaussian likelihood function. It avoids calculating jj by simly ignoring it. It simli es the calculation of y 0 () y; by the trick of "backcasting." Let " be the vector (" m ; " m+ ; ; " ) 0 where m is some large ositive integer. hen the vector of observations y = (y ; y ) 0 can be aroximated by y = D" where D is a triangular matrix of moving average coe cients. It follows that Eyy 0 DD 0 and y 0 y " 0 D 0 (DD 0 ) D": It is easy to verify that, under normality, the conditional exectation of " given y is " c E("jy) = (E"y 0 )(Eyy 0 ) y D 0 (DD 0 ) y D 0 (DD 0 ) D": Since D 0 (DD 0 ) D is idemotent, it follows that y 0 y " 0 D 0 (DD 0 ) D" " 0 c" c : hus we nd that estimation method is equivalent to minimizing the sum of squares of the exected current and ast innovations. Box and Jenkins suggested a very clever way of doing this calculation. hey noted that the backwards ARMA model A(L )y t = B(L ) t has the same autocorrelation roerties as the original model. hus, given tentative estimates of the arameters, one can backcast the observations y 0 ; y ; ; y m : From these values and using the original di erence equation, one can comute best linear redictors of " given y: Hence the Gauss-Newton algorithm can be emloyed to minimize y 0 y just as it is emloyed to minimize e 0 e. Details of the calculation can be found in the Granger and Newbold text. 5 Asymtotic heory Proofs of the large-samle results given in the revious sections are nontrivial and rely on a general asymtotic theory for sums of deendent variables, a theory that is well beyond the scoe of these notes. Here we shall resent just a few of the basic ideas. 5

6 Suose fy t g is a stationary AR() rocess generated by y t = y t + " t with jj < : If is estimated by least squares from the observations y 0 ; y ; :::; y, then the standardized estimator can be written as ( ) = = P y t " t P y t = N D where the summations are from to. If N converges in distribution to a N(0, A) variable and if D converges in robability to the constant B, then the standardized estimator has a limiting N(0, A=B ) distribution. If A b and B b are consistent estimates, then we might aroximate the distribution of b by a normal with mean and variance A= b B b : Note that ED = =( ): o show that D converges in robability to its exectation, it is su cient to show that its variance tends to zero as! : But var(d ) = X X t=0 s=0 Suose the covariances do not deend on t so cov(y t ; y s) = X r= + X r t=0 cov(y t ; y t+r) var(d ) = X r= + ( jrj )g r where g r = cov(y t ; y t+r): If g r! 0 as r!, then this variance necessarily converges to zero. he g r can be comuted from the moving average reresentation for y t : If the " t are i.i.d. and ossess nite fourth moment, a little algebra shows that 4 g r = [ ( ) ]r where 4 is the fourth cumulant of " t : Since jj < ;we conclude that indeed D! B = =( ): Using a more delicate argument, the assumtion of a nite fourth moment can be droed. Indeed, the i.i.d. assumtion can be relaced by the assumtion that " t and " t are martingale di erence sequences, although the argument is somewhat more tedious. Showing that the numerator N is asymtotically normal is a bit more comlicated. Let U t = y t " t so N = = P U t : Suose that both " t and " t are stationary martingale di erence sequences. hen fu t g is also a stationary martingale di erence sequence and P Ut converges in robability to its exectation X E(" t y t ) = 4 = B as long as su cient moments exist. hen, using the aylor series exansion of the exonential function, the characteristic function for N can be written as Y Y () = E e iut= = E + i U t t= t= Y E + i U t U t E t= Y t= + i U t e P U s = e B = E 6 Ut + O( 3= ) Y t= + i U t ;

7 where some delicate arguments are needed to justify the indicated aroximations. Since U s is a martingale di erence sequence, iterated exectations can be used to show that the nal exected roduct equals one. hus the limiting characteristic function of N is that of a N(0, B) and hence (^ ) has a limiting normal distribution with variance =B =. [Note: if E(" t jast) is not a constant but deends on ast variables, it may still be the case that N is asymtotically normal. However, its asymtotic variance will not equal B. hus, in the resence of conditional heteroskedasticity, the usual standard errors are incorrect. Robust standard errors are available and are discussed in Hamilton s text.] For general ARMA models, the nonlinear least squares estimator satis es the rst order conditions Z( b ) 0 e( b ) = 0: Exanding e( b ) around the true arameter value 0 ; we obtain Z( b ) 0 [e( 0 ) + Z( )( b 0 )] = 0: Often we can show that ( b 0 ) = [Z( b ) 0 Z( )] Z( b ) 0 e( 0 ) = Z(0 ) 0 Z( 0 ) D N + o (): Z( 0 ) 0 e( 0 ) + o (): Using techniques similar to those used above, one can show that D converges in robability to a nonrandom nonsingular matrix B and that, under conditional homoskedasticity, N is a standardized sum whose summands are a martingale di erence sequence and is asymtotically N(0, B). Hence, ( b 0 ) is asymtotically normal with variance B : Since B can be estimated by Z( b ) 0 Z( b )=; one says that b is aroximately normal with mean 0 and variance s [Z( b ) 0 Z( b )] General conditions under which samle moments constructed from a strictly stationary time series converge in robability (or almost surely) to oulation moments are often referred to as ergodic theorems. hese rather dee results arise from statistical mechanics and make assumtions about the robabilities of events that are invariant with resect to a time shift. For examle, the event fy t > 0 for all tg is invariant since it holds if and only if the event fy t+ > 0 for all tg holds; but the event fy 5 > 0g is not invariant since it does not imly fy 6 > 0g: A stationary rocess is called indecomosable if all invariant events occur with robability one or zero. A stationary rocess is called ergodic if every samle moment converges almost surely to its exectation. he basic ergodic theorem then says that indecomosablity is a necessary and su cient condition for ergodicity. It can be shown that stationary normally distributed rocesses whose autocovariances r tend to zero (as r tends to in nity) are indecomosable and hence ergodic. Strong mixing conditions also can be used. In general, however, excet for the case of discrete Markov chains, it is rather di cult to nd economically meaningful conditions that imly indecomosability. In ractice, we usually just make assumtions on the summability of autocorrelation or moving average coe cients to rove convergence of samle moments. 6 Regression with Autocorrelated Errors As noted in section, ARMAX models of the form A(L)y t = + F (L)x t + ::: + F k (L)x kt + B(L)" t 7

8 can be estimated by conditional (nonlinear) least squares as long as the lag olynomials A; B; F ; :::; F k are all of low order or are ratios of low order olynomials. his assumes that the orders of the olynomials are known or can be easily determined. Often we desire an alternative aroach that is robust to seci cation error in olynomial order. First we consider the secial case where A(L) = I; that is, we have a linear regression model y = X + u where the errors are assumed to be indeendent of the regressors. he errors are a stationary time series with mean zero and second-moment matrix. If the errors are aroximately normal and is known, a natural estimator of is the GLS estimator (X 0 X) X 0 y: Of course, in ractice, is unknown so a multi-ste rocedure is often used. First one rewhitens the data by icking a nonrandom matrix D such that Du is thought to be roughly white noise. Second, one regresses Dy on DX obtaining residual vector e: (In ractice, most emirical economists seem to use the identity matrix for D so e is the OLS residual.) Fit an ARMA model to the residuals and use this estimated model to comute the matrix ^. he nal estimate of is ^ = (X 0 ^ X) X 0 ^ y: Under regularity conditions on the exogenous variables and some assumtions on the rate at which the high-order autocovariances tend to zero, it can be shown that if the number of arameters in the ARMA model for fu t g are allowed to increase slowly to in nity as the samle size increases, ^ is asymtotically equivalent to the GLS estimator with known : A similar result is available for rational distributed lag models. For examle, suose y t = L x t + u t ; where u t is stationary with mean zero but covariance matrix : If were known one could estimate the remaining aameters by minimizing u 0 u: Instead one can rocede as follows. Find reliminary consistent estimates of and ; say by instrumental variables. Fit an ARMA model to the estimated ^u t and use that model to comute ^: hen estimate the arameters by minimizing u 0 ^ u: Again, if the number of arameters in the ARMA model for fu t g are allowed to increase slowly to in nity as the samle size increases and some regularity conditions are satis ed, this rocedure is asymtotically equivalent to the estimator using the true : 8