12: AUTOREGRESSIVE AND MOVING AVERAGE PROCESSES IN DISCRETE TIME. Σ j =

Transcription

1 1: AUTOREGRESSIVE AND MOVING AVERAGE PROCESSES IN DISCRETE TIME Moving Averages Recall ha a whie noise process is a series { } = having variance σ. The whie noise process has specral densiy f (λ) = of uncorrelaed zero mean random variables σ on [, ]. A moving average process is he series = a. (1) = Noe ha is a linear combinaion (i.e., "moving average") of pas presen and fuure values of he innovaions or shocks,. A more useful model for forecasing purposes is he one-sided moving average = a. () = 0 Boh (1) and () are said o be infinie moving averages, since here are an infinie number of parameers a needed o deermine he process. A finie moving average resuls if only finiely many parame ers are used: n = a. = m The moving average { } (given by (1), say) is he oupu of a linear filer wih inpu { } and ransfe funcion r B (λ) = ae. = i λ Thus, and { } has specral densiy i λ = e B(λ)dZ (λ),

2 -- σ f (λ) = B (λ) f (λ) = ae = i λ. The variance of is Var = f (λ)d λ=σ a = (Exercise). KE FACT: Every weakly saionary ime series wih coninuous specrum can be represened as a (possibly) infinie moving average. We will prove his imporan resul for he process { }, assumed o have a sricly posiive specral densiy f (λ). Le σ be a posiive number, and define he process = e A(λ)dZ (λ), i λ where A (λ) = σ () 1/ 1. 1/ (f (λ)) The maching condiion is saisfied (i.e., { } is a well-defined weakly saionary process wih zero mean and finie variance) since Now, { } has specrum A (λ) f (λ) =σ. f (λ) = A (λ) f (λ) = σ. I follows ha is a whie noise process. By invering he filer, { } can be recovered from { } : Now, = i λ e 1 dz (λ). A (λ) 1 is a square inegrable funcion on [, ] and hence has a Fourier series expansion A (λ)

3 -3-1 = ae i λ, A (λ) = where {a } are consans wih a <. (See Example 1. of Koopmans for he relevan Hilber space heory.) Thus, i λ = e [ ae ] dz (λ) = a e dz (λ) = i λ i λ( ) a. = = = This is he desired moving average represenaion of { }. Auoregressive Processes A discree ime weakly saionary process { } is said o be an auoregression of order p ( AR(p)) if here exiss a whie noise process { } wih variance σ, and consans b,...,b 1 p such ha + b b = (all ), (3) 1 1 p p E = 0 (all s, wih s < ). (4) s Equaion (3) expresses as a linear regression on is own pas; hence he erm "auoregression". Equa ion (4) is a condiion o ensure ha he "innovaion" a ime, namely, really represens new infor - maion in he sense of being uncorrelaed wih pas values of he series { }. The definiion of an k k = 0 infinie order auoregression is he same as above excep ha (3) is replaced by b =, where b 0 is = 1 and b <. Since { } is he resul of passing { } hrough a linear filer wih ransfer func- k = 0 k =0 k k i λk ion B (λ) = be, i follows ha dz (λ) = B (λ)dz (λ), and hence he specrum of he AR proces f s (λ) = σ 1. i λk be k = 0 k k For AR(p) models, a key quesion is wheher here exiss a weakly saionary soluion { }ohe sochasic difference equaions (3) and (4). These equaions define he AR(p) process implicily, and a

4 -4- soluion canno be guaraneed o exis unless an addiional condiion is imposed on he parameer values b,...,b. The condiion involves he zeros (roos) of he funcion 1 p β(z ) = 1 + bz+ 1 bz bp z p, considered as a polynomial in he complex variable z : For a finie auoregression o exis, i is sufficien ha all he zeros of β(z ) lie ouside he uni circoefficiens equal o hose of he power series expansion of 1/β(z ). cle. The soluion will be a one-sided moving average of he defining whie noise process wih This is an imporan resul, and a skech of he proof follows. Firs, noe ha he ransfer func- ion is B (λ) =β(e i λ ). If β(z ) has no zeros on he uni circle, hen i λ = e [1/B (λ)] dz (λ) saisfies he maching condiion, and hence is a weakly saionary finie-variance soluion o (3). I fol- lows ha has he moving average represenaion (see previous secion) = a, (5) = where he a are he Fourier coefficiens of 1/B (λ): i λ 1/B (λ) = ae. (6) = Equaion (5) gives an explici form for he soluion o (3). The quesion ha remains is wheher his soluion saisfies (4). Assuming ha he zeros of β(z ) lie ouside he uni circle, here mus be some real number ρ>1 such ha 1/β(z ) is analyic for z <ρ. I follows ha for z <ρ, 1/β(z ) can be represened as a power series (Lauren series) involving posiive powers of z only. On he uni circle ( i.e., if z = 1), his power series mus be idenical o he Fourier series (6). I follows ha a = 0 for < 0, and hence = a, i.e., can be wrien as a one sided moving average. Hence, if s <, = 0

5 -5- E [ ] = E [ a ] = ae[ ] = 0. s = 0 s = 0 Thus, { } saisfies condiion (4), and he proof is complee. s We have us shown ha he condiion ha he zeros of β(z ) lie ouside he uni circle is sufficien for he exisence of a finie auoregression. I can be shown ha his condiion is also necessary. Thus, a finie auoregression exiss if and only if he zeros of β(z ) lie ouside he uni circle. Parameer Esimaion Suppose we are willing o assume ha he process { } obeys some finie-parameer model of known dimension. Two examples are he p h order auoregressive (AR (p )) model, p =0 b = (b = 1,var =σ ), 0 and he q h order moving average (MA (q )) model, q = a (a = 1,var =σ ). = 0 0 Since in pracice we will have only a limied amoun of daa x,...,x 0 n 1 from he sochasic 1 p 1 q process { }, he parameers (b,...,b, σ for he AR (p ), a,...,a, σ for he MA (q )) will ypically be unknown, and hence mus be esimaed from he available daa. If we assume ha he whie noise process is Gaussian, hen we may esimae he parameers by maximum likelihood. Le denoe he vecor of model parameers, and le x = (x,.., x 0 n 1 T ance funcion {c } (r = 0, 1,...) of he process. Le ). I can be shown ha deermines he covarir be he resuling n n covariance marix of 0 n 1 i,...,. The marix has (i, ) enry (i, ) = c for i, = 0,...,n 1. Then he (Gaussian) likelihood funcion is he oin mulivariae Gaussian densiy funcion n / L (x ) = () 1/ exp( x T 1 x /). The maximum likelihood esimae (MLE) ˆ is obained by minimizing he quaniy log L (x ) = n log () + log + x x, T 1 where he righ hand side is reaed as a funcion of, wih x held fixed a is observed value.

6 -6- The chief drawback of he MLE is ha i may be quie difficul o compue. An explici formula for ˆ does no exis, so he esimaor mus be calculaed by numerical opimizaion mehods. These mehods are inherenly difficul o use unless he model dimension is small. Furhermore, he likelihood funcion iself is ofen very rough, and conains many local opima. As a resul, he compuaion of he MLE may require a grea deal of compuer ime. The compuaion of he likelihood funcion even for a single given value of is a non-rivial ask in iself, since in principle i requires he inversion of he n n marix. This problem has largely been solved, however, by aking advanage of he Toepliz naure of he covariance marix, or by using so- called Sae Space or Kalman Filer mehods. To save ime, compuaionally efficien mehods of parameer esimaion have been developed. Examples include he Burg and ule-walker mehods for AR models, and he leas squares and Hannan-Rissanen esimaors for more general models. Unforunaely, some of he compuaionally efficien esimaors do no perform as well as he MLE in erms of generally acceped performance cri- eria, such as (saisical) asympoic efficiency. Sill, hese esimaors are useful as saring values in he ieraive search for he MLE. Model Selecion Anoher imporan pracical problem arising in he applicaion of finie-parameer models is ha of model selecion. I is clear ha in mos pracical siuaions boh he model ype (for example, AR or MA ) as well as he corresponding model dimension (p or q ) will be unknown. Insead, we have a number of candidae models we are willing o ry, and we wan o selec he one which is "bes". Box and Jenkins refer o his as he model idenif icaion problem, a erminology which seems o imply ha one of he candidae models is in fac he rue model (i.e., he model which acually generaed he daa.) Box and Jenkins propose o idenify AR or MA models by examining plos of he sample auocorrelaion funcion {ρˆ }={ĉ /ĉ }, and anoher quaniy called he sample parial auocorrelaion funcr r 0 ion. I can be shown ha if he process is MA (q ), hen he populaion auocorrelaion funcion {ρ } = {c /c } will be zero for all lags exceeding q. Thus, aside from sampling variabiliy, he sample r r 0

7 -7- auocorrelaion funcion should display his same "cu-off" a lag q. Similarly, if he process is AR (p ), hen he sample parial auocorrelaion funcion should cu off a lag p, aside from sampling variabiliy. We, on he oher hand, do no wish o assume ha any of he candidae models is necessarily he rue model. In fac, we do no hink i is usually reasonable o assume ha he rue model is even finie-dimensional. Neverheless, i may be quie useful (especially for rapid on-line forecasing) o fi finie-dimensional models o he daa, as long as i is clearly recognized ha such models can only provide approximaions o he ruh. This poin of view is shared by many ime series analyss, including Tukey, Bloomfield, Akaike, Parzen and Shibaa. In his conex, an imporan and useful mehod for model selecion has been developed by Akaike. The mehod is called he Akaike Informaion Crierion ( AIC ). The mehod proceeds by compuing he crierion f uncion AIC = log (maximized likelihood) + (# parameers) for each of he candidae models, where (# parameers) denoes he number of parameers esimaed in he model. (For example, he AR (p ) model has p +1 parameers and he MA (q ) model has q +1 parame- ers, since he innovaion variance σ is included separaely in hese models.) The model which minimizes AIC is hen seleced for final use. If he models are no esimaed by maximum likelihood, hen he firs erm of AIC may be replaced by n log σˆ, where σˆ is he esimaed whie noise variance. I has been shown by Shibaa ha if he rue model is Gaussian AR () and does no degenerae o a finie-order auoregression, hen AIC provides an asympoically opimal model selecion, assuming ha he candidae models are finie-dimensional Gaussian AR models. Oher model selecion mehods include he CAT crierion of Parzen (for AR models only), he AIC C crierion of Hurvich and Tsai (his is a correcion o AIC o provide improved performance in small samples) and he Bayesian Informa- ion Crierion, BIC, of Schwarz (which assumes ha one of he candidae models is he rue model.) If m denoes he number of parameers in he candidae model, hen AIC C and BIC are given by IC = n log σˆ + (m + 1) n n m A C, BIC = n log σˆ + m log n.