Chpter 4 Models for Sttionry Time Series This chpter discusses the bsic concepts of brod clss of prmetric time series models the utoregressive-moving verge models (ARMA. These models hve ssumed gret importnce in modeling rel-world processes. 4. Generl Liner Processes We will lwys let { } denote the observed time series. From here on we will lso let { t } represent n unobserved white noise series, tht is, sequence of identiclly distributed, zero-men, independent rndom vribles. For much of our work, the ssumption of independence could be replced by the weker ssumption tht the { t } re uncorrelted rndom vribles, but we will not pursue tht slight generlity. A generl liner process, { }, is one tht cn be represented s weighted liner combintion of present nd pst white noise terms: = + t ψ + t ψ + t (4.. If the right-hnd side of this expression is truly n infinite series, then certin conditions must be plced on the ψ-weights for the right-hnd side to be mthemticlly meningful. For our purposes, it suffices to ssume tht (4.. We should lso note tht since { t } is unobservble, there is no loss in the generlity of Eqution (4.. if we ssume tht the coefficient on is, effectively, ψ =. An importnt nontrivil exmple to which we will return often is the cse where the ψ s form n exponentilly decying sequence: ψ j = φ j where φ is number between nd +. Then For this exmple i = ψ i < = + t φ + t φ + t EZ ( t = E ( + t φ t φ t =
Chpter 4 Models for Sttionry Time Series so tht { } hs constnt men of zero. Also, Furthermore, Vr( = Vr( + t φ + t φ + t = Vr( t + φ Vr( t + φ 4 Vr( t + = σ ( + φ + φ 4 + σ = -------------- φ (by summing geometric series Cov, = Cov ( φ t t φ t, + t φ t φ t 3 = Cov( φ t, t + Cov( φ t, φ t + = φσ + φ 3 σ + φ 5 σ + = φσ ( + φ + φ + φσ = -------------- φ (gin summing geometric series Thus φσ -------------- φ Corr(, = -------------- = φ σ -------------- φ φ k σ In similr mnner we cn find Cov(, k = -------------- φ nd thus Corr(, k = φ k (4..3 It is importnt to note tht the process defined in this wy is sttionry the utocovrince structure depends only on lgged time nd not on bsolute time. For generl liner process, = + t ψ + t ψ + t, clcultions similr to those done bove yield the following results: EZ ( t = γ = k Cov(, k = σ ψ i ψ k i + k (4..4 i = with ψ =. A process with nonzero men µ my be obtined by dding µ to the right-hnd side of Eqution (4..4. Since the men does not ffect the covrince
4. Moving Averge Processes properties of process, we ssume zero men until we begin fitting models to dt. 4. Moving Averge Processes In the cse where only finite number of the ψ-weights re nonzero, we hve wht is clled moving verge process. In this cse we chnge nottion somewht nd write = t θ t θ t θ q t q (4.. We cll such series moving verge of order q nd bbrevite the nme to MA(q. The terminology moving verge rises from the fct tht is obtined by pplying the weights, θ, θ,..., θ q to the vribles t, t, t,.., t q nd then moving the weights nd pplying them to t+, t, t,.., t q+ to obtin + nd so on. Moving verge models were first considered by Slutsky (97 nd Wold (938. The First-Order Moving Averge Process We consider in detil the simple, but nevertheless importnt moving verge process of order, tht is, the MA( series. Rther thn specilize the formuls in Eqution (4..4, it is instructive to rederive the results. The model is = t θ t. Since only one θ is involved, we drop the redundnt subscript. Clerly EZ ( t = nd Vr ( = σ ( + θ. Now Cov, = Cov ( θ t t, t θ t = Cov( θ t, t = θσ nd Cov, = Cov ( θ t t, t θ t 3 = since there re no s with subscripts in common between nd. Similrly, Cov(, k = whenever k ; tht is, the process hs no correltion beyong lg. This fct will be importnt lter when we need to choose suitble models for rel dt. In summry: for n MA( model = t θ t, The reson for this chnge will become pprent lter on in Section XX. 3
Chpter 4 Models for Sttionry Time Series EZ ( t = γ = Vr( γ = θσ ρ = ( θ ( + θ (4.. γ k = ρ k = for k Some numericl vlues for ρ versus θ in Exhibit (4. help illustrte the possibilities. Note tht the ρ vlues for θ negtive cn be obtined by simply negting the vlue given for the corresponding positive θ-vlue. Exhibit 4. Lg Autocorreltion for n MA( Process θ. ρ = θ ( + θ...99..9.3.75.4.345.5.4.6.44.7.47.8.488.9.497..5 A clculus rgument shows tht the lrgest vlue tht ρ cn ttin is.5 when θ is nd the smllest vlue is.5, which occurs when θ is +. (See Exercise (4.3 Exhibit (4. displys grph of the lg utocorreltion vlues for θ rnging from to +. 4
4. Moving Averge Processes Exhibit 4. Lg Autocorreltion of n MA( Process for Different θ.5.5 ρ. -.5 -.5 -.9 -.6 -.3..3.6.9 θ Exercise (4.4 sks you to show tht when ny nonzero vlue of θ is replced by /θ, the sme vlue for ρ is obtined. For exmple, ρ is the sme for θ =.5 s for θ = /.5 =. If we knew tht n MA( process hd ρ =.4 we still could not tell the precise vlue of θ. We will return to this troublesome point when we discuss invertibility in Section 4.5 on pge 6. Exhibit (4.3 shows time plot of simulted MA( series with θ =.9 nd normlly distributed white noise. Recll from Exhibit (4. tht ρ =.497 for this model; thus there is modertely strong positive correltion t lg. This correltion is evident in the plot of the series since consecutive observtions tend to be closely relted. If n observtion is bove the men level of the series then the next observtion lso tends to be bove the men. (A horizontl line t the theoreticl men of zero hs been plced on the grph to help in this interprettion. The plot is reltively smooth over time with only occsionl lrge fluctutions. 5
Chpter 4 Models for Sttionry Time Series Exhibit 4.3 Time Plot of n MA( Process with θ =.9 4 3 Z(t - - -3 4 6 8 Time The lg utocorreltion is even more pprent in Exhibit (4.4 which plots versus. Note the modertely strong upwrd trend in this plot. Exhibit 4.4 Plot of Z(t versus Z(t- for MA( Series in Exhibit (4.3 4 3 Z(t - - -3-3 - - 3 4 Z(t- The plot of versus in Exhibit (4.5 gives strong visuliztion of the zero utocorreltion t lg for this model. 6
4. Moving Averge Processes Exhibit 4.5 Plot of Z(t versus Z(t- for MA( Series in Exhibit (4.3 4 3 Z(t - - -3-3 - - 3 4 Z(t- A somewht different series is shown in Exhibit (4.6. This is simulted MA( series with θ = +.9. Recll from Exhibit (4. tht ρ =.497 for this model; thus there is modertely strong negtive correltion t lg. This correltion cn be seen in the plot of the series since consecutive observtions tend to be on opposite sides of the zero men. If n observtion is bove the men level of the series then the next observtion tends to be below the men. The plot is quite jgged over time especilly when compred to the plot in Exhibit (4.3. Exhibit 4.6 Time Plot of n MA( Process with θ = +.9 4 3 Z(t - - -3-4 4 6 8 Time 7
Chpter 4 Models for Sttionry Time Series The negtive lg utocorreltion is even more pprent in the lg plot of Exhibit (4.7. Exhibit 4.7 Plot of Z(t versus Z(t- for MA( Series in Exhibit (4.6 4 3 Z(t - - -3-4 -4-3 - - 3 4 Z(t- The plot of versus in Exhibit (4.8 displys the zero utocorreltion t lg for this model. Exhibit 4.8 Plot of Z(t versus Z(t- for MA( Series in Exhibit (4.6 4 3 Z(t - - -3-4 -4-3 - - 3 4 Z(t- MA( processes hve no utocorreltion beyond lg but by incresing the order of the process we cn obtin higher order correltions. The Second-Order Moving Averge Process Consider the moving verge process of order two: = t θ t θ t Here γ = Vr( = Vr( t θ t θ t = ( + θ + θ σ, 8
4. Moving Averge Processes γ = Cov (, Z Cov θ t = ( t t θ t, t θ t θ t 3 = Cov( θ t, t + Cov( θ t, θ t = [ θ + ( θ ( θ ]σ = ( θ + θ θ σ nd γ = Cov, = Cov ( θ t t θ t, t θ t 3 θ t 4 = Cov( θ t, t = θ σ Thus for n MA( process, θ + θ θ ρ = ---------------------------- + θ + θ ρ = θ --------------------------- + θ + θ ρ k = for k = 3, 4,... (4..3 For the specific cse = t t +.6 t, we hve + ( (.6.6 ρ = -------------------------------------------- + ( + (.6 = --------- =.678.36 nd.6 ρ = --------- =.54.36 A time plot of simultion of this MA( process is shown in Exhibit (4.9. The series tends to move bck nd forth cross the men in one time unit. This reflects the firly strong negtive uotcorreltion t lg. Once more, horizontl reference line t the theoreticl process men of zero is shown. 9
Chpter 4 Models for Sttionry Time Series Exhibit 4.9 Time Plot of n MA( Process with θ = nd θ =.6 5..5 Z(t. -.5-5. 4 6 Time 8 The plot in Exhibit (4. reflects tht negtive utocorreltion quite drmticlly. Exhibit 4. Plot of Z(t versus Z(t- for MA( Series in Exhibit (4.9 5..5 Z(t. -.5-5. -5. -.5. Z(t-.5 5. The wek postive sutocorreltion t lg two is displyed in Exhibit (4..
4. Moving Averge Processes Exhibit 4. Plot of Z(t versus Z(t- for MA( Series in Exhibit (4.9 5..5 Z(t. -.5-5. -5. -.5. Z(t-.5 5. Finlly, the lck of utocorreltion t lg three is pprent from the sctterplot in Exhibit (4.. Exhibit 4. Plot of Z(t versus Z(t-3 for MA( Series in Exhibit (4.9 5..5 Z(t. -.5-5. -5. -.5. Z(t-3.5 5. The Generl MA(q Process For the generl MA(q process, clcultions show tht nd = t θ t θ t θ q t q, similr γ = ( + θ + θ + + θ q σ θ k + θ θ k + + θ θ k + + + θ q k θ q ------------------------------------------------------------------------------------------------------ for k =,,..., q ρ k = + θ + θ + + θ q for k> q (4..4 (4..5
Chpter 4 Models for Sttionry Time Series where the numertor of ρ k is just θ q. The utocorreltion function cuts off fter lg q, tht is, it is zero. Its shpe cn be most nything for the erlier lgs. Another type of process, the utoregressive process, provides models for lternte utocorreltion ptterns. 4.3 Autoregressive Processes Autoregressive processes re s their nme suggests regressions on themselves. Specificly, pth-order utoregressive process { } stisfies the eqution = φ Z + t φ Z + t + φ p Z + t p t (4.3. The current vlue of the series is liner combintion of the p most recent pst vlues of itself plus n innovtion term t which incorportes everything new in the series t time t tht is not explined by the pst vlues. Thus for every t we ssume tht t is independent of,, 3,... Yule (97 crried out the originl work on utoregressive processes. The First-Order Autoregressive Process Agin, it is instructive to consider the first-order model, bbrevited AR(, in detil. Assume the series is sttionry nd stisfies = φ + t (4.3. where we hve dropped the subscript from the coeffient φ for simplicity. As usul, in these initil chpters we ssume tht the process men hs been subtrcted out so tht the the series men is zero. The conditions for sttionrity will be considered lter. We first tke vrinces of both sides of Eqution (4.3. nd obtin γ = φ γ + σ Solving for γ yields σ γ = -------------- (4.3.3 φ Notice the immedite impliction tht φ < or tht φ <. Now tke Eqution (4.3., multiply both sides by k (k =,,..., nd tke expected vlues. EZ ( t k Z t = φez ( t Z k t + E ( t k or γ k = φγ k + E ( t k Since the series is ssumed to be sttionry with zero men, nd since t is independent of k, we obtin E ( t Z E t k = ( EZ ( t t k =
4.3 Autoregressive Processes nd so γ k = φγ k for k =,, 3,... (4.3.4 Setting k =, we get γ = φγ = φσ ( φ. With k =, we obtin γ = φ σ ( φ. Now it is esy to see tht in generl γ k φ k σ = -------------- (4.3.5 φ nd thus γ k ρ k = ---- = φ k for k =,, 3,... (4.3.6 γ Since φ <, the mgnitude of the utocorreltion function decreses exponentilly s the number of lgs, k, increses. If < φ <, ll correltions re positive; if < φ <, the lg utocorreltion is negtive (ρ = φ nd the signs of successive utocorreltions lternte from positive to negtive with their mgnitudes decresing exponentilly. Portions of the grphs of severl utocorreltion functions re displyed in Exhibit (4.3 3
Chpter 4 Models for Sttionry Time Series Exhibit 4.3 Autocorreltion Functions for Severl AR( Models..8 φ =.9..8 φ =.4.6.6 ρ k ρ k.4.4... 3 4 5 6 Lg 7 8 9. 3 4 5 6 7 Lg 8 9. φ =.7.5 ρ k. -.5 -. 3 4 5 6 7 8 9 Lg Notice tht for φ ner ±, the exponentil decy is quite slow (for exmple, (.9 6 =.53, but for smller φ the decy is quite rpid (for exmple, (.4 6 =.4. With φ ner ±, the strong correltion will extend over mny lgs nd produce reltively smooth series if φ is positive nd very jgged series if φ is negtive. Exhibit (4.4 displys the time plot of simulted AR( process with φ =.9. Notice how infrequently the series crosses its zero theoreticl men. There is lot of inerti in the series it hngs together remining on the sme side of the men for extended periods. An observer might clim tht the series hs severl trends. We know tht, in fct, the theoreticl men is zero for ll time points. The illusion of trends is due to the strong utocorreltion of neighboring vlues of the series. 4
4.3 Autoregressive Processes Exhibit 4.4 Time Plot of n AR( Series with φ =.9 5 4 3 Z(t - - -3-4 3 4 5 6 Time The smoothness of the series nd the strong utocorreltion t lg is depicted in the lg plot shown in Exhibit (4.5. Exhibit 4.5 Plot of Z(t versus Z(t- for AR( Series of Exhibit (4.4 5 4 3 Z(t - - -3-4 -4-3 - - 3 4 5 Z(t- This AR( model lso hs strong, positive utocorreltion t lg, nmely ρ = (.9 =.8. Exhibit (4.6 shows this quite well. 5
Chpter 4 Models for Sttionry Time Series Exhibit 4.6 Plot of Z(t versus Z(t- for AR( Series of Exhibit (4.4 5 4 3 Z(t - - -3-4 -4-3 - - 3 4 5 Z(t- Finlly, t lg three the utocorreltion is still quite high: ρ 3 = (.9 3 =.79. Exhibit (4.7 confirms this for this prticulr series. Exhibit 4.7 Plot of Z(t versus Z(t-3 for AR( Series of Exhibit (4.4 5 4 3 Z(t - - -3-4 -4-3 - - 3 4 5 Z(t-3 The Generl Liner Process Version of the AR( Model The recursive definition of the AR( process given in Eqution (4.3. is extremely useful for interprettion of the model. For other purposes it is convenient to express the AR( model s generl liner process s in Eqution (4... The recursive definition is vlid for ll t. If we use this eqution with t replced by t, we get = φ + t. Substituting this into the originl expression gives 6
4.3 Autoregressive Processes = φφz ( t + t + t = t + φ t + φ If we repet this substitution into the pst, sy k times, we get = t + φ t + φ t + + φ k t k+ + φ k k (4.3.7 Assuming φ < nd letting k increse without bound, it seems resonble (this is lmost rigorous proof tht we should obtin the infinite series representtion = + t φ + t φ + t φ 3 + t 3 (4.3.8 This is in the form of the generl liner process of Eqution (4.. with ψ j = φ j which we lredy investigted in Section 4. on pge. Note tht this representtion reemphsizes the need for the restriction φ <. Sttionrity of n AR( Process It cn be shown tht, subject to the restriction tht t be independent of,, 3,... nd tht σ >, the solution of AR( defining recursion = φ + t will be sttionry if, nd only if, φ <. The requirement φ < is usully clled the sttionrity condition for the AR( process [See Box, Jenkins, nd Reinsel (994, p. 54, Nelson, 973, p. 39, nd Wei (99, p. 3] even though more thn sttionrity is involved. See lso Exercises (4.6, (4.8, nd (4.5. At this point we should note tht the utocorreltion function for the AR( process hs been derived in two different wys. The first method used the generl liner process representtion leding up to Eqution (4..3. The second method used the defining recursion = φ + t nd the development of Equtions (4.3.4, (4.3.5, nd (4.3.6. A third derivtion is obtined by multiplying both sides of Eqution (4.3.7 by k, tking expected vlues of both sides, nd using the fct tht t, t, t,..., t (k re independent of k. The second method should be especilly noted since it will generlize nicely to higher-order processes. The Second-Order Autoregressive Process Now consider the series stisfying = φ + φ + t (4.3.9 where, s usul, we ssume tht t is independent of,, 3,... To discuss sttionrity we introduce the AR chrcteristic polynomil φ( x = φ x φ x nd the corresponding AR chrcteristic eqution 7
Chpter 4 Models for Sttionry Time Series φ x φ x = We recll tht qudrtic eqution lwys hs two roots (possibly complex. Sttionrity of the AR( Process It my be shown tht, subject to the condition tht t is independent of,, 3,..., sttionry solution to Eqution (4.3.9 exists if, nd only if, the roots of the AR chrcteristic eqution exceed in bsolute vlue (modulus. We sometimes sy tht the roots should lie outside the unit circle in the complex plne. This sttement will generlize to the pth-order cse without chnge. In the second order cse the roots of the qudrtic chrcteristic eqution re esily found to be φ ± φ + 4φ ------------------------------------- (4.3. φ For sttionrity we require tht these roots exceed in bsolute vlue. In Appendix A we show tht this will be true if, nd only if, three conditions re stisfied: φ + φ <, φ φ <, nd φ < (4.3. As with the AR( model, we cll these the sttionrity conditions for the AR( model. This sttionrity region is displyed in Exhibit (4.8. It lso pplies in the first-order cse, where the AR chrcteristic eqution is just φx = with root /φ, which exceeds one in bsolute vlue if nd only if φ <. 8
4.3 Autoregressive Processes Exhibit 4.8 Sttionrity Region for AR( Process Prmeters rel roots φ + 4φ = φ complex roots - - - φ The Autocorreltion Function for the AR( Process To derive the utocorreltion function for the AR( cse, we tke the defining recursive reltionship of Eqution (4.3.9, multiply both sides by k, nd tke expecttions. Assuming sttionrity, zero mens, nd tht t is independent of k, we get γ k = φ γ k + φ γ k for k =,, 3,... (4.3. or, dividing through by γ, ρ k = φ ρ k + φ ρ k for k =,, 3,... (4.3.3 Equtions (4.3. nd/or (4.3.3 re usully clled the Yule-Wlker equtions especilly the set of two equtions obtined for k = nd. Setting k = nd using ρ = nd ρ = ρ, we get ρ = φ + φ ρ nd so φ ρ = -------------- (4.3.4 φ Using the now known vlues for ρ (nd ρ, Eqution (4.3.3 cn be used with k = to obtin ρ = φ ρ + φ ρ φ (4.3.5 ( φ + φ = -------------------------------------- φ Successive vlues of ρ k my be esily clculted numericlly from the recursive reltionship of Eqution (4.3.3. Although Eqution (4.3.3 is very efficient for clculting utocorreltion vlues numericlly from given vlues of φ nd φ, for other purposes it is desirble to hve more explicit formul for ρ k. The form of the explicit solution depends criticlly on the 9
Chpter 4 Models for Sttionry Time Series roots of the chrcteristic eqution φ x φ x =. Denoting the reciprocls of these roots by G nd G, it is shown in Appendix A tht For the cse φ φ + 4φ φ + φ + 4φ G = ------------------------------------- nd G = ------------------------------------- G G, it cn be shown tht we hve ( G Gk + ( G Gk + ρ k = -------------------------------------------------------------------------------- for k =,,,... ( G G ( + G G (4.3.6 If the roots re complex, tht is, if φ + 4φ <, then ρ k my be rewritten s ρ k R k sin( Θk + Φ = ------------------------------ for k =,,,... (4.3.7 sin( Φ where R = φ, nd Θ nd Φ re defined by cos( Θ = φ ( φ, nd tn( Φ = [( φ ( + φ ]. For completeness we note tht if the roots re equl, ( φ + 4φ =, then we hve + φ ρ k + --------------k φ ----- k = for k =,,,... (4.3.8 φ A good discussion of the derivtions of these formuls cn be found in Fuller (9XX, Sections XX The specific detils of these formuls re of little importnce to us. We need only note tht the utocorreltion function cn ssume wide vriety of shpes. In ll cses, the mgnitude of ρ k dies out exponentilly fst s the lg k increses. In the cse of complex roots, ρ k displys dmped sine wve behvior with dmping fctor R, R <, frequency Θ, nd phse Φ. Illustrtions of the possible shpes re given in Exhibit (4.9.
4.3 Autoregressive Processes Exhibit 4.9 Autocorreltion Functions for Severl AR( Models. φ =.5, φ =.5. φ =., φ =.5.75.75 ACF.5 ACF.5.5.5.. 3 4 5 6 7 8 9 Lg 3 4 5 6 7 8 9 Lg..75 φ =.5, φ =.75..75 φ =., φ =.6.5.5 ACF.5 ACF.5.. -.5 -.5 -.5 -.5 3 4 5 6 7 8 9 Lg 3 4 5 6 7 8 9 Lg Exhibit (4. displys the time plot of simulted AR( series with φ =.5 nd φ =.75. The periodic behvior of ρ k shown in Exhibit (4.9 is clerly reflected in the nerly periodic behvior of the series with the sme period of 36/3 = time units.
Chpter 4 Models for Sttionry Time Series Exhibit 4. Time Plot of n AR( Series with φ =.5 nd φ =.75 5 Series -5 4 36 48 6 7 84 96 8 Time The Vrince for the AR( Model The process vrince γ cn be expressed in terms of the model prmeters φ, φ, nd σ s follows: Tking the vrince of both sides of Eqution (4.3.9 yields γ = ( φ + φ γ + φ φ γ + σ (4.3.9 Setting k = in Eqution (4.3. gives second liner eqution γ nd γ, γ = φ γ + φ γ, which cn be solved simultneously with Eqution (4.3.9 to obtin ( φ σ γ = -------------------------------------------------------------------------- ( φ ( φ φ φ φ (4.3. φ σ = -------------- ---------------------------------- + φ ( φ φ The ψ-coefficients for the AR( Model The ψ-coefficients in the generl liner process representtion for n AR( series re more complex thn for the AR( cse. However, we cn substitute the generl liner process representtion using Eqution (4.. for, for, nd for into
4.3 Autoregressive Processes = φ + φ + t. If we then equte coefficients of j we get the recursive reltionships ψ = ψ φ ψ = (4.3. ψ j φ ψ j φ ψ j = for j =, 3,... These my be solved recursively to obtin ψ =, ψ = φ, ψ = φ + φ, nd so on. These reltionships provide excellent numericl solutions for the ψ-coefficients for given numericl vlues of φ nd φ. One cn lso show tht for G G n explcit solution is Gj + ψ Gj + j = ---------------------------------- (4.3. G G where, s before, G nd G re the reciprocls of the roots of the AR chrcteristic eqution. If the roots re complex Eqution (4.3. my be rewritten s ψ j R j sin( ( j + Θ = --------------------------------, (4.3.3 sin( Θ dmped sine wve with the sme dmping fctor R nd frequency Θ, s in Eqution (4.3.7 for the utocorreltion function. For completeness, we note tht if the roots re equl then ψ j = ( + jφj (4.3.4 The Generl Autoregressive Process Consider now the pth-order utoregressive model = φ Z + t φ Z + t + φ p Z + t p t (4.3.5 with AR chrcteristic polynomil φ( x = φ x φ x φ p x p (4.3.6 nd corresponding AR chrcteristic eqution φ x φ x φ p x p = (4.3.7 As noted erlier, ssuming tht t is independent of,, 3,..., sttionry solution to Eqution (4.3.7 exists if, nd only if, the p roots of the AR chrcteristic eqution exceed in bsolute vlue (modulus. Numericlly finding the roots of pth degree polynomil is nontrivil tsk, but simple lgorithm bsed on Schur s Theorem cn be used to check on the sttionrity condition without ctully finding the roots. See Appendix XX. Other reltionships between polynomil roots nd coefficients my be used to show tht the following two inequlities re neccesry for sttionrity. Tht is, for the roots to be greter thn in modulus it must be true tht both: 3
Chpter 4 Models for Sttionry Time Series φ + φ + + φ p < nd φ p < (4.3.8 Assuming sttionrity nd zero mens we my multiply Eqution (4.3.5 by k, tke expecttions, nd obtin the importnt recursive reltionship ρ k = φ ρ k + φ ρ k + + φ p ρ k p for k (4.3.9 Putting k =,,..., p into Eqution (4.3.9 nd using ρ = nd ρ k = ρ k, we get the generl Yule-Wlker equtions ρ = φ + φ ρ + φ 3 ρ + + φ p ρ p ρ = φ ρ + φ ρ + φ 3 ρ 3 + + φ p ρ p (4.3.3 ρ p = φ ρ p φ ρ p φ 3 ρ p 3 φ p Given numericl vlues for φ, φ,..., φ p, these liner equtions cn be solved to obtin numericl vlues for ρ, ρ,..., ρ p. Then Eqution (4.3.9 cn be used to obtin numericl vlues for ρ k t ny number of higher lgs. Noting tht E ( t = E [ t ( φ φ φ p p t ] = E ( t σ we my multiply Eqution (4.3.5 by, tke expecttions, nd find which, using ρ k = γ k /γ, cn be written s = γ = φ γ + φ γ + + φ p γ p + σ σ γ = ---------------------------------------------------------------------- φ ρ φ ρ φ p ρ p (4.3.3 nd express the process vrince γ in terms of the prmeters σ, φ, φ,..., φ p, nd the now-known vlues of ρ, ρ,..., ρ p. Of course, explicit solutions for ρ k re essentilly impossible in this generllity but we cn sy tht ρ k will be liner combintion of exponentilly decying terms (corresponding to the rel roots of the chrcteristic eqution nd dmped sine wve terms (corresponding to the complex roots of the chrcteristic eqution. Assuming sttionrity, the process cn lso be expressed in the generl liner process form of Eqution (4.., but the ψ-coefficients re complicted functions of the prmeters φ, φ,..., φ p. The coefficients cn be found numericlly: see XXXXX. 4
4.4 The Mixed Autoregressive-Moving Averge Model 4.4 The Mixed Autoregressive-Moving Averge Model If we ssume tht the series is prtly utoregressive nd prtly moving verge, we obtin quite generl time series model. In generl, if = φ Z + t φ Z + t + φ p Z + t p t θ t θ t (4.4. θ q t q we sy tht { } is mixed utoregressive-moving verge process of orders p nd q, respectively; we bbrevite the nme to ARMA(p,q. As usul, we discuss n importnt specil cse first. The ARMA(, Model The defining eqution cn be written = φ + t θ t To derive Yule-Wlker type equtions, we first note tht E ( t = E [ t ( φ + t θ t ] = σ nd E ( t = E [ t ( φ + t θ t ] = φσ θσ (4.4. = ( φ θσ If we multiply Eqution (4.4. by k nd tke expecttions, we hve γ = φγ + [ θφ ( θ ]σ γ = φγ θσ (4.4.3 γ k = φγ k for k Solving the first two equtions yields ( φθ + θ γ = -----------------------------------σ (4.4.4 φ nd solving the simple recursion gives ( θφ ( φ θ ρ k = ------------------------------------- (4.4.5 θφ θ φ k for k + Note tht this utocorreltion function decys exponentilly s the lg k increses. The dmping fctor is φ, but the decy strts from initil vlue ρ, which lso depends on θ. This is in contrst to the AR( utocorreltion, which lso decys with dmping fctor is φ, but lwys from initil vlue ρ =. For exmple, if φ =.8 nd θ =.4, then 5
Chpter 4 Models for Sttionry Time Series ρ =.53, ρ =.48, ρ 3 =.335 nd so on. Severl shpes for ρ k re possible, depending on the sign of ρ nd the sign of φ. The generl liner process form of the model cn be obtined in the sme mnner tht led to Eqution (4.3.8. We find tht is, = t + ( φ θ φ j t j, (4.4.6 j = ψ j = ( φ θφ j for j We should now mention the obvious sttionrity condition φ, or equivlently, the root of the AR chrcteristic eqution φx = must exceed unity in bsolute vlue. For the generl ARMA(p,q model, we stte the following fcts without proof: Subject to the condition tht t is independent of,, 3,..., sttionry solution to Eqution (4.4. exists, if, nd only if, ll the roots of the AR chrcteristic eqution φ(x = exceed unity in modulus. If the sttionrity conditions re stisfied, then the model cn lso be written s generl liner process with ψ-coefficients determined from ψ = ψ = θ + φ ψ = θ + φ + φ ψ ψ j = θ j + φ p ψ j p + φ p ψ j p+ + + φ ψ j where we tke ψ j = for j < nd θ j = for j > q. (4.4.7 Agin ssuming sttionrity, the utocorreltion function cn esily be shown to stisfy ρ k = φ ρ k + φ ρ k + + φ p ρ k p for k > q (4.4.8 Similr equtions cn be developed for k =,, 3,..., q tht involve θ, θ,..., θ q. An lgorithm suitble for numericl computtion of the complete utocorreltion function in given in Appendix XX. 4.5 Invertibility We hve seen tht for the MA( process we get excty the sme utocorreltion function if θ is replced by /θ. In the exercises we find similr problem with nonuniqueness for the MA( model. This lck of uniqueness of MA models given their utocorreltion functions must be ddressed before we try to infer the vlues of 6
4.5 Invertibility prmeters from observed time series. It turns out tht this nonuniqueness is relted to the seemingly unrelted question stted next. An utoregressive process cn lwys be reexpressed s generl liner process through the ψ-coefficients so tht n AR process my lso be thought of s n infinite-order moving verge process. However, for some purposes the utoregressive representtions re lso convenient. Cn moving verge model be reexpressed s n utoregression? To fix ides, consider n MA( model: = t θ t (4.5. First rewriting this s t = + θ t nd then replcing t by t nd substituting for t bove, we get t = + θ( + θ t = + θ + θ t If θ <, we my continue this substitution infinitely into the pst nd obtin the expression [compre with Equtions (4.3.7 nd (4.3.8.] t = Z + t θz + t θ Z + t or = ( θ θ θ 3 3 + t (4.5. If θ <, we see tht the MA( model cn be inverted into n infinite-order utoregressive model. We sy tht the MA( model is inverible if, nd only if, θ <. For generl MA(q or ARMA(p,q model, we define the MA chrcteristic polynomil s θ( x = θ x θ x θ 3 x 3 θ q x q (4.5.3 nd the corresponding MA chrcteristic eqution θ x θ x θ 3 x 3 θ q x q = (4.5.4 It cn be shown tht the MA(q model is invertible, tht is, there re coefficients π j such tht = π Z + t π Z + t π 3 Z + t 3 + t (4.5.5 if, nd only if, the roots of the MA chrcteristic eqution exceed in modulus. (Compre with sttionrity of n AR model. It my lso be shown tht there is only one set of prmeter vlues tht yield n invertible MA process with given utocorreltion function. For exmple, = t + t nd = t +(/ t both hve the sme utocorreltion function but only the second one with root is invertible. From here on we will resrict our ttention to the phsyiclly sensible clss of invertible models. For generl ARMA(p,q model we require both sttionrity nd invertibility. 7
Chpter 4 Models for Sttionry Time Series Chpter 4 Exercises 4. Find the utocorreltion function for the sttionry process defined by = 5 + t -- t + -- 4 t 4. Sketch the utocorreltion functions for the following MA( models with prmeters s specified: ( θ =.5 nd θ =.4 (b θ =. nd θ =.7 (c θ = nd θ =.6 4.3 Verify tht for n MA( process mx ρ < θ< =.5 nd min ρ < θ< 4.4 Show tht when θ is replced by /θ the utocorreltion function for n MA( process does not chnge. 4.5 Clculte nd sketch the utocorreltion functions for ech of the following AR( models. Use sufficient lgs tht the utocorreltion function hs nerly died out. ( φ =.6 (b φ =.6 (c φ =.95 (Do out to lgs. (d φ =.3 4.6 Suppose tht { } is n AR( process with < φ < +. ( Find the utocovrince function for W t = = in terms of φ nd σ. (b In prticulr, show tht Vr(W t = σ /(+φ..5 4.7 Describe the importnt chrcteristics of the utocorreltion function for the following models: ( MA(, (b MA(, (c AR(, (d AR(, nd (e ARMA(,. 4.8 Let { } be n AR( process of the specil form = φ + t. Use first principles to find the rnge of vlues of φ for which the process is sttionry. = 8
4.5 Invertibility 4.9 Use the recursive formul of Eqution (4- to clculte nd then sketch the utocorreltion functions for the following AR( models with prmeters s specified. In ech cse specify whether the roots of the chrcteristic eqution re rel or complex. If complex, find the dmping fctor, R, nd frequency, Θ, for the corresponding utocorreltion function when expressed s in Eqution (4-4. ( φ =.6 nd φ =.3 (b φ =.4 nd φ =.5 (c φ =. nd φ =.7 (d φ = nd φ =.6 (e φ =.5 nd φ =.9 (f φ =.5 nd φ =.6 4. Sketch the utocorreltion functions for ech of the following ARMA models: ( ARMA(, with φ =.7 nd θ =.4. (b ARMA(, with φ =.7 nd θ =.4. 4. For the ARMA(, model =.8 + t +.7 t +.6 t show tht ( ρ k =.8ρ k for k >. (b ρ =.8ρ +.6 σ /γ. 4. Consider two MA( processes, one with θ = θ = /6 nd nother with θ = nd θ = 6. ( Show tht these processes hve the sme utocorreltion function. (b How do the roots of the corresponding chrcteristic polynomils compre? 4.3 Let { } be sttionry process with ρ k = for k >. Show tht we must hve ρ.5. (Hint: Consider Vr(Z n+ + Z n +...+ Z nd then Vr(Z n+ Z n + Z n...± Z. Use the fct tht both of these must be nonnegtive for ll n. 4.4 Suppose tht { } is zero men, sttionry process with ρ <.5 nd ρ k = for k >. Show tht { } must be representble s n MA( process. Tht is, show tht there is white noise sequence { t } such tht = t θ t where ρ is correct nd t is uncorrelted with k for k >. (Hint: Choose θ such tht θ < nd ρ = θ/(+θ ; then let t = θ j Z. If we ssume tht { } is norml j = t j process, t will lso be norml nd zero correltion is equivlent to independence. 4.5 Consider the AR( model = φ + t. Show tht if φ = the process cnnot be sttionry. (Hint: Tke vrinces of both sides. 9
Chpter 4 Models for Sttionry Time Series 4.6 Consider the nonsttionry AR( model = 3 + t. ( Show tht = -- stisfies the AR( eqution. 3 j j = t + j (b Show tht the process defined in prt ( is sttionry. (c In wht wy is this solution unstisfctory? 4.7 Consider process tht stisfies the AR( eqution =.5 + t. ( Show tht = (.5 t + t +.5 t + (.5 t + is solution of the AR( eqution. (b Is the solution given in prt ( sttionry? 4.8 Consider process tht stisfies the zero-men, sttionry AR( eqution = φ + t with <φ<+. Let c be ny non-zero constnt nd define Y t = + cφ t. ( Show tht E(Y t = cφ t so tht {Y t } is nonsttionry. (b Show tht {Y t } stisfies the sttionry AR( eqution Y t = φy t + t. (c Is {Y t } sttionry? 4.9 Consider n MA(6 model with θ =.5, θ =.5, θ 3 =.5, θ 4 =.65, θ 5 =.35, nd θ 6 =.565. Find much simpler model tht nerly the sme ψ weights. 4. Consider n MA(7 model with θ =, θ =.5, θ 3 =.5, θ 4 =.5, θ 5 =.65, θ 6 =.35, nd θ 7 =.565. Find much simpler model tht nerly the sme ψ weights. 4. Consider the model = t t +.5 t 3. ( Find the utocovrince function for this process. (b Show tht this certin ARMA(p,q process in disguise. Tht is, identify vlues for p nd q, nd for the θ s nd φ s such tht the ARMA(p,q process hs the sme sttisticl properties s { }. 4. Show tht the sttement The roots of φ x φ x φ p x p = re greter thn in bsolute vlue is equivlent to the sttement The roots of x p φ x p φ x p φ p = re less thn in bsolute vlue. (Hint: If G is root of one eqution, is /G root of the other? 4.3 Suppose tht { } is n AR( process with ρ = φ. Define the sequence {b t } s b t = φ+. ( Show tht Cov(b t,b t k = for ll t nd k. (b Show tht Cov(b t,+k = for ll t nd k >. 3
4.5 Invertibility 4.4 Let { t } be zero men, unit vrince white noise process. Consider process tht begins t time t = nd is defined recursively s follows. Let Z = c nd Z = c Z +. Then let = φ + φ + t for t > s in n AR( process. ( Show tht the process men is zero. (b For prticulr vlues of φ nd φ within the sttionrity region for n AR( model, show how to choose c nd c so tht both Vr(Z = Vr(Z nd the lg utocorreltion between Z nd Z mtches tht of sttionry AR( process with prmeters φ nd φ. (c Once the process { } is generted, show how to trnsform it to new process tht hs ny desired men nd vrince. (This exercise suggests convenient method for simulting sttionry AR( processes. 4.5 Consider n AR( process stisfying = φ + t where φ cn be ny number nd { t } is white noise process such tht t is independent of the pst {,, }. Let Z be rndom vrible with men µ nd vrince σ. ( Show tht for t > we cn write = t + φ t + φ t + φ 3 t 3 + + φ t + φ t Z. (b Show tht for t > we hve E( = φ t µ. (c Show tht for t > Vr( = φ ---------------- t φ σ + φ t σ for φ tσ + σ for φ = (d Suppose now tht µ =. Argue tht, if { } is sttionry, we must hve φ. (e Continuing to suppose tht µ =, show tht, if { } is sttionry, then Vr( = σ ( φ nd so we must hve φ <. Appendix A: The Sttionrity Region for n AR( Process In the second order cse the roots of the qudrtic chrcteristic polynomil re esily found to be φ ± φ + 4φ ------------------------------------- (4.A. φ For sttionrity we require tht these roots exceed in bsolute vlue. We now show tht this will be true if, nd only if, three conditions re stisfied: 3
Chpter 4 Models for Sttionry Time Series φ + φ <, φ φ <, nd φ < Proof: Let the reciprocls of the roots be denoted G nd G. Then (4.A. φ G = ------------------------------------------ = φ φ + 4φ φ ------------------------------------------ φ φ + 4φ φ + φ + 4φ ------------------------------------------ φ + φ + 4φ = φ ( φ + φ + 4φ --------------------------------------------------------- φ ( φ + 4φ = φ φ + 4φ ------------------------------------- φ + φ + 4φ Similrly, G = ------------------------------------- We now divide the proof into two cses corresponding to rel nd complex roots. The roots will be rel if, nd only if, φ + 4φ. I. Rel Roots: G i < for i = nd if, nd only if, or φ φ + 4φ φ + φ + 4φ < ------------------------------------- < -------------------------------------- < < φ φ + 4φ < φ + φ + 4φ <. Consider just the first inequlity. Now < φ φ + 4φ if, nd only if, φ + 4φ < φ + if, nd only if, φ + 4φ < φ + 4φ + 4, if, nd only if, φ < φ +, or φ φ <. The inequlity φ + φ + 4φ < is treted similrly nd leds to φ + φ <. These equtions together with φ + 4φ define the sttionrity region for the rel root cse shown in Exhibit (4.8. II. Complex Roots: Now φ + 4φ <. Here G nd G will be complex conjugtes nd G = G < if, nd only if, G <. But G = ( φ + ( φ 4φ 4 = φ so tht φ >. This together with the inequlity φ + 4φ < defines the prt of the sttionrity region for complex roots shown in Exhibit (4.8 nd estblishes Eqution (4.3.. This completes the proof. 3