1 Linear Difference Equations

Transcription

1 ARMA Handout Jialin Yu 1 Linear Difference Equations First order systems Let {ε t } t=1 denote an input sequence and {y t} t=1 sequence generated by denote an output y t = φy t 1 + ε t t = 1, 2,... with y 0 given The equation is a first order difference equation. It is easily solved by recursive substitution p-th order systems Consider a p-th order system t 1 y t = φ t y 0 + φ i ε t i y t = φ 1 y t 1 + φ 2 y t φ p y t p + ε t To solve this system it is useful to write it in vector first order form (called companion form) as or y t y t 1 y t 2... y t p+1 φ 1 φ 2 φ 3... φ p = Z t = ΦZ t 1 + e t y t 1 y t 2 y t 3... y t p ε t

2 with Z t = y t y t 1 y t 2... y t p+1 etc. By recursive substitution, t 1 Z t = Φ t Z 0 + Φ i e t i so that a solution is obtained given Z 0 = (y 0, y 1,..., y t p+1 ). Stability: The linear difference equation Z t = ΦZ t 1 + e t is stable if, when e t = e for all t and Z 0 is an arbitrary constant, them lim t Z t = Z where Z does not depend on Z 0 In the first-order case, stability is achieved if φ < 1 In the higher order case, what is required is that lim t Φt = 0 and t 1 Φ i converges These two conditions are equivalent to the conditions that all eigenvalues of Φ are less than one in modulus (absolute value). To see this, assume for the moment that the eigenvalues of Φ are distinct. In this case we can decompose Φ = P ΛP 1 where the columns of P are eigenvectors of Φ and Λ is a diagonal matrix with the eigenvalues of Φ on the diagonal (see Hamilton). 2

3 Thus Φ 2 = P ΛP 1 P ΛP 1 = P Λ 2 P 1 and Φ t = P Λ t P 1 When the eigenvalues of Φ are not distinct, a similar argument can be applied to the Jordan decomposition of Φ (see Hamilton). 2 Stochastic Processes In stochastic models, the input sequence {Y t } t=1 of random variables form an example of stochastic processes. Some useful definitions: Stochastic process: the probability law governing {Y t } t=1 Realization: One draw from the process, i.e. {y t } t=1. Note that {y t } t=1 is only ONE observation instead of an infinite number of observations of the stochastic process. What is the implication and how to deal with this problem? Strict Stationarity: The process is strict stationary if the probability distribution of (Y t, Y t+1,..., Y t+k ) is identical to the probability distribution of (Y τ, Y τ+1,..., Y τ+k ) for all t, τ, k. (Thus, all joint distributions are time invariant) Autocovariances: The autocovariances are λ t,k = cov ( Y t, Yt+k T ) Autocorrelations: The autocorrelations are ρ t,k = corr ( Y t, Y T t+k). If Y 1 correlates with Y 2, Y 2 correlates with Y 3, does Y 1 correlate with Y 3? Covariance Stationarity: The process is covariance stationary if µ t = E (Y t ) = µ for all t λ t,k = λ k for all t 3

4 Thus the means and autocovariances do not depend on time. When Y is scalar, covariance stationary implies λ k = λ k and when Y is a vector, covariance stationarity implies λ k = λ T k Why? consider the example where ( ) ( xt+1 a = y t+1 c }{{} Z t+1 b d ) (xt ) + ε t+1 y t }{{} Z t Assuming Z t is i.i.d. standard bi-variate normal. In this case, ( ) a c λ 1 = Cov (Z t, Z t+1 ) = b d ( ) a b λ 1 = c d Trend Stationary: The process is trend stationary if Y t f (t) is stationary, for appropriately chosen f (.). In many cases, f (t) is taken to be f (t) = α + βt. In the case of stock price, the trend can represent for example equity premium. Difference Stationary: A process is difference stationary if Y t Y t 1 is stationary. For example, if Y t is stock price, does difference stationary mean the stock price follows a random walk? (No, there might be 4

5 other variables that can predict stock return, see the various notions of market efficiency.) What is the implication for proving stock market efficiency? It is really equivalent to prove a negative statement there does not exist a variable that can predict stock return. Let denote the differencing operator so that Y t = Y t Y t 1, then sometimes a series is said to be Integrated of order d written as I (d) if d Y t is stationary, but d 1 Y t is not stationary. White noise: A process is called white noise if it is covariance stationary, µ = 0 and λ k = 0 for k > 0 Martingale Process: Y t is a martingale if E [Y t Ω t 1 ] = Y t 1 where Ω t 1 Ω t is the time t 1 information set. Often Ω t = {Y τ } t τ=0. Martingale Difference Process: Y t is a martingale difference process if E [Y t Ω t 1 ] = 0 Markov process: {X t } is Markov if, given X t, the distribution of X s for s > t does not depend on X u for u < t. Is a Markov process martingale? Is a martingale a markov process? 3 Autoregressive Processes An example of stochastic process is Y T = φ 1 Y T 1 + φ 2 Y T φ p Y T p + ε t where ε t iid ( 0, σ 2) and (Y 0, Y 1,...Y p+1 ) is independent of {ε t } t=1 with mean µ 0 and variance σ0 2. This process is called Autoregressive Process of 5

6 Order p, abbreviated as AR(p). In the AR(1) model t 1 Y t = φy t 1 + ε t implies Y t = φ t Y 0 + φ i ε t i It can be verified that µ t = EY t = φ t µ 0 λ t,0 = V ary t = φ 2t σ φ2t 1 φ 2 σ2 for φ < 1 = σ tσ 2 for φ = 1 And these are time invariant if 1. φ < 1 2. µ 0 = 0, σ 2 0 = σ2 1 φ 2 It then can be verified that λ t,k = φk def σ2 = λ 1 φ2 k which does not depend on t. Hence condition 1 and 2 are necessary and sufficient for covariance stationarity. given 1, Y 0 only has a transitory effect on the process. Often only 1 is mentioned because, But if you want to simulate a truely stationary process, the initial observation of an AR(1) model needs to be drawn from its stationary distribution. For AR(p) model, there are similar restrictions that lead to covariance stationarity. Like in AR(1), there are two sets of restrictions. The first is that the process is stable, that is all the eigenvalues of Φ are less than 1 in modulus. The second involves the mean and variance of the initial conditions. These are worked out in detail in Hamilton. 6

7 4 The Lag Operator A useful notational device is the lag operator, denoted by L (some authors use B). In general, L is an operator that maps the sequence {y t } t= into another sequence {x t } t=. Specifically, L lags the sequence one period. Thus If b denotes a constant Ly t = y t 1, L 2 y t = y t 2,..., L p y t = y t p bly t = by t 1 = Lby t We can use this operator to write AR(p) model as Y t = φ 1 Y t 1 + φ 2 Y t φ p Y t p + ε t = φ 1 LY t + φ 2 L 2 Y t φ p L p Y t + ε t or ( 1 φ1 L φ 2 L 2... φ p L p) Y t = ε t or φ (L) Y t = ε t with φ (L) = ( 1 φ 1 L φ 2 L 2... φ p L p). Notice that the operator φ (L) is a p th order polynomial in the lag operator L. This polynomial is called the autoregressive polynomial. As an exercise, you should show that the roots of the autoregressive polynomial are the reciprocals of the eigenvalues of the companion matrix Φ (the zeros of the autoregressive polynomial are the values of z that makes φ (z) = 0). Thus, often the conditions for covariance stationary are stated as The roots of AR polynomial are greater than 1 in modulus. The lag operator is useful because it can be manipulated in familiar algebraic ways to simplify calculations. For example, in the AR(1) process, we can write each observation as the sum of lagged shocks t 1 Y t = φ t Y 0 + φ i ε t i 7

8 It is quite tedious to do this for AR(p) process. With the lag operator, this task can be accomplished in a less tedious way. To begin, for AR(1) process, one can write (1 φl) Y t = ε t and we want to find the operator that maps {ε t } into {Y t }, that is we seek a c (L) such that Y t = c (L) ε t Since (1 φl) Y t = ε t, it is natural to write c (L) = (1 φl) 1. What does this mean? Recall (1 x) 1 = 1 + x + x 2 + x when x < 1. This suggests writing (1 φl) 1 = 1 + φl + φ 2 L 2 + φ 3 L which yields the solution Y t = φ i ε t i which is the same one would obtain from infinite recursive substitution. Thus, inverting (1 φl) algebraically gives the right answer. A word of warning however: The algebraic inverse of the polynomial (1 φz) where z is a variable, is not unique. To see this, note (1 φz) φ 1 z 1 = φ 1 z 1 1 so that and thus 1 φz = φ 1 z 1 1 φ 1 z 1 (1 φz) 1 = φ 1 z 1 ( 1 φ 1 z 1) 1 = φ 1 z 1 ( 1 + φ 1 z 1 + φ 2 z ) 8

9 which suggests writing (1 φl) 1 = φ 1 L 1 ( 1 + φ 1 L 1 + φ 2 L ) which implies the solution Y t = φ i ε t+i i=1 Where does this come from? Y t+1 = φy t + ε t+1 hence Y t = φ 1 Y t+1 + φ 1 ε t+1 = φ 2 Y t+2 + φ 2 ε t+2 + φ 1 ε t+1 =... This is the forward solution to the difference equation which could be deduced, for example, by forward recursive substitution. How do we choose between the two solutions Y t = φ i ε t i and Y t = φ i ε t+i i=1 One approach is to impose a side condition on the solution: bounded input sequences {ε t } must lead to bounded output sequences. This would rule out the forward solution if φ < 1 since φ t will explode as t grows large. Analogously, it would rule out the backward solution if φ > 1. This yields a solution rule: Solve stable roots backwards and unstable roots forward. 9

10 5 Moving Average Models We say that Y t follows a moving average process of order q if Y t = ε t θ 1 ε t 1 θ 2 ε t 2... θ q ε t q where ε t iid ( 0, σ 2). Example: where might you see moving average models. Market microstructure models. E.g., assume the observed price equals the true fundamental (which follows random walk) plus a bid-ask bounce then the return p t = p t + ε t r t = p t p t 1 = p t p t 1 + ε t ε t 1 which is in the form of a MA model (for now, this is easy to see by assuming p t p t 1 = 0, later we ll show that this assumption does not matter). Some properties Covariance Stationary MA(1) model: Y t = ε t θε t 1 E (Y t ) = 0 V ar (Y t ) = σ 2 ( 1 + θ 2) Cov (Y t, Y t 1 ) = θσ 2 (given this, is stock return positively or negatively auto-correlated if there is bid-ask bounce?) Cov (Y t, Y t k ) = 0 for k > 1 Hence the process is covariance stationary MA(q) model: Y t = ε t θ 1 ε t 1 θ 2 ε t 2... θ q ε t q E (Y t ) = 0 V ar (Y t ) = σ 2 ( 1 + q i=1 θ2 i 10 )

11 Invertibility Cov (Y t, Y t k ) = σ 2 ( θ k + q k j=1 θ jθ k+j ) for k q Cov (Y t, Y t k ) = 0 for k > q Hence the process is covariance stationary Motivation: Consider forecasting in MA(1) model Y t = ε t θε t 1. Thus, the forecast of Y t constructed at time t 1 would be θε t 1. The problem is that ε t 1 is not directly observed; it must be constructed from the lagged values of Y t. How can this be done? Note t 1 ε t = θ t ε 0 + θ i Y t i Thus, the sequence of ε s could be formed from present and lagged Y s if ε 0 were known. But of course ε 0 is unknown. The process is invertible (meaning that the ε s can be determined from lagged Y s) if t 1 ε t = θ i qm Y t i εt as t i.e., assuming ε 0 = 0 has no lasting effect on the quality of the forecasts. For the MA(1) model, invertibility requires θ < 1. Equivalently, writing Y t = (1 θl) ε t, invertibility requires that the roots of θ (z) = 1 θz are greater than 1 in modulus For the MA(q) model, we get an analogously result, namely that the process is invertible if the roots of θ (z) = 1 θ 1 z θ 2 z 2... θ q z q are greater than 1 in modulus Another motivation for the invertibility restriction comes from considering the autocovariances of the MA(1) model λ 0 = σ 2 ( 1 + θ 2) λ 1 = θσ 2 11

12 Note we could also describe these two autocovariances as ( λ 0 = σ θ 2) λ 1 = θ σ 2 with θ = θ 1 and σ 2 = σ 2 (1 + θ 2 ) 1 (1 + θ 2 ). That is, the two MA(1) processes Y t = ε t θε t 1 with V ar (ε t ) = σ 2 and Y t = ε t θ ε t 1 with V ar ( ε t ) = σ 2 have exactly the same autocovariances. Thus, given data on Y t, we can t tell the processes apart, at least using the first two moments of the data. These two models are Observationally Equivalent for their first two moments. Since θ = θ 1, one way to (arbitrarily) choosen between them is to impose the restriction that the MA parameter is 1 in absolute value. 6 Autocovariance Generating Functions The autocovariance generating function for a covariance stationary process is given by λ (z) = λ j z j j= so that the autocovariances are given by the coefficients on the argument z j. It s purpose (or one of the purposes) is the same as the moment generating function - namely it is a convenient way to store the autocovariances of a covariance stationary stochastic process. For the MA process, the ACGF is particularly easy to construct. Suppose Y t = θ (L) ε t 12

13 then λ (z) = σ 2 θ (z) θ ( z 1) We will verify this formula for MA(1) process - you should verify it for higher order MA processes For an MA(1) process θ (z) = (1 θz), so that θ (z) θ ( z 1) = (1 θz) ( 1 θz 1) = θz 1 + ( 1 + θ 2) θz which implies autocovariances λ 1 = θσ 2 λ 0 = σ 2 ( 1 + θ 2) λ 1 = θσ 2 with all other autocovariances equal to 0. correct answer for an MA(1) process. Thus, the formula yields the Now, return to the issue of invertibility. Consider the MA polynomial for the MA(q) model θ (z) = 1 θ 1 z θ 2 z 2... θ q z q Suppose that this polynomial has zeros at z = γ 1 we can factor the polynomial as and so that ACGF is given by 1, γ 1 2 θ (z) = (1 γ 1 z) (1 γ 2 z)... (1 γ q z),..., γ 1 q. In this case, λ (z) = σ 2 (1 γ 1 z) (1 γ 2 z)... (1 γ q z) ( 1 γ 1 z 1) ( 1 γ 2 z 1)... ( 1 γ q z 1) But since (1 γz) ( 1 γz 1) is proportional to ( 1 γ 1 z ) ( 1 γ 1 z 1) (recall the discussion of invertibility in the MA(1) model), we can flip or invert the roots of the MA polynomial, change σ 2 to adjust for the factor of porportionality, and obtain the same ACGF hence a model with the same 13

14 autocovariances. Thus, for example, in the MA(2) model, if θ (z) = (1 γ 1 z) (1 γ 2 z) then models with are observationally equivalent. θ 1 (z) = ( 1 γ 1 1 z) (1 γ 2 z) θ 2 (z) = (1 γ 1 z) ( 1 γ 1 2 z) θ (z) = ( 1 γ 1 1 z) ( 1 γ 1 2 z) 7 Autoregressive-Moving Average (ARMA) Models Autoregressive-Moving Average models combine the simple AR and MA models. The ARMA(p,q) model is Y t = φ 1 Y T 1 +φ 2 Y T φ p Y T p +ε t θ 1 ε t 1 θ 2 ε t 2... θ q ε t q or φ (L) Y t = θ (L) ε t with and φ (L) = 1 φ 1 L φ 2 L 2... φ p L p θ (L) = 1 θ 1 L θ 2 L 2... θ q L q Conditions for covariance stationary and invertibility are just the same as in the simple models: the roots of φ (z) and θ (z) are greater than 1 in modulus. 14

15 The ACGF for the ARMA model can be derived as follows. Since φ (L) Y t = θ (L) ε t then Y t = c (L) ε t with c (L) = φ (L) 1 θ (L) which is a well defined (mean square convergent) polynomial in positive powers of L (i.e. backward looking) if the roots of φ (z) are greater than 1 in absolute value. Thus Y t has the MA representation Y t = c (L) ε t so that λ (z) = σ 2 c (z) c ( z 1) = σ 2 φ (z) 1 θ (z) φ ( z 1) 1 ( θ z 1 ) 8 ARIMA Models Suppose Y t is integrated of order d so that Y t must be differenced d times to be stationary. Let X t = (1 L) d Y t and suppose that X t follows an ARMA(p,q) model. Then we say that Y t follows an ARIMA(p,d,q) model. (The extra I in ARIMA stands for integrated ). 15