Identifiability, Invertibility Defn: If {ǫ t } is a white noise series and µ and b 0,..., b p are constants then X t = µ + b 0 ǫ t + b ǫ t + + b p ǫ t p is a moving average of order p; write MA(p). Q: From observations on X can we estimate the b s and σ 2 = Var(ǫ t ) accurately? NO. Defn: Model for data X is family {P θ ; θ Θ} of possible distributions for X. Defn: Model is identifiable if θ θ 2 implies P θ P θ2 ; different θ s give different distributions for data. Unidentifiable model: there are different values of θ which make exactly the same predictions about the data. So: data do not distinguish between these θ values. 34
Example: Suppose ǫ is an iid N(0, σ 2 ) series and that X t = b 0 ǫ t +b ǫ t. Then the series X has mean 0 and covariance C X (h) = (b 2 0 + b2 )σ2 h = 0 b 0 b σ 2 h = 0 otherwise Fact: normal distribution is specified by its mean and its variance. Consequence: two mean 0 normal time series with the same covariance function have the same distribution. Observe: if you multiply the ǫ s by a and divide both b 0 and b by a then the covariance function of X is unchanged. Thus: cannot hope to estimate all three parameters, b 0, b and σ. Arbitrary choice: b 0 = 35
Are parameters b and σ identifiable? We try to solve the equations and C(0) = ( + b 2 )σ 2 C() = bσ 2 to see if the solution is unique. Divide the two equations to see or C() C(0) = b + b 2 b 2 C(0) C() b + = 0 which has the solutions (C(0) C(0) C() ± ) 2 C() 4 2 36
You should notice two things:. If C(0) C() < 2 there are no solutions. Since C(0) = Var(X t )Var(X t+ ) we see C()/C(0) is the correlation between X t and X t+. So: have proved that for an MA() process this correlation cannot be more than /2 in absolute value. 2. If C(0) C() > 2 there are two solutions. 37
Note: two solutions multiply together to give the constant term in the quadratic equation. If two roots are distinct it follows that one of them is larger than and the other smaller in absolute value. Let b and b denote the two roots. Let α = C()/b and α = C()/b. Let ǫ t be iid N(0, α) and ǫ t be iid N(0, α ). Then and X t ǫ t + bǫ t X t ǫ t + b ǫ t have identical means and covariance functions. Observing X t you cannot distinguish the first of these models from the second. We will fit M A() models by requiring our estimated b to have ˆb. 38
Reason: manipulate model equation for X as for autoregressive process: ǫ t = X t bǫ t = X t b(x t bǫ t 2 ) =. 0 ( b) j X t j This manipulation makes sense if b <. If so then we can rearrange the equation to get X t = ǫ t ( b) j X t j which is an autoregressive process. 39
If, on the other hand, b > then we can write X t = b bǫ t + bǫ t Let ǫ t = bǫ t; ǫ is also white noise. We find ǫ t = X t b ǫ t = X t b (X t+ b ǫ t+ ) which means =. 0 ( b )j X t+j X t = ǫ t ( b )j X t+j This represents the current value as depending on the future which seems physically far less natural than the other choice. 40
Defn: An M A(p) process is invertible if it can be written in the form X t = a j X t j + ǫ t Defn: A process X is an autoregression of order p (written AR(p)) if X t = p a j X t j + ǫ t (so an invertible MA is an infinite order autoregression). Defn: The backshift operator transforms a time series into another time series by shifting it back one time unit; if X is a time series then BX is the time series with (BX) t = X t. The identity operator I satisfies IX = X. We use B j for j =,2,... to denote B composed with itself j times so that (B j X) t = X t j For j = 0 this gives B 0 = I. 4
Now use B to develop a formal method for studying the existence of a given AR(p) and the invertibility of a given MA(p). An AR() process satisfies (I a B)X = ǫ Think of I a B as infinite dimensional matrix; get formal identity X = (I a B) ǫ So how will we define this inverse of an infinite matrix? We use the idea of a geometric series expansion. If b is a real number then ( ab) = ab = j=0 (ab) j so we hope that (I a B) can be defined by (I a B) = a j Bj j=0 42
This would mean X = j=0 a j Bj ǫ or looking at the formula for a particular t and remembering the meaning of B j we get X t = j=0 a j ǫ t j This is the formula I had in lecture 2. Now consider a general AR(p) process: (I p a j B j )X = ǫ We will factor the operator applied to x. Let φ(x) = p a j x j Then φ is degree p polynomial so it has (theorem of C. F. Gauss) p roots /b,...,/b p. (None of the roots is 0 because the constant term in φ is.) This means we can factor φ as φ(x) = p ( b j x) 43
Now back to the definition of X: p (I b j B)X = ǫ can be solved by inverting each term in the product (in any order the terms in the product commute) to get X = p (I b j B) ǫ The inverse of I b B will exist if the sum b k j Bk k=0 converges; this requires b j <. Thus a stationary AR(p) solution of the equations exists if every root of the characteristic polynomial φ is larger than in absolute value (actually the roots can be complex and I mean larger than in modulus). 44
Summary An MA(q) process X t = ǫ t q j= b jǫ t j is invertible iff all roots of characteristic polynomial ψ(x) = q j= b jx j lie outside unit circle in complex plain. For given covariance function of an MA(q) process there is only one set of coefficients b,..., b q for which the process is invertible. An AR(p) process X t q j= a jx t j = ǫ t is asymptotically stationary iff all roots of characteristic polynomial φ(x) = p a jx j lie outside unit circle in complex plain. (Asymptotically stationary means: make X, X 2,..., X p anything; use equation defining AR(p) to define rest of X values; then as t the process gets closer to being stationary. Asymptotic stationarity is equivalent to existence of an exactly stationary solution of equations.) 45
Defn: A process X is an ARMA(p, q) (mixed autoregressive of order p and moving average of order q) if it satisfies φ(b)x = ψ(b)ǫ where ǫ is white noise and and φ(b) = I ψ(b) = I p q a j B j b j B j The ideas we used above can be stretched to show that the process X is identifiable and causal (can be written as an infinite order autoregression on the past) if the roots of ψ(x) lie outside the unit circle. A stationary solution, which can be written as an infinite order causal (no future ǫs in the average) moving average, exists if all the roots of φ(x) lie outside the unit circle. 46