Introduction to Time Series Analysis. Lecture 7.

Transcription

1 Last lecture: Introduction to Time Series Analysis. Lecture 7. Peter Bartlett 1. ARMA(p,q) models: stationarity, causality, invertibility 2. The linear process representation of ARMA processes: ψ. 3. Autocovariance of an ARMA process. 4. Homogeneous linear difference equations. 1

2 Introduction to Time Series Analysis. Lecture 7. Peter Bartlett 1. Review: ARMA(p,q) models and their properties 2. Review: Autocovariance of an ARMA process. 3. Homogeneous linear difference equations. Forecasting 1. Linear prediction. 2. Projection in Hilbert space. 2

3 Review: Autoregressive moving average models An ARMA(p,q) process {X t } is a stationary process that satisfies φ(b)x t = θ(b)w t, where φ, θ are degree p, q polynomials and {W t } WN(0, σ 2 ). We ll insist that the polynomials φ and θ have no common factors. 3

4 Review: Properties of ARMA(p,q) models Theorem: If φ and θ have no common factors, a (unique) stationary solution {X t } to φ(b)x t = θ(b)w t exists iff φ(z) = 1 φ 1 z φ p z p = 0 z 1. This ARMA(p,q) process is causal iff φ(z) = 1 φ 1 z φ p z p = 0 z > 1. It is invertible iff θ(z) = 1 + θ 1 z + + θ q z q = 0. z > 1. 4

5 Review: Properties of ARMA(p,q) models so φ(b)x t = θ(b)w t, X t = ψ(b)w t θ(b) = ψ(b)φ(b).. 1 = ψ 0, θ 1 = ψ 1 φ 1 ψ 0, θ 2 = ψ 2 φ 1 ψ 1 φ 2 ψ 0,. We need to solve the linear difference equation θ j = φ(b)ψ j, with θ 0 = 1, θ j = 0 for j < 0, j > q. 5

6 Review: Autocovariance functions of ARMA processes φ(b)x t = θ(b)w t, X t = ψ(b)w t, q γ(h) φ 1 γ(h 1) φ(2)γ(h 2) = σw 2 θ j ψ j h. j=h We need to solve the homogeneous linear difference equation φ(b)γ(h) = 0 (h > q), with initial conditions γ(h) φ 1 γ(h 1) φ p γ(h p) = σ 2 w q θ j ψ j h, j=h h = 0,...,q. 6

7 Introduction to Time Series Analysis. Lecture Review: ARMA(p,q) models and their properties 2. Review: Autocovariance of an ARMA process. 3. Homogeneous linear difference equations. Forecasting 1. Linear prediction. 2. Projection in Hilbert space. 7

8 Homogeneous linear diff eqns with constant coefficients a 0 x t + a 1 x t a k x t k = 0 ( a0 + a 1 B + + a k B k) x t = 0 a(b)x t = 0 auxiliary equation: a 0 + a 1 z + + a k z k = 0 (z z 1 )(z z 2 ) (z z k ) = 0 where z 1, z 2,...,z k C are the roots of this characteristic polynomial. Thus, a(b)x t = 0 (B z 1 )(B z 2 ) (B z k )x t = 0. 8

9 Homogeneous linear diff eqns with constant coefficients a(b)x t = 0 (B z 1 )(B z 2 ) (B z k )x t = 0. So any {x t } satisfying (B z i )x t = 0 for some i also satisfies a(b)x t = 0. Three cases: 1. The z i are real and distinct. 2. The z i are complex and distinct. 3. Some z i are repeated. 9

10 Homogeneous linear diff eqns with constant coefficients 1. The z i are real and distinct. a(b)x t = 0 (B z 1 )(B z 2 ) (B z k )x t = 0 for some constants c 1,...,c k. x t is a linear combination of solutions to (B z 1 )x t = 0, (B z 2 )x t = 0,..., (B z k )x t = 0 x t = c 1 z t 1 + c 2 z t c k z t k, 10

11 Homogeneous linear diff eqns with constant coefficients 1. The z i are real and distinct. e.g., z 1 = 1.2, z 2 = c 1 z 1 t + c2 z 2 t c 1 =1, c 2 =0 c 1 =0, c 2 =1 c 1 = 0.8, c 2 =

12 Reminder: Complex exponentials a + ib = re iθ = r(cosθ + isinθ), where r = a + ib = a 2 + b 2 ( ) b θ = tan 1 ( π, π]. a Thus, r 1 e iθ 1 r 2 e iθ 2 = (r 1 r 2 )e i(θ 1+θ 2 ), z z = z 2. 12

13 Homogeneous linear diff eqns with constant coefficients 2. The z i are complex and distinct. As before, a(b)x t = 0 x t = c 1 z t 1 + c 2 z t c k z t k. If z 1 R, since a 1,...,a k are real, we must have the complex conjugate root, z j = z 1. And for x t to be real, we must have c j = c 1. For example: x t = c z t 1 + c z 1 t = r e iθ z 1 t e iωt + r e iθ z 1 t e iωt = r z 1 t ( e i(θ ωt) + e i(θ ωt)) = 2r z 1 t cos(ωt θ) where z 1 = z 1 e iω and c = re iθ. 13

14 Homogeneous linear diff eqns with constant coefficients 2. The z i are complex and distinct. e.g., z 1 = i, z 2 = 1.2 i c 1 z 1 t + c2 z 2 t c= i c= i c= i

15 Homogeneous linear diff eqns with constant coefficients 2. The z i are complex and distinct. e.g., z 1 = i, z 2 = 1 0.1i c 1 z 1 t + c2 z 2 t c= i c= i c= i

16 Homogeneous linear diff eqns with constant coefficients 3. Some z i are repeated. a(b)x t = 0 (B z 1 )(B z 2 ) (B z k )x t = 0. If z 1 = z 2, (B z 1 )(B z 2 )x t = 0 (B z 1 ) 2 x t = 0. We can check that (c 1 + c 2 t)z t 1 is a solution... More generally, (B z 1 ) m x t = 0 has the solution ( c1 + c 2 t + + c m 1 t m 1) z t 1. 16

17 Homogeneous linear diff eqns with constant coefficients 3. Some z i are repeated. e.g., z 1 = z 2 = (c 1 + c 2 t) z 1 t c 1 =1, c 2 =0 c 1 =0, c 2 =2 c 1 = 0.2, c 2 =

18 Solving linear diff eqns with constant coefficients Solve: a 0 x t + a 1 x t a k x t k = 0, with initial conditions x 1,...,x k. Auxiliary equation in z C: a 0 + a 1 z + + a k z k = 0 (z z 1 ) m 1 (z z 2 ) m2 (z z l ) m l = 0, where z 1, z 2,...,z l C are the roots of the characteristic polynomial, and z i occurs with multiplicity m i. Solutions: c 1 (t)z1 t + c 2 (t)z2 t + + c l (t)z t l, where c i (t) is a polynomial in t of degree m i 1. We determine the coefficients of the c i (t) using the initial conditions (which might be linear constraints on the initial values x 1,...,x k ). 18

19 Autocovariance functions of ARMA processes: Example ( B 2 )X t = ( B)W t, X t = ψ(b)w t, ( ψ j = 1, 1 5, 1 4, 1 20, 1 16, 1 80, 1 64, 1 ) 320,.... γ(h) φ 1 γ(h 1) φ 2 γ(h 2) = σ 2 w γ(h) γ(h 2) = q h j=0 θ h+j ψ j σ 2 w (ψ ψ 1 ) if h = 0, 0.2σ 2 wψ 0 if h = 1, 0 otherwise. 19

20 Autocovariance functions of ARMA processes: Example We have the homogeneous linear difference equation γ(h) γ(h 2) = 0 for h 2, with initial conditions γ(0) γ( 2) = σw 2 (1 + 1/25) γ(1) γ( 1) = σw/

21 Autocovariance functions of ARMA processes: Example Homogeneous lin. diff. eqn: The characteristic polynomial is γ(h) γ(h 2) = z 2 = 1 4 ( ) 4 + z 2 = 1 (z 2i)(z + 2i), 4 which has roots at z 1 = 2e iπ/2, z 1 = 2e iπ/2. The solution is of the form γ(h) = cz h 1 + c z 1 h. 21

22 Autocovariance functions of ARMA processes: Example z 1 = 2e iπ/2, z 1 = 2e iπ/2, c = c e iθ. We have γ(h) = cz h 1 + c z 1 h ( = 2 h c e i(θ hπ/2) + c e i( θ+hπ/2)) ( ) hπ = c 1 2 h cos 2 θ. And we determine c 1, θ from the initial conditions γ(0) γ( 2) = σ 2 w (1 + 1/25) γ(1) γ( 1) = σ 2 w/5. 22

23 Autocovariance functions of ARMA processes: Example We determine c 1, θ from the initial conditions: We plug γ(0) = c 1 cos(θ) γ(1) = c 1 2 sin(θ) γ(2) = c 1 4 cos(θ) into γ(0) γ(2) = σ 2 w (1 + 1/25) 1.25γ(1) = σ 2 w/5. 23

24 Autocovariance functions of ARMA processes: Example 1.2 γ(h)

25 Introduction to Time Series Analysis. Lecture Review: ARMA(p,q) models and their properties 2. Review: Autocovariance of an ARMA process. 3. Homogeneous linear difference equations. Forecasting 1. Linear prediction. 2. Projection in Hilbert space. 25

26 Review: least squares linear prediction Consider a linear predictor of X n+h given X n = x n : f(x n ) = α 0 + α 1 x n. For a stationary time series {X t }, the best linear predictor is f (x n ) = (1 ρ(h))µ + ρ(h)x n : E (X n+h (α 0 + α 1 X n )) 2 E (X n+h f (X n )) 2 = σ 2 (1 ρ(h) 2 ). 26

27 Linear prediction Given X 1, X 2,...,X n, the best linear predictor X n n+m = α 0 + n α i X i i=1 of X n+m satisfies the prediction equations E ( X n+m X n n+m) = 0 E [( X n+m X n n+m) Xi ] = 0 for i = 1,...,n. This is a special case of the projection theorem. 27

28 Projection Theorem If H is a Hilbert space, M is a closed linear subspace of H, and y H, then there is a point Py M (the projection of y on M) satisfying 1. Py y w y for w M, 2. Py y < w y for w M, w y 3. y Py, w = 0 for w M. Py M y Py y 28

29 Hilbert spaces Hilbert space = complete inner product space: Inner product space: vector space, with inner product a, b : a, b = b, a, α 1 a 1 + α 2 a 2, b = α 1 a 1, b + α 2 a 2, b, a, a = 0 a = 0. Norm: a 2 = a, a. complete = limits of Cauchy sequences are in the space Examples: 1. R n, with Euclidean inner product, x, y = i x iy i. 2. {random variables X: EX 2 < }, with inner product X, Y = E(XY ). (Strictly, equivalence classes of a.s. equal r.v.s) 29

30 Projection theorem Example: Linear regression Given y = (y 1, y 2,...,y n ) R n, and Z = (z 1,...,z q ) R n q, choose β = (β 1,...,β q ) R q to minimize y Zβ 2. Here, H = R n, with a, b = i a ib i, and M = {Zβ : β R q } = sp {z 1,...,z q }. 30

31 Projection theorem If H is a Hilbert space, M is a closed subspace of H, and y H, then there is a point Py M (the projection of y on M) satisfying 1. Py y w y 2. y Py, w = 0 for w M. Py M y Py y 31

32 Projection theorem Py y Py y y Py, w = 0 y Z ˆβ, z i = 0, i M Z Z ˆβ = Z y ˆβ = (Z Z) 1 Z y normal equations. 32

33 Projection theorem Example: Linear prediction Given 1, X 1, X 2,...,X n { r.v.s X : EX 2 < }, choose α 0, α 1,...,α n R so that Z = α 0 + n i=1 α ix i minimizes E(X n+m Z) 2. Here, X, Y = E(XY ), M = {Z = α 0 + n i=1 α ix i : α i R} = sp {1, X 1,...,X n }, and y = X n+m. 33

34 Projection theorem If H is a Hilbert space, M is a closed subspace of H, and y H, then there is a point Py M (the projection of y on M) satisfying 1. Py y w y 2. y Py, w = 0 for w M. Py M y Py y 34

35 Projection theorem: Linear prediction Let Xn+m n denote the best linear predictor: Xn+m n X n+m 2 Z X n+m 2 for all Z M. The projection theorem implies the orthogonality Xn+m n X n+m, Z = 0 for all Z M Xn+m n X n+m, Z = 0 for all Z {1, X 1,...,X n } E ( X n n+m X n+m ) = 0 E [( X n n+m X n+m ) Xi ] = 0 That is, the prediction errors (X n n+m X n+m ) are uncorrelated with the prediction variables (1, X 1,...,X n ). 35

36 Linear prediction Error (Xn+m n X n+m ) is uncorrelated with the prediction variable 1: E ( ) Xn+m n X n+m = 0 ( E α 0 + ) α i X i X n+m = 0 i ( µ 1 ) α i = α 0. i 36

37 Linear prediction... µ ( 1 i α i ) = α 0. Substituting for α 0 in X n n+m = α 0 + i α i X i, we get X n n+m = µ + i α i (X i µ). So we can subtract µ from all variables: X n n+m µ = i α i (X i µ). Thus, for forecasting, we can assume µ = 0. So we ll ignore α 0. 37

38 One-step-ahead linear prediction Write Xn+1 n = φ n1 X n + φ n2 X n φ nn X 1 Prediction equations: E ( (Xn+1 n ) X n+1 )X i = 0, for i = 1,...,n n φ nj E (X n+1 j X i ) = E(X n+1 X i ) j=1 n φ nj γ(i j) = γ(i) j=1 Γ n φ n = γ n, 38

39 One-step-ahead linear prediction Prediction equations: Γ n φ n = γ n. γ(0) γ(1) γ(n 1) γ(1) γ(0) γ(n 2) Γ n = γ(n 1) γ(n 2) γ(0), φ n = (φ n1, φ n2,...,φ nn ), γ n = (γ(1), γ(2),..., γ(n)). 39

40 Mean squared error of one-step-ahead linear prediction Pn+1 n = E ( X n+1 Xn+1 n ) 2 = E (( X n+1 Xn+1)( n Xn+1 Xn+1 n )) = E ( ( X n+1 Xn+1 Xn+1 n )) = γ(0) E (φ nxx n+1 ) = γ(0) γ nγ 1 n γ n, where X = (X n, X n 1,...,X 1 ). 40

41 Mean squared error of one-step-ahead linear prediction Variance is reduced: P n n+1 = E ( X n+1 X n n+1 = γ(0) γ nγ 1 n γ n ) 2 = Var(X n+1 ) Cov(X n+1, X)Cov(X, X) 1 Cov(X, X n+1 ) = E (X n+1 0) 2 Cov(X n+1, X)Cov(X, X) 1 Cov(X, X n+1 ), where X = (X n, X n 1,...,X 1 ). 41

42 Introduction to Time Series Analysis. Lecture 7. Peter Bartlett 1. Review: ARMA(p,q) models and their properties 2. Review: Autocovariance of an ARMA process. 3. Homogeneous linear difference equations. Forecasting 1. Linear prediction. 2. Projection in Hilbert space. 42