STAD57 Time Series Analysis. Lecture 14

Transcription

1 STAD57 Time Series Analysis Lecure 14 1

2 Maximum Likelihood AR(p) Esimaion Insead of Yule-Walker (MM) for AR(p) model, can use Maximum Likelihood (ML) esimaion Likelihood is join densiy of daa {x 1,,x n } as funcion of parameers:, φ, 1,, n;, φ, L f x x ML esimaors are values of parameers ha maximize likelihood funcion Wha is he likelihood of he AR(p) model?

3 Maximum Likelihood AR(p) Esimaion For ML assume AR(p) model is Gaussian X 1 X 1 p ( X p ) W, { W } ~ N(0, ) Thus, condiional disribuion of X becomes X X,, X ~ N X ( X ), This allos us o rie likelihood as: 1 p 1 1 p p, φ, 1,, n;, φ, L f x x f ( x, x,, x ) f ( x x,, x ) f ( x x,, x ) 1 p p1 p 1 n n1 n p n f ( x, x,, x ) f ( x x,, x ) 1 p p1 1 p 3

4 Maximum Likelihood AR(p) Esimaion Densiies f ( x x 1,, x p), p 1,, n are rivial o find, bu iniial densiy f ( x1, x,, x p ) can be complicaed funcion of parameers. Since all densiies are Normal, likelihood funcion ill have form: n/ S, φ L, φ, ( ) g( φ) exp here S(µ,φ) is sum of squares n, φ (, φ) 1 1 p p S h x x x p1 and g(φ), h(µ, φ) are some funcions of µ & φ 4

5 Example Find he likelihood of he AR(1) model: X X W W N 1 1, { } ~ (0, ) 5

6 6

7 Maximum Likelihood AR(p) Esimaion To find ML esimaors maximize likelihood e need o ˆ, φˆ, ˆ L, φ, Can sho alays given by S ˆ, φˆ / n ˆ Bu here is no closed-form soluion for ˆ, φˆ Tha s because of complicaed form of g( φ), h(, φ) In pracice, use numerical echniques for finding ˆ, φˆ (maximizing L) Common mehods are Neon-Raphson or Fisher scoring algorihms 7

8 ML Esimaion in R R funcion ar.mle() performs AR(p) model ML esimaion E.g. daa AR order (i.e. p) ml.fi = ar.mle( rec, order=) ( ˆ ) ( φˆ ) ˆ ( ) ( Cov( φˆ )) To check esimaes: ml.fi$x.mean [1] ml.fi$ar [1] ml.fi$var.pred [1] ml.fi$asy.var [,1] [,] [1,] [,]

9 Condiional Leas Squares AR(p) Esimaion AR(p) model likelihood is: L, φ, f ( x 1, x,, x ) f ( x p x 1 1,, x p ) p Wha complicaes hings is iniial erm f ( x1, x,, x p ) For large n, effec of iniial densiy is small n relaive o produc f ( x x 1,, x p ) p1 In his case, can look a condiional likelihood L x x f x x x Condiion on firs #p values, o remove n n, φ, 1,, p ( 1 1,, p) p f x1 x x p (,,, ) 9

10 Condiional Leas Squares AR(p) Esimaion Condiional likelihood simplifies o n p S c, φ L, φ, x1,, xp ( ) exp here S c (µ,φ) is condiional sum of squares n, φ 1 1 S x x x c p1 p p Maximize condiional likelihood by minimizing S c (µ,φ) using ordinary leas square (OLS) from regression In TS, his is called condiional leas squares (LS) 10

11 Condiional Leas Squares AR(p) Esimaion Condiional LS esimaors for AR(p) model are given by OLS esimaion of X ˆ ˆ X ˆ ˆ ( X ˆ ) 1 1 p p X ˆ ˆ ˆ X, p 1 n p p here: ˆ ˆ, j 1,, j j p ˆ ˆ 1ˆ ˆ 0 1 p ˆ S c n ˆ, φˆ p 11

12 Condiional LS Esimaion in R R funcion ar.ols() performs AR(p) model condiional ML E.g. ls.fi = ar.ols( rec, order=) To check esimaes: daa AR order (i.e. p) ( ˆ ) ( φˆ ) ˆ ( ) ls.fi$x.mean [1] ls.fi$ar [1] ls.fi$var.pred [1]

13 AR(p) Esimaion As n, all AR(p) esimaion mehods (Yule- Walker, ML, condiional LS) give same resuls For large n, prefer Yule-Walker / condiional LS They are faser han ML (no need for numerical opimizaion) and heir resuls are no very differen Hoever for small sample sizes, and especially if TS is Gaussian, ML esimaion performs beer han he oher o mehods For small n, prefer ML esimaion 13

14 ARMA Esimaion Have seen esimaion for pure AR(p) using: Yule-Walker (MM) Condiional Leas Squares (LS) Maximum Likelihood (ML) beer for large n beer for small n For esimaion in general ARMA model, only rely on ML & LS MM can be problemaic / subopimal for models ha conain MA componen 14

15 Example Consider MA(1) model X W W MM: esimae (, ) by maching firs o sample & heoreical nd order momens 1 15

16 Maximum Likelihood ARMA Esimaion Consider Gaussian ARMA(p,q) model: X X X W W W 1 1 p p 1 1 q q Likelihood funcion: ih parameer vecor β, 1,, n; β, L f x x β 1 p 1 q There are o complicaions ih L(β): (,,,,,, ) WN sequence {W } is no observed canno use condiional disribuion X X,, X, W,, W ~ 1 p 1 q 1 1 p p 1 1 q q ~ N ( X ) ( X ) W W W, W depends on enire pas {X 1,,X 1 } (by inveribiliy) 16

17 Maximum Likelihood ARMA Esimaion Wrie ARMA likelihood as: f ( x ) f ( x x ) f ( x x, x ) f x x,, x n n1 1 n β, 1,, 1 L f x x x 1 For Gaussian ARMA models, X is X 1,, X1 1 equal o BLP X, hich is Normally disribued 1 (as linear funcion of Normals) ih variance P!! X X,, X ~ N X, P, L β f x x x ( x x ) 1 1,, 1 exp 1 1 P P 17

18 Maximum Likelihood ARMA Esimaion Given daa { x 1,, x n }, every BLP 1 is funcion of he parameers β rie x ( ) 1 β 1 1 Also P (0) (1 jj ) r ( β), here Thus, here: x j1 1( β) (1 ) ( β) & 1( β) (0) / j0 j r r r n ( β, ) 1 1,, 1 L f x x x 1/ S( β) r β r β r β n/ ( ) 1( ) ( ) n( ) exp n [ x x 1( β)] S( β) 1 r ( β) 18

19 Maximum Likelihood ARMA Esimaion ˆ β, ˆ L( β, ) Can sho alays given by ˆ S( βˆ ) / n There is no closed-form soluion for ˆβ To find ML esimaors maximize likelihood e need o In pracice, use numerical echniques for finding ˆβ (maximizing L) Common mehods are Neon-Raphson or Fisher scoring algorihms 19

20 Condiional Leas Squares ARMA Esimaion Likelihood funcion can be complicaed Need 1-sep-ahead BLP Can simplify esimaion by condiioning on firs p values & using runcaed predicion: Esimae squares p X 1, 1,, n ( β) ( x ) ( x ) ( β), p 1 j1 j j k 1 k k here p p 1 1 q ( β) ( β) ( β) 0 ˆβ Sc ( ) ( ) p 1 q by minimizing condiional sum of n β β and ˆ by S ( βˆ ) / n p c 0

21 Large Sample Behavior of ARMA Esimaion As n, ARMA esimaors (eiher ML or LS) behave as: ˆ ˆ ( φ, θ) ~ N ( φ, θ), Γ n, ˆ Γφφ Γφθ here Γ and p, q Γθφ Γθθ Γ Γ Γ φφ θθ φθ p 1 p, q ( i j) for AR( p) model: ( B) Y W Y i, j1 q ( i j) for AR( q) model: ( B) W i, j1 p, q Γ ( i j) for cross-covariance ( h) θφ Y i1, j1 Noe: Γ is p p, Γ is q q, and Γ is p q φφ θθ φθ Y 1

22 Example Find 1 Γ p, q for ARMA(1,1) model

23 ML Esimaion in R R funcion arima() performs ARMA esimaion For ML esimaion: daa ARMA order as (p,0,q) ml.fi = arima( soi, order=c(,0,) ) To check esimaes: ( βˆ ) ˆ ( ) ( Cov( βˆ )) ml.fi$coef ar1 ar ma1 ma inercep ml.fi$sigma [1] ml.fi$var.coef ar1 ar ma1 ma inercep ar e e e e e-06 ar e e e e e-06 ma e e e e e-06 ma e e e e e-06 inercep e e e e e-04 3

24 LS Esimaion in R For large n, perform condiional LS esimaion Similar resuls o ML, bu simpler ( faser) o run condiional LS: minimize Condiional Sum of Squares (CSS) ls.fi = arima( soi, order=c(,0,), mehod= CSS ) To check esimaes: ( βˆ ) ˆ ( ) ( Cov( βˆ )) ls.fi$coef ar1 ar ma1 ma inercep ls.fi$sigma [1] ls.fi$var.coef ar1 ar ma1 ma inercep ar e e e e-06 ar e e e e-06 ma e e e e-05 ma e e e e-05 inercep e e e e-04 4