STA 6857 Estimation ( 3.6)

Transcription

1 STA 6857 Estimation ( 3.6)

2 Outline 1 Yule-Walker 2 Least Squares 3 Maximum Likelihood Arthur Berg STA 6857 Estimation ( 3.6) 2/ 19

3 Outline 1 Yule-Walker 2 Least Squares 3 Maximum Likelihood Arthur Berg STA 6857 Estimation ( 3.6) 3/ 19

4 Yule-Walker Equations The Yule-Walker equations give us a means of estimating the coefficients φ 1, φ 2,..., φ p in an AR(p) model. George Udny Yule ( ) is a Scottish statistician who taught at Cambridge University. The Yule distribution, a discrete power law, is named after him. Arthur Berg STA 6857 Estimation ( 3.6) 4/ 19

5 Yule-Walker Equations "Every cell phone call solves the Yule-Walker equations every ten microseconds." Thierry Dutoit Sir Gilbert Thomas Walker ( ) was a British physicist and statistician known for his description of the Southern Oscillation. Arthur Berg STA 6857 Estimation ( 3.6) 5/ 19

6 Start with the mean zero AR(p) model: x t = φ 1 x t 1 + φ 2 x t φ p x t p + w t ( ) Multiply both sides of ( ) by x t h for h = 1,..., p: x t x t h = φ 1 x t 1 x t h + φ 2 x t 2 x t h + + φ p x t p x t h + w t x t h ( ) Take expectations throughout: γ(h) = φ 1 γ(h 1) + φ 2 γ(h 2) + + φ p γ(h p) (1) Now take the expectation of ( ) with h = 0: γ(0) = φ 1 γ(1) + φ 2 γ(2) + + φ p γ(p) + E(x t z t ) }{{} σ 2 ( ) Rearranging ( ) gives σ 2 = γ(0) φ 1 γ(1) φ p γ(p) (2) Equations (1) and (2) are the Yule-Walker Equations. Arthur Berg STA 6857 Estimation ( 3.6) 6/ 19

7 Y-W Equations in Matrix Form From the recurrence γ(h) = φ 1 γ(h 1) + φ 2 γ(h 2) + + φ p γ(h p) (1) for extract the p equations γ(1) = φ 1 γ(0) + φ 2 γ(1) + + φ p γ(p 1) γ(2) = φ 1 γ(1) + φ 2 γ(0) + + φ p γ(p 2). γ(p) = φ 1 γ(p 1) + φ 2 γ(p 2) + + φ p γ(0) γ(1) γ(2) γ(0) γ(1) γ(p 1) = γ(1) γ(0) γ(p 2) γ(p 1) γ(p 2) γ(0) γ(p) }{{}}{{} Γ p γ p Hence Γ p φ = γ p where Γ p = {γ(k j)} p j,k=1. φ 1 φ 2. φ p } {{ } φ Arthur Berg STA 6857 Estimation ( 3.6) 7/ 19

8 Estimation Using the Y-W Equations Under the method of moments approach, we estimate φ = (φ 1, φ 2,..., φ p ) with φ = Γ p 1 γp and subsequently σ 2 = γ(0) φ 1 γ(1) φ 2 γ(2) φ p γ(p) = γ(0) φ γ p ) = γ(0) ( Γp 1 γp γp = γ(0) γ p Γp 1 γp where Γ p = { γ(k j)} p j,k=1 and γ p = ( γ(1),..., γ(p)). Arthur Berg STA 6857 Estimation ( 3.6) 8/ 19

9 Other Estimation Methods Burg s Method Method of Moments (Example 3.27, p. 125) Least Squares Conditional Maximum Likelihood Maximum Likelihood Arthur Berg STA 6857 Estimation ( 3.6) 9/ 19

10 Outline 1 Yule-Walker 2 Least Squares 3 Maximum Likelihood Arthur Berg STA 6857 Estimation ( 3.6) 10/ 19

11 Treating Autoregression as Regression For the mean zero AR(p) model x t = φ 1 x t 1 + φ 2 x t φ p x t p + w t we can regress x t 1, x t 2,..., x t p on x t by writing the data {x t } n t=1 as n x t, x }{{} t 1, x t 2,..., x t p }{{} y x t=p+1 Now we can perform least squares regression of x on y. Arthur Berg STA 6857 Estimation ( 3.6) 11/ 19

12 Outline 1 Yule-Walker 2 Least Squares 3 Maximum Likelihood Arthur Berg STA 6857 Estimation ( 3.6) 12/ 19

13 Simple MLE Example Let x 1, x 2,..., x n iid N (µ, 1). ˆµ = x = 1 n MLE (Maximum Likelihood Estimator) UMVUE (Uniformly Minimum Variance Unbiased Estimator) admissible (Can t be beaten uniformly.) Maximize L(µ) = L(µ x) = f µ (x) where n n f µ (x) = i=1 f µ (x i ) = i=1 n i=1 1 2π e (x i µ)2 2 = Maximizing f µ (x) with respect to µ amounts to minimizing n g(µ) = (x i µ) 2 i=1 x i ( ) 1 (2π) n/2 e 1 n 2 i=1 (x i µ) 2 Arthur Berg STA 6857 Estimation ( 3.6) 13/ 19

14 Simple MLE Example (cont.) To minimize g(µ), set its derivative equal to zero, i.e. g (µ) = 2 n (x i µ) = 2n x 2nµ = 0 i=1 Therefore we see µ = x minimizes g(µ) thus maximizes the likelihood L(µ). Hence the maximum likelihood estimator of µ is x. Assumption in this derivation: independence Arthur Berg STA 6857 Estimation ( 3.6) 14/ 19

15 MLE for AR(1) For the AR(1) process x t = φx t 1 + w t where w t iid N (0, σ 2 ), we have the likelihood equation L(φ, σ) = n f σ (x t x t 1, x t 2,..., x 1 ) t=1 Note that x t x t 1 φx t 1 + w t so that = f (x 1 )f (x 2 x 1 ) f (x n x n 1 ) ( ) f (x t x t 1 ) = f w (x t φx t 1 ) where f w is the pdf of N (0, σ 2 ). Hence from ( ) we have L(φ, σ) = f (x 1 ) n f w (x t φx t 1 ) t=2 Arthur Berg STA 6857 Estimation ( 3.6) 15/ 19

16 MLE for AR(1) (cont.) From the causal representation of x 1, x 1 = φ j w 1 j we see that x 1 is normally distributed with mean zero and variance var(x 1 ) = var(φ j w 1 j ) = σ 2 φ 2j = σ2 1 φ 2 j=0 Therefore the Likelihood can written as where j=0 j=0 L(φ, σ) = (2πσ 2 ) n/2 (1 φ 2 ) 1/2 exp S(φ) = (1 φ 2 )(x 1 µ) 2 + S(φ) is called the unconditional sum of squares. [ S(φ) ] 2σ 2 n (x t φx t 1 ) 2. Arthur Berg STA 6857 Estimation ( 3.6) 16/ 19 t=2

17 MLE for AR(1) (cont.) Take the partial derivative of L(φ, σ) with respect to σ 2 to see maximizes the likelihood. Therefore σ 2 = S(φ) n σ 2 ML = S( φ ML ) n After taking logs, we see that the MLE of φ is the minimizer of ( ) S(φ) l(φ) = ln ln(1 φ2 ) n n Numerical optimization must be performed here, i.e. no closed form solution of φ ML exists. Arthur Berg STA 6857 Estimation ( 3.6) 17/ 19

18 Conditional MLE of AR(1) The problem becomes much simpler if we condition on x 1, namely the likelihood becomes where L(φ, σ) = f (x 2 x 1 ) f (x n x n 1 ) n = f w (x t φx t 1 ) t=2 = (2πσ 2 ) n/2 exp S(φ) = n (x t φx t 1 ) 2. t=2 [ S(φ) ] 2σ 2 Minimizing S(φ) over φ yields the conditional MLE which is also the least squares estimate! Arthur Berg STA 6857 Estimation ( 3.6) 18/ 19

19 Next Time Asymptotics Bootstrap ARIMA introduction ( 3.7) Arthur Berg STA 6857 Estimation ( 3.6) 19/ 19