Likelihood Ratio Test of a General Linear Hypothesis. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

Transcription

1 Likelihood Ratio Test of a General Linear Hypothesis Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

2 Consider the Likelihood Ratio Test of H 0 : Cβ = d vs H A : Cβ d. Suppose y N(Xβ, σ 2 I). The likelihood function is L(β, σ 2 y) = (2πσ 2 ) n/2 e 1 2σ 2 (y Xβ) (y Xβ) for β R p and σ 2 > 0. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

3 Note that L(β, σ 2 y) = (2πσ 2 ) n/2 e 1 2σ 2 Q(β), where Q(β) = (y Xβ) (y Xβ) = y Xβ 2. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

4 The parameter space under the null hypothesis H 0 : Cβ = d is Ω 0 = {(β, σ 2 ) : Cβ = d, σ 2 > 0}. The parameter space corresponding to the union of the null and alternative parameter spaces is Ω = {(β, σ 2 ) : β R p, σ 2 > 0}. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

5 The likelihood ratio test rejects H 0 iff is sufficiently small. Λ(y) = sup Ω 0 L(β, σ 2 y) sup Ω L(β, σ 2 y) Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

6 To conduct a significance level α likelihood ratio test, we reject H 0 iff Λ(y) c α, where c α satisfies sup{p(λ(y) c α β, σ 2 ) : (β, σ 2 ) Ω 0 } α. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

7 To find Λ(y), we must maximize the likelihood over Ω 0 and Ω. For any fixed β R p, we can find the value of σ 2 > 0 that maximizes L(β, σ 2 y) as follows. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

8 Because log is a strictly increasing function, the value of σ 2 that maximizes L(β, σ 2 y) is the same as the value of σ 2 that maximizes l(β, σ 2 y) log L(β, σ 2 y). Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

9 l(β, σ 2 y) = n 2 log(2πσ2 ) 1 2σ 2 Q(β) l(β, σ 2 y) σ 2 = n 2σ 2 + Q(β) 2σ 4. Equating to 0 and solving for σ 2 yields ˆσ 2 (β) = Q(β) n. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

10 Furthermore, examination l(β,σ2 y) as a function of σ 2 shows that σ 2 l(β, σ 2 y) as a function of σ 2 is increasing to the left of Q(β)/n and decreasing to the right of Q(β)/n. Thus, for any fixed β R p, the likelihood is maximized over σ 2 > 0 at Q(β)/n. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

11 Now note that L(β, ˆσ 2 (β) y) = (2πQ(β)/n) n/2 e n 2Q(β) Q(β) = (2πeQ(β)/n) n/2, which is clearly maximized over β by minimizing Q(β) over β. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

12 Let ˆβ be any minimizer of Q(β) over β R p. We know from previous results that ˆβ minimizes Q(β) over β R p iff ˆβ is a solution to NE. (X X) X y is one solution. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

13 Thus, the maximum likelihood estimator (MLE) of σ 2 is ˆσ 2 MLE = Q(ˆβ)/n = y Xˆβ 2 /n, where ˆβ is any solution to the NE. Recall that Xˆβ = P X y is the same for all solution to NE. Thus, ˆσ 2 MLE is the same for any ˆβ that solves the NE. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

14 Note that the MLE of σ 2 ˆσ 2 MLE = Q(ˆβ) n Q(ˆβ) n rank(x) = n rank(x) n = y (I P X )y n rank(x) n rank(x) n = ˆσ 2 n rank(x). n Recall Thus, E(ˆσ 2 ) = σ 2. E(ˆσ MLE) 2 = n rank(x) σ 2 < σ 2. n Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

15 If X is of full-column rank, then ˆβ = (X X) 1 X y is the maximum likelihood estimator (MLE) of β. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

16 We have sup L(β, σ 2 y) = L(ˆβ, ˆσ MLE y) 2 Ω = (2πeQ(ˆβ)/n) n/2. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

17 If we let β denote the minimizer of Q(β) over β R p satisfying Cβ = d, then sup L(β, σ 2 y) Ω 0 = L( β, Q( β)/n y) = (2πeQ( β)/n) n/2. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

18 Thus Λ(y) = = (2πeQ( β)/n) n/2 (2πeQ(ˆβ)/n) n/2 [ ] n/2 Q( β). Q(ˆβ) Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

19 Now note that Λ(y) c α [ ] Q( β) n/2 c α Q(ˆβ) Q( β) Q(ˆβ) c 2/n α Q( β) Q(ˆβ) 1 c 2/n α 1 Q( β) Q(ˆβ) Q(ˆβ) c 2/n α 1. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

20 [Q( β) Q(ˆβ)]/q Q(ˆβ)/(n r) n r q (c 2/n α 1), where q = rank(c) and n r = n rank(x). opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

21 Example: Suppose where y = Xβ + ε, ε N(0, σ 2 I). Furthermore, suppose rank( n p X ) = p. Partition X = [X 1, X 2 ] and β = where X 1 is n p 1 and β 1 is p 1 1. [ β1 β 2 ], Suppose we wish to test H 0 : β 2 = 0. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

22 H 0 : β 2 = 0 is a GLH H 0 : Cβ = d with C = [ 0, q p 1 q q I ] and d = 0, q 1 where q = p p 1. This GLH is testable because q p C has rank q and Cβ is estimable due to full-column rank of X. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

23 Q( β) = min{q(β) : Cβ = d} = min{ y Xβ 2 : β R p β 2 = 0} = min{ y X 1 β 1 2 : β 1 R p 1 } = y X 1 ˆβ1 2 = y P X1 y 2 = (I P X1 )y 2 = y (I P X1 )y = SSE Reduced. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

24 Q(ˆβ) = y Xˆβ 2 = y P X y 2 = (I P X )y 2 = y (I P X )y = SSE Full. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

25 The DF for SSE Full is DF F = rank(i P X ) = n p. The DF for SSE Reduced is DF R = rank(i P X1 ) = n p 1. DF R DF F = p p 1 = q. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

26 We have shown that, in the general case, the likelihood ratio test (LRT) of H 0 : Cβ = d rejects for sufficiently large [Q( β) Q(ˆβ)]/q. Q(ˆβ)/(n r) Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

27 In this example, [Q( β) Q(ˆβ)]/q Q(ˆβ)/(n r) = [SSE Reduced SSE Full ]/(DF R DF F ) SSE Full /DF F, which should look familiar. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

28 We now show that for a testable GLH H 0 : Cβ = d, the GLT statistic F = (Cˆβ d) (C(X X) C ) 1 (Cˆβ d)/q ˆσ 2 = [Q( β) Q(ˆβ)]/q. Q(ˆβ)/(n r) Thus, the GLT is equivalent to the LRT. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

29 Note that ( ) I Q(ˆβ)/(n r) = y PX y n r = ˆσ 2. Thus, it remains to show Q( β) Q(ˆβ) = (Cˆβ d) (C(X X) C ) 1 (Cˆβ d). From 3.10, we know that β is leading subvector of solution to RNE. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

30 Theorem 6.1: If Cβ = d is testable and β is the leading subvecctor of a solution to the RNE [ ] [ ] [ ] X X C b X y =, C 0 λ d then Q( β) Q(ˆβ) = (ˆβ β) X X(ˆβ β) = (Cˆβ d) (C(X X) C ) 1 (Cˆβ d). Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

31 Proof of Result 6.1: First show that Q( β) Q(ˆβ) = (ˆβ β) X X(ˆβ β). Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

32 Now let λ denote the trailing subvector of the solution to RNE whose leading subvector is β; i.e., suppose ] [ β λ is solution to RNE. Show X X(ˆβ β) = C λ. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

33 Now show λ = (C(X X) C ) 1 (Cˆβ d). opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

34 We have established (1) Q(ˆβ) Q( β) = (ˆβ β) X X(ˆβ β) (2) X X(ˆβ β) = C λ (3) λ = (C(X X) C ) 1 (Cˆβ d). Use these results to finish the proof. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42

35 Corollary 6.4: Suppose Cβ = d is testable, and suppose ˆβ is a solution to the NE. Then the leading subvector of a solution to the RNE with constraint Cβ = d can be found by solving for b in the equations X Xb = X y C (C(X X) C ) 1 (C ˆβ d). opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 42