Distributions of Quadratic Forms. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

Transcription

1 Distributions of Quadratic Forms Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

2 Under the Normal Theory GMM (NTGMM), y = Xβ + ε, where ε N(0, σ 2 I). By Result 5.3, the NTGMM = y N(Xβ, σ 2 I). Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

3 Mean of y determined by β through Xβ. Variance of y determined by σ 2. y = P X y + (I P X )y P X y C(X), (I P X )y C(X). opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

4 We use ŷ = P X y to estimate Xβ We use ˆε = (I P X )y to estimate σ 2. (P X y = Xˆβ) ( ˆσ 2 = ˆε ˆε ) n rank(x). Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

5 Also, recall that y y = ŷ ŷ + ˆε ˆε SSTO = SSR + SSE. Under the NTGMM, what can we say about the distribution of these sums of squares? opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

6 Lemma 5.1: A p p symmetric matrix A is idempotent with rank s iff a p s matrix G with orthogonal columns such that A = GG. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

7 Proof of Lemma 5.1: (= ) By the Spectral Decomposition Theorem, A = QΛQ = p λ i q i q i, i=1 where Q = [q 1,..., q p ], Q Q = I, Λ = diag(λ 1,..., λ p ). Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

8 Because A is idempotent, λ i {0, 1} i = 1,..., p. Because rank(a) = s, exactly s of λ 1,..., λ p equal to 1 and p s of λ 1,..., λ p equal to 0. Let i 1,..., i s be λ i1 = = λ is = 1. Let G = [q i1,..., q is ]. Then p s A = λ i q i q i = λ ij q ij q i j = i=1 j=1 s q ij q i j = GG and G G = I. j=1 Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

9 ( =) If A = GG, where G is a p s matrix with orthonormal columns. Then Thus, rank(a) = s. rank(gg ) = rank(g ) = rank(g) = rank(g G) = rank( s s I ) = s. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

10 Furthermore, A = (GG ) = GG = A and AA = (GG )(GG ) = G(G G)G = GIG = GG = A. A is also symmetric and idempotent. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

11 Result 5.14: Let X N(µ, p p I ) and let p p A be a symmetric matrix. Then A is idempotent with rank s = X AX χ 2 s (µ Aµ/2). opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

12 Proof of Result 5.14: By Lemma 5.1, G A = GG and G G = s s I. Then G X N(G µ, G IG = G G = s s I ). Thus, by Result 5.9, (G X) (G X) χ 2 s ((G µ) G µ/2). (G X) (G X) = X GG X = X AX (G µ) G µ = µ GG µ = µ Aµ. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

13 Result 5.15: Suppose X N(µ, Σ), with p p Σ of rank p. Suppose A is p p and symmetric. Then AΣ is idempotent of rank s = X AX χ 2 s (µ Aµ/2). Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

14 Proof of Result 5.15: Let W = Σ 1/2 X. Then W N(Σ 1/2 µ, Σ 1/2 ΣΣ 1/2 = I). Let B = Σ 1/2 AΣ 1/2. Then B is symmetric by symmetry of Σ 1/2 and A. Furthermore, rank(b) = rank(σ 1/2 AΣ 1/2 ) = rank(a) = rank(aσ) = s. Σ 1/2 and Σ are full-rank. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

15 Finally, note that B is idempotent: AΣAΣ = AΣ Σ 1/2 AΣAΣ = Σ 1/2 AΣ Σ 1/2 AΣAΣΣ 1/2 = Σ 1/2 AΣΣ 1/2 Σ 1/2 AΣAΣ 1/2 = Σ 1/2 AΣ 1/2 Σ 1/2 AΣ 1/2 Σ 1/2 AΣ 1/2 = Σ 1/2 AΣ 1/2 BB = B. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

16 Thus, by Result 5.14, W BW χ 2 s ((Σ 1/2 µ) B(Σ 1/2 µ)/2). Now note W BW = X Σ 1/2 Σ 1/2 AΣ 1/2 Σ 1/2 X = X AX and likewise (Σ 1/2 µ) B(Σ 1/2 µ) = µ Aµ. X AX χ 2 s (µ Aµ/2). Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

17 Find the distribution of SSE. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

18 Let Then SSE = ˆε ˆε = y (I P X )y ( ) I = σ 2 y PX σ 2 y. A = I P X σ 2 and Σ = σ 2 I = Var(y). AΣ = I P X σ 2 σ 2 I = I P X. Thus, AΣ is idempotent and rank(aσ) = n rank(x). opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

19 By Result 5.15, ( ) ( I y PX 1 σ 2 y χ 2 n rank(x) 2 β X (I P X )X = X P X X = X X = 0, we have Thus, ( I PX ( ) I y PX σ 2 y χ 2 n rank(x). SSE σ 2 χ 2 n rank(x), which is a scaled Chi-Square distribution. σ 2 ) ) Xβ. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

20 Similarly, SSR σ 2 χ 2 rank(x) ( ) 1 2 β X Xβ/σ 2. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

21 Result 5.16: Suppose X N(µ, Σ) and A is symmetric with rank s. Then BΣA = 0 = BX and X AX are independent. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

22 By the SDT, we have A = QΛQ, p p where Q is square with orthonormal columns q 1,..., q p and Λ = diag(λ 1,..., λ p ) with exactly s of λ 1,..., λ p not equal to zero. Because p QΛQ = λ i q i q i, i=1 we can without loss of generality (WLOG) assume λ 1,..., λ s 0. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

23 Thus, where A = s λ i q i q i = Q 1 Λ 1 Q 1, i=1 Q 1 = [q 1,..., q s ], Q 1 Q 1 = I s s Λ 1 = diag(λ 1,..., λ s ) and Λ 1 1 = diag ( 1 λ 1,..., 1 λ s ). opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

24 [ ] [ ] BX B Now consider Q 1 X = X. Then Q 1 [ ] ([ ] ) B B X N µ, V, Q 1 Q 1 where V = = [ ] B ] Σ [B Q 1 Q 1 [ BΣB BΣQ 1 Q 1 ΣB Q 1 ΣQ 1 ]. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

25 BΣA = 0 = BΣQ 1 Λ 1 Q 1 = 0 = BΣQ 1 Λ 1 Q 1 Q 1 = 0Q 1 = BΣQ 1 Λ 1 = 0 = BΣQ 1 Λ 1 Λ 1 1 = 0Λ 1 1 = BΣQ 1 = 0 = BX and Q 1X independent by Result 5.4. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

26 Now BX and Q 1X are independent, = BX and (Q 1 X) Λ 1 Q 1 X independent = BX and X Q 1 Λ 1 Q 1 X independent = BX and X AX independent. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

27 Corollary 5.4: Suppose X N(µ, Σ), A symmetric with rank r, B symmetric with rank s. Then BΣA = 0 = X AX and X BX are independent. Proof: HW problem. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

28 Find the distribution of SSR/r, where r = rank(x). SSE/(n r) Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

29 We have seen that SSR σ 2 χ 2 r ( ) 1 2 β X Xβ/σ 2 and SSE σ 2 χ 2 n r. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

30 Thus, SSR/r SSE/(n r) = SSR σ 2 /r SSE σ 2 /(n r) will have the distribution F r,n r ( 1 2 β X Xβ/σ 2) if SSR and SSE independent. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31

31 SSR = y P X y, SSE = y (I P X )y (I P X )(σ 2 I)P X = σ 2 (I P X )P X = σ 2 (P X P X P X ) = σ 2 (P X P X ) = 0. By Corollary 5.4, SSR and SSE are independent. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 31