Constraints on Solutions to the Normal Equations. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

Transcription

1 Constraints on Solutions to the Normal Equations Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

2 If rank( n p X) = r < p, there are infinitely many solutions to the NE X Xb = X y. For (X X) any GI of X X, the set of all solutions is { (X X) X y + (I (X X) X X)z : z R p}. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

3 It is possible to place additional constraints on a solution to the NE so that a unique solution that satisfies the constraints. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

4 Example: Suppose y ij = µ + τ i + ε ij (i = 1,..., t; j = 1,..., n i ) where E(ε ij ) = 0 i, j. ˆµ Consider the following constraints on a solution to the NE ˆβ ˆτ 1 =.. ˆτ 2 (Note that constraints are on ˆβ not β.) opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

5 Four common choices are 1. ˆτ 1 = 0 (set first to zero) 2. ˆτ t = 0 (set last to zero) 3. t i=1 ˆτ i = 0 (sum to zero) 4. t i=1 n iˆτ i = 0 (weighted sum to zero) opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

6 Any one of these constraints may be imposed by insisting that a solution to the NE ˆβ satisfies Aˆβ = 0 for some matrix A whose rows ˆµ define linear constraints on ˆβ ˆτ 1 =.. ˆτ 2 Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

7 For example, 1. A = [0, 1, 0,..., 0]. 2. A = [0, 0, 0,..., 1]. 3. A = [0, 1, 1,..., 1]. 4. A = [0, n 1, n 2,..., n t ]. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

8 Note that solution ˆβ satisfies both the NE and the constraint equations Aˆβ = 0 iff [ ] [ ] X X X ˆβ y =. A 0 Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

9 Furthermore, we know that X Xb = X y Xb = P X y. Thus, we have ˆβ a solution to NE satisfying the constraints iff [ ] [ ] X PX y ˆβ =. A 0 Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

10 We now know that if [ we ] want a unique solution to NE and constraint X equations, we need to be of full-column rank; that is, we need A ([ ]) X rank = p. A Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

11 rank( n p X) = r < p. Thus, we know X has r LI rows. If we can find s p r p 1 vectors, when these vectors are combined with r LI rows of X, the set of p vectors is LI, then we can use the s vectors as rows of A to get ([ ]) X rank = p. A Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

12 Suppose x 1,..., x r denote r LI rows of X (written as column vectors) or, equivalently, r LI columns of X. We seek a 1,..., a s (s = p r) so that {x 1,..., x r, a 1,..., a s } is a set of p = r + s LI vectors in R p. [ ] X Then, with A = [a 1,..., a s ], will have full-column rank. A Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

13 Obviously, a 1,..., a s must be LI and a k / C(X ) k = 1,..., s. Show by example that these conditions are not sufficient to guarantee ([ ]) X rank = p. A opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

14 Example: 1 Suppose r = 1 and x 1 = x r = If a 1 = 1 and a 2 = 0, then a 1, a 2 LI, a 1 / C(X ), and a 2 / C(X ). 0 1 [ ] X However, = A has rank 2 < 3 = p Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

15 The problem arises because, although neither a 1 nor a 2 is in C(X ), LCs of a 1 and a 2 (e.g., a 1 + a 2 = x 1 ) that are in C(X ). Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

16 Prove the following results: Suppose x 1,, x r LI in R p. Suppose a 1,..., a s LI in R p. Suppose r + s = p. Then x 1,..., x r, a 1,..., a s LI span{x 1,..., x r } span{a 1,..., a s } = {0}. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

17 Proof: (= ) x 1,..., x r, a 1,..., a s LI iff c 1 x c r x r + d 1 a d s a s = 0 c 1 = = c r = d 1 = = d s = 0. If z span{x 1,..., x r } span{a 1,..., a s }, then c 1,..., c r and d 1,..., d s z = c 1 x c r x r = d 1 a 1 d s a s c 1 x c r x r + d 1 a d s a s = 0 c 1 = = c r = d 1 = = d s = 0 z = 0. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

18 ( =) span{x 1,..., x r } span{a 1,..., a s } = {0} c 1 x c r x r = d 1 a 1 d s a s only when r s c i x i = ( d j )a j = 0. i=1 j=1 x 1,..., x r LI and a 1,..., a s LI, r s c i x i = ( d j )a j = 0 i=1 j=1 c 1 = = c r = d 1 = = d s = 0. c 1 x c r x r + d 1 a d s a s = 0 only when c 1 = = c r = d 1 = = d s = 0. x 1,..., x r, a 1,..., a s are LI. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

19 Note that the condition span{x 1,..., x r } span{a 1,..., a s } = {0} (d 1 a d s a s ) β is not estimable whenever d 1,..., d s are not all 0. Thus, the constraints that we add, as well as all nontrivial LCs of those constraints, must correspond to nonestimable functions. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

20 Note that a 1,..., a s with the desired properties always exist. Why is that? opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

21 We know C(X ) has dim p r = s and satisfies C(X ) C(X ) = {0}. We can take a 1,..., a s to be basis vectors of C(X ) = N (X). opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

22 Although choosing basis vectors from N (X) is one possibility, we don t need a 1,..., a s N (X). Demonstrate this with an example. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

23 Example: 1 Suppose rank(x) = 1 and x 1 = x r = If we take a 1 = 0 and a 2 = 1, then a 1 / N (X), a 2 / N (X) and rank = 3 = p Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

24 The result that we proved can be alternatively stated as follows: Suppose rank( n p X) = r < p. Suppose rank( s p A) = s where s = p r. Then ([ ]) X C(X ) C(A ) = {0} rank = p. A Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

25 Lemma 3.1: For s p A rank(a) = s = p rank( n p X) and C(X ) C(A ) = {0}, [ ] [ ] X X X y b = A 0 is equivalent to [ ] [ ] X X X y b =, A A 0 which is equivalent to (X X + A A)b = X y. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

26 Proof of Lemma 3.1: Ab = 0 A Ab = 0 A Ab = 0 b A Ab = 0 Ab = 0. and Thus, the first equivalence holds. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

27 Now [ ] [ ] X X X y b = X Xb = X y and A Ab = 0 A A 0 X Xb + A Ab = X y (X X + A A)b = X y. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

28 Now note that (X X + A A)b = X y X Xb X y = A Ab X (Xb y) = A ( Ab). Now note that X (Xb y) C(X ) and A ( Ab) C(A ). Thus, X (Xb y) = A ( Ab) C(X ) C(A ), which implies X (Xb y) = A ( Ab) = 0. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

29 Therefore, we have X (Xb y) = 0 X Xb = X y A Ab = 0 A Ab = 0. [ ] [ ] X X X y b =. A A 0 Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

30 Result 3.6: Suppose rank( n p X) = r, rank( s p A) = s = p r, and C(X ) C(A ) = {0}. Then (i) X X + A A is nonsingular. (ii) (X X + A A) 1 X y is unique solution to X Xb = X y and Ab = 0. (iii) (X X + A A) 1 is GI of X X. (iv) A(X X + A A) 1 X = 0. (v) A(X X + A A) 1 A = I. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

31 Proof of Result 3.6: (i) ([ ]) X p = rank A ( [ ]) X = rank [X, A ] A = rank(x X + A A). (By Corollary 2.2) Thus, X X + A A is full rank p p matrix and is nonsingular. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

32 (ii) By Lemma 3.1, solution to (X X + A A)b = X y satisfies X Xb = X y and Ab = 0 and vice versa. X X + A A is nonsingular, the unique solution is (X X + A A) 1 X y, which is obtained by multiplying (X X + A A)b = X y on the left by (X X + A A) 1. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

33 (iii) By (ii), (X X + A A) 1 X y is a solution to X Xb = X y y R n. X X(X X + A A) 1 X y = X y y R n X X(X X + A A) 1 X = X X X(X X + A A) 1 X X = X X (X X + A A) 1 is GI of X X. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

34 (iv) By (ii), (X X + A A) 1 X y solves Ab = 0 y R n. A(X X + A A) 1 X y = 0 y R n. A(X X + A A) 1 X = 0. (v) HW Problem. See exercise opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

35 Returning to our example, y ij = µ + τ i + ε ij (i = 1,..., t; j = 1,..., n i ) where E(ε ij ) = 0 i, j. µ τ 1 β =.. Let s consider the constraint τ t t n iˆτ i = 0. i=1 Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

36 Find ˆβ that satisfies NE and the constraint. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

37 1 n1 1 n1 0 n1 0 n1 1 n2 0 n2 1 n2 0 n2 X = nt 0 nt 0 nt 1 nt n n 1 n 2 n t n 1 n X X = n 2 0 n n t 0 0 n t Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

38 nˆµ + n 1ˆτ n tˆτ t X n 1ˆµ + n 1ˆτ 1 Xˆβ =. n t ˆµ + n tˆτ t y X y 1 y =.. y t Equating X Xˆβ and X y leads to Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

39 nˆµ + n 1ˆτ n tˆτ t = y n 1ˆµ + n 1ˆτ 1 = y 1. n t ˆµ + n tˆτ t = y t. This system of equations has an infinite number of solutions. However, if we insist that n 1ˆτ n tˆτ t = 0, the first equation becomes opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

40 nˆµ = y ˆµ = ȳ. Substituting ˆµ = ȳ in the other equations yields n i ȳ + n iˆτ i = y i ȳ + ˆτ i = ȳ i ˆτ i = ȳ i ȳ Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41

41 For the general case, we can compute ˆβ = (X X + A A) 1 X y. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 41