Ann Inst Stat Math (2010) 62:929 941 DOI 10.1007/s10463-008-0199-8 Weighted least-squaes estimatos of paametic functions of the egession coefficients unde a geneal linea model Yongge Tian Received: 9 Januay 2008 / Revised: 9 June 2008 / Published online: 9 Septembe 2008 The Institute of Statistical Mathematics, Tokyo 2008 Abstact The weighted least-squaes estimato of paametic functions Kβ unde a geneal linea egession model {y, Xβ, σ 2 } is defined to be K ˆβ, whee ˆβ is a vecto that minimizes (y Xβ) V(y Xβ) fo a given nonnegative definite weight matix V. In this pape, we study some algebaic and statistical popeties of K ˆβ and the pojection matix associated with the estimato, such as, thei anks, unbiasedness, uniqueness, as well as equalities satisfied by the pojection matices. Keywods Geneal linea egession model Paametic functions WLSE Pojection matix Unbiasedness of estimato Uniqueness of estimato 1 Intoduction Thoughout this pape, R m n denotes the set of all m n eal matices. The symbols A, (A) and R(A) stand fo the tanspose, the ank and the ange (column space) of amatixa R m n, espectively. The Mooe Penose invese of A R m n, denoted by A +, is defined to be the unique solution G to the fou matix equations (i) AGA = A, (ii) GAG = G, (iii)(ag) = AG, (iv) (GA) = GA. AmatixG is called a genealized invese of A, denoted by A,ifitsatisfies(i). Futhe, let P A, E A and F A stand fo the thee othogonal pojectos P A = AA +, E A = I m AA + and F A = I n A + A. Y. Tian (B) China Economics and Management Academy, Cental Univesity of Finance and Economics, 100081 Beijing, China e-mail: yongge.tian@gmail.com
930 Y. Tian Conside a geneal linea egession model o in the compact fom y = Xβ + ε, E(ε) = 0, Cov(ε) = σ 2, (1) {y, Xβ, σ 2 }, (2) whee y R n 1 is an obsevable andom vecto, X R n p is a known matix of abitay ank, β R p 1 is a vecto of unknown paametes, R n n is a known o unknown nonnegative definite matix, and σ 2 is an unknown positive paamete. Let K R k p be a given matix. Then the poduct Kβ is called a vecto of paametic functions of β in (2). The vecto Kβ is said to be estimable unde (2) if thee exists a matix L R k n such that E(Ly) = Kβ holds. It is well known that Kβ is estimable unde (2) if and only if R(K ) R(X ), (3) see, e.g., Alalouf and Styan (1979) and Tian et al. (2008). The most popula method fo appoaching estimations of paametic functions unde a linea model is to apply vaious citeia of minimizing the (weighted) sum of squaed deviations unde the model. Let V R n n be a given nonnegative definite (nnd) matix, i.e., V can be witten as V = ZZ fo some matix Z. Then the weighted least-squaes estimato (WLSE) of β in (2), denoted by WLSE(β), is defined to be WLSE(β) = agmin β (y Xβ) V(y Xβ); (4) the WLSE of the paametic functions Kβ unde (2) is defined to be WLSE(Kβ) = KWLSE(β). As is well known, the nomal matix equation coesponding to (4) is given by X β = X Vy. (5) This equation is always consistent. Solving the equation gives the following esult. Lemma 1 Let K R k p. Then the geneal expession of the WLSE of Kβ unde (2) can be witten as whee WLSE(Kβ) = P K;X;V y, (6) P K;X;V = K(X ) + X V + KF U (7)
Weighted least-squaes estimatos of paametic functions 931 is called the pojection matix associated with WLSE(Kβ), in which U R p n is abitay. The expectation and covaiance matix of WLSE(Kβ) ae given by EWLSE(Kβ)=P K;X;V Xβ, (8) CovWLSE(Kβ)=σ 2 P K;X;V P K;X;V. (9) In paticula, the WLSE of the mean vecto Xβ in (2) is given by whee In this case, the matix P X;V in (11) is squae. WLSE(Xβ) = P X;V y, (10) P X;V = X(X ) + X V + XF U. (11) The WLSEs given in Lemma 1 ae a most useful class of estimatos in egession analysis. Many estimatos unde a geneal linea model can be egaded as special cases of the WLSEs with espect to some given weight matices. If the covaiance matix in (2) is known, the two commonly used choices of the weight matix V in (4) aegivenby (i) V = ( +XTX ), whee T is a symmetic matix such that (V) = X,, (ii) V = 1 if is nonsingula. In these cases, the WLSEs of Kβ ae the best linea unbiased estimatos (BLUEs) of Kβ unde (2) fo the estimable Kβ. If the covaiance matix is unknown, the weight matix V in (4) may be chosen accoding to some estimations of in (2), o some assumptions on the patten of. In what follows, we assume that the weight matix V in (4)isgiven. It can be seen fom (6), (8), (9) and (10) that algebaic and statistical popeties of WLSE(Kβ) and WLSE(Xβ) ae mainly detemined by the pojection matices P K;X;V, P X;V and the matix poducts P K;X;V X, P K;X;V, P X;V X and P X;V. These facts pompt us to give a compehensive appoach to the pojection matices P K;X;V, P X;V and the poducts P K;X;V X, P K;X;V, P X;V X and P X;V fom mathematical point of view. The pojection matix P X;V was studied by some authos, see, e.g., Baksalay and Puntanen (1989), Mita and Rao (1974), and Rao (1974) among othes. Takane et al. (2007) and Tian and Takane (2008a,b) ecently econsideed the pojection matix P X;V and the coesponding estimato WLSE(Xβ) = P X;V y by making use of the matix ank method, and obtained some new esults on P X;V and WLSE(Xβ). In this pape, we conside the following poblems on the pojection matix P K;X;V in (6) and the coesponding WLSE(Kβ) in (6): (I) (II) (III) The maximal and minimal possible anks of P K;X;V, P K;X;V X and P K;X;V, and thei ank invaiance. Uniqueness of P K;X;V, P K;X;V X and P K;X;V. Identifying conditions fo P K;X;V X = K to hold.
932 Y. Tian (IV) Equalities satisfied by P K;X;V. (V) Uniqueness of WLSE(Kβ). (VI) Unbiasedness of WLSE(Kβ). (VII) Rank of CovWLSE(Kβ). The poofs of the main esults in the pape ae given in Appendix. In what follows, we give some well-known esults on anks of matices and a linea matix equation. These esults will be used in Sects. 2 and 3 fo deiving popeties of P K;X;V and WLSE(Kβ) = P K;X;V y. The esults in the following lemma wee shown by Masaglia and Styan (1974). Lemma 2 Let A R m n, B R m k and C R l n. Then A, B =(A) + (E A B) = (B) + (E B A), (12) A = (A) + (CF C A ) = (C) + (AF C ). (13) The following esult was given by Tian and Styan (2001). Lemma 3 Any pai of idempotent matices A and B of the same size satisfy ( A B ) = A B + A, B (A) (B). (14) Hence A = B if and only if R(A) = R(B) and R(A ) = R(B ). We also need two fomulas shown by De Moo and Golub (1991) on the extemal anks of the matix expession A BZC; see also Tian (2002) and Tian and Cheng (2003). Lemma 4 Let A R m n, B R m k and C R l n be given. Then { } A max ( A BZC ) = min A, B,, (15) Z Rk l C A AB min ( A BZC ) = A, B +. (16) Z R k l C C0 The following esult is well known; see Penose (1955). Lemma 5 Let A R m n, B R m k and C R l n be given. Then the matix equation BZC = A is solvable fo Z if and only if R(A) R(B) and R(A ) R(C ), o equivalently, BB + AC + C = A. In this case, the geneal solution can be witten as Z = B + AC + + F B U 1 + U 2 E C, whee U 1, U 2 R k l ae abitay. To simplify some matix opeations in Sects. 2 and 3, we use the following simple esults on the Mooe Penose inveses: A + = (A A) + A = A (AA ) +, (17) (X )(X ) + = (X V)(X V) +,(X ) + (X ) = () + (). (18)
Weighted least-squaes estimatos of paametic functions 933 2 Popeties of the pojection matix P K;X;V Notice that P K;X;V, P K;X;V X and P K;X;V in (7), (8) and (9) ae special cases of a geneal poduct P K;X;V Q. We fist show some algebaic popeties of the poduct P K;X;V Q. Theoem 1 Let P K;X;V be as given in (7) and let Q R n q. Then Hence, (a) max (P K;X;V Q) = min P K;X;V min P K;X;V ( P K;X;V Q ) = { X VQ X X VQ X } (), (Q), (19) K X. (20) The ank of the poduct P K;X;V Q is invaiant if and only if R(K ) R(X V) o X VQ X = K X + (Q). (b) The poduct P K;X;V Q is unique if and only if R(K ) R(X V) o Q = 0. Applying Theoem 1 to P K;X;V, P K;X;V X and P K;X;V leads to the following coollay. Coollay 1 Let P K;X;V be as given in (7). Then, (a) The maximal and minimal anks of P K;X;V ae given by max (P K;X;V ) = (K), (21) P K;X;V K min (P K;X;V ) = (K) + (). (22) P K;X;V (b) The ank of P K;X;V is invaiant if and only if R(K ) R(X V). (c) P K;X;V is unique if and only if R(K ) R(X V), in which case, P K;X;V = K(X ) + X V. (d) The maximal and minimal anks of the poduct P K;X;V X ae given by max (P K;X;V X) = min{ (K), (X) }, (23) P K;X;V K min ( P K;X;V X ) = (K) + (). (24) P K;X;V (e) The ank of P K;X;V X is invaiant if and only if R(K ) R(X V) o (K) + () = K, X V +(X). (f) The poduct P K;X;V X is unique if and only if R(K ) R(X V) o X = 0.
934 Y. Tian (g) The maximal and minimal anks of P K;X;V ae given by max (P K;X;V ) = min P K;X;V min P K;X;V ( P K;X;V )= { X V X X V X } (), ( ), (25) K X. (26) (h) The poduct P K;X;V is unique if and only if R(K ) R(X V) o = 0. (i) The ank of P K;X;V is invaiant if and only if R(K ) R(X V) o X V X = K X + ( ). 3 Existence of unbiased WLSE(Kβ) and its popeties One of the most impotant and desiable popeties of WLSE(Kβ) is its unbiasedness fo Kβ unde (2). It can be seen fom (8) that the estimato WLSE(Kβ) is unbiased fo Kβ unde (2) if and only if P K;X;V Xβ = Kβ. (27) Lemma 6 Let WLSE(Kβ)be as given in (6). Then the estimato is unbiased fo Kβ unde (2) if and only if P K;X;V Xβ = Kβ. In paticula, if P K;X;V X = K, (28) then the estimato WLSE(Kβ) in (6) is unbiased fo Kβ unde (2). Solving the equation in (28), we obtain the following esult. Theoem 2 Let P K;X;V be as given in (7). Then, (a) Thee exists a P K;X;V such that (28) holds if and only if R(K ) R(X ), namely, Kβ is estimable unde (2). (b) Unde (a), the geneal expession of P K;X;V satisfying (28) can be witten as o equivalently, ˆP K;X;V = P K;X;V + KF X + + KF WE X, (29) ˆP K;X;V = KX + P K;X;V ( I n XX ), (30) whee P K;X;V = K(X ) + X V, W R p n is abitay, and X is any genealized invese of X. Moe popeties of the pojection matix ˆP K;X;V in (29) ae given below.
Weighted least-squaes estimatos of paametic functions 935 Theoem 3 Let ˆP K;X;V be as given in (29). Then, (a) Z {ˆP K;X;V } if and only if ZX = K and Z, =(X). (b) ( ˆP K;X;V ) = (K). (c) R( ˆP K;X;V ) = R(K). (d) ˆP K;X;V is unique fo any X if and only if R(K ) R(X V) o (X) = n. In this case, (i) ˆP K;X;V = K(X ) + X V if R(K ) R(X V), (ii) ˆP K;X;V = KX + if (X) { = n. X (e) max ( ˆP K;X;V ) = min X } V X (), (X). ˆP K;X;V K 0 K0 X (f) min ( ˆP K;X;V ) = X X 0 V X + 0. ˆP K;X;V K 0 0 (g) ˆP K;X;V is unique if and only if R(K ) R(X V) o R( ) R(X). Applying Theoems 2 and 3 to (11) gives the following esults. Coollay 2 Let P X;V be as given in (11). Then, (a) Thee always exists a P X;V such that P X;V X = X (31) holds. Moeove, the geneal expession of P X;V satisfying (31) can be witten as o equivalently, ˆP X;V = P X;V + XF X + + XF WE X, (32) ˆP X;V = XX + P X;V ( I n XX ), (33) whee P X;V = X(X ) + X V, W R p n is abitay, and X is any genealized invese of X. (b) ˆP X;V is idempotent fo any X. (c) ( ˆP X;V ) = (X). (d) (e) (f) R( ˆP X;V ) = R(X). ˆP X;V is unique fo any X if and only if R(X ) = R(X V) o (X) = n. In this case, ˆP X;V = X(X ) + X V fo R(X ) = R(X V), o ˆP X;V = I n fo (X) = n. The poduct V ˆP X;V is unique fo any ˆP X;V, and V ˆP X;V = (X ) + X V = V 1/2 P V 1/2 XV 1/2, R(V ˆP X;V ) = R(), (V ˆP X;V ) = (X V ˆP X;V ) = (), whee V 1/2 is the nnd squae oot of V.
936 Y. Tian (g) (h) (i) max ( ˆP X;V ) = min {(X) + ( ) (), ( )}. ˆP X;V X min ( ˆP X;V ) = (X) + ( ) + ( ). ˆP X;V 0 ˆP X;V is unique if and only if R(X ) = R(X V) o R( ) R(X). Let W = 0. Then (32) educes to ˆP X;V = P X;V + XF X + = P X + P X;V ( I n P X ). (34) Some popeties of the pojecto ˆP X;V in (34) ae given below. Theoem 4 Let ˆP X;V be as given in (34). Then the following statements ae equivalent: (a) ˆP X;V = ˆP X;V. (b) ˆP X;V = P X. (c), X =(X), i.e., R() R(X). Some popeties of WLSE(Kβ) and WLSE(Xβ) coesponding to ˆP K;X;V and ˆP X;V in (29) and (32) ae given in the following two theoems. Theoem 5 Assume Kβ is estimable unde (2), and let ˆP K;X;V be as given in (29). Then, (a) The coesponding estimato WLSE(Kβ) in (6) can be witten as WLSE(Kβ) = ˆP K;X;V y = ( P K;X;V + KF X + + KF WE X )y, (35) (b) whee P K;X;V = K(X ) + X V, and W R p n is abitay. The expectation and the covaiance matix of WLSE(Kβ) in (35) ae given by EWLSE(Kβ)=Kβ and CovWLSE(Kβ)=σ 2 ˆP K;X;V ˆP K;X;V. (36) (c) (d) (e) (f) WLSE(Kβ) in (35) is unique if and only if R(K ) R(X V) o y R(X). In this case, (i) WLSE(Kβ) = K(X ) + X Vy if R(K ) R(X V), (ii) WLSE(Kβ) = KX + y if y { R(X). X max (CovWLSE(Kβ)) = min X V (), K 0 } X (X). K0 X min (CovWLSE(Kβ)) = X X 0 V X + 0. K 0 0 CovWLSE(Kβ) is unique if and only if R(K ) R(X V) o R( ) R(X).
Weighted least-squaes estimatos of paametic functions 937 Equations (35) and (36) eveal such an unexpected fact that fo any given weight matixv and any estimable paametic functions Kβ, thee always exists a WLSE(Kβ) that is unbiased fo Kβ unde (2). In such case, we can constuct the weight matix V and the abitay matix W in (35) such that the unbiased WLSE(Kβ) has some pescibed popeties. Theoem 6 Let ˆP X;V be as given in (32). Then, (a) The coesponding estimato WLSE(Xβ) = ˆP X;V y can be witten as WLSE(Xβ) = ˆP X;V y = ( P X;V + XF X + + XF WE X )y, (37) (b) whee P X;V = X(X ) + X V, and W R p n is abitay. The expectation and the covaiance matix of WLSE(Xβ) in (37) ae given by EWLSE(Xβ)=Xβ and CovWLSE(Xβ)=σ 2 ˆP X;V ˆP X;V. (38) (c) (d) (e) (f) WLSE(Xβ) in (37) is unique if and only if R(X ) = R(X V) o y R(X). In this case, (i) WLSE(Xβ) = X(X ) + X Vy if R(X ) = R(X V), (ii) WLSE(Xβ) = P X y if y R(X). max (CovWLSE(Xβ)) = min{ ( ) + (X) (), ( )}. X min (CovWLSE(Xβ)) = ( ) + (X) + ( ). 0 CovWLSE(Xβ) is unique if and only if R(X ) = R(X V) o R( ) R(X). Acknowledgments The autho is gateful to an anonymous efeee fo helpful comments and suggestions on an ealie vesion of this pape. Appendix Poof of Theoem 1 It is easily seen fom (7) that the poduct P K;X;V Q can be witten as P K;X;V Q = K(X ) + X VQ + KF UQ, (39) whee the matix U is abitay. Recall that elementay block matix opeations (EBMOs) do not change the ank of a matix. Applying (15) and (16) and EMBOs to this expession gives max ( P K;X;V Q ) = max P K;X;V U K(X ) + X VQ + KF UQ = min{ K(X ) + X VQ, KF, (Q) }, min ( P K;X;V Q ) = min P K;X;V U K(X ) + X VQ + KF UQ = K(X ) + X VQ, KF (KF ),
938 Y. Tian wheeby(13) K(X ) + X K(X VQ, KF = ) + X VQ K 0 = (X ) + X VQ = X VQ X (). () () Hence we have (19) and (20). Result (a) follows fom (19) and (20). Result (b) is obvious fom (39). Poof of Theoem 2 Substituting (7) into(28) and simplifying by (18) gives KF UX = KF. (40) Fom Lemma 5, the equation in (40) is solvable fo U if and only if Applying EBMOs gives X KF = (X). (41) X X = KF K K() + () = X K, so that (41) is equivalent to R(K ) R(X ). In this case, it can deived fom Lemma 5 that the geneal solution of (40) is U = X + +I n (KF ) + (KF ) U 1 + U 2 E X, (42) whee U 1, U 2 R p n ae abitay. Substituting (42) into(7) yields ˆP K;X;V = P K;X;V + KF X + + KF U 2 E X, (43) establishing (29). Eq. (30) is obtained by setting U = X in (7): ˆP K;X;V = P K;X;V +K K() + () X = KX + P K;X;V ( I n XX ). In fact, ecall that the geneal expession of X can be witten as X = X + +F X U 1 + U 2 E X, whee U 1 and U 2 ae abitay matices. Substituting this X into (30) yields ˆP K;X;V = P K;X;V + KF X = P K;X;V + KF X + + KF U 2 E X, which is the same as (29).
Weighted least-squaes estimatos of paametic functions 939 Poof of Theoem 3 It can be seen fom (29) that Z {ˆP K;X;V } if and only if the following matix equation P K;X;V + KF X + + KF WE X = Z (44) is solvable fo W. By Lemma 5, the equation in (44) is solvable fo W if and only if G G, KF =(KF ) and = (E X ), (45) EX whee G = Z P K;X;V KF X +. It can be deived fom (12), (13), (18) and EMBOs that (XF ) = (X) (), (E X ) = n (X), and Z P K;X;V KF X + K G, KF = () 0 Z K = (X ) + X () V Z K ZX = (X ) + X () V 0 Z K ZX = X (), V 0 G Z P K;X;V KF X + 0 = (X) E X I n X 0 ZX + K(X ) + X + KF X + X = (X) I n 0 ZX = (X) I n 0 = n + (K ZX) (X). Z K ZX Hence (45) is equivalent to X = (X) and ZX = K, establishing V 0 (a). It follows fom ˆP K;X;V X = K that ( ˆP K;X;V ) (K). Also note fom (29) that ( ˆP K;X;V ) (K). Thus ( ˆP K;X;V ) = (K) fo any W in (29), as equied fo (b). The ange equality in (c) is obvious fom (29) and (b). Result (d) follows diectly fom (29). Let G = P K;X;V + KF X +. Then it can be deived fom (15), (16) and EBMOs that
940 Y. Tian max ( ˆP K;X;V ) = max ( G + KF WE X ) ˆP K;X;V W { } G = min G, KF,, (46) E X min ˆP K;X;V ( ˆP K;X;V ) = min W ( G + KF WE X ) = G, KF + G E X G KF. (47) E X 0 Applying (12), (13), (18) and simplifying by EMBOs gives ise to K(X G, KF = ) + X V + KF X + K () 0 = (X ) + X () V X = X V (), K 0 G K(X = ) + X V + KF X + 0 (X) E X X X = (X), K 0 K(X ) + X V + KF G X + K 0 KF = 0 X () (X) E X 0 0 0 0 = 0 X () (X) (X ) + X V 0 X 0 = 0 () (X). 0 Substituting these ank equalities into (46) and (47) gives (e) and (f). It can be deived fom (29) that ˆP K;X;V is unique if and only if KF = 0 o E X = 0. Hence (g) follows. Poof of Coollay 2 Results (a), (c), (d), (e), (g), (h) and (i) follow fom Theoem 3. Note that (XX ) 2 = XX and XX P X;V ( I n XX ) = P X;V ( I n XX ). Thus it is easy to deive fom (30) that
Weighted least-squaes estimatos of paametic functions 941 ˆP 2 X;V =XX + P X;V ( I n XX ) 2 = XX + P X;V ( I n XX ) = ˆP X;V, establishing (b). Result (f) follows fom (29). Poof of Theoem 4 Since both ˆP X;V and ˆP X;V ae idempotent, it can be deived fom (14) that ( ˆP X;V ˆP X;V ) = 2 ˆP X;V, ˆP X;V 2( ˆP X;V ) = 2 X, P X + ( I n P X )(X ) + X 2(X) = 2 X, (X ) + X 2(X) = 2 X, 2(X). It can also be deived fom (14) that ( ˆP X;V P X ) = ˆP X;V, P X + ˆP X;V, P X ( ˆP X;V ) (X) = P X + ( I n P X )(X ) + X, P X (X) =, X (X). Thus (a), (b) and (c) ae equivalent. Refeences Alalouf, I. S., Styan, G. P. H. (1979). Chaacteizations of estimability in the geneal linea model. Annals of Statistics, 7, 194 200. Baksalay, J. K., Puntanen, S. (1989). Weighted-least-squaes estimation in the geneal Gauss Makov model. In Y. Dodge (Ed.), Statistical data analysis and infeence (pp. 355 368). Amstedam: Elsevie. De Moo, B., Golub, G. H. (1991). The esticted singula value decomposition: Popeties and applications. SIAM Jounal on Matix Analysis and Applications, 12, 401 425. Masaglia, G., Styan, G. P. H. (1974). Equalities and inequalities fo anks of matices. Linea and Multilinea Algeba, 2, 269 292. Mita, S. K., Rao, C. R. (1974). Pojections unde seminoms and genealized Mooe Penose inveses. Linea Algeba and Its Applications, 9, 155 167. Penose, R. (1955). A genealized invese fo matices. Poceedings of the cambidge philosophical society, 51, 406 413. Rao, C. R. (1974). Pojectos, genealized inveses and the BLUE s. Jounal of the Royal Statistical Society- Seies B, 36, 442 448. Takane, Y., Tian, Y., Yanai, H. (2007). On constained genealized inveses of matices and thei popeties. Annals of the Institute of Statistical Mathematics, 59, 807 820. Tian, Y. (2002). The maximal and minimal anks of some expessions of genealized inveses of matices. Southeast Asian Bulletin of Mathematics, 25, 745 755. Tian, Y., Cheng, S. (2003). The maximal and minimal anks of A BXC with applications. New Yok Jounal of Mathematics, 9, 345 362. Tian, Y., Styan, G. P. H. (2001). Rank equalities fo idempotent and involutay matices. Linea Algeba and Its Applications, 335, 101 117. Tian, Y., Takane, Y. (2008a). Some popeties of pojectos associated with the WLSE unde a geneal linea model. Jounal of Multivaiate Analysis, 99, 1070 1082. Tian, Y., Takane, Y. (2008b). On V-othogonal pojectos associated with a semi-nom. Annals of the Institute of Statistical Mathematics, inpess. Tian, Y., Beisiegel, M., Dagenais, E., Haines, C. (2008). On the natual estictions in the singula Gauss Makov model. Statistical Papes, 49, 553 564.