Generalized Least-Squares Regressions V: Multiple Variables

Transcription

1 City University of New York (CUNY) CUNY Academic Works Publications Research Kinsborouh Community Collee -05 Generalized Least-Squares Reressions V: Multiple Variables Nataniel Greene CUNY Kinsborouh Community Collee How does access to this work benefit you? Let us know! Follow this additional works at: Part of the Mathematics Commons, Numerical Analysis Computation Commons, the Statistics Probability Commons Recommended Citation N Greene "Generalized Least-Squares Reressions V: Multiple Variables," in New Developments in Pure Applied Mathematics, 05, pp 7-5 This Article is brouht to you for free open access by the Kinsborouh Community Collee at CUNY Academic Works It has been accepted for inclusion in Publications Research by an authorized administrator of CUNY Academic Works For more information, please contact AcademicWorks@cunyedu

2 Generalized Least-Squares Reressions V: Multiple Variables Nataniel Greene Abstract The multivariate theory of eneralized least-squares is formulated here usin the notion of eneralized means The multivariate eneralized least-squares problem seeks an m dimensional hyperplane which minimizes the averae eneralized mean of the square deviations between the data the hyperplane in m + variables The numerical examples presented suest that a multivariate eneralized least-squares method can be preferable to ordinary least-squares especially in situations where the data are illconditioned Keywords Generalized least-squares, eometric mean reression, least-squares, multivariate reression, multiple reression, orthoonal reression II MULTIVARIATE REGRESSIONS The theory of eneralized least-squares was already described by this author for the case of two variables [4] [7] The extension of this theory to multiple variables is now beun A The Explicit Error Formula Solution for Ordinary Least-Squares The multivariate ordinary least-squares problem is dened as follows Denition : (Multivariate Ordinary Least-Squares Problem) An m dimensional hyperplane I OVERVIEW ORDINARY least-squares reression in multiple variables x 0 ; x ; :::; x m suffers from a fundamental lack of symmetry It beins with the choice of one variable, x 0 ; as the dependent variable x ; :::; x m as the independent variables It then minimizes the distance between the data the reression hyperplane in the x 0 variable alone However, the reression hyperplane formed by minimizin the distance between the data the hyperplane in the variable x k is not the same as solvin for x k in the hyperplane formed by minimizin the distance in the variable x j for j 6= k For each of the variables x k ; minimizin the distance between the data the hyperplane in the variable x k is called ordinary least-squares (OLS) x k j fx 0 ; :::; x m n fx k reression To predict the value of x k based on the data usin OLS, one must use the OLS x k j fx 0 ; :::; x m n fx k reression hyperplane It is not valid to take the OLS x j j fx 0 ; :::; x m n fx j hyperplane solve for x k when j 6= k The fact that there are m + OLS hyperplanes to model a sinle set of data in m + variables is problematic One wishes to have a sinle linear model for the data, for which it is valid to solve for any one of the variables for prediction purposes Multivariate eneralized leastsquares solves this problem by seekin to minimize the averae eneralized mean of the square deviations between the data the hyperplane in all the variables simultaneously For the resultin reression hyperplane, it is valid to solve for any of the variables for prediction purposes N Greene is with the Department of Mathematics Computer Science, Kinsborouh Community Collee, City University of New York, 00 Oriental Boulevard, Brooklyn, NY 5 USA (phone: ; nreenemath@mailcom) ISBN: x 0 = b 0 + b x + b x + ::: + b m x m () is souht which minimizes the error function dened by where E = N (x 0i ) () i= x 0i = b 0 + b x i + b x i + ::: + b m x mi x 0i : () This is called OLS x 0 j fx ; :::; x m reression In eneral, OLS x k j fx 0 ; :::; x m n fx k reression seeks a hyperplane which minimizes E = (x ki ) (4) N i= where b 0 b b k x ki = x 0i x i ::: x (k )i b k b k b k b k b k+ b m x (k+)i ::: x mi x ki (5) b k b k The deviation x ki at the ith data point (x 0i ; x i ; :::x mi ) is the difference between the hyperplane solved in terms of the variable x k evaluated at the data point the data value x ki : Stard subscript notations are employed for dealin with means, stard deviations, correlation coefcients covariances in multiple variables The ith data value for the kth variable x k is denoted by x ki, which is short for (x k ) i The means, stard deviations, correlation coefcients are denoted as follows: k = xk, k = xk, jk = xj;x k The covariance notation jk = jk j k is preferred in this paper because it makes many of the complex formulas presented here more manaeable The notation y = x 0 y i = x 0i can also be used However, denotin the y- variable always usin the zero subscript x 0 allows one to

3 easily identify it when there are any number of variables naturally ts with the subscript convention just described The notation x ki reatly simplies workin with the error function The next lemma describes a fundamental relation between the reression coefcient b k, x ki ; which is the deviation of the variable x k from the hyperplane at the ith data value x 0i ; which is the deviation of x 0 from the hyperplane at the ith data value Lemma : (Fundamental Relation) b k = x 0i (6) x ki or x ki = x 0i b k (7) The proof is straihtforward alebra This lemma will play a fundamental role further on in allowin one to always extract a weiht function from any eneralized least-squares error expression The explicit bivariate formula for the ordinary least-squares error described by Ehrenber [] has a known eneralization usin covariance notation It is written here as a matrix-vector equation also explicitly Theorem : (Explicit Multivariate Error Formula) Let b = (b ; :::; b m ) T, = ( ; :::; m ) T, s 0 = ( 0 ; :::; m0 ) T S = [ jk ] mm Then the multivariate ordinary least-squares error written in matrix-vector notation is E = b T Sb b T s b b T explicitly it is mx mx mx E = jk b j b k k0 b k (9) Alternatively write E = Proof: as follows E = N = N mx j= k= b k= kk b k + X j<k b k= ()! mx b k k : (0) k= jk b j b k mx k0 b k k=! mx b k k : () k= Bein with the error expression manipulate (b 0 + b x i + ::: + b m x mi x 0i ) i= (b (x i ) + ::: + b m (x mi m ) i= (x 0i 0 ) + (b b + ::: + b m m )) Square the summ distribute the summation onto each term To simplify, utilize the covariance notation jk = P N N i= x ji j (xki k ) utilize the fact that ISBN: P N i= x ji j = 0 for all j = 0:::m The result is then obtained Corollary 4: The explicit error formula written in covariance notation for two variables x x 0 is E = b 0 b (b b ) : () This is equivalent to Ehrenber's formula For three variables x ; x x 0, the formula is E = b + b + b b 0 b 0 b (b b + b ) () For four variables x ; x, x x 0, the formula is E = b + b + b + b b + b b + b b 0 b 0 b 0 b (b b + b + b ) (4) The explicit formula for the multivariate OLS reression coefcients is now written simply in matrix-vector form Theorem 5: (OLS explicit solution) The vector b of OLS x 0 j fx ; :::; x m reression coefcients is iven explicitly by b = S s 0 (5) b 0 = 0 b T (6) Proof: Let r = (@=@b ; m ) T denote the radient operator Take the radient of the error with respect to b set it equal to zero: re = 0 Use the matrix-vector form of the error distribute the radient onto each term r b T Sb b T s b b T = 0 Set b 0 = 0 B The Hessian Matrix Sb s 0 b b T = 0 b T ; obtain Sb = s 0 b = S s 0 : Since it is already known that the ordinary least-squares solution vector b minimizes the error function, the Hessian matrix H of second-order partial derivatives of E must be positive denite Recall that H is positive denite when det H > 0 when the determinants of all the upper-left submatrices of H are positive Alternatively, H is positive denite when all the eienvalues of H are positive It is instructive here to compute the Hessian matrix Theorem 6: The Hessian matrix is iven by m + m + m H = : (7) m m + m mm + m

4 Proof: Form all second order partial derivatives of the error function H (j+)(k+) j for j; k = 0:::m Verify that H = (b0;b ;:::;b m) H (k+) = H (k+) = k H (j+)(k+) = H (k+)(j+) = jk + j k : Theorem 7: The Hessian determinant is iven by det H = m+ det S Proof: Takin a multiple of one row addin it to another does not affect the determinant For k = :::m, multiply the rst row by k add it to row k + After this has been done to every row, perform a cofactor expansion alon the rst column obtain the result m det H = m+ + m + m m m + m mm + m m = m+ 0 m 0 m mm m = m+ m mm Theorem : The Hessian matrix H the covariance matrix S are both positive denite Proof: Since det H = m+ det S, the Hessian matrix H is positive denite if only if the covariance matrix S is positive denite However, it is already known that the OLS solution vector b minimizes the error function Therefore, conclude that H S are both positive denite C Generalized Means Denition 9: A function M (x 0 ; x ; :::; x m ) denes a eneralized mean for all x i > 0 if it satises Properties -5 below If it satises Property 6 it is called a homoenous eneralized mean The properties are: (Continuity) M (x 0 ; x :::; x m ) is continuous in each variable (Monotonicity) M (x 0 ; x :::; x m ) is non-decreasin in each variable (Symmetry) M (x 0 ; x ; :::; x m ) = M x s(0) ; x s() ; :::; s s(m) where s (i) is any permutation of the indices 0 throuh m ISBN: (Identity) 5 (Intermediacy) M (x; x; :::; x) = x: min (x 0 ; :::; x m ) M (x 0 ; :::; x m ) max (x 0 ; :::; x m ) 6 (Homoeneity) M (tx 0 ; tx ; :::; tx m ) = tm (x 0 ; x ; :::; x m ) for all t > 0: All the special multivariate means are included in this denition Note that m+ variables are used in this denition in order that it share the same form as the m + reression variables XMR notation is used here to name eneralized reressions: if `X' is the letter used to denote a iven eneralized mean, then XMR is the correspondin eneralized mean square reression Example 0: The multivariate harmonic mean is iven by m + H (x 0 ; :::; x m ) = x 0 + ::: + : () x m This mean enerates multivariate orthoonal reression (HMR) Example : The multivariate eometric mean is iven by G (x 0 ; :::; x m ) = (x 0 ::: x m ) =(m+) : (9) This mean enerates multivariate eometric mean reression (GMR) Example : The multivariate arithmetic mean is iven by A (x 0 ; :::; x m ) = m + (x 0 + ::: + x m ) : (0) This mean enerates multivariate arithmetic mean reression (AMR) Example : The selection mean is iven by S (k) (x 0 ; :::; x m ) = x k () after x 0 ; :::x m are arraned in increasin order This mean enerates OLS x k j fx 0 ; :::; x m n fx k reression Reressions based on these special cases are used in this work further on The eneralized means in the next examples have free parameters which can be used to parameterize these special cases Example 4: The power mean of order p is iven by =p M p (x 0 ; :::; x m ) = m + (xp 0 + ::: + xp m) : () Example 5: The weihted arithmetic mean with positive weihts satisfyin ::: + m = ; is iven by M (0; ;:::; m) (x 0 ; x ; :::; x m ) = 0 x 0 + x + ::: + m x m () after x 0 ; :::x m are arraned in increasin order Example 6: The weihted eometric mean with positive weihts satisfyin ::: + m = ; is iven by M (0 ; ;:::; m ) (x 0 ; x ; :::; x m ) = x 0 0 x :::x m m (4) after x 0 ; :::x m are arraned in increasin order

5 D Two Generalized Least-Squares Problems the Equivalence Theorem The multivariate symmetric least-squares problem is formulated as follows Denition 7: (The Multivariate Symmetric Least-Squares Problem) An m dimensional hyperplane x 0 = b 0 + b x + b x + ::: + b m x m (5) is souht which minimizes the error function dened by E = N M i= (x 0i ) ; (x i ) ; :::; (x mi ) (6) where M (x 0 ; x ; :::; x m ) is any eneralized mean A more eneral related problem is the weihted ordinary least-squares problem Denition : (The Weihted Ordinary Least-Squares Problem) An m dimensional hyperplane x 0 = b 0 + b x + b x + ::: + b m x m (7) is souht which minimizes the error function dened by E = (b ; :::; b m ) N (x 0i ) () or simply E = E OLS The next theorem states that every multivariate symmetric least-squares problem is equivalent to a weihted multivariate ordinary least-squares problem with weiht function (b ; :::; b m ) Theorem 9: (Equivalence Theorem) Every eneral symmetric least-squares error function can be written equivalently as E = (b ; :::; b m ) (x 0i ) (9) N or simply E = E OLS with (b ; :::; b m ) = M i= i= ; b ; :::; b m E OLS expressed usin the explicit error formula Proof: Write E = N = N = N M (x 0i ) ; (x i ) ; :::; (x mi ) i= M i= (x 0i ) ; b (x 0i ) ; :::; (x 0i ) M i= where the fundamental relation ; b ; :::; b m b m (0) (x 0i ) (x ki ) = b (x 0i ) k is used Let (b ; :::; b m ) = M ; ; :::; b b factor it m out from the summation ISBN: Example 0: The weiht function correspondin to HMR is iven by m + (b ; :::; b m ) = + b + ::: + : () b m Example : The weiht function correspondin to GMR is iven by (b ; :::; b m ) = (b ::: b m ) =(m+) : () Example : The weiht function correspondin to AMR is iven by (b ; :::; b m ) = + m + b + ::: + b : () m Example : The weiht function correspondin to the kth selection mean is iven by (b ; :::; b m ) = b : (4) k Example 4: The weiht function correspondin to the power mean is iven by p (b ; :::; b m ) = =p + b p + ::: + bm p : m + (5) Example 5: The weiht function correspondin to the weihted arithmetic mean is (b ; :::; b m ) = 0 + b + ::: + m b m : (6) Example 6: The weiht function correspondin to the weihted eometric mean is iven by (b ; :::; b m ) = b ::: b m m : (7) E Solvin for the Generalized Reression Coefcients The fundamental practical question of multivariate eneralized reression is how to solve for the coefcients b ; :::; b m The next theorem describes the procedure in eneral The procedure is applied further on to produce the specic reression equations in three four variables for several cases of interest Theorem 7: (System of Equations for Generalized Reression Coefcients) Let E denote the ordinary least-squares error function, k k, r = ( ; :::; m ) T F = b T Sb b T s : () Then the vector b = (b ; :::; b m ) T of reression coefcients is obtained by solvin the nonlinear matrix-vector equation for b b 0 = 0 system F r + (Sb s 0 ) = 0 (9) j= k= b T Explicitly, one solves the nonlinear k F + F k = 0 (40) for b ; :::; b m where k = :::m; mx mx F = jk b j b k mx k0 b k + 00 (4) k= F k k is iven by mx F k = jk b j k0 : (4) j=

6 Proof: Let E = E be the eneralized reression error where E = E OLS Use the matrix-vector form of the error, take the radient of the error with respect to b set it equal to zero Substitute Set b 0 = 0 r (E) = 0 Er + re = 0 E = b T Sb b T s b b T re = (Sb s 0 ) b b T : b T, obtain (Sb s 0 ) + F r = 0 (4) Alternatively, take the partial derivative of E with respect to b k for k = :::m set the resultin expressions equal to zero F The Hessian Matrix Determinant In order for the reression coefcients b 0 ; :::b m to minimize the error function be admissible, the Hessian matrix of second order partial derivatives must be positive denite when evaluated at b 0 ; :::; b m The eneral Hessian matrix is calculated next As in the bivariate case, certain combinations of its rst second partial derivatives appear in the matrix One combination is denoted here by J jk another is denoted by G jk They are called indicative functions Denition : Dene the indicative functions J jk = jk G jk = j j k (44) jk k : (45) The two indicative functions are related by the equation G jk F k = J jk F Theorem 9: (Hessian matrix) The Hessian matrix H of second order partial derivatives of the error function iven by H (j+)(k+) (E) j (b0;b ;:::;b m) for j; k = 0:::m It is computed explicitly as follows H = (47) H (k+) = H (k+) = k (4) H (j+)(k+) = H (k+)(j+) = J jk F + jk + j k Alternatively, (49) H (j+)(k+) = G jk F k + jk + j k : (50) ISBN: Since Proof: Take the second order (E) = ( j E + E j k = jk E + j E k + k E j + E jk : k E + E k = 0 substitute E k = k E E j = j E into the two middle terms, simplify, jk j k (E) j E + E jk which is the rst form of the Hessian Now substitute E = k E k obtain the j (E) = j jk E k + E jk : k As before, upon substitutin for b 0, E = F, E k = F k E jk = F jk = jk + j k Theorem 0: (Hessian determinant) The Hessian determinant is iven by det H = m+ det (F J + S) (5) where J = [J jk ] mm explicitly by det H = F J + F J m + m m+ : F J m + m F J mm + mm (5) Alternatively, det H = m+ det (K + S) (5) where K = [G jk F k ] mm explicitly by det H = G F + G m F m + m m+ : G m F + m G mm F m + mm (54) Proof: Bein with the (m + ) (m + ) determinant of H reduce it to an equivalent m m determinant The (m + ) (m + ) determinant is iven by det H = m+ G F + + m G m F + m + m m G m F m + m + m (55) G m F m + mm + m

7 or alternatively det H = m+ J F + + m J m F + m + m m J m F + m + m : (56) J m F + mm + m As was done for the OLS Hessian, multiply the rst row by k add it to row k + for k = :::m After this has been done to every row, perform a cofactor expansion alon the rst column obtain the result G Least-Volume Hybrid Least-Volume Reression A multivariate reression method is described by Tofallis [9] [] which minimizes the hypervolumes of the simplices formed by the data the reression hyperplane It is referred to here as Tofallis' least-volume reression (LVR) Denition : (LVR) Let V = x 0 x ::: x m denote the hypervolume of dimension m + for all x k > 0 An m dimensional hyperplane x 0 = b 0 +b x +:::+b m x m is souht which minimizes the averae hypervolume formed by the data points the hyperplane The error function is iven by E = V (x 0i ; x i ; :::; x mi ) : (57) N i= For the case of bivariate reression (m = ), least-volume reression is the same as eometric mean reression Multivariate GMR, rst described in a paper by Draper Yan [], is equivalent to minimizin the averae of the hypervolumes raised to the = (m + ) power For m 6=, LVR is not equivalent to GMR it is not a eneralized least-squares reression It is instead a eneralized least-pth power reression with p = m + Theorem : Least-volume reression in m + variables is equivalent to a weihted least-pth power reression with p = m + : Proof: Write E = N = N = N = V (jx 0i j ; jx i j ; :::; jx mi j) i= V i= jx 0i j ; jx 0i j m+ V i= jb ::: b m j N jb j jx 0ij ; :::; ; jb j ; :::; jb m j jx 0i j m+ i= The weiht function for LVR is iven by (b ; :::; b m ) = = jb ::: b m j : jb m j jx 0ij ISBN: A related weihted least-squares method is now dened ivin close results to LVR called hybrid least-volume reression It is a hybrid reression method alon the lines of the hybrid methods discussed in the rst paper of this series [4] Denition : (Hybrid LVR) An m dimensional hyperplane is souht which minimizes the hybrid error function where E = (b ; :::; b m ) N (b ; :::; b m ) = (x 0i ) (5) i= jb ::: b m j : (59) The error function is a product of the LVR weiht function the multivariate OLS error function The hybrid LVR coefcients obtained in the second example below are seen to differ from the actual LVR coefcients only in the hundredths place H Specic Reression Equations for the Case of Three Variables The eneral formula for the reression coefcients is applied here to the problem of determinin the coefcients in the equation x 0 = b 0 + b x + b x (60) for certain special cases Aain, covariance notation jk is used in order to obtain equations that are as simple as possible to write In all cases, b 0 = 0 b b For OLS x 0 j fx ; x ; which is stard OLS reression correspondin to the selection mean S (0) (x 0 ; x ; x ) a weiht function (b ; b ) =, the followin linear system of equations in b b is obtained b + b = 0 (6) b + b = 0 For OLS x j fx ; x 0 correspondin to the selection mean S () (x 0 ; x ; x ) the weiht function (b ; b ) =, the b followin system of equations is obtained b + b b 0 b 0 b + 00 = 0 (6) b + b 0 = 0 For OLS x j fx ; x 0 correspondin to the selection mean S () (x 0 ; x ; x ) the weiht function (b ; b ) =, the b followin system of equations is obtained b + b 0 = 0 b (6) + b b 0 b 0 b + 00 = 0 For HMR, which is orthoonal reression, the followin system of equations is obtained >: b + ( ) b b b b + 0 b + 0 b b 0 b ( 00 ) b + b 0 = 0 b ( ) b b b b + 0 b + 0 b b 0 b ( 00 ) b + b 0 = 0 (64)

8 For GMR the followin system of equations is obtained b b + b b 0 b + 0 b 00 = 0 b b b b 0 b + 0 b + 00 = 0 (65) For AMR the followin system of equations is obtained b 4 b + b b 0 b b + b 4 b 4 + b b b b + 0 b 0 b + 0 b b 00 b = 0 b >: b 4 + b b 0 b b b 4 + b 4 + b b b b + 0 b 0 b + 0 b b 00 b = 0 (66) For hybrid LVR the followin system of equations is obtained b b + 0 b 00 = 0 b b 0 b + 00 = 0 (67) I Specic Reression Equations for the Case of Four Variables The eneral formula for the reression coefcients is applied here to the problem of determinin the coefcients in the equation x 0 = b 0 + b x + b x + b x (6) for certain special cases In all cases, b 0 = 0 b b b For OLS x 0 j fx ; x ; x reression, which is stard ordinary least-squares, the followin system of equations is obtained < : b + b + b = 0 b + b + b = 0 b + b + b = 0 (69) For OLS x j fx ; x ; x 0 reression, the followin system of equations is obtained >: b + b + b b + b b + b b 0 b 0 b 0 b + 00 = 0 b + b + b 0 = 0 b + b + b 0 = 0 (70) For OLS x j fx ; x ; x 0 reression, the followin system of equations is obtained >: b + b + b 0 = 0 b + b + b b + b b + b b 0 b 0 b 0 b + 00 = 0 b + b + b 0 = 0 (7) For HMR the followin system of equations is obtained ( ) b b + b b + b b b b b b + b b b b + b b b b b 0 b + 0 b + 0 b 0 b b 0 b b + ( 00 ) b b b + 0 = 0 b b b b b b b b b + ( ) b b + b b + b b + b b + b + b 0 b + 0 b 0 b + 0 b b + 0 b b + b + ( 00 ) b + b 0 = 0 b b + ( ) b b b b b b b b b + b + b + b b + b b + b b 0 b 0 b + 0 b >: + 0 b b + 0 b b + b + b + ( 00 ) b 0 = 0 (7) For GMR, the followin system of equations is obtained b b b + b b b b + b b 0 b + 0 b + 0 b 00 = 0 b b + b b b b b + b b 0 b + 0 b 0 b + 00 = 0 b >: + b b + b b b b b b 0 b 0 b + 0 b + 00 = 0 (74) For AMR the followin system of equations is obtained b 4 b b + b b b + b b b 0 b b b + b b b + b b b b b b b b b b 4 b b 4 b b b 4 b b + b b + b 4 b + b b + 0 b b b + 0 b b + 0 b b 0 b b 0 b b 00 b b = 0 b b 4 b + b b b + b b b 0 b b b b b b b b b + b b b + b b b b 4 b + b 4 b + b b + b b b b b b 4 + b b b b b 0 b b + 0 b b + 0 b b 0 b b 00 b b = 0 b b b + b b b + b b b 4 0 b b b b b b + b b b b b b + b b b + b b 4 + b b + b b b 4 b b b + b >: b 4 b b 4 0 b b + 0 b b 0 b b + 0 b b + 0 b b b 00 b b = 0 (75) For hybrid LVR the followin system of equations is obtained b b b b b + 0 b + 0 b 00 = 0 b b + b + b b (76) 0 b 0 b + 00 = 0 b >: + b b + b b 0 b 0 b + 00 = 0 For OLS x j fx ; x ; x 0 reression, the followin system of equations is obtained >: b + b + b 0 = 0 b + b + b 0 = 0 b + b + b b + b b + b b 0 b 0 b 0 b + 00 = 0 (7) ISBN: III NUMERICAL EXAMPLES Example 4: (Cement Data) This example is taken from Hald's statistics text [] (p 66) The data describe the heat evolved in the curin of cement as a function of the percentae in weiht of certain compounds in the mixture The y variable, called here x 0 ; is the heat measured in calories per ram

9 The variables x x are the percentaes by weiht of two different cement compounds The data are as follows x x x The reader can verify that 0 = 95:40, 00 = 0:9045, 7:4654 = = ; (77) 4: :695 s 0 = = ; (7) 0 76:06 S = :940 9:6 : (79) 9:60 :5479 The data are well-conditioned: cond S = 7:547: Also det S = 6:766 0 The stard OLS x 0 j fx ; x reression plane is iven by x 0 = 5:577 + :46x + 0:66x : (0) The OLS x j fx ; x 0 reression plane is iven by x 0 = 5:466 + :565x + 0:656x : () The OLS x j fx ; x 0 reression plane is iven by x 0 = 5:97 + :449x + 0:6940x : () The HMR plane is iven by x 0 = 5:0 + :57x + 0:6607x : () The GMR plane is iven by x 0 = 5: :4950x + 0:6700x : (4) The AMR plane is iven by x 0 = 5:764 + :4699x + 0:6799x : (5) The hybrid LVR plane is iven by x 0 = 5:766 + :505x + 0:679x : (6) This example suests that all the methods yield reression planes that are reasonably close to each other when the data are well-conditioned The next example illustrates that when the data are ill-conditioned, ordinary least-squares can perform poorly while eneralized least-squares methods can still perform well Example 5: This example is chosen from the work of Tofallis [0], [] where the data are used to compare leastvolume reression to ordinary least-squares The data are from a model problem in Belsley's book on collinearity [] (p 5) are ill-conditioned Suppose an underlyin linear model of some physical relationship is known iven by x 0 = : 0:4x + 0:6x + 0:9x + " (7) where " has a normal distribution with mean 0 variance 0:0 Suppose two persons A B wish to estimate the ISBN: linear relationship for themselves Suppose they both share the same x 0 data but they take independent measurements of variables x, x, x Their results are presented in a table A x x x x B x x x x The oal is to try recover the actual coefcients b 0, b, b b from the data usin reression The reader can verify that for Person A, 0 = :950, 00 = :746, = 4 5 = 4 0: : ; () 0:005 s 0 = = :6706 0:4069 0: ; (9) 9:54 0: :065 S = 4 0: :76 0:0405 : (90) 0:065 0:040 0:0607 The data suffer from multicollinearity, which is the near linear dependence of one of the variables on the remainin variables This is evidenced by the hih condition number of the covariance matrix: cond S = : Also note that the determinant is nearly sinular: det S =6: : For Person B, 0 = :950, 00 = :746; = 4 5 = 4 0:7065 0:575 5 ; (9) 0:0075 s 0 = = 4 :6644 0: ; (9) 0: : 0: :07 S = 4 0: :79 0:04075 : (9) 0:07 0:0407 0:0609 Aain the covariance matrix is ill-conditioned nearly sinular: cond S = :40 7 det S=: : The stard OLS x 0 j fx ; x ; x reression planes for the data of Person A Person B are A : x 0 = : :974x + 9:09x :4400x (94) B : x 0 = :75 + 0:470x + 4:56x 7:646x : (95) The discrepancy between the OLS reression coefcients the model coefcients is strikin The OLS reression

10 coefcients disaree with the model coefcients in both sin manitude Therefore OLS reression does not appear to be useful in a case such as this In comparison, several eneralized reression methods presented next appear to do a much better job in recoverin the model coefcients, areein with the model coefcients in both sin manitude The AMR planes are A : x 0 = :479 0:4069x + 0:55x + 0:9767x (96) B : x 0 = :47 0:4070x + 0:55x + 0:9766x : The GMR planes are (97) A : x 0 = :440 0:445x + 0:4x + :095x (9) B : x 0 = :4 0:44x + 0:47x + :00x : The hybrid LVR planes are (99) A : x 0 = :06 0:470x + 0:67x + :95x (00) B : x 0 = :00 0:47x + 0:6x + :9x : The LVR planes are computed by Tofallis as (0) A : x 0 = :0 0:4x + 0:7x + :97x (0) B : x 0 = :0 0:4x + 0:7x + :9x : (0) In this example, AMR comes the closest to recoverin the model coefcients GMR, hybrid LVR LVR appear to perform comparably well The calculations of the three remainin OLS reressions HMR require further study They appear to have a neative Hessian determinant, makin them inadmissible The hybrid LVR coefcients obtained here are in ood areement with the LVR coefcients presented by Tofallis, differin only in the hundredths place This suests that hybrid LVR can be a useful least-squares alternative to LVR IV SUMMARY The extension of the bivariate theory of eneralized leastsquares to multivariate reression is beun in this paper The multivariate symmetric least-squares problem in m + variables seeks an m dimensional hyperplane which minimizes the averae eneralized mean of the square deviations between the data hyperplane in each of the variables The weihted multivariate ordinary least-squares problem in m + variables is also dened usin an explicit formula for the ordinary leastsquares error As in the bivariate case, every symmetric leastsquares problem is shown to be equivalent to a weihted ordinary least-squares problem The weiht function (b ; :::; b m ) characterizes the reression method The multivariate eneralized least-squares error is then a product of the weiht ISBN: function the explicit multivariate error function Partial derivatives of this analytic expression for the error are then taken with respect to each of the coefcients b ; :::; b m set equal to zero The result is a nonlinear system of equations in b ; :::; b m involvin only the covariances jk which can then be solved to yield the reression coefcients for any eneralized least-squares method In order for the solution to minimize the error function be admissible, the Hessian matrix must be computed found to be positive denite The specic system of equations for the reression coef- cients is presented for OLS, HMR, GMR, AMR, for three four variables A related least-squares alternative to Tofallis' least-volume reression (LVR) called hybrid LVR is presented here as well Numerical evidence suests that when the data are illconditioned ordinary least-squares reressions may not succeed in uncoverin an underlyin linear model In comparison, certain eneralized least-squares methods can come closer to uncoverin the model coefcients REFERENCES [] D A Belsley, Conditionin Dianostics, New York: John Wiley & Sons, 99 [] N R Draper Y Yan, "Generalization of the mean functional relationship," Computational Statistics Data Analysis, vol, pp 55-7, 997 [] S C Ehrenber, "Derivin the Least-Squares Reression Equation," The American Statistician, vol 7, p, Au 9 [4] N Greene, "Generalized Least-Squares Reressions I: Efcient Derivations," in Proceedins of the st International Conference on Computational Science Enineerin (CSE'), Valencia, Spain, 0, pp 45-5 [5] N Greene, "Generalized Least-Squares Reressions II: Theory Classication," in Proceedins of the st International Conference on Computational Science Enineerin (CSE '), Valencia, Spain, 0, pp [6] N Greene, "Generalized Least-Squares Reressions III: Further Theory Classication," in Proceedins of the 5th International Conference on Applied Mathematics Informatics (AMATHI '4), Cambride, MA, 04, pp 4- [7] N Greene, "Generalized Least-Squares Reressions IV: Theory Classication Usin Generalized Means," in Mathematics Computers in Science Industry, Varna, Bularia, 04, pp 9-5 [] A Hald, Statistical Theory with Enineerin Applications, New York: John Wiley & Sons, 95 [9] C Tofallis, "Model Fittin Usin the Least Volume Criterion," in Alorithms for Approximation IV, JC Mason J Levesly, Eds, University of Hudderseld Press, 00 [0] C Tofallis, "Model Fittin for Multiple Variables by Minimizin the Geometric Mean Deviation," in Total least squares errors-invariables modellin: alorithms analysis applications, S Van Huffel P Lemmerlin, Eds, Dordrecht: Kluwer Academic Publishers, 00 [] C Tofallis, "Multiple Neutral Data Fittin," Annals of Operations Research, vol 4, pp 69-79, 00