Generalized Least-Squares Regressions II: Theory and Classication

Transcription

1 Recent Advances in Intelligent Control, Modelling Comutational Science Generalized Least-Squares Regressions II Theor Classication NATANIEL GREENE Deartment of Mathematics Comuter Science Kingsorough Communit College, CUNY 00 Oriental Boulevard, Brookln, NY 35 UNITED STATES Astract In the rst aer of this series, a variet of known new smmetric weighted least-squares regression methods were resented with efcient derivations This aer continues generalizes the revious work with a theor for deriving, analzing, classifing all smmetric weighted least-squares regression methods Ke Words Least-squares, smmetric least-squares, weighted ordinar least-squares, orthogonal regression, geometric mean regression Overview Ordinar least-squares regression suffers from a fundamental lack of smmetr the regression line of given the regression line of given are not inverses of each other Two alternative smmetric regression methods which overcome this rolem are orthogonal regression geometric mean regression In the rst aer of this series [], a variet of alternative smmetric weighted regression methods were resented analzed The derivations were efcient in their use of Ehrenerg's formula for the ordinar least-squares error [], avoiding the cumersome algeraic maniulations with summations which would otherwise have een necessar The derivations also etracted a unique weight function g () from the error eression in ever case Clearl there are innitel man cases of smmetric weighted regressions, a similar efcient derivation will al in ever case With the attern of derivation now clearl estalished, this aer generalizes the rocedures in a theor for comuting classifing an generalized least-squares regression In the theor, the general smmetric least-squares rolem the general weighted least-squares rolem are formall de- ned Since generalized regressions are characterized their weight function g (), this aer derives formulas for the regression coefcients a in terms of g () as well as general formulas for the Hessian matri determinant In the rocess, formulas for general classes of weight functions emerge, all the regression cases derived reviousl are categorized as elonging to various weight function classes Theor Classication The theor for classifing least-squares regressions is detailed now General Framework Coefcient Formulas Denition Dene a function (; ) that is (i) Non-negative (; ) 0 (ii) Smmetric in (iii) Even in (; ) = (; ) ( ; ) = (; ) (iv) Non-decreasing in For differentiale this means 0 0 (v) Homogeneous with degree in (; ) = (; ) ISBN

2 Recent Advances in Intelligent Control, Modelling Comutational Science Denition (The General Smmetric Least-Squares Prolem) Values of a are sought which minimize an error function dened E = N i= a + i i ; a + i i () Denition 3 (The General Weighted Ordinar Least- Squares Prolem) Values of a are sought which minimize an error function dened or E = g () N (a + i i ) () i= E = g () + ( ) + a + (3) where g () is a ositive even function that is nondecreasing for < 0 non-increasing for > 0 The net theorem is fundamental it is alread anticiated from the revious work It states that ever generalized smmetric least-squares rolem is equivalent to a weighted ordinar least-squares rolem with weight function given g () Theorem 4 The general smmetric least-squares error function can e written equivalentl as or where E = g () N (a + i i ) (4) i= E = g () + ( ) + a + g () = (5) ; (6) Proof Sustitute a + i i with (a + i i ) then use the homogeneit roert E = N N = N i= i= i= a + i i ; a + i i a + i i ; (a + i i ) (a + i i ) ; Dene write g () = E = g () N ; (a + i i ) i= Sustitute using Ehrenerg's formula otain E = g () + ( ) + a + From the characterization of the generating function it follows that the weight function must e a ositive even function of that is non-decreasing for < 0 non-increasing for > 0 The following tale summarizes the non-hrid smmetric regressions detailed earlier Case Generating OLS Weight Function Function ψ (, ) g() OLS OLS Orthogonal + + GMR Pthagorean + Least Perimeter Squared Squared Harmonic Mean ( + ) ( + ) Theorem 5 Ever generating function (; ) can e recovered from its corresonding weight function g () using (; ) = g = g (7) Proof B denition homogeneit g = ; = = ; = = (; ) ISBN

3 Recent Advances in Intelligent Control, Modelling Comutational Science the second formula follows smmetr The rocedure for nding a is to again set artial derivatives of the error function equal to zero then solve the resulting equations for a This is done in the net theorem for the general case The result is a general equation for the sloe in terms of,,, g () g 0 () which can e used to verif the secic formulas for in the rst aer with greater ease The formula for is called here the First Discreanc Formula ecause the left h side is the discreanc etween the generalized least-squares coefcient the ordinar least-squares coefcient given Theorem 6 The -intercet a the sloe of the generalized least-squares regression line satisf d d a = (8) n g () + ( ) o = 0 Proof Begin with the error function E = g () + ( ) + a + (9) Take the rst artial derivative with resect to a set it equal to zero E a = g () a + = 0 Solve for a otain a = Net, take the rst artial derivative of the error function with resect to set it equal to zero 0 = E = g 0 () + ( ) + a + +g () ( ) + a + Sustitute a = 0 = g 0 () + ( ) +g () ( ) Oserve that the same result is otained rst sustituting a = eliminating a from the error function Then one can solve the equation d n g () + ( ) o = 0 d for This is the simler comutation to erform Theorem 7 (First Discreanc Formula) The discreanc etween the generalized least-squares coef- cient the ordinar least-squares coefcient is given imlicitl = g 0 () g () +! (0) Proof Perform the differentiation indicated the revious formula Corollar 8 The general regression coefcient alwas has the same sign as Denote the ordinar least-squares coefcient OLS When is ositive > OLS when is negative < OLS Proof The aove equation for calculates the discreanc etween ordinar least squares sloe the least squares sloe ased on the function g Oserve that the eression + is alwas ositive since jj The function g () is alwas ositive g 0 () is alwas negative for ositive g 0 () is alwas ositive for negative Conclude that when > 0, the right h side is ositive therefore > Similarl, when is negative the right h side is negative < This imlies that sgn = sgn The First Discreanc Formula is useful for deriving the formulas for in the cases that were alread worked out, ut it is also rolematic in that it is an imlicit formula in In the net theorem an elicit formula for is given Theorem 9 (Second Discreanc Formula) An elicit formula for the discreanc is given = g () g 0 () s g () sgn g 0 () ( ) Proof Use the quadratic formula to solve for Choose the sign in front of the radical to e sgn () ISBN

4 Recent Advances in Intelligent Control, Modelling Comutational Science Lemma 0 The following inequalit is true for all for which the eression is dened g 0 () g () () Proof The quantit under the radical in the second discreanc formula must necessaril e nonnegative Therefore g () g 0 () g 0 () g () g 0 () g () It is shown now that g () can grow or deca at most eonentiall over an interval Theorem Let g () e dened over an interval [ 0 ; ] Then g () can grow or deca at most eonentiall over this interval More secicall where ke g () Ke (3) k = g ( 0 ) e 0 K = g ( 0 ) e 0 Proof Rewrite the revious inequalit as or g0 () g () d da ln g () Integrate all three sides of the inequalit over [ 0 ; ] otain 0 ln g () ln g ( 0) 0 Eonentiate all three sides, solve for g () otain the inequalit The Weight Function Relative Error It was seen that the error function for generalized least-squares regression is the roduct of the weight function g () the ordinar least-squares error function It follows that the weight function g () is the ratio of generalized least-squares error to ordinar least-squares error g () = E g E OLS (4) For several regressions the weight function g () is strictl less than In these cases E g < E OLS (5) which ma e a desirale feature for a regression to have If ordinar least-squares regression is viewed as the stard, then the relative error etween the two regression errors is given je OLS E g j = j g ()j (6) E OLS If the generalized least-squares regression is viewed as the stard, then the relative error is given je g E OLS j = E g g () (7) More generall, for an two regressions with weight functions g () g () the error relative to the g () regression is given je g E g j = E g g () g () (8) Multiling 00% ields the equivalent ercent errors The methods can e ranked according to the ercent error relative to ordinar least-squares regression, relative to geometric mean regression, or relative to an other generalized regression In this wa the fundamental role of the weight function in analzing generalized regressions is further underscored 3 The Indicative Function the Hessian Matri In order for the values of a to minimize the error function, the Hessian matri of second order artial derivatives evaluated at a must also e ositive denite The general Hessian matri is calculated in the net theorem For a given weight function g () a articular comination of g its rst second derivatives ISBN

5 Recent Advances in Intelligent Control, Modelling Comutational Science alwas occurs in the calculation of H det H It is denoted here G () It las a fundamental role in indicating whether the Hessian matri will e ositive denite It also las a fundamental role in determining the common form which all weight functions ossess Denition Dene the indicative function G () = g0 () g () g 00 () g 0 () (9) call the equation the indicative equation Theorem 3 The general Hessian matri can e written comactl as where " # H = g () det H + 4g() det H = 4g () Proof Begin with + G () (0) () n E = g () + ( ) o + a + take second order artial derivatives E g () a + = g () E g () a + Relace a with = g () + g 0 () a + otain E a = E a = g () The second derivative with resect to is given fg () ( ( ) + a + +g 0 () + ( ) o + a + = g 0 () ( ) + a + +g 00 () + ( ) + a + +g () + + g 0 () ( ) + a + Relace a with otain E = g () + + 4g 0 () +g 00 () ( ) + o = g () + + g 0 () g () + g 00 () g () +!) g() g 0 () Now use the First Discreanc Formula to relace + with otain E = g () + g + 0 () g 00 () g () g 0 () Finall det H = E aa E Ea = 4 (g ()) + g + 0 () g 00 () g () g 0 () 4 (g ()) = 4 (g ()) + g 0 () g () g 00 () g 0 () ISBN

6 Recent Advances in Intelligent Control, Modelling Comutational Science The net theorem is a simle corollar of the revious theorem However, it is called a theorem in order that the reader not miss its signicance It gives the simlest wa to check for ositive-deniteness use the indicative function Theorem 4 Suose g () g 0 () are not zero Then the Hessian matri is ositive denite the Hessian determinant is ositive if onl if G () > () An function G () satisfing this condition is admissile as an indicative function For eamle, an G () such that sgn G () = sgn is admissile as an indicative function, since sgn = sgn ensures that G () > 0 More generall, an G () of the form G () = () where () is a ositive function is admissile To aid the reader in the rocess of verifing the Hessian determinant formulas resented earlier, a tale of indicative functions is resented Case Weight Function g() OLS NA OLS Orthogonal + GMR 0 Indicative Function G() Pthagorean Least Perimeter 3sgn Squared Squared Harmonic sgn Mean ( + ) + Hrid Least Perimeter Hrid Harmonic Mean Hrid Pthagorean Hrid Orthogonal Eonentia l e( ) sgn sgn have the same indicative function Furthermore, several indicative functions differ from each other onl a multilicative constant To consolidate further generalize the aove tale, the inverse rolem of determining all weight functions corresonding to a given indicative function must e solved The solution to the inverse rolem reveals the common form that all the weight functions share Theorem 5 Let G () e an indicative function Then a general solution g () to the indicative equation is given g () = c + k R e where c k are aritrar constants Proof The indicative equation G () = g0 () g () is solved for g () Write Therefore R (3) G () d d g 00 () g 0 () G () = g0 () g 00 () g () g 0 () = d d ln jg ()j d d ln g 0 ()! = d (g ()) ln d jg 0 ()j ln! Z (g ()) jg 0 = ()j (g ()) g 0 () Z = K e G () d + C G () d where K = e C The resulting differential equation is now a Bernoulli equation g 0 Z () (g ()) = K e G () d g 0 () (g()) Su- To solve it, let u = g() so that u0 = stituting for u ields Z u 0 () = K e G () d There is clearl redundanc in this tale It is aarent, for eamle, that different weight functions can Z u () = k e Z G () d d + c ISBN

7 Recent Advances in Intelligent Control, Modelling Comutational Science where k = K Sustituting ack for g () ields the result The net theorem shows that linear cominations of indicative functions roduce valid weight functions Theorem 6 (Linear Cominations of Indicative Functions) Let G 0 (), G () G () e indicative functions with corresonding weight functions The tale resented net is a consolidation a generalization of the revious chart Ever regression is categorized using a common indicative function To ever general indicative function there is a corresonding class of weight functions Man of the regressions derived reviousl their weight functions are now seen to e instances of the same general weight function g fg0 g () = g fg g () = g fg g () = c 0 + k 0 R 0 () d ; c + k R () d c + k R () d Then The weight functions corresonding to multilication of G 0 () a constant are given g fg0 g () = c + k R ( 0 ()) d (4) The weight functions corresonding to a sum G () + G () are given g fg +G g () = c + k R () () d (5) 3 The weight functions corresonding to a difference G () G () are given g fg G g () = c + k R (6) () () d Indicative Function G() 0 3 sgn sgn + General Weight Function g() c c + k c + k c + k e( ) / + k ( + ) d c + k( + ) c + k( + ) / 7 sgn + + c + k( + ) (( ) + ) Secific Cases GMR c=0, k=, g( ) = Hrid Harmonic Mean c=, k=, g ( ) = /( + ) OLS c=0, k=, =, g ( ) = Orthogonal c=, k=, =, g ( ) = /( + ) Previous cases =0 Eonential c=0, k=, 0 < < 0 g( ) = e( ) Hrid Pthagorean c=0, k=, =3, g ( ) + = Pthagorean c=, k=, =4, g ( ) = + Hrid Orthogonal c=0, k=, =, g ( ) + = Squared Harmonic Mean c=0, k=, =, g ( ) = /( + ) Hrid Least Perimeter c=, k=, =, g( ) = + Least Perimeter Squared c=, k=, =3, g ( ) = ( + ) 4 The weight functions corresonding to a general linear comination G () + qg () are given g fg +qg g () = c + k R ( ()) ( ()) q d (7) With the aove theorems, an admissile function G () can now e used to construct a weighted ordinar least-squares regression rolem Linear cominations of reviousl known indicative functions can also e formed the resulting weight functions are more easil constructed As alwas, the value for which minimizes the error is determined solving the First Discreanc Formula for setting a = The tale reveals the hidden relationshis the underling unit ehind the disarate regressions resented reviousl It also oens u man more secic cases of weighted ordinar least-squares regression for future eloration Further generalizations additions to this tale are ossile as well The detailed construction of generalized least-squares rolems ased on other choices for G () is a suject for future work 3 Summar The derivation of least-squares regressions involves constructing the summation eression for the mean squared error etween the data the line, denoted ISBN

8 Recent Advances in Intelligent Control, Modelling Comutational Science here E In the stard derivation, E a E are set equal to zero, the equations are solved for minimizing solution (a; ) To check that the solution is actuall a minimum, the Hessian determinant must e comuted found to e ositive In the rst aer of this series, efcient derivations for a variet of generalized least-squares regressions were resented on a case--case asis The novelt of the derivations lied in their use of Ehrenerg's formula for the ordinar least-squares error, avoiding cumersome algeraic maniulations with summation smols The derivations also related the deviations of the data to the sloe of the line in order to etract a weight function g () from the error eression With the attern of derivation now clearl estalished, this aer generalizes the rocedure into a theor for comuting classifing an generalized least-squares regression In the theor, ever smmetric least-squares regression egins from a generating function denoted (; ) The generating function is a ositive, even, non-decreasing homogenous function of the deviations The deviations are then related to the sloe of the line a weight function g () = ; is etracted from the error eression In this wa it is shown that ever generalized smmetric least-squares rolem is equivalent to a weighted ordinar least-squares rolem, since the generalized error function is a roduct of g () Ehrenerg's ordinar least-squares error formula All cases of smmetric regression are classi- ed in terms of a secic generating function (; ) a corresonding weight function g () Even when a weight function g () does not stem from a smmetric least-squares regression, as was the case with the hrid smmetric regressions the eonential regressions, one can still solve the weighted ordinar least-squares rolem In all cases, setting E a = 0 E = 0 leads to an imlicit formula an elicit formula for the discreanc etween the ordinar least-squares sloe the generalized least-squares sloe, OLS These formulas for are called discreanc formulas The formula for a in all cases is given a = The general calculation of the Hessian matri determinant roduces a articular comination of g () its rst second derivatives This eression is called the indicative function denoted G () The indicative function streamlines the comutation of the Hessian matri determinant It susequentl also gives a simle wa to test whether a regression arising from a weight function g () has a minimizing solution check whether G () ( OLS ) > A tale of indicative functions G () for the secic regressions alread descried is resented Finall, the indicative equation (the differential equation eressing G () in terms of g ()) is solved, a general integral formula eressing g () in terms of G () is otained The integral formula for g () contains aritrar constants, so that man different regressions actuall have the same general weight function Linear cominations of indicative functions are shown to roduce valid weight functions A tale of indicative functions is resented with the corresonding integral weight functions worked out The tale reveals man disarate regressions elonging to the same general weight function class having the same indicative function In this wa, all smmetric least-squares regressions are categorized a generating function, a weight function, an indicative function All weighted ordinar least-squares regressions, of which smmetric regressions are a art, are categorized grouing them into classes with the same general indicative function the same general weight function References [] A S C Ehrenerg, Deriving the Least-Squares Regression Equation, The American Statistician, Vol 37, No 3 (Aug 983), 3 [] N Greene, Generalized Least-Squares Regressions I Efcient Derivations, Proceedings of the st International Conference on Comutational Science Engineering (CSE '3), Valencia, Sain, August 6-8, 03 [3] S B Martin, Less than the Least An Alternative Method of Least-Squares Linear Regression, Undergraduate Honors Thesis, Deartment of Mathematics, McMurr Universit, Ailene, Teas, 998 [4] P A Samuelson, A Note on Alternative Regressions, Econometrica, Vol 0, No (Jan 94) [5] R Taageera, Making Social Sciences More Scientic The Need for Predictive Models, Oford Universit Press, New York, 008 [6] E B Woolle, The Method of Minimized Areas as a Basis for Correlation Analsis, Econometrica, Vol 9, No, (Jan 94), 38-6 ISBN