Generalized Least-Squares Regressions III: Further Theory and Classication

Transcription

1 Generalized Least-Squares Regressions III: Further Theor Classication NATANIEL GREENE Department of Mathematics Computer Science Kingsborough Communit College, CUNY Oriental Boulevard, Brookln, NY UNITED STATES Abstract: This paper continues the work of this series with two results. The rst is an eponential equivalence theorem which states that ever generalized least-squares regression line can be generated b an equivalent eponential regression. It follows that ever generalized least-squares line has an effective normalized eponential parameter between which classies the line on the spectrum between ordinar least-squares the etremal line for a given set of data. The second result is a fundamental formula for the generalized least-squares slope: b = ( + )( = ) where is a measure of scatter is an increasing function of ranging from to. According to the formula it is the nature of generalized least-squares regression to incorporate the scatter of the data into the slope. The regression methods differ in how much the scatter should be weighted. The parameter measures this varies from one regression method to another. Ke Words: Least-squares, generalized least-squares, smmetric least-squares, weighted ordinar least-squares, orthogonal regression, geometric mean regression. Overview In the rst two papers of this series [, ], the known cases of smmetric least-squares regression, as well as a variet of least-squares regression methods which ma not have been known or full eplicated were derived b this author. The derivation of each method was made efcient through the use of Ehrenberg's formula for the ordinar least-squares error [] through the etraction of a weight function g(b) which characterizes the regression. For ever case of generalized least-squares, the error between the line the data was shown to be a product of the weight function g(b) Ehrenberg's error formula. The re-derivation of orthogonal regression was notable in showing that orthogonal regression minimizes the average harmonic mean of the square deviations, allowing it to also be categorized alternativel as harmonic mean regression. This work was then generalized into a theor for deriving, analzing, classifing all smmetric weighted least-squares regression methods. All smmetric least-squares regressions were categorized b a generating function (; ) that is positive, even, homogenous of degree two in, a weight function g(b) = (; =b), an indicative function G(b) = g (b)=g(b) g (b)=g (b). All weighted ordinar least-squares regressions, of which smmetric regressions are a part, were categorized b grouping them into classes with the same general indicative function G(b) the same general weight function g(b) = =(c + k R R ep( G(b)db)db). This paper continues the development of this theor. It is shown here that ever generalized leastsquares regression line can be generated b an equivalent eponential regression. All generalized leastsquares regression lines fan out from the mean point are bounded between the OLS j line the etremal eponential line. The normalized parameter in the weight function g (b) = ep ( p jbj), is then used to numericall classif the regression lines for a given set of data on the spectrum between the ordinar least-squares line the etremal line. A fundamental formula for the generalized leastsquares slope is then derived based on. It is clear from the formula that ever generalized least-squares method incorporates the scatter of the data into the slope. The degree to which the do so is measured b a parameter that is a function of also classies the lines on the spectrum between the ordinar leastsquares line the etremal line. ISBN:

2 The Equivalence of Generalized Regression Eponential Regression A variet of smmetric weighted least-squares regression methods were derived the various regression lines were observed empiricall to fan out between the OLS j line the OLS j line. Eponential regressions were also dened with weight function given b g (b) = ep ( p jbj) : Eponential regression lines fan out between the OLS j line the etremal line = a + b where a = b b = + sgn p. Eponential regressions were treated as just another interesting case of weighted ordinar least-squares regressions. Here their importance is made clear in that eponential regressions subsume all possible generalized leastsquares. Coefcient of Scatter Recall that measures the scatter of the data cloud awa from the ordinar least-squares regression line. It follows that the square root of this is also a measure of scatter. It is useful here to have a notation for this quantit for its sign to agree with. Denition Dene = sgn p () to be the coefcient of scatter satisfing + = : When the correlation coefcient is parametrized as cos then is parameterized as sgn sin. The notation makes the epressions in the formulas developed earlier simpler more more intuitive. In notation the slope of the etremal line can be rewritten as b = ( + ).. The Eponential Parameter Classies Non-eponential Regressions An eplicit formula for the eponential regression slope b was alread given [] is quoted in the net theorem. Theorem (Eponential Slope Formula) Let b be the slope of an eponentiall weighted least-squares regression with parameter p. Then s b = + sgn where p p p = = jj : p A () The eponential regression lines fan out from the mean point ; var continuousl from the ordinar least-squares j line with p = to the etremal line with p = p : The negative sign in front of the radical was chosen in order that the Hessian determinant be positive. In the second part of the series [] the Second Discrepanc Formula was derived. It is re-written here in parallel form in the net theorem. Theorem (Second Discrepanc Formula) The slope of a generalized least-squares regression line is given b b = g g (b) s g (b) A : g (b) () The similarit between the two formulas is striking. The two formulas become the same when one sets g (b) = p sgn b () g (b) where again sgn b = sgn. Denote the weight function satisfing this equation b g (b) = ep ( p jbj). It is now clear that ever least-squares regression line with arbitrar weight function g (b) is the solution to a corresponding eponential regression problem with weight function g (b) = ep ( p jbj) parameter p. Since p varies from problem to problem it is useful to have a normalized eponential regression parameter satisfing p = p where. Denition Dene = p p () to be the normalized eponential parameter write g (b) = ep ( p jbj) : (6) As varies between the weight function g (b) = ep ( p jbj) generates all the possible regression The net theorem gives the formula for the parameter p the normalized parameter once the slope is known. Theorem (Eponential Equivalence Theorem) Let b be the slope of a generalized least-squares regression line with associated weight function g (b). ISBN:

3 Then this line can be generated from an equivalent eponentiall weighted least-squares regression with weight function g (b) = ep ( p jbj) = ep ( p jbj) effective parameters given b b p = b + (7) = p : (8) Proof. Solve for p in terms of b in the eponential slope formula. Once a generalized regression line has been computed, one can alwas go back compute the effective parameters p or corresponding to the equivalent eponential regression line. Ever generalized regression line can now be assigned an effective value between. The net formula reveals the simple form which all generalized least-squares regression lines have. Theorem 6 (General Slope Intercept Formulas) Let b be the slope of a generalized least-squares regression line with effective normalized eponential parameter, then b = ( + ) (9) where are related b the equations = + p () The -intercept is = + : () a = b : () Proof. Begin with the eponential slope formula, substitute p = =( = ) simplif. This fundamental formula makes clear that ever generalized least-squares regression method seeks to incorporate the scatter of the data into the slope. The regression methods differ in how much the scatter should be weighted. The scatter parameter measures this varies from one regression method to another. Corollar 7 (Third Discrepanc Formula) generalized least-squares regression b For an b OLS = : () In words: the discrepanc between the generalized least-squares slope the ordinar least-squares slope is given b a weighted scatter term. Equivalentl, the generalized least-squares slope is the sum of the ordinar least-squares slope plus a weighted scatter term The functions = () = () are both increasing over [; ]. When = then = the line is the OLS j line. When = then = the line is the etremal line. The graph of as a function of is displaed, allowing one to visuall estimate the normalized eponential parameter corresponding to a particular value vice versa. It is clear from the graph that is alwas less than or equal to with equalit onl at the endpoints. λ Plot of λ as a function of γ 6 8 γ In the net table, quarter values the corresponding eact values for are calculated here as a reference. These reference values are used for comparison purposes in the numerical eamples below. γ λ In those cases where an eplicit formula for b was derived one can substitute write specic formulas for pertaining to that regression. This is done now for geometric mean regression OLS j regression since their parameters are compact epressions in. OLS GMR OLS ( ρ ) Etremal γ κ κρ λ κ κ + ρ ρ ISBN:

4 In all cases, including those cases where the slope was a solution to a cubic or higher-degree polnomial equation, the effective eponential parameter can be found numericall once the slope is known. This is done in the eamples in the net section. Numerical Eamples This section revisits the eamples eplored in the previous work [] taking into account the eponential equivalence theorem. In addition to placing smmetric, hbrid smmetric eponential regressions on the same tables, the corresponding numerical values for, are now computed for each regression line. Eample Si data values are given: (; 6), (; ), (; ), (; ), (; ), (; ). The reader can verif that = :97, = :9, = :, = :, = :778, = :986. The effective eponential parameter the corresponding scatter parameter are computed along with the equation of each line. The reader can verif in each case that b = ( + ) a = b. The graph shows the generalized regression lines together with the etremal line thereb displaing the region containing all possible generalized regression 6 Generalized Regressions Compared graph. Regression Tpe Etremal Eponentia l Eponentia l Eponentia l OLS Eponentia l Eponentia l Pthagorean Least Perimeter Squared GMR Squared Harmonic Orthogonal Eponentia l Hbrid Pthagorean Hbrid Least Perimeter Hbrid Harmonic Hbrid Orthogonal OLS = a + b Eponentia l Scatter Parameter γ Parameter λ = = = = = = = = = = = = = = = = = = In the last column, is again the fraction of the scatter that regression method contributes to the slope. It is seen, for eample, that OLS j contributes approimatel % of the scatter to the slope whereas GMR contributes % orthogonal regression contributes 9%. In this eample all the non-eponential methods contribute less than % of the scatter to the slope. Eample Ten data values are given: (; :), (; ), (:; :), (; ), (; ), (6:; ), (6:; ), (7; ), (9; ), (; :). The reader can verif that = :868, = :6, = :, = :, = :77, = :698. The effective eponential parameter the corresponding scatter parameter are again computed along with the equation of each line. The reader can verif in each case that b = ( + ) a = b. Again, in the graph, the generalized regression lines are plotted together with the etremal line thereb displaing the region containing all possible generalized regression.... Additional eponential regressions with = ; are included in the table. The eponential regressions with = 8 7 ; corresponding to = ; respectivel are also included here for comparison purposes. The are not shown in the ISBN:

5 6 - Generalized Regressions Compared ponential equivalence theorem, which states that an generalized least-squares regression line with arbitrar weight function g (b) can be generated b an equivalent eponentiall weighted least-squares problem. It follows that ever generalized least-squares line can be assigned an effective eponential parameter which classies it on the spectrum between the ordinar least-squares line ( = ) the etremal line ( = ) : The second result is a fundamental formula for the generalized least-squares slope: b = ( + ) + p where =. The formula eplains what all generalized least-squares methods do how the differ from one another. Ever generalized least-squares method incorporates the scatter of the data into the slope. The degree to which it does this is measured b the parameter, again classifing ever generalized regression line on the spectrum between the ordinar least-squares line the etremal line. Additional eponential regressions are included in the table for comparison purposes. Regression Tpe Etremal Eponentia l OLS Eponentia l Pthagorean Eponentia l Least Perimeter Squared GMR Eponentia l Eponentia l Hbrid Pthagorean Squared Harmonic Hbrid Least Perimeter Eponentia l Orthogonal Hbrid Harmonic Hbrid Orthogonal OLS = a + b Eponentia l Scatter Parameter γ Parameter λ = = = = = = = = = = = = = = = = = =. + 6 Here OLS j contributes approimatel 68% of the scatter to the slope whereas GMR contributes % orthogonal regression contributes %. In this eample all the non-eponential methods contribute less than 7% of the scatter to the slope. Summar Two fundamental results on generalized least-squares regression have been presented. The rst is the e- References [] S. C. Ehrenberg, Deriving the Least-Squares Regression Equation, The American Statistician, Vol. 7, No. (Aug. 98), p.. [] N. Greene, Generalized Least-Squares Regressions I: Efcient Derivations, in: Recent Advances in Intelligent Control, Modelling Computational Science, Proceedings of the st International Conference on Computational Science Engineering (CSE'), Valencia, Spain, August 6-8,, pp. -8. [] N. Greene, Generalized Least-Squares Regressions II: Theor Classication, in: Recent Advances in Intelligent Control, Modelling Computational Science, Proceedings of the st International Conference on Computational Science Engineering (CSE'), Valencia, Spain, August 6-8,, pp [] R. Taagepera, Making Social Sciences More Scientic: The Need for Predictive Models, Oford Universit Press, New York, 8. In Part II of this series [], page 9, Denition, Part (iv) should read as follows: Non-decreasing in for : For differentiable this means when On page 6, the second line of the proof of Theorem should read d ln g (b). In the chart on page 6, db Row should read g (b) = p + =b. ISBN: