Generalized Least-Squares Regressions I: Efcient Derivations

Transcription

1 Generlzed Lest-Squres Regressons I: Efcent Dervtons NATANIEL GREENE Deprtment of Mthemtcs nd Computer Scence Kngsough Communt College, CUNY 00 Orentl Boulevrd, Brookln, NY 35 UNITED STATES Astrct: Ordnr lest-squres regresson suffers from fundmentl lck of smmetr: the regresson lne of gven nd the regresson lne of gven re not nverses of ech other. Alterntve smmetrc regresson methods hve een developed to ddress ths concern, notl: thogonl regresson nd geometrc men regresson. Ths pper presents n detl vret of lest-squres regresson methods whch m not hve een known full eplcted. The dervton of ech method s mde efcent through the use of Ehrenerg's fmul f the dnr lest-squres err nd through the etrcton of weght functon g() whch chrcterzes the regresson. F ever cse of generlzed lest-squres, the err etween the lne nd the dt s shown to e product of the weght functon g() nd Ehrenerg's err fmul. Ke Wds: Lest-squres, smmetrc lest-squres, weghted dnr lest-squres, thogonl regresson, geometrc men regresson. Ordnr nd Alterntve Lest- Squres Regresson. Overvew Ordnr lest-squres regresson dtes ck to the wk of Legendre nd Guss nd t s stll the most commonl used method f ndng lne of est t through gven set of dt ponts. The method mnmzes the verge vertcl dstnce etween the dt nd the lne. Its eclusve menton n elementr sttstcs tets ensures tht t s the onl method tht mn users of regresson know. Despte ts wdespred use, dnr lest-squres regresson suffers from some fundmentl lmttons. One mj lmtton s the lck of smmetr. Ordnr lest-squres egns wth decson of the prt of the user tht one of the vrles s the ndependent vrle (cll t ) nd the other vrle s the dependent vrle (cll t ). The ndependent vrle s ssumed to e perfectl known, wheres the dependent vrle hs err. In ths cse, one computes the dnr lest-squres regresson lne f gven (OLS j). The equton s = + where = nd =. Whle t s vld to use ths regresson lne to predct the -vlue whch cresponds to gven -vlue, t s not vld to use ths lne to predct the -vlue whch cresponds to gven -vlue. When s known ectl nd s suject to err one must compute the dnr lestsqures regresson lne of gven (OLS j). Onl from ths regresson lne s t vld to predct the - vlue crespondng to gven -vlue: Ths prolem of there eng two regresson lnes f gven set of dt cn e restted s follows: the OLS j lne nd the OLS j lne re not nverses of ech other. Indeed, the nverse fm of the OLS j lne s gven = where 0 = 0 nd 0 =. A me generl nd roust ssumpton, s to ssume tht oth nd vrles hve err, nd to llow f the posslt of functonl nterdependence etween the vrles. In ths cse one cn construct smmetrc lest-squres regressons n whch the errs n oth the nd vrles nd the lne re mnmzed. Two known emples of smmetrc regressons re thogonl regresson nd geometrc men regresson (GMR). Orthogonl regresson mnmzes the verge perpendculr dgonl err etween the dt ponts nd the lne nd GMR mnmzes the verge re of the trngles fmed the dt ponts nd the lne. In contrst, OLS j mnmzes the verge vertcl err etween the dt nd the lne nd OLS j mnmzes the verge hzontl err etween the dt nd the lne. Another lmtton of stndrd dnr lestsqures s the lck of trnstvt: the OLS regresson lne f zj s not the composton of the OLS regresson lne f zj wth the OLS lne f j. Orthog- ISBN:

2 onl regresson, whle smmetrc, s not trnstve. GMR s oth smmetrc nd trnstve whch mkes t n del choce f roust modelng. GMR regresson s lso scle nvrnt, whch mens tht multplng the vlues of the dt constnt fct nd regressng s equvlent to multplng vrle of the gnl regresson lne tht sme constnt. The fmul f the GMR regresson lne s lso prtculrl smple: = + where = nd = sgn : All these ssues re dscussed n greter detl n Tgeper [8]. Despte the gret dvntges of GMR f roust modellng, the gol of ths pper nd the ppers whch follow s to consder other possle generlzed lestsqures regressons s well, nd to develop the f dervng nd clssfng these methods. Ths pper hs severl specc gols: to re-derve the known cses of generlzed lest-squres n mnner tht s s ref nd efcent s possle; to eple the dervton of severl new cses of smmetrc regresson; to see n the process tht ever smmetrc regresson s ctull weghted dnr lestsqures regresson; nd to eple the dervton of severl weghted dnr lest-squres regressons. The choce of specc cses wked out here s necesst dosncrtc nd cnnot e comprehensve. Ths prolem s rected n the pper to follow [3], where ths wk s generlzed.. Ordnr Lest-Squres Gven : Vertcl Regresson Ordnr lest-squres regresson seeks coefcents nd whch mnmze the nverge squre o vertcl devton etween the dt ( ; ) N = nd the lne = +. The vertcl devton etween dt pont ( ; ) nd the lne = + s gven n solute vlue j ( + )j. F convenence ths s wrtten s j + j. The verge squre vertcl devton s therefe gven E = N ( + ) : () = F nottonl smplct the err s represented usng E nd t s cler tht one cn lso wrte E = ". To mnmze ths err functon, the usul procedure s to set prtl dervtves E = 0 nd E = 0 nd solve f nd : The result s nd = () = (3) where nd re the men vlues of nd gven = (4) N nd = N = (5) = nd nd re stndrd devtons gven nd v u = t N v u = t N ( ) (6) = = : (7) Fnll, s the coefcent of crelton gven = N ( ) = ( ) : (8) = Note tht populton prmeter notton,,,, nd N s used throughout ths wk, ut the sme results hold when the re replced wth smple sttstcs,, s, s, r nd n. One must verf tht these vlues f nd mnmze the err functon E (; ). The test s to check tht the Hessn mtr of second der prtl dervtves E E H = (9) E E s postve dente. Ths occurs when ts rst entr nd ts determnnt re oth postve. Snce H = + (0) nd det H = 4, these vlues f nd mnmze the err..3 Alterntve Dervton: Ehrenerg's Fmul There s n lterntve dervton of dnr lestsqures regresson descred n Ehrenerg [] whch does not requre clculus. One rst epresses E n terms of, ;, nd. Then one nds the vlues of nd whch mnmzes the err usng elementr lger. Ths smple pproch hs pprentl een overlooked eposts of lest-squres. ISBN:

3 Theem (Ehrenerg's fmul) E = + ( ) + + Proof. E = N = N ( + ) = = ( ) + + () = = + ( ) + + The mnmzng vlues f nd cuse the lst two terms to equl zero. The err s then governed. Recll tht quntt mesures the sctter of the dt cloud w from the lne of est t. The quntt s clled the coefcent of determnton nd t mesures the strength of the lner reltonshp. When =, = 0 nd there s no sctter ecuse ll the dt fll long the lne = +. When = 0, =, the dt re uncrelted nd there s mmum sctter. In ths cse = 0 nd the regresson lne s gven =. In the (; ) plne the level curves of E re concentrc ellpses wth es rotted nd trnslted w from the gn nd the err functon tself s n ellptcl prolod n (; ; E) spce. Ehrenerg's fmul wll pl ke role throughout ths pper, gretl smplfng the dervtons of ll susequent regresson fmuls..4 Ordnr Lest-Squres Gven : Hzontl Regresson In hzontl regresson coefcents nd re sought whch mnmze the verge n squre ohzontl devton etween the dt ( ; ) N = nd the lne = +, whch cn e wrtten equvlentl s =. The hzontl devton etween dt pont ( ; ) nd the lne s gven n solute vlue +. The verge hzontl devton s therefe gven E = N = + () One could mnmze ths err functon drectl. However, t s me frutful to utlze the relton + = ( + ) (3) whch s lgercll strghtfwrd. Geometrcll, t mens tht the vertcl devton etween dt vlue nd lne nd the hzontl devton etween dt vlue nd the lne re constrned the slope of the lne ccdng to the relton = Vertcl devton Hzontl devton : (4) Ths oservton s used n Woolle's dervton of geometrc men regresson [0]. The relton s used repetedl n ths pper to smplf ever dervton ecuse t lws llows f the etrcton of weght functon from the err epresson. The result s n equvlent lest-squres err functon E = N ( + ) (5) = E = g () + ( ) + + (6) g () = : (7) Settng E = 0 nd E = 0 nd solvng f nd elds the desred epressons. The smplest w to do ths n prctce s to rst susttute = (8) nd elmnte from the err functon. Then solve f n d d + ( ) = 0: (9) The result s = : (0) The Hessn mtr H = s postve dente snce det H = () () ISBN:

4 nd hs the sme sgn s. In fct, susttute f nd otn det H = (3) Therefe these vlues f nd mnmze the err..5 Orthogonl Lest-Squres: Dgonl Regresson An lterntve known regresson method whch tkes nto ccount oth nd devtons mplctl s clled thogonl regresson. Other nmes nclude perpendculr regresson Demng regresson [4,5]. In ths method one nds nd whch mnmze the verge squre dgonl devton etween the dt nd the lne. Recll tht the dstnce d etween pont ( ; ) nd the lne = + s gven 4 d = j + j p : (4) + Thus the verge squre dgonl devton s gven E = + N ( + ) (5) = E = g () + ( ) + + (6) g () = + : (7) Set E = 0 nd E = 0 nd solve f nd to otn the desred vlues of nd : The smplest w to do ths s to gn susttute = nto the err functon nd then solve d d + + ( ) = 0 (8) f. The result s qudrtc equton n : + = 0. (9) Solvng f elds two eplct solutons when 6= 0: = q + 4 : (30) Snce + =, one vlue of the slope s lws postve nd one vlue s lws negtve. The Hessn mtr H = + s postve dente when det H = + (3) 4 ( + ) > 0 (3) whch occurs when hs the sme sgn s. Therefe n der to mnmze the err functon, lws choose the soluton wth the sme sgn s. When = 0 t s nturl to dene = 0, snce lm 0 = 0. Ths s lso consstent wth settng = 0 n the qudrtc equton. F n rght trngle wth sdes A; B nd C, there s the Recprocl Pthgen Theem [,6] whch s not well-known. It sttes tht A + B = L (33) where L s the lttude to the hpotenuse C. Ths s equvlent to the sttement tht the lttude squred s hlf the hrmonc men of the squres of the two shtest sdes. Smolcll, L = H A ; B (34) where H (; ) = + s the hrmonc men of nd. The ove fmul f the dstnce etween pont nd lne s now derved usng ths result. Oserve tht d = H ( + ) ; + = ( + ) + ( + ) + + : (35) Replcng + wth ( + ) elds the stndrd epresson d = (+ ) : In ths w, + thogonl lest-squres cn e understood s mnmzng the verge hrmonc men of the squre devtons n nd. Although the termnolog s not used, thogonl regresson could e rghtfull clled hrmonc men regresson (HMR) n prllel to the method of geometrc men regresson (GMR), whch s dscussed net. ISBN:

5 .6 Geometrc Men Regresson: Lest Asolute Ares The method of geometrc men regresson (GMR) [4,7,8,0], s rgul the most roust of ll smmetrc lest-squres regressons n tht t hs the smplest fmuls f the coefcents nd, the regresson remns vld f chnges n unts n the nd vrles, one cn compose two me regresson lnes nd otn vld result. The method mnmzes the verge solute product of the devtons n nd : Ths s the sme s the geometrc men of the squre devtons n nd. Oserve tht d = G ( + ) ; + s = ( + ) + = ( + ) + (36) where G (; ) = p s the geometrc men of nd. Wrte E = N ( + ) + : = (37) Replce + wth ( + ) nd otn lest-squres err functon E = jj N ( + ) (38) = E = g () + ( ) + + Susttute = (39) g () = jj : (40) nd solve d + ( ) = 0 d jj f. The result s = (4) The Hessn mtr H = jj s postve dente ecuse det H = 4 + = 44 (4) (43) s lws postve. Choose the sgn of to gree wth the sgn of nd wrte = sgn : (44) It s nturl to dene = 0 when = 0. Ths s lso consstent wth the use of the sgn functon n the fmul, snce sgn 0 = 0. Alterntve Smmetrc Lest- Squres Regressons In ths secton lterntve methods of lest-squres regresson re epled whch m not hve een known full eplcted. The dervton f ech method uses the sme stremlned steps tht were emploed to derve the known methods. Ths s done wth n ee towrd the net pper n ths seres, where the dervton s generlzed nd ll possle smmetrc lest-squres regressons re nlzed nd clssed.. Pthgen Regresson An lterntve method to thogonl regresson nd GMR clled Pthgen lest-squres s now nvestgted. The method s smmetrc n tht t mnmzes the verge squre devton n oth nd smultneousl. The err functon s gven ( + ) + + : E = N = (45) The term Pthgen s pproprte ecuse the method mnmzes the verge squre hpotenuse of the trngles fmed the dt ponts nd the lne. One cn crr the usul mnmzton procedure on ths epresson gn replce + ( + ) nd wrte E = + N wth ( + ) (46) = ISBN:

6 E = g () + ( ) + + (47) g () = + : (48) As efe, susttute = nto the err functon nd then solve d + d + ( ) = 0 (49) f. The result s qurtc equton n : = 0: (50) F prctcl purposes, ths qurtc polnoml must e solved numercll. It s now proved tht there s lws unque rel soluton to ths equton wth the sme sgn s whch mnmzes the err functon. The dscrmnnt f monc qurtc polnoml 4 +B 3 +C +D+E s gven [9] = B C D 4B C 3 E 4B 3 D 3 +8B 3 CDE 7B 4 E 4C 3 D +6C 4 E + 8BCD 3 80BC DE 6B D E + 44B CE 7D 4 +44CD E 8C E 9BDE +56E 3 : (5) When < 0 there re lws two rel solutons nd two comple conjugte solutons. In our cse = (5) whch s less thn zero f jj, the nturl rnge of. Ths mples tht there re ectl two rel solutons. Descrtes' Rule of Sgns cn e used to further deduce the nture of the solutons. Descrtes' Rule of Sgns sttes tht f polnoml f () n stndrd fm, the numer of postve roots equls the numer of chnges n sgn etween consecutve nonzero coefcents, t s less thn ths numer multple of two. The numer of negtve roots of f equls the numer of chnges n sgn etween consecutve nonzero coefcents of f ( ), t s less thn ths numer multple of two. F > 0, there re three chnges n sgn n f (), ut snce there re onl two rel solutons, the numer of postve solutons must dffer two, nd so there s one nd onl one postve rel soluton. The other rel soluton s negtve. Smlrl f < 0, there s one chnge n sgn n f ( ), nd so there s ectl one negtve rel soluton. The other rel soluton s postve. It s shown now tht the rel soluton to the qurtc equton s lws greter thn f > 0 nd less thn f < 0. Re-epnd the polnoml n the vrle q = nd cll the resultng polnoml h (q). The vrle q mesures the dscrepnc etween the Pthgen slope nd the dnr lestsqures slope. The result s the equvlent equton 0 = q q q q : (53) Accdng to Descrtes' Rule of Sgns, when > 0 there s one chnge n sgn n h (q)nd therefe one postve soluton. When < 0 there s one chnge n sgn n h ( q) nd therefe one negtve soluton. Therefe there s one postve soluton f tht s greter thn when s postve nd one negtve soluton f tht s less thn when s negtve. The Hessn mtr H = det H + 5 (54) 4 + s postve dente snce det H = = + ; (55) s lws postve f these vlues of. Therefe to perfm Pthgen regresson, the user solves the qurtc polnoml ove numercll nd chooses the one nd onl rel soluton f wth the sme sgn s. Ths vlue f s lws greter n mgntude thn OLS nd t mnmzes the err functon long wth the crespondng vlue f. ISBN:

7 . Lest Permeter Squred Regresson Consder mnmzng the verge squred permeter of the trngles fmed the dt vlues nd the regresson lne. The err functon s E = j + j + N + = (56) nd the resultng method s clled lest permeter squred regresson. Replce gn + wth ( + ) nd otn E = + jj N ( + ) (57) = E = g () + ( ) + + g () = (58) + : (59) jj Susttute = nto the err functon nd then solve ( d + d jj + ( ) ) = 0 (60) f. The result s cuc equton n 3 + sgn sgn = 0: (6) Denote ths polnoml f (). It s now proved tht there s lws unque rel soluton to ths equton wth the sme sgn s. Accdng to Descrtes' Rule of Sgns there re three chnges n sgn n f () when > 0 mplng ether three postve solutons one postve soluton. There re lso three chnges n sgn n f ( ) when < 0 mplng ether three negtve solutons one negtve soluton. However, the dscrmnnt f monc cuc polnoml 3 + B + C + D s gven [9] = B C 4C 3 4B 3 D + 8BCD D : (6) When s negtve the cuc hs one rel soluton nd two comple conjugte solutons. When s postve there re three rel solutons. In our cse = (sgn ) (sgn ) (63) nd t s s gurnteed to e negtve when sgn = sgn. Therefe there s lws unque rel soluton to the polnoml equton wth the sme sgn s. The Hessn mtr H = + jj " s postve dente snce det H = 4 + jj 3 sgn + det H 4(+=jj) jj + # (64) 9 = > 0: (65) ; The polnoml equton cn e wrtten unmguousl susttutng sgn = sgn nd (sgn ) = jj nto the coefcent epressons, otnng 3 + jj sgn = 0: (66) In ths fm, the user solves the cuc polnoml f the one nd onl rel soluton. Ths soluton s gurnteed to hve the sme sgn s, to e greter n mgntude thn OLS nd to mnmze the err functon. It s nturl to dene = 0 when = 0 whch s consstent wth the sgn functon whch ppers n the polnoml..3 Squred Hrmonc Men Regresson It ws seen tht thogonl lest-squres cn e thought of s mnmzng the verge hrmonc men of the squre devtons n nd. Consder nsted mnmzng the verge squre of the hrmonc mens of the solute devtons n nd. Tht s mnmze E = N = Replce + E = j + j + j + j + + wth ( + ( + jj) N : (67) ) nd otn ( + ) (68) = ISBN:

8 E = g () + ( ) + + g () = (69) ( + jj) : (70) Susttute = nto the err functon nd then solve ( d ) + ( ) d ( + jj) = 0 (7) f. The result s the followng fmul f : = sgn + jj : (7) jj + The Hessn mtr H = ( + jj) s postve dente snce det H = " 4 4 det H (+jj) jj # (73) ( + jj) 5 (74) s lws greter thn zero. Therefe these fmuls f nd mnmze the err. 3 Weghted Ordnr Lest-Squres Regressons It s pprent from the prevous cses tht smmetrc lest-squres prolem cn e wrtten s weghted dnr lest-squres prolem where weght functon multples the dnr lest-squres err. Ths secton eples some weghted dnr lest-squres regresson prolems chosen from relted smmetrc lest solute devton regresson prolem. These prolems re lso referred to here s hrd smmetrc lest-squres prolems. 3. Hrd Lest Permeter Regresson An lterntve to lest permeter squred regresson s to smpl mnmze the verge permeter of the trngles fmed the dt vlues nd the regresson lne The err functon s then D = N j + j + + = Replce gn + otn D = + jj N wth ( + : (75) ) nd j + j : (76) = Ths err functon cn certnl e mnmzed, however t s not prolem n lest-squres ut n lest solute devton. Consder nsted hrd smmetrc lest-squres prolem wth n err functon tht comnes the weght functon from the lest solute devton prolem nd the dnr lest-squres err. The functon s E = + jj N ( + ) = E = g () + ( ) + + (77) g () = + jj : (78) Cll the mnmzton of ths err hrd lestpermeter regresson. Ths s generlzed lestsqures prolem whch cn e solved usng the purel nltc methods used so fr. Elmnte s usul nd solve d + d jj + ( ) = 0: Ths results n cuc equton (fter choosng sgn = sgn ) 3 + sgn ( jj ) sgn = 0: (79) It s now proved usng Descrtes' Rule of Sgns tht there s lws one postve rel soluton to ths equton when s postve nd one negtve rel soluton to ths equton when s negtve. Let f () denote the polnoml. When s postve nd ( jj ) s postve, there s one chnge n sgn nd therefe one postve soluton. When ISBN:

9 s postve nd ( jj ) s negtve, there s gn one chnge n sgn nd therefe one postve soluton. When s negtve nd ( jj ) s postve, there s one chnge n sgn n f ( ) nd therefe one negtve soluton. When s negtve nd ( jj ) s negtve, there s one chnge n sgn n f ( ) nd therefe one negtve soluton. The Hessn mtr H = + " # det H jj + (80) 4(+=jj) s postve dente snce 8 det H = 4 + < jj : + sgn jj + (8) s lws greter thn zero. Therefe these fmuls f nd mnmze the err. 3. Hrd Pthgen Regresson Ths method reconsders Pthgen regresson ut nsted of mnmzng the verge squre hpotenuse of the trngles from the dt pont nd lne, t mnmzes the verge hpotenuse. The crespondng err functon s D = N s ( + ) + = Replce + D = r wth ( + + N + 9 = ; (8) ) nd otn j + j : (83) = Ths s nother generlzed lest solute devton prolem whch cn e solved usng lgthms f lest solute devton. The nterest here however, s n generlzed lest-squres prolems snce the re menle to nltc solutons. Consder nsted hrd smmetrc lest-squres prolem wth err functon r E = + ( + ) : (84) N = Replce wth Ehrenerg's fmul nd wrte E = g () + ( ) + + (85) r g () = + : (86) Cll the mnmzton of ths err functon hrd Pthgen regresson. Elmnte s usul nd solve "r d + d + ( ) # = 0: (87) The result s qurtc polnoml = 0: (88) Denote ths polnoml f (). Accdng to Descrtes' Rule of Sgns there re three chnges n sgn n f () when > 0 mplng ether three postve solutons one postve soluton. There re lso three chnges n sgn n f ( ) when < 0 mplng ether three negtve solutons one negtve soluton. However, re-epnd ths polnoml n the vrle q = nd cll the resultng polnoml h (q). The vrle q mesures the dscrepnc etween the hrd Pthgen slope nd the dnr lest-squres slope. The result s the equvlent equton 0 = q q q q : (89) Accdng to Descrtes' Rule of Sgns, when > 0 there s one chnge n sgn n h (q) nd therefe one postve soluton. When < 0 there s one chnge n sgn n h ( q) nd therefe one negtve soluton. Therefe there s one postve soluton f tht s greter thn when s postve nd one negtve soluton f tht s less thn when s negtve. The Hessn mtr " # H = q + det H + (90) 4 + s gurnteed to e postve dente snce 8 < det H = : = ; > 0 (9) ISBN:

10 f these vlues of. Therefe to perfm hrd Pthgen regresson, the user solves the qurtc polnoml ove numercll nd chooses the unque rel soluton wth the sme sgn s tht s greter n mgntude thn OLS. Ths vlue f nd the crespondng vlue f mnmze the err. 3.3 Hrd Orthogonl Regresson Consder gn thogonl regresson ut nsted of mnmzng the verge squre dgonl dstnce etween the dt ponts nd lne, mnmze the verge dgonl dstnce. Snce the dstnce d etween pont ( ; ) nd the lne = + s gven d = j + j p : + Thus the verge solute dgonl devton s gven D = p + N j + j : (9) = Ths s nother generlzed lest solute devton prolem whch cn e solved usng lgthms f lest solute devton. Consder here nsted hrd err functon E = p + N ( + ) (93) = E = g () + ( ) + + g () = (94) p + : (95) Cll the mnmzton of ths err functon hrd thogonl regresson. Elmnte s usul nd solve d p d + + ( ) = 0: (96) The result s cuc polnoml equton 3 + = 0: (97) Let f () denote the polnoml. Accdng to Descrtes' Rule of Sgns, when > 0 nd > 0, f () hs one chnge n sgn nd there s one postve soluton. When > 0 nd < 0, f () hs one chnge of sgn nd there s one postve soluton. When < 0 nd > 0, f ( ) hs one chnge of sgn nd there s one negtve soluton. Fnll when < 0 nd < 0, f ( ) hs one chnge of sgn nd there s one negtve soluton. The Hessn mtr " # H = p (98) + s postve dente snce det H = 8 < 4 ( + ) : det H 4(+ ) = ; > 0: 3.4 Hrd Hrmonc Men Regresson The smplest regresson nvolvng hrmonc mens of solute devtons n nd s to mnmze D = N = N = = j + j + j + j + + H j + j ; + (00) (99) Therefe these fmuls f nd mnmze the err. where H denotes the hrmonc men s efe. Replcng + wth ( + ) elds D = ( + jj) N j + j : (0) = Now consder here nsted the hrd err functon E = ( + jj) N ( + ) (0) = E = g () + ( ) + + (03) g () = ( + jj) : (04) ISBN:

11 Cll the mnmzton of ths err functon hrd hrmonc men regresson. Elmnte s usul nd solve d d ( + jj) + ( ) = 0: (05) The result s (fter lettng sgn = sgn ) 0 s = + + jj + ( ) A : The Hessn mtr H = ( + jj) s postve dente snce det H = + (06) (07) 4 > 0: (08) ( + jj) Therefe these fmuls f nd mnmze the err. 3.5 Eponentll-Weghted Ordnr Lest-Squres Regresson An nterestng cse of weghted dnr lest-squres s to consder n eponentl weght functon nd mnmze the err functon g () = e pjj (09) where sgn = sgn s usul. Ths equton cn rewrtten s s p = ( ) p : (4) Snce the epresson nsde the rdcl cnnot e negtve, ths fces restrcton on the dmssle vlues f p, nmel: 0 p p 0 (5) where The Hessn mtr H = e pjj " p 0 = p : (6) p s postve dente snce det H = 4 e pjj p = 4 e pjj s ( ) # + (7) p (8) whch s dened nd postve f 0 p < p 0 : When p = 0 the regresson degenertes nto dnr lest-squres. When p = p 0 the coefcent fmuls produce the etreml lne = where 0 = 0 nd E = e pjj N ( + ) (0) = E = e pjj + ( ) + + : () 0 = + sgn p : (9) However, det H = 0 when p = p 0 so tht the etreml lne's coefcents do not mnmze the err functon. Nevertheless, t s useful lne to compute snce ll eponentl regresson lnes re ounded etween the OLS j lne nd the etreml lne. f sutle vlues of p. solve n e pjj d d The result s = + sgn p Elmnte s usul nd + ( ) o = 0: s ( ) () p A (3) 4 Numercl Emples In the emples whch follow, ll the dfferent regresson lnes re plotted smultneousl f the sme set of dt. All generlzed lest-squres lnes = + pss through the men pont ; snce f ll the lnes =. The outermost regresson lnes re the dnr lest-squres j nd j lnes. In ll cses the smmetrc nd hrd smmetrc lnes fn out from the men pont nd occup the spce etween the dnr lest-squres lnes. ISBN:

12 Emple S equspced dt vlues re consdered: (0; 6), (; 4), (; 3), (3; 4), (4; ), (5; ). The dt n ths emple re tken from Mrtn's stud of thogonl regresson [4] where thogonl regresson ws compred wth OLS j regresson. Here ll regresson lnes re compred smultneousl Generlzed Regressons Compred Regresson Tpe OLS Pthgen Lest Permeter Squred GMR Squred Hrmonc Men Orthogonl Hrd Pthgen Hrd Lest Permeter Hrd Hrmonc Men Hrd Orthogonl OLS Equton = + = = = = = = = = = = = The second grph s the sme s the rst wth the regon round the -ntercepts mgned Generlzed Regressons Compred From the grphs nd the equtons t s evdent tht the smmetrc regressons nd the hrd smmetrc regressons re nded n seprte groups. The smmetrc regressons re closer to the OLS j lne nd the hrd smmetrc regressons re closer to the OLS j lne. The eponentll-weghted regressons re treted here seprtel. In ths emple the dmssle vlues f p re: 0 p p 0 where p 0 = :6584. Recll tht p = 0 reduces to OLS j regresson nd p = p 0 s the etreml cse. When p = p 0 the equton s = 5:78 0:95793 nd when p = p 0 the etreml lne s = 6:466 :333: All the regresson lnes re ounded etween the OLS j lne nd the etreml lne The regresson lnes re lsted n der of decresng -ntercept whch s how the pper n the grphs. Emple Ten dt vlues re consdered: (0; 0:5), (; 0), (:5; :5), (3; ), (5; 4), (6:5; 4), (6:5; ), (7; 3), (9; 5), (0; 4:5). Lke efe the re plotted n sctter plot nd ll the smmetrc nd hrd smmetrc regresson lnes re supermposed. ISBN:

13 5 Generlzed Regressons Compred Regresson Tpe OLS Hrd Orthogonl Hrd Hrmonc Men Equton = + = = = Orthogonl Hrd Lest Permeter Squred Hrmonc Men Hrd Pthgen GMR Lest Permeter Squred Pthgen OLS = = = = = = = = The net grph s mgncton of the prevous grph round the -ntercepts Generlzed Regressons Compred In ths emple, the grphs nd the tle lso suggest tendenc f smmetrc nd hrd smmetrc regresson lnes to remn seprte, however n ths cse there s some overlp. Here the dmssle prmeter vlues f eponentll-weghted regressons re: 0 p p 0, where p 0 = 3:393. Agn p = 0 cresponds to OLS j. When p = p 0 the regresson lne s = 0: :503. When p = p 0 the regresson lne s etreml nd ts equton s = : :749. All the lnes re ounded etween the OLS j lne nd the etreml lne. In these emples the crelton coefcent s close n solute vlue to nd so s lso close n solute vlue to. As result the OLS j nd j lnes re vsull ner ech other nd oth lnes re seen to pss through the dt. However, whenever the crelton coefcent s smll, the OLS j lne wll dverge notcel from the other lnes nd from the dt s well. Ths s not shown here The equtons f the lnes re gn lsted n der of decresng -ntercept, s the pper on the grphs. 5 Summr The dervton of lest-squres regressons nvolves constructng the summton epresson f the men squred err etween the dt nd the lne, denoted here E. In the stndrd dervton, E nd E re set equl to zero, nd the equtons re solved f mnmzng soluton (; ) : To check tht the soluton s ctull mnmum, the Hessn determnnt must e computed nd found to e postve. The pproch of ths pper hs severl notle fetures whch mde the dervtons of generlzed ISBN:

14 lest-squres regressons s efcent nd uncomplcted s possle. Frst, ths pper utlzes hgh-level fmul f the dnr lest-squres err E due to Ehrenerg whch s lred epressed n terms of,,, nd. Ths fmul s not wdel-known ut t deserves to e. The fmul llows f n elementr dervton of dnr lest-squres nd t llows ll the generlzed lest-squres methods descred here to e derved wthout complcted summton mnpultons. In ll cses, the computtons f E nd E ecome clculus-level prolems. The clculton of the Hessn mtr nd determnnt ecomes trctle now n ll cses s well. A second notle feture n ll the dervtons s the use of relton epressng the devton etween dt pont nd the lne wth respect to n terms of the devton of the dt pont wth respect to. It s used here n ever dervton to etrct weght functon g () from the err epresson E. In ths w, the generlzed lest-squres err s lws product of the weght functon nd Ehrenerg's fmul f dnr lest-squres err. Wth the pttern of dervton now smpled nd stremlned n the known cses, we were le to eple vret of new generlzed lest-squres methods n the sme efcent mnner. Whenever possle, eplct fmuls f the new regresson coefcents nd were determned. In those cses where the slope ws soluton to cuc qurtc equton, Descrtes' Rule of Sgns nd sometmes the polnoml dscrmnnt were nvoked to prove the estence of unque soluton hvng the sme sgn s. Numercl emples show the smmetrc nd hrd smmetrc regresson lnes fnnng out from the men pont nd occupng the spce etween the dnr lest-squres lnes. The emples lso revel tendenc f the smmetrc regresson lnes to remn seprte from the hrd regresson lnes. In future wk, the ccurc of these regresson lnes wll e nlzed. The wk of ths pper s generlzed nto the f dervng nd clssfng generlzed lest-squres regressons n the pper to follow. of the st Interntonl Conference on Computtonl Scence nd Engneerng (CSE '3), Vlenc, Spn, August 6-8, 03. [4] L. Leng, T. Zhng, L. Klenmn, nd W. Zhu, Ordnr Lest Squre Regresson, Orthogonl Regresson, Geometrc Men Regresson nd ther Applctons n Aerosol Scence, Journl of Phscs: Conference Seres, 78, (007), pp.- 5. [5] S. B. Mrtn, Less thn the Lest: An Alterntve Method of Lest-Squres Lner Regresson, Undergrdute Hons Thess, Deprtment of Mthemtcs, McMurr Unverst, Alene, Tes, 998. [6] R. B. Nelson, Proof Wthout Wds: A Recprocl Pthgen Theem, Mthemtcs Mgzne, Vol. 8, No. 5, Dec. 009, p.370. [7] P. A. Smuelson, A Note on Alterntve Regressons, Econometrc, Vol. 0, No. (Jn. 94) pp [8] R. Tgeper, Mkng Socl Scences Me Scentc: The Need f Predctve Models, Ofd Unverst Press, New Yk, 008. [9] E.W. Wessten, Dscrmnnt (Polnoml) n: CRC Concse Enccloped of Mthemtcs, nd. Edton, Chpmn & Hll/CRC Boc Rton, 003, p [0] E. B. Woolle, The Method of Mnmzed Ares s Bss f Crelton Anlss, Econometrc, Vol. 9, No., (Jn 94), pp References [] A. S. C. Ehrenerg, Dervng the Lest-Squres Regresson Equton, The Amercn Sttstcn, Vol. 37, No. 3 (Aug. 983), p.3. [] V. Ferln, Mthemtcs wthout (mn) wds, College Mth Journl, No. 33, (00), p. 70. [3] N. Greene, Generlzed Lest-Squres Regressons II: The nd Clsscton, Proceedngs ISBN: