An Algebraic Connection between Ordinary Lease-square Regression and Regression though the Origin

Size: px

Start display at page:

Download "An Algebraic Connection between Ordinary Lease-square Regression and Regression though the Origin"

Marjory Wheeler
5 years ago
Views:

1 03 Hawa Uverst Iteratoal Cofereces Educato & Techolog Math & Egeerg Techolog Jue 0 th to Jue th Ala Moaa Hotel, Hoolulu, Hawa A Algebrac Coecto betwee Ordar Lease-square egresso ad egresso though the Org Dr. Xaohu Zhog Uverst of Detrot Merc, Detrot, M

2 A Algebrac Coecto betwee Ordar Lease-square egresso ad egresso though the Org Xaohu Zhog Uverst of Detrot Merc, Detrot, USA Kewords: egresso; egresso through the org; Breakdow pots; squared. Abstract Ordar least-square regressools or regresso through the orgto? That s the questo. Ths paper tres to establsh a algebrac relato of the slopes ad squares betwee these two models ad stud the coecto betwee them. Oe of the results ca be used as a pedagogcal tool to costruct data set wth breakdow pot. Itroducto I the frst course of statstcs, OLS model s alwas the oe beg troduced. However, the realt, especall some phscal model, TO ma be more apprecate. For ths specal case of smple lear regresso, most tetbooks dscuss t the same as a specal case of the OSL b droppg the costat term whle others cautousl war that ot to use ths model uless ecessar. Ma lteratures dcted that these models are ot comparable. But s there a coecto betwee them? Tradtoall, the classcal smple lear regresso model α β was used to ft the observatos for two varables ad for coeffcets ca be estmated as β α β 3 b the ordar least squares OLS method, wth otato for the estmator of α ad β for smplct.,,...,, where the,. Here we use the same

3 Ths model was wdel used a wde rage of areas cludg egeerg, socal studes, boscece, etc. I ths model, oe mportat statstcs for checkg whether the model s a good ft s SSSSSS SSSSSS SSSSSS SSSSSS where SSSSSS, SSSSSS ıı ad SSSSSS ıı. Ths dett s true because ıı ıı 0. A geeral rule of thumb s that the closer the 4 s to, the better ft the model s to the observatos. O the other had, the same set of data ca also be ftted to the lear model: β 5 Such a model s called regresso through the org TO. I ths model the coeffcet s gve b wth the correspodg β 6 s gve b SS SST However dett 4 s o loger true. ˆ I ma practcal stuatos, TO s ecessar. Such ecesst was well dscussed b Casella[983] ad Esehauer [003]. The approprateess of adoptg ether OLS or TO model was also dscussed these two papers. It was cocluded that regresso through the org s a mportat ad useful tool appled statstcs, but t remas a subject of pedagogcal eglect, cotrovers ad cofuso Esehauer [003] because the squares ad other statstcs are ot comparable. Whle calculatg the basc regresso model t s ow farl eas usg a statstcal software or eve hadheld calculator, the TO model s also just oe clck awa. Practtoers other felds are stll ot clear the dffereces betwee these two models. Eve SPSS there s such dsclamer: For regresso through the org the o-tercept model, square measures the proporto of the varablt the depedet varable about the org eplaed b regresso. Ths CANNOT be compared to Square for models whch clude a tercept. Some also observed that most of the cases, both squares ad F both crease from OLS to TO. Is ths alwas true? If so, what codtos wll make them larger? What s the relatoshp betwee these statstcs? What else ca we look for whle selectg a approprate model? These are few questos we wat to aswer ths paper usg smple algebra ad calculus techques. 7

4 Augmeted Data Oe wa to stud the relato betwee these two models s to add a specal augmeted pot,,, Casella[983]. The leverage mpact of ths pot was studed ad was made use for comparg the two models. To lear the coecto betwee the OLS ad TO models, we vestgate a broader model b troducg a geeral augmeted pot, kk, kk to the data set wth k beg a arbtrar costat. If we deote ad, the smple algebra shows that r where k r 8 The OLS model wth the augmeted pot wll have the followg parameters: r r r β ] [ ] [ r r r SST SS r If r, the k, ad β, ad ] [ whch agree wth ad 4 for the OLS. Ths meas that the parameters above are those for the OLS wth the orgal data set. Whe 0 r, the k, ad 0 β, ad ] [ 0

5 whch are eactl 6 ad 7 for the TO. Thus we restrct r to be betwee 0 ad wth k [, ] ad the two models ca be obtaed b a ufed OLS method wth a parameter r [0,]. We wll stud how the model parameters chage from OLS to TO b varg r from to 0. The slopes of the regresso models ad breakdow pot I both models, OSL ad TO, the slope β r plas a mportat role the predcto. The sg of the slope dcates the drecto of the depedet varable learl correlated wth the depedet varable ad the magtude tells the rate of chage. For the purpose of ths stud, we wll ol cosder the cases whe 0 ad 0. Actuall, f both of these meas are 0, the OSL wll be eactl TO alread. Now we ca wrte where aa r r a β r m 9 r r b, bb, ad m. Wthout lost of geeralt, we wll cosder the case whe M > 0 the results wll be opposte whe M < 0. I addto, we wll avod the rare case whe a b. Notce that b >, we wll aalze how the slope chages from OSL to TO b smpl lookg at the graph of the ratoal b a fucto β r m`. r b

6 From the graph, t ca be see that β r s alwas a mootoc fucto of r [0,]. The locato of the statstcs a relatve to ths terval determes the sg of β r ad whether t s creasg or decreasg. The dfferet cases ca be summarzed as follows. Case a: The slope of OLS s postve ad greater tha the slope of TO, or 0 < β 0 < β r < β for r 0,, ff a > b. We see that the graph of the ratoal fucto 9 s above m ad creasg for r [0,]. Traslatg to the OSL ad TO model, t ca be terpreted as that the slope of OSL s greater tha m, as t decreases toward the slope of TO whch s also greater tha m. Notce that the le obtaed wth r 0 s ot the real TO, but t s a le parallel to the TO. Case b: The slope of OLS s postve ad smaller tha that of the TO, or β 0 > β r > β > 0 for r 0,, ff < a < b. Case c: The slope of OLS s egatve ad smaller tha that of the TO, or 0 > β 0 > β > β for r 0,, ff a < 0.

7 Case d: Ths s the most terestg case: The sg of the slope of OLS s egatve ad the sg of the slope of TO s postve, or β 0 < 0 < β, ff 0 < a <. Furthermore β a 0, ths partcular regresso le wll have a slope 0. These observatos ma ot be ver sgfcat ow. But the last case ca be used as a pedagog method to costruct the data set wth breakdow pots. Wth ths method, smple algebra ca be used costruct the data. There are ma was to defe breakdow pots. The are the pots a set of data so cotamated such that the model does ot cove useful formato. Oe smple defto s foud Cha[00]. Defto: The breakdow pot of a procedure for fttg a le to pars of bvarate data s defed as the mmum proporto of the sample that, whe replaced b sutabl chose outlers, ca drve the slope to ft or zero, Cha [00]. It was cocluded that the slope β model caot tolerate wth eve oe outler wth a breakdow pot of. Cha[00] gave a algebrac method b tral ad error of asmptotc breakdow pot. Here we ca provde aother method of costructg a set of data wth a ver good OLS ft, but wth oe outler, the regresso le becomes totall useless, meag that the slope becomes eactl zero. To costruct such a eample, we ca start wth a set that obvousl have a egatve slope OLS ad postve slope TO. Such data set ca be geerated wth a le such β α ε, where β < 0 ad α > 0, ad ε ~ N0, σ for postve pars of,. A TO for ths set wll have a postve slope because egatve slope wll produce all egatve predcted values. O the other had the slope of OLS should be egatve because t s estmatg β 0. Thus, from the result case d, t ca be cocluded that the statstcs a must be betwee 0 ad. Followg eample llustrates the process. Eample: We start wth the smplest set of data: 8, 0, 9, 99, 0, 98. The OLS model s

8 .5.83 wth whch s cosdered a ver good ft. If TO s troduced, we wll have Usg Tredle commet Ecel wll produce a meagless result Ths pheomea s the result of ccorect mplemetato of the squared formula dscussed Esehauer [003]. However, the statstcs a wll help fdg a outler. Lettg r a ad solve for k from 8, we have k a The augmeted pot s k, 9.6, 06. Now the OSL o the set of data 8, 0, 9, 99, 9.6, 06, 0, 98 wll have a slope 0 whch o oe wats to have OLS ad TO of Three pots ² ² OLS wth oe augmeted pot as the breakdow pot ² The advatage of ths method s to have algebrac steps to create a set that show that the breakpot of the slope of a SLO has a break pot of. Sce the slope β r vared cotuousl betwee β 0 ad β, ths meas that we ca also create a set of data wth oe etra pot to have the slope that s betwee these two slopes a smlar fasho. But how well s the fttg? We wll dscuss ths b vestgatg chage of a dcator et. The Applcators alwas get ected whe the saw that becomes bgger whle movg from OLS to TO most of the observatos [quote about the webste]. However, t ma ot be ver meagful to have a bgger. Followg, we wll fd out some cases that we are certa that

9 the becomes bgger, whle other cases t ca be affrmed that t s smaller. More mportat, we wll use the correct defto so that The formula of wll ot become egatve. should be ver famlar to most studets eve though t s slghtl complcated tha the slope formula. ecall that for a set of data wth augmeted pot defed secto, r r r a 0 r r r b r c where a ad b are defed last secto, ad c. For r, the square r s that for the a set of data, thus t s alwas true that 0 r. Usg ths defto, the square the eample s To stud ths ratoal fucto, we wll cosder all r ecept the values of b ad c where ths fucto has verdcal asmptotes, but the value of the fucto wll eceed or become egatve whe r >. We wll b c also avod the ver rare ad specal cases whe a b, b c, a c ad a, because the are all equvalet ad t happes f ad ol f m for all,,..,. Wthout lost of geeralt, we assume that b < c. Elemetar calculus reveals that r should have two crtcal pots r a ad r bc a b c. We wll see that the locato of these two crtcal b c a pots determes how r chages o the terval [0, ]. Frst, we see that oe ad ol oe of the crtcal pots must be betwee b ad c. The other crtcal must be smaller tha b, otherwse, r wll become greater tha for all r <. As a b-product, ths result ca be summarzed as a theorem: Theorem: Let {,,,.. } be pars of pots of reals such that 0, 0, ad m for all,,...,, the oe of the followg must be true: bc a b c a < m b, c < < ma b, c b c a or bc a b c < m b, c < a < ma b, c b c a

10 where a, b, ad c. Wth the result of ths theorem, the chages of squares from OLS to TO ca be summarzed as follows b vestgatg the graph of the ratoal fucto defed b 0. Case I: square s smaller whe movg from OLS to TO, meag r s creasg whe r chages from 0 to ff rr < 0, or. Case II: square s greater whe movg from OLS to TO, meag r s decreasg whe r chages from 0 to ff rr >, ad. Case III: If 0 < r <, or, the r wll reach a mmum at r. We wll ot be able to judge whch square s greater wthout actual calculatg them. But such a case, t s possble to costruct a data set that both 0 ad are the same. Cocluso: OLS ad TO models rema to be a mth. However, wth basc algebra, we ca establsh certa relatoshp betwee ther slopes ad square. Such relatoshp ca be used as a pedagog tool to reveal to studet the fact that square s ot alwas bgger wth TO. The future work wll be to establsh smlar relatoshp betwee the F statstcs. efereces Casella, G Leverage ad egresso through the Org. Amerca Statstca,

11 37, Cha, W. 00, Teachg the Cocept of Breakdow Pot Smple Lear egresso, Iteratoal Joural of Mathematcal Educato Scece ad Techolog, Volume 3, No.5, Eshehauer, J. Autum 003, egresso through the Org, Teachg Statstcs, Volume 5, Number 3, 76-80

Objectives of Multiple Regression

Objectives of Multiple Regression Obectves of Multple Regresso Establsh the lear equato that best predcts values of a depedet varable Y usg more tha oe eplaator varable from a large set of potetal predctors {,,... k }. Fd that subset of