Elementary Slopes in Simple Linear Regression. University of Montana and College of St. Catherine Missoula, MT St.

Size: px

Start display at page:

Download "Elementary Slopes in Simple Linear Regression. University of Montana and College of St. Catherine Missoula, MT St."

Beverley Shelton
5 years ago
Views:

1 Elemetar Slopes Smple Lear Regresso Rud Gdeo Adele Mare Rotha, CSJ Uverst of Motaa ad College of St. Cathere Mssoula, MT 598 St. Paul, MN 5505 I a bvarate data plot, ever two pots determe a elemetar slope. For pots wth dstct -values, there are / elemetar slopes. These elemetar slopes are eamed uder the two classcal regresso assumptos: the regressor varable values are fed ad the error s depedet ad ormal, ad the data s bvarate ormal. For case, t s demostrated that a weghted average of the elemetar slopes gves the stadard least squares estmate. I case, t s show that the elemetar slopes have a rescaled Cauch dstrbuto; ths Cauch dstrbuto s the used to estmate bvarate ormal parameters. Two oparametrc correlato coeffcets, Kedall s τ ad the Greatest Devato correlato coeffcet GD, are used wth elemetar slopes regresso estmato. Smulatos show the robustess of the oparametrc method of estmato usg Kedall s τ ad GD. Kewords: bvarate ormal, Cauch dstrbuto, Kedall s τ, Greatest Devato correlato coeffcet Ths work depeds part o earler upublshed work of Gdeo ad s avalable o hs web ste: Some of the refereces wll refer to papers posted at ths web ste.. Smple Lear Regresso wth fed regressor varable data Let the regresso equato model be α + β + ε, errors depedet wth V ε ad ε 0.,,,, K be the data wth dstct - E Let { }, values. The the set of -/ elemetar slopes are to the model, α + β + ε α + β + ε ε β + ε Because E ε ε 0, each slope s ubased for β. Also.,. Accordg Y Y V ε ε V. I U-Statstcs methods Radles ad Wolfe 979, t would be suggested that the elemetar slopes be averaged to obta a ubased estmate. However, ths s slghtl chaged here b takg a weghted average wth the weghts beg the recprocals of the varaces of the elemetar slopes. Elemetar Slopes 6//00 0: AM --

2 Elemetar Slopes 6//00 0: AM -- Lemma : The weghted average of the elemetar slopes gves the usual least squares estmate of the slope. Proof: Let A be the set of dces,,. The the sum of the weghts W s gve b: W A ε,.. Result. s related to oe-sample U-statstcs Radles ad Wolfe 979, pp. 6-6 ad follows from the demostrato of the equalt., show below, whe s take as. For dstct -values ad, the weghted-average estmate s A W ε β, ˆ A W ε, W.. Fall, substtutg for W gves: ˆ β `. To demostrate equalt., a eample s show for. The the set A has 6 pots; },,,,,,,,,, {, A. A ε, + + Add ad subtract so that A ε, Note that So. becomes A ε,

3 Recogzg [ ], εa, above, as the computatoal form of. Thus W, εa Or for geeral postve teger, W., elds W, εa W. If ot all are dstct, the formulas ad equatos wll hold f whe, s terpreted as zero sce ts lmt as 0 s zero. These terms must appear the summatos. B etedg the dea of a U-Statstc to a weghted average, t has bee show that the classcal least squares estmate of slope s a weghted average of elemetar slopes.. Smple Regresso wth Bvarate Normal Data I ths secto, the data X, Y have a bvarate ormal dstrbuto. Aga the elemetar slopes are aalzed, but ow both the umerator ad deomator are radom varables. It s show that the elemetar slopes for a bvarate ormal dstrbuto have a Cauch dstrbuto. It s the show how to use the Cauch dstrbuted elemetar slopes to estmate the regresso parameters for the bvarate ormal. The equal dstrbuto otato d defed Radles ad Wolfe 979, p. s used. Lemma : The elemetar Slopes for a Bvarate Normal Dstrbuto have a rescaled Cauch Dstrbuto. Proof: Frst, let X, Y have a stadardzed bvarate ormal dstrbuto wth correlato coeffcet ρ. The for two depedet observatos X, Y ad X, Y, let U Y Y ad V X X so that U/V R R for rato s the elemetar slope. The radom varable U, V has a bvarate ormal dstrbuto wth meas 0, varaces, ad correlato coeffcet ρ. I order to obta the ot Elemetar Slopes 6//00 0: AM --

4 dstrbuto of R, S, let RU/V ad SU. Obta the ot dstrbuto of R, S ad tegrate out S to obta that the dstrbuto of R s Cauch wth locato parameter ρ ad scale parameter ρ f r, < r <. r ρ π ρ + ρ. The dest for R s Let R deote ths Cauch wth locato parameter ρ ad scale parameter ρ. The the dstrbuto of ρ R ρ R 0 s the stadard Cauch. For the geeral bvarate ormal, let X d N µ,, Y d N µ, wth correlato coeffcet ρ. It follows that Y Y d Y Y d R ad that the elemetar slope R. Lastl, a X X X X sample of elemetar slopes s related to a stadard Cauch b Y X Y d ρ + ρ R0 X. I the case where X, Y has a bvarate ormal dstrbuto, the regresso model s Y µ + ρ µ.. It s ow eas to estmate ρ ad the slope parameter, ρ, b oparametrc methods. The slope parameter s estmated as a tercept a regresso that uses the elemetar slopes as the depedet varable; see equato.. The correlato coeffcet s estmated usg both the slope ad tercept ths regresso. For a complete developmet of the work that follows, the reader s referred to problem.. Radles ad Wolfe 979, p. ad papers through 7 at the web ste. A sopss of ecessar materal follows. Let r p, be the otato for the calculato of Pearso s correlato coeffcet o a set of data,. Let GD be the Greatest Devato correlato coeffcet Gdeo ad Hollster 987 ad GD, ts value o a set of data. For smple lear regresso, the least squares estmate of slope s obtaed b solvg for b the equato r p, b 0 ; that s, b makg the correlato betwee the depedet varable ad the ucetered resduals zero. The GD slope estmate s smlarl Elemetar Slopes 6//00 0: AM --

5 obtaed b solvg GD, b 0. I fact a correlato coeffcet ca be used ths maer as eplaed Gdeo 99 ad Gdeo ad Rummel 99. That s, for a correlato coeffcet r, solve for b r, b 0.. Ths same tpe of correlato coeffcet equato s used for locato ad scale estmato. Gdeo ad Rotha 00. The form of the equato remas the same; ol the argumets chage. I ths correlato method of estmato, scale must be estmated frst ad the locato. Whereas the orgal sample sze s, the sample sze of the elemetar slopes s. Let m ad let q be the ordered quatles correspodg to equall spaced probabltes from the assumed dstrbuto: the tegers through m each dvded b m +. Here q comes from the stadard Cauch, R 0, dstrbuto. These quatles are pared wth ordered sample data. Let vector v be the ordered set of elemetar slopes,. The sample sze m, defg q above, s /. The a estmate of scale usg r p s foud b solvg for s q, v sq 0, where v equals the vector of ordered slopes. r p The locato estmate comes from takg the mea of v sq. Classcal methods are ot vald for the Cauch dstrbuto; the method used here s smlar to that Radles ad Wolfe 979, problem.., p. ecept that the work s defed through correlato coeffcets ad uses the ordered data. For a robust ad vald estmate of scale based o GD, solve for s GD q, v sq 0. The locato estmate comes from takg the meda of v sq. The geeral scale equato for a correlato coeffcet r usg ordered quatles correspodg to the ordered data s r q, ordered data sq 0. The same umercal routes used. suffce to solve for s equato.. The estmate of the slope parameter, ρ,. s the soluto s.. The estmate of the locato parameter, ρ,. s the meda of v sq ; call ths estmate c. Alteratvel stated, the estmate of the regresso slope parameter ρ. for the orgal data comes from the tercept estmate c. where Elemetar Slopes 6//00 0: AM -5-

6 the elemetar slopes are the depedet varable. Sce s estmates c estmates ρ ρ, the rato u of s to c estmates. The equato, ρ ρ ad ρ u, s ow solved for ρ ; so the estmate of ρ s sg c. B ρ + u dvdg c b ths estmate of ρ, oe has a estmate of the rato of the stadard devatos.. Regresso Estmato usg Kedall s τ ad Elemetar Slopes The basc regresso estmato equato for Kedall sτ s equato. wth τ replacg r, τ, b 0, ad solve for b. For fed b the cocordaces ad dscordaces are couted from the sgs for all pars of dces,, <, of b b b. For τ, b 0, the umber of cocordat pars must equal the umber of dscordat pars ad ths mples b s determed so that # b < 0 # 0. b > Ths occurs whe b s the meda of the elemetar slopes. If a plot s made of b agast τ, b, t s a mootoc decreasg step fucto that ol decreases at each elemetar slope. The fucto GD, b behaves a smlar maer ecept t decreases at some but ot all of the elemetar slopes thus assumg fewer values.. Illustrato b Smulato of the Estmato Regresso b Correlato Coeffcets ad Elemetar Slopes I order to estmate α ρ ad β ρ., the elemetar slopes are the data to be ordered ad regressed equato. agast the Cauch quatles, q. The elemetar slopes for ths data umber where s the orgal sample sze. For 0, 0, 00, the umber of elemetar slopes s 990, 9,95 ad,8,775, respectvel. The computer laguage C route used solvg equato. or. for GD uses these elemetar slopes as the data. The Elemetar Slopes 6//00 0: AM -6-

7 route does a sstematc search o these elemetar slopes; ad so for greater tha 0 or so, t s overwhelmed. However, a feature of equato. allows good estmato whe the ordered data s trucated at each ed. Computer smulatos demostrate the practcalt of these methods. Two tpes of data were cosdered: bvarate ormal BN ad bvarate ormal wth some outler cotamato the Y varable to demostrate the robustess of GD ad Kedall sτ methods. Ths outler data was lmted to a radom amout; amel, the umber of outlers each sample was bomal wth equal to the sample sze ad probablt of 0.0 for outlers. The outlers were geerated from a N0, 5 dstrbuto. For each case BN ad BN wth Outlers, sample szes of 0 ad 00 were used ad 000 smulatos were ru. Estmates of the slope parameter the BN, ρ, were obtaed usg three methods: the usual least squares or, correlato laguage, Pearso s r estmate, Kedall s τ method, ad the GD method o the elemetar slopes. Estmates of the correlato parameter ρ were obtaed b: the usual Pearso s r, the GD regresso usg the elemetar slopes ad takg the rato of the slope to the tercept ad the usg u as detaled Secto, ad the greatest devato usg the ormal trasformato s π GD /, see Gdeo ad Hollster 987. The meas ad stadard devatos for the estmators the smulatos are gve Tables I VIII below. Table IX dsplas a summar of the GD estmate of the stadard devato of the orgal -data foud b usg equato. ad ormal quatles. Followg the tables, a short dscusso of the prcple results s gve. Elemetar Slopes 6//00 0: AM -7-

8 Table I Estmato of Slope, o outlers, 0 ad 000 smulatos each τ mea SD LS or P mea SD GD wth the mea elem slopes SD Table II Estmato of Slope, wth outlers, 0 ad 000 smulatos each τ mea SD LS or P mea SD GD wth the mea elem slopes SD Table III Estmato of Correlato ρ, o outlers, 0 ad 000 smulatos LS or P mea SD GD wth mea elem Slopes SD GD wth the mea se trasf SD Table IV Estmato of Correlato ρ, wth outlers 0, ad 000 smulatos LS or P mea SD GD wth mea elem Slopes SD GD wth the mea se trasf SD Elemetar Slopes 6//00 0: AM -8-

9 The stadard devato parameter of the X varable was ad dd ot chage over the smulatos. Ol the Y varable was cotamated b outlers as descrbed earler. For sample sze 0, the sample mea of the 000 smulatos of the GD estmate of the stadard devato parameter of X was usuall about.09 wth a sample stadard devato of 0.6. There apparetl s a slght upward bas. Stated aother wa, the estmate ± two stadard errors s.09 ± 0.0. Table V Estmato of Slope, o outlers 00, ad 000 smulatos each τ mea SD LS or P mea SD GD wth the mea elem slopes SD Table VI Estmato of Slope, wth outlers 00, ad 000 smulatos each τ mea SD LS or P mea SD GD wth the mea elem slopes SD Table VII Estmato of Correlato ρ, o outlers 00, ad 000 smulatos LS or P mea SD GD wth mea elem Slopes SD GD wth the mea se trasf SD Elemetar Slopes 6//00 0: AM -9-

10 Table VIII Estmato of Correlato ρ wth outlers, 00 ad 000 smulatos LS or P mea SD GD wth mea elem Slopes SD GD wth the mea se trasf SD For these rus of sample sze 00, the sample mea of the 000 smulatos of the GD estmate of the stadard devato parameter of X was usuall about.0 wth a sample stadard devato of Thus the creased sample sze reduced the bas. Here the estmate ± two stadard errors s.0 ± 0.0. A geeral summar of the results ow follows. For both outler ad o outler data, two sample szes are used, 0 ad 00. For the sample sze 00, the GD method o the elemetar slopes GD-ES used the mddle 00 elemetar slopes for the regresso equato.. These mddle 00 elemetar slopes 50 o each sde of the meda of the elemetar slopes are pared wth the mddle 00 Cauch quatles. It would seem that ths data reducto would be detrmetal to the estmato process, but the results seem lttle flueced b ths reducto. From Tables I ad V, o outlers, all methods are earl ubased for the slope parameter. The GD-ES ad τ methods are ver comparable ad ol margall less effectve tha the least squares method terms of slghtl larger varato. From Tables II ad VI, wth outlers, for 0, the best method for the slope s τ, the comes GD-ES, ad fall LS. For 00, GD-ES ad τ are earl the same ad qute superor to LS. From Tables III ad VII, o outlers, the estmato of ρ, the GD-ES method appears ubased for all correlatos. GD-se s ubased ear 0, but uderestmates for larger ρ. LS ad GD-ES are farl comparable wth respect to varato. From Tables IV ad VIII, wth outlers, the estmato of ρ, GD-ES has substatall less bas for all ρ ad for ρ > 0 less varato ad s preferable to Pearso s r. Ecept ear 0, GD-se s also preferable over the classcal method. From Table IX the estmato of the stadard devato of X b the GD method va equato. s see as slghtl based. Some results are take from Fraser 976 to ad a comparso. Let s be the usual classcal stadard devato ad s gd be the GD estmate. The we use Elemetar Slopes 6//00 0: AM -0-

11 V s s Table IX Comparso of the Stadard Devato of X, 0 Es.96 Es gd.09 SDs 0.8 SDs gd 0.6, 00 Es.99 SDs 0. Es gd.0 SDs gd 0.66 E s + ad E s E to costruct the followg table. The values for s are from Fraser 976. The values for s gd are estmates from the 000 smulatos. 5. A Further Note o Estmato wth Fed Regressor Varable Data Equato. ca be used to estmate the slope ad tercept a smple lear regresso wth a correlato coeffcet. Ths s detaled the smple lear regresso papers Gdeo 99, Gdeo ad Rummel 99. Smple lear regresso has bee carred out ma tmes usg GD. As classcal smple lear regresso, the resduals ca be computed. However, the regresso stadard error s computed b usg equato. wth q beg ormal quatles ad the ordered data the ordered resduals. The slope of the le s the stadard error of the regresso ft. I the same maer as ormal theor or classcal methods, the estmates of resdual stadard error ad the stadard devato of ca be used to compute a regresso correlato coeffcet, ρ ˆ s s. The s statstcs all come from GD methods. Ths method s detaled Gdeo ad Mller 99. If the error s Cauch the fed- model, the GD method usg Cauch quatles wll gve regresso parameter estmates as well as locato ad scale estmates of all the volved parameters. I addto, estmato of parameters wth the bvarate Cauch ca also be carred out. I these stuatos, of course, classcal methods caot be used. 6. A Fal Summar Ths paper llustrates some terestg facts about the use of elemetar slopes smple lear regresso. A rak-based correlato estmator, GD, was used o the elemetar slopes to demostrate that the Cauch dstrbuto ca be proftablt used estmato. Wth outlers dstrbuted smmetrcall, both Kedall sτ ad GD operatg o the orgal data ad GD operatg o the elemetar slopes of the bvarate data are show to be robust. It appears that researchers estmato have prevousl overlooked the Cauch dstrbuto, but ths paper has show that the Cauch dstrbuto s essetal the estmato procedure bvarate ormal stuatos. Elemetar Slopes 6//00 0: AM --

12 Ths work s a eteso of a etre sstem of estmato based o a correlato coeffcet ad, partcular, o oparametrc correlato coeffcets. Several papers eplorg ths sstem are avalable o the web ste. It s hoped that more of the web ste materal wll be publshed ad the web ste materal elarged to show more of the methodolog of the correlato coeffcet sstem of estmato. The ma computer programs are wrtte C code ad are terfaced wth the S-Plus statstcal package. Refereces Fraser, D. A. S. 976, Probablt & Statstcs, Theor ad Applcatos, problem, page 06, North Sctuate, MA: Dubur Press. Gdeo, R. A. 99, Radom Varables, Regresso, ad the GD, upublshed paper URL: Uverst of Motaa, Dept. of Mathematcal Sceces. Gdeo, R. A., ad Hollster, R. A. 987, "A Rak Correlato Coeffcet Resstat to Outlers," Joural of the Amerca Statstcal Assocato, 8, Gdeo, R. A., ad Mller, J. M. 995, Multple Regresso Techque wth Asmptotcs, upublshed paper URL: Uverst of Motaa, Dept. of Mathematcal Sceces. Gdeo, R. A., ad Rotha, A. M. 00, Locato ad Scale Estmato wth Correlato Coeffcets, upublshed paper URL: Uverst of Motaa, Dept. of Mathematcal Sceces. Gdeo, R. A., ad Rummel, S. E. 99, Correlato Smple Lear Regresso, upublshed paper URL: SPACE-REG.pdf, Uverst of Motaa, Dept. of Mathematcal Sceces. Radles, RH ad Wolfe, D.A. 979, Itroducto to the Theor of Noparametrc Statstcs, New York: Joh Wle ad Sos. R. A. Gdeo: Ackowledgmets Ths work has bee progress for ma ears wth ver few publshed papers avalable to ackowledge all the facult ad studet help. These people are also lsted at the web ste, ad I hope o oe has bee mssed. I thak these people for all the help the have gve me. The are the oes who have kept ths research alve. Elemetar Slopes 6//00 0: AM --

13 Apped: A Eample of the Process Usg Bvarate Normal Data wth Some Cotamato the Y varable. I ths eample, a sample of sze 0 was geerated from a bvarate ormal dstrbuto wth parameters 5,, µ 6, 6, ad ρ 0. 6 wth the Y varable subect to µ radom outler cotamato; the umber of outlers was bomal wth equal to 0 ad the probablt of 0. for outlers. I ths ru, fve outlers were geerated from a N0, 5 dstrbuto. Ol three of these fve outlers are apparet o the scatterplot of the data. The data ad the mddle 0 elemetar slopes out of the 90, 0 choose, are gve below. Fgure dsplas the scatterplot ad three regresso les: the classcal least squares, the ordar GD regresso le from., ad the GD elemetar slope regresso le from., the darkest le. Fgure shows a plot of 0 ordered elemetar slopes, order statstcs 6 to 6, versus the correspodg quatles for the stadard Cauch. These 0 order statstcs of elemetar slopes were used as the depedet varable ad the correspodg quatles for the stadard Cauch radom varable as the regressor varable.. The slope of ths regresso le s the soluto s to.; here s.085. The tercept of ths regresso le s the meda, c, of the ucetered resduals, amel c The GD elemetar slope estmate of ρ s Thus the GD elemetar slope regresso le, for the orgal data, show Graph s Y X. The tercept, c, of the regresso le Graph s the estmate of the slope ad meda of.5708 usg the orgal data s the tercept. A summar of the three fts s ow gve the form Methodtercept, slope: LS -0.76,.; GD -.,.7; GD o elemetar slopes -.56, The two GD methods are ver smlar ad preferable to the LS method. The theoretcal regresso le, wthout outlers, was E Y. wth the stadard devato of the resduals beg.8. The correlato parameter was 0.6. The GD elemetar slope method of estmatg correlato was ecellet ths eample, ad the slope estmate was superor to the classcal least squares method ad the ordar GD regresso. Elemetar Slopes 6//00 0: AM --

14 Fgure The ordered -data: The correspodg -data: The mddle 0 elemetar slopes: Elemetar Slopes 6//00 0: AM --

15 Fgure Elemetar Slopes 6//00 0: AM -5-

( ) = ( ) ( ) Chapter 13 Asymptotic Theory and Stochastic Regressors. Stochastic regressors model

( ) = ( ) ( ) Chapter 13 Asymptotic Theory and Stochastic Regressors. Stochastic regressors model Chapter 3 Asmptotc Theor ad Stochastc Regressors The ature of eplaator varable s assumed to be o-stochastc or fed repeated samples a regresso aalss Such a assumpto s approprate for those epermets whch