Bivariate regression. dependent and independent variables. Which way is the relationship?

Size: px

Start display at page:

Download "Bivariate regression. dependent and independent variables. Which way is the relationship?"

Ethel Haynes
5 years ago
Views:

1 Bvarate regresso the correlato coeffcet measures the assocato betwee sets of pared varates, but t does ot tell us the way the two varables are related does ot allow us to predct the value of oe varable wth kowledge of the value of the other varable does t sgal aomales the relatoshp betwee dvdual pars bvarate regresso lets us do all of these thgs Probably dates back to 5 depedet ad depedet varables Whch way s the relatoshp? regresso allows us to suggest (hypothesze) causal relatoshps ad ther drecto - substatated by prevous research ad commo sese scattergram - used to plot depedet alog y axs, depedet alog x axs regresso volves plottg a best-ft le betwee the pots o a scattergram coveto s to treat the depedet varable as PREDICTED ad the depedet varable as the PREDICTOR predcto/terpolato s oe of the ma uses because x ad y are sampled. As we do t have complete formato o values for a gve x we wat to terpolate termedate values from the best ft le o the scattergram

2 ..6.4 Scattergram or scatterplots Iterpretato: how close to a sgle le?..6.4 Descrbg relatoshps (scatterplots) vdeo To see the uedted verso go to: Epsode. Descrbg Relatoshps dervato of best ft le example : y x easy to place best le through these pots as the assocato s perfect correlato coeffcet = there are o resduals/aomales/o devatos of pots from geeral relatoshp sce every pot s o the regresso le however varables are rarely perfectly correlated because of ) poor/theory/uderstadg or ) measuremet error ca place best-ft le through pots although r< ad so pots represetg varates do form a straght le devatos/aomales /resduals from regresso are show as resduals: why plot them vertcally rather tha perpedcular to the regresso le? Because resduals are the dfferece betwee the actual/observed values of the depedet varable (y values) ad the expected/predcted value of the depedet varable ( $y ) for a partcular value of x

3 fttg the regresso le by least square method ay straght le draw o a x y coordate system ca be represeted by a equato of the form Equato revew Slope/Y-tercept equato Idepedet Varable Slope, or coeffcet Y = a + bx Depedet varable Y-tercept Equato revew Slopes Slope rse/ru Y/ X The tercept least square approach - objectve s to fd the combato of tercept ad slope values whch mmze the sum of squares of the resdual values, that s, mmze the dfferece betwee the actual ad predcted values at partcular values of x There s o tercept f the data values are stadardzed before they re used The tercept s oly meagful f t makes sese for the depedet varable take o a value of There should be some values recorded ear zero f t s to be terpreted 3

4 Sample regresso fucto We ve bee talkg about populato characterstcs; we wll measure sample: Sample regresso fucto We ve bee talkg about populato characterstcs; we wll measure sample: Hats always dcate sample ˆ Y-hat: codtoal mea value of Y the sample: E(Y X ) Our estmate of real (populato) value of Y ˆ ˆ Y = β + β X ˆ ˆ ˆ Y = β + β X uˆ = Y Yˆ Ths s the defto of a resdual Sample Regresso fucto We ve bee talkg about populato characterstcs; we wll measure sample: Sample Regresso fucto We ve bee talkg about populato characterstcs; we wll measure sample: Ths gves us the stochastc sample regresso fucto. We wll use ths to fer thgs about the relatoshp betwee X ad Y populato ˆ ˆ ˆ Y = β + β X uˆ = Y Yˆ Y = ˆ β ˆ β + X + uˆ Actual Y s our sample; correspod wth X s Y = ˆ β ˆ β Sample coeffcets: defe the shape of the le betwee X ad Y our sample + X + uˆ Actual X s our sample; correspod wth Y s Resdual or error term How we do the math How we do the math We choose ordary least squares: mmze the squared vertcal dfferece betwee le ad pots regresso le: spedg= 7 +.5*statetax Ordary Least Squares (OLS): Mathematcally mmze the squared vertcal dstace betwee all the pots ad the le Why squared? statetax 4

5 Why we lke OLS Mathematcal propertes Pot estmators Pass through sample meas Mea of resduals = Resduals ucorrelated wth X ad Y Why we lke OLS Mathematcal propertes Statstcal propertes: There s some real relatoshp (+, -, or zero) betwee X s ad Y s the populato We evaluate the relatoshp betwee X ad Y our sample OLS assumptos ad Gauss- Markov The relatoshp betwee our sample parameters ad the populato parameters s smlar to relatoshp betwee sample statstcs ad populato parameters That s: our sample beta-hats are draw from a samplg dstrbuto aroud the real populato parameters For populato beta samplg dstrbuto: GM theorem proves that βˆ s are BLUE: Best: least varace Lear (descrptve of process) Ubased: cetered aroud real β Cosstet: as N, βˆ β Iterpretato of Coeffcets Recall our basc equato: β Y = ˆ β + ˆ β X + uˆ = E( Y X ) The costat, or Y-tercept Predcted value of Y whe X = zero (why?) Iterpretato of Coeffcets Recall our basc equato: Y = ˆ β + ˆ β X + uˆ = E( Y X ) β The costat, or Y-tercept Predcted value of Y whe X = zero (why?) Ths may or may ot be logcally useful cocept Always exercse cauto payg too much atteto to costat predctos ayway 5

6 Iterpretato of Coeffcets Recall our basc equato: β Y = ˆ β + ˆ β X + uˆ = E( Y X ) Slope coeffcet; recall Iterpretato of Coeffcets Recall our basc equato: β Y = ˆ β + ˆ β X + uˆ = E( Y X ) So what does that mea for a slope of.44 (say)? β rse Y = slope = = ru X β rse Y = slope = = ru X Iterpretato of Coeffcets Uts matter Always terpret equatos lght of uts o both sdes Should always use logcal uts, but may choose the logcal ut whch makes regresso most tractable: Am for coeffcets betwee zero ad Avod (o-zero) coeffcets <.5 or so Thgs we caot draw wth a Y=a+bX equato: Ifte slope Iterrupted le (dscotuous fucto) No-fuctos Fuctos may be o-lear though Coceptual overvew Regresso le shows relatoshp betwee fxed values of X ad average values of Y Regresso assumes relatoshp cotas stochastc elemet Outcomes follow probablty dstrbuto Regresso: coceptual overvew Relatoshp betwee fxed depedet varables ad average values of stochastc depedet varable:.75.7 Obvously cotrved data P(votg) Educato 6

7 Regresso: coceptual overvew For each value of X, Y follows a probablty dstrbuto.75.7 Obvously cotrved data Regresso: coceptual overvew For each value of X, Y follows a probablty dstrbuto: The probablty dstrbuto of a dscrete radom varable s a lst of probabltes assocated wth each of ts possble values We would lke these dstrbutos to be ormal P(votg) Educato Regresso: coceptual overvew Regresso: coceptual overvew Why s the depedet varable stochastc? Icomplete model o theoretcal level Not possble to collect quattatve data o everythg Measuremet errors Parsmoy: ot worth tryg to develop perfect model Wrog fuctoal form Regresso: coceptual overvew Regresso vs. Correlato Correlato s symmetrc Lear assocato No depedet ad depedet varables Both varables are assumed to be radom Regresso aalyss s asymmetrc Idepedet varable ot radom, but fxed repeated samples We assume oly depedet varable s radom, or that t follows probablty fucto Iferece for relatoshps vdeo To vew the vdeo o your ow go to: 5.html Epsode 5 7

8 How we do the math: OLS The whole exercse s about fttg a le to pots our sample scatterplot, usg the equato Y = a +bx By chagg values of a ad b, ths equato wll gve us ay straght le whch exsts a plae We wll use OLS to fgure out whch values of a ad b provde the best ft The slope b = the amout of chage y wth a chage ut of x b = gradet of the regresso le b = = x = xy ( x ) = x = = y = covarato varato x The tercept y x a = b = = = tercept wth y axs whe x = The tercept s ormally of lttle terest Ofte the y rage does t eve clude the tercept There are occasos whe the tercept s relevat Example would be a regresso of crop yeld versus fertlzer use, whe a= would deote o use of fertlzer How we do the math: OLS Example We wll use OLS to fgure out whch values of a ad b best ft the dots our pcture Note that depedg o how we defe best, we ll ed up wth dfferet les for same pcture Rver Nle Amazo Chag Jag Huag Ho Yellow) Mackeze bas sq km (x) Dscharge km 3 (y) Msssspp Idus Nelso- Saskatchewa

9 SPSS output ( Σy = 76 Σx = y (Gx ) =3449 Σx =757 ( y y$) /( ) ) = 495. x ( x) / Coeffcets Ustadardzed Coeffcets Stadardzed Coeffcets t Sg. Σx y = B Std. Error Beta (Costat) BASIN Depedet Varable: DISCHARG Revsed output wth bas rescaled 7 amaz Coeffcets 6 Ustadardzed Coeffcets Stadardzed Coeffcets t Sg B Std. Error Beta (Costat) BASIN DISCHARG chag huagsask dus mack - mss le Depedet Varable: DISCHARG BASIN 966.( 4 76) b = = = amaz 76 a = ( 4 ) = DISCHARG chag huagsask dus mack - mss le BASIN 9

10 Predcted values $y =a+bx = = [.99*33.7] [.99*75] = = the stadard method of measurg the goodess of ft of a regresso s to calculate the extet to whch the regresso accouts for the varato the observed values of the depedet varable ths s doe by calculatg the varace of the observed value of y = [.99*] = 45.7 s s r y$ = y = regresso varace y y = y = total varace s = =coeffcet of determato s y tests o the resduals a complemetary test of goodess of ft volves lookg at the resduals there should be o systematc varato the resduals Coeffcet of determato Rver Nle Amazo Chag Jag (Yagtze) Huag Ho (Yellow) Mackeze Msssspp Idus Nelso- Saskatchew a Bas km (s) x km 3 y y y hat resd s y = y = 6 = s y y 4577 = y = 6 = s r = = = s y Total

11 Regresso dagostcs Regresso dagostcs Fgure (a) - s a reasoable descrpto of y ad x. Regresso dagostcs Regresso dagostcs Fgure (b) - s obvously curvlear. Regresso dagostcs Regresso dagostcs Fgure (c) - oe data pot has udue fluece. Fgure (d) - ca oly ft a le to last pot.

12 resduals ted to crease as we crease the value of x there s o systematc varato apparet the resduals a curved le would be a better ft autocorrelato test f there s o correlato betwee the absolute values of the resduals ths s kow as seral correlato or autocorrelato the calculato for autocorrelato makes use of pars of values the frst resdual pared wth the secod, the secod wth the thrd ad so o absolute values of resduals a = =. 6 7 st 7 last 7 a b ab b = = s b b b 453. ( ) = sa = a 579. a = (. 6 ) = =7

13 3793. ab ab. 6( 675. ) r = = 7 =. ss 377.( ) a b a value close to or - suggests a relatoshp betwee successve resduals a complete absece of a relatoshp would gve a value of. A Cofdece Iterval s the estmato of a mea respose for a gve X. Cofdece Bads show a terval estmate for the etre regresso le. The Predcto Iterval s the predcto of a respose of a sgle ew observato of a gve X. Cofdece Iterval Vdeo clp o cofdece tervals Web vdeo avalable at: html Epsode 9, Cofdece Itervals How "wde" you have to cast your "et" to be sure of capturg the true populato parameter. I mght say that my 95% Cofdece Iterval s plus or mus %, meag that odds are 95 out of hudred that the true populato parameter s somewhere betwee ad %. Cofdece bad Measuremet of the certaty of the shape of the ftted regresso le. A 95% cofdece bad mples a 95% chace that the true regresso le fts wth the cofdece bads. It s a measuremet of ucertaty. The sample regresso equato s a estmate of the populato regresso equato. Lke ay other estmate, there s a ucertaty assocated wth t. The ucertaty s expressed cofdece bads about the regresso le. They have the same terpretato as the stadard error of the mea, except that the ucertaty vares accordg to the locato alog the le. 3

14 Cofdece bad for rver dscharge (95%) The ucertaty s least at the sample mea of the Xs ad gets larger as the dstace from the mea creases. The regresso le s lke a stck aled to a wall wth some wggle to t. dscharge cubc km BASIN Cofdece bad for rver dscharge Cofdece bad formula dscharge cubc km BASIN BASIN At 99% At 95% dscharge cubc km ts e + * ( x x) ( x) x S e s stadard error of the estmate X* s the s the locato alog the X-axs where the dstace s beg calculated The dstace s smallest whe x* = mea of x Predcto Bad (or Predcto Iterval) Measuremet of the certaty of the scatter about a certa regresso le. A 95% predcto bad dcates that, geeral, 95% of the pots wll be cotaed wth the bads. Used to estmate for a sgle value, ot the mea of Y Predcto terval for a dvdual y (based o exstg data) y$ ± t + + ( x x) ( x) x 4

15 y± ( ) = y± predcto bad or predcto terval 95% predcto bad Predcto tervals estmate a radom value where cofdece lmts estmate populato parameters t s possble to establsh lmts wth whch predctos are made from regresso equatos whe the regresso equato s used to predct a cofdece terval for the expected value ca be calculated Summary of predcto ssues We caot be certa of the mea of the dstrbuto of Y. Predcto lmts for Y (ew) must take to accout: varato the possble mea of the dstrbuto of Y varato the resposes Y wth the probablty dstrbuto Predcto terval for a ew respose the predcto terval s gve by: t e ( xo x) [ + ] x x where Σe s the sum of squares of the resduals from the regresso x s the mea of the values of the depedet varable x X s the partcular value for whch $y s beg predcted s the umber of pars of measuremets t s the partcular value take from the t table 5

16 95% tervals Implcatos o precso The greater the spread the x values, the arrower the cofdece terval, the more precse the predcto of E(Y o ). Gve the same set of x values, the further x o s from the (sample) mea of the x, the wder the cofdece terval, the less precse the predcto of E(Y o ). Commets o assumptos x h s a value wth scope of model, but t s ot ecessary that t s oe of the x values the data set. The cofdece terval formula for E(Y h ) works okay eve f the error terms are oly approxmately ormally dstrbuted. If you have a large sample, the error terms ca eve devate substatally from ormalty wthout greatly affectg approprateess of the cofdece terval. Cofdece bad most applcable for causal modelg Predcto terval most applcable for predctve uses sgfcace of b s the sample coeffcet (a estmate) sgfcatly dfferet from the populato coeffcet b=.99 s a estmate of the populato parameter H : Y ad X ( the populato) are ot related,.e. b s ot sgfcatly dfferet from H : Y ad X ( the populato) are related, b s sgfcatly dfferet tha T-test for b b B t = s eb where B = populato parameter Ho: B =.. b so t = wth df = s eb.. seb.. = = = ( y y$) ( x x) 6

17 the deomator ca be calculated va the formula To gve seb.. = x x ( ) ( y y$) / ( x) x seb = = t = = 55.. we ca reject H, b s sgfcatly dfferet tha Trasforms trasforms olear regresso semlog trasform: y=α+βlogx 7

18 y double log log y=α + βlogx or Y=AX B log y=α - βlogx or Y=AX -B α=.5, β= recprocal trasform Y=α + β/x Y=α - β/x α=.5, β= x y double log trasform 4 recprocal trasform Value 4. X Y LOGY Value Y. X X why trasform? To approach ormalty the data a) f data s postvely skewed (that s a log tal to the rght) a square root trasform mght be the aswer f more extreme, a logarthm trasform mght be ecessary f eve more extreme hgher roots mght be ecessary b) f data s egatvely skewed a power trasform mght be the aswer Cofdece Itervals If the correlato s perfect, the predctos are completely accurate; f the correlato s ot perfect, what s our level of cofdece our predcto? Calculate the Stadard Error of the Estmate, t expresses the degree of spread of the observatos (y values) aroud the regresso le uts of y. sey.. = = ( y y$) 5946 = = % of the observatos wth S.E. s. 95 % of the observatos wth S.E. s. y = a + bx + ( x = y = (-33.3) + (.99*) = ± 95.7 Ths expresso s essetally a average error for the regresso. The stadard error of the estmate s useful determg the rage of potetal Y values for a partcular X value.

19 95 % of y values (9 out of ) wll le wth (-336 to rage. Ths s hgh as ths example,.e. If you predcted a dscharge at uts of catchmet area, the predcto would be 649.7, but could be much hgher or lower. Cofdece Iterval Predcto Iterval Dummy varables Y = α + βd + u How does ths work f D = {,}? If hemsphere =, dummy = If hemsphere =s, dummy= A smple regresso usg a dummy varable s smlar to a oe-way aalyss of varace Dummy varables Rver Nle Amazo bas sq km (x) Dscharge km 3 (y) d hemsphere S S Y = α + βd + e How does ths work f D = {,}? Whe D = : Chag Jag 9 N Y = α + β* + e Huag Ho Mackeze N N Y = α + e Whe D = : Msssspp N Y = α + β* + e Idus 3 46 N Y = (α + β) + e Nelso- Saskatchewa 94 7 N 9

20 Dummy varables Y = α + βd + u How does ths work f D = {,}? Thus: α s average value of Y whe D = α + β s average value whe D = Statstcal sgfcace of β s t-test for dfferece of meas betwee the two categores Outlers I partcular outlers (.e., extreme cases) ca serously bas the results by "pullg" or "pushg" the regresso le a partcular drecto, thereby leadg to based regresso coeffcets. Ofte, excludg just a sgle extreme case ca yeld a completely dfferet set of results. Outlers Ifluetal observato If a pot les far from the other data the horzotal drecto, t s kow as a fluetal observato. Ther removal may substatally chage the regresso equato Lurkg varables A lurkg varable exsts whe the relatoshp betwee two varables s sgfcatly affected by the presece of a thrd varable whch has ot bee cluded the modelg effort. Such a varable mght be a factor of tme (for example, the effect of poltcal or ecoomc cycles) Sometmes the lurkg varable s a 'groupg' varable of sort. Ths s ofte examed by usg a dfferet plottg symbol to dstgush betwee the values of the thrd varables. For example, cosder the followg plot of the relatoshp betwee salary ad years of experece for urses. The dvdual les show a postve relatoshp, but the overall patter whe the data are pooled, shows a egatve relatoshp.

21 Lurkg varable Extrapolato Wheever a lear regresso model s ft to a group of data, the rage of the data should be carefully observed. Attemptg to use a regresso equato to predct values outsde of ths rage s ofte approprate, ad may yeld credble aswers. For example, a lear model whch relates weght ga to age for youg chldre. Applyg such a model to adults, or eve teeagers, would be absurd, sce the relatoshp betwee age ad weght ga s ot cosstet for all age groups.

Simple Linear Regression

Simple Linear Regression Statstcal Methods I (EST 75) Page 139 Smple Lear Regresso Smple regresso applcatos are used to ft a model descrbg a lear relatoshp betwee two varables. The aspects of least squares regresso ad correlato