Revsed: v3 Ordnar Least Squares (OLS): Smple Lnear Regresson (SLR) Analtcs The SLR Setup Sample Statstcs Ordnar Least Squares (OLS): FOCs and SOCs Back to OLS and Sample Statstcs Predctons (and Resduals) wth OLS Estmates Examples n Excel and Stata The SLR Setup You have a dataset consstng of n observatons of (, ) : { } x, =,, n You beleve that except for random nose n the data, there s a lnear relatonshp between the x s and the s: β + βx You are nterested n estmatng the unknown parameters β and β 3 If there was no nose n the data, then snce = β + βx for all observatons, we can easl determne β and β But tpcall, the relatonshp s not exact n the observed data 4 Call our parameter estmates ˆβ and ˆβ, and our predcted values ˆ ˆ ˆ = β + βx 5 We call the dfference between the observed and the predcted value ˆ ˆ ˆ = β + βx the resdual, ˆ uˆ ˆ ˆ ˆ = = β + βx u : ( ) 6 Here s an example (negatve resduals for # and #; postve resduals for #3 and #4): β s just the slope of the lne connectng an two dataponts, and β β = x, for an datapont
OLS: SLR Analss (Rev d v3) Sample Statstcs 7 The sample mean: x = x and n 8 Devatons from means: ( x x)' s and ( )' s = Note that x = nx n a B constructon, the total of the devatons from the means wll be zero: ( x x) = and ( ) = ( ) = ( + ) ( ) ( ) n ( ) n n ( ) n and lkewse for the 9 Squared devatons from means: = + = + = x s n n average squared devaton from the mean (except we dvde b n-, not n the reason for ths wll become clear when we consder unbased estmaton) The sample varance: S ( ) ( ) = S = = n almost the The sample standard devaton: varance S = ( ) n Sum of the product of the devatons from means: ( x x)( ) = ( x x + ) = ( x ) n + n = ( ) n, the square root of the sample n n almost the average product of the devatons from the means (except we agan dvde b n-, not n and es, ths s also related to unbased estmaton) 3 The sample covarance: S = ( x x)( ) = ( x ) n agan, a Some examples: In the followng examples, x = and = On the left, most of the data are n quadrants I and III, where ( x x)( ) >, and so when ou sum those products ou get a postve sample covarance Most of the acton on the rght s n quadrants II and IV where ( x x)( ) <, and so those products sum to a negatve covarance
OLS: SLR Analss (Rev d v3) S 4 The sample correlaton: ˆ ρ =, the rato of the sample covarance to the product of the SS sample standard devatons x a It ma not be so obvous, but b constructon, ˆ ρ, or ˆ ρ b If S =, the sample covarance s and the sample correlaton s also And f the sample covarance s negatve (postve), then so s the sample correlaton (snce sample standard devatons are alwas postve, so long as the are well defned and not zero) c If ˆ ρ s close to then the relatonshp between x and wll look qute lnear (wth a postve slope f ˆ ρ, and a negatve slope f ˆ ρ If = β + βx for all, so there s n fact an exact lnear relatonshp between the x s and s, then sample correlaton s + or d And as ˆ ρ gets closer to, the relatonshp between x and looks less and less lnear e So: Correlaton captures the extent to whch x and are movng together n a lnear fashon Ordnar Least Squares (OLS): FOCs and SOCs 5 Mnmze Sum (of the) Squared Resduals (SSRs) a We can estmate the ntercept and slope parameters, β and β, b mnmzng the sum of the squared resduals (SSRs) or errors ths explans where the least squares part of OLS comes from (We square the resduals so that postve and negatve resduals don t offset one another) b So the challenge s to fnd the slope and ntercept parameter estmates that mnmze ( ( β )) β ( ) SSR = u = + x n n S = β ( x x) = βsxx And snce S = ( ) =β Sxx, S = β Sx n n S βsxx β So: ˆ ρ = SS = β SS = β = ± Proof: Snce S = ( x x)( ) = ( x x) [ β + βx ( β + βx) ], x x x 3
OLS: SLR Analss (Rev d v3) c Fndng the mnmum: Frst Order Condtons (FOCs): A standard approach to fndng the mnmum of a functon s to evaluate ts (partal) dervatves and explot the fact that at the mnmum, the frst dervatves wll be zero (ths s called the frst order condton, FOC) Second Order Condtons (SOCs): And we ll also tpcall want to ensure that we have a mnmum and not, sa, a maxmum, b checkng as well the second dervatves At the mnmum, the second (partal) dervatves at the pont satsfng the FOC wll be postve We wll be skppng ths step as t can get a bt complcated d Focusng on the FOCs for our mnmzaton problem: ( ) wth respect to (wrt) β and β mnmze SSR = ( β + β x ) FOC : Dfferentatng wrt β : SSR = ˆ ˆ ( β βx) n + nβ + βnx =, and so β SSR = ˆ β ˆ = βx β The estmate of the ntercept assures that the average predcted value, ˆ β ˆ + β x, s ˆ β + ˆ β x = ˆ β x + ˆ β x = the same as the average observed value, snce ( ) FOC : Dfferentatng wrt β : ( ) and Snce SSR = ( β + βx) ˆ β ˆ β x SSR = ( β x + β x ) = ( ) β ( x x ) SSR =, we want to mnmze ( x x) ( ) β ( x x) = = β So ( x x)( ) ( x x) ( )( ) ˆ x ( ) x = β x x, and ˆ β = e The SOCs are more complcated, but t s eas to see that at the coeffcent values satsfng the two FOCs, the two second partal dervatves (wrt β and wrt β ) are both SSR SSR postve, suggestng that we have a mnmum: = n > and = x > β β 4
OLS: SLR Analss (Rev d v3) 6 OLS coeffcent estmates: a For the gven sample, the OLS estmates of the unknown ntercept and slope parameters are: ˆ β ( x x)( ), and ˆ β ˆ = βx = ( x x) b ˆβ and sample means Snce ˆ β ˆ = βx, the estmated ntercept s the sample mean of the s mnus ˆβ tmes the sample means of the x s c ˆβ and sample varances, covarance and correlaton Usng our sample statstcs notaton: ˆ β ( x x)( ) ( x x)( ) / ( n ) S = = = ( x x) ( x x) / ( n ) Sxx Thus, the OLS slope estmator s just the rato of the sample covarance of x s and s and the sample varance of the x s: ˆ Sample Covarance( x, ) β = Sample Varance( x) S Recall that the sample correlaton s defned b: ˆ ρ =, where S x and SS the square roots of the respectve sample varances S S Sx v Snce ˆ ρ = =, we have: ˆ S S β ˆ = = ρ SS S S S S x xx v So whle the regresson slope coeffcent s the product of the sample correlaton between the x s and s and the rato of the two estmated standard devatons: ˆ Sample StdDev( ) β = Sample Correlaton( x, ) Sample StdDev ( x ) If the two sample standard devatons are the same then the estmated slope coeffcent wll be the estmated correlaton between the x s and s d Important asde: Snce ( x x) =, Accordngl, ˆ ( x x) β = ( x x) ( x x)( ) = ( x x) ( x x) = ( x x) xx Ths wll prove useful later x x S are 5
OLS: SLR Analss (Rev d v3) e ˆβ s a weghted average of slopes: The estmated slope coeffcent s a weghted average of slopes of lnes jonng the varous dataponts to the sample means: ˆ ( ) β = w ( x x ) Ths result holds because ( x )( ) x ( x x) ( ) ( ) w ( n ) Sxx ( n ) Sxx ( x x) ( x x) ˆ β = = = ( ) s the slope of the lne connectng ( x, ) to ( x, ) ( x x) ( x x) ( x x) w = = ( n ) S ( x x) xx j j are non-negatve weghts, whch sum to so the slopes are weghted proportonall to ( x x), the square of the varous x-dstances from the x mean v In ths nterpretaton, dataponts are not weghted equall Those that are farther awa from x (n the x dmenson) get greater weght, and that weght ncreases wth the square of the x-dstance from x v See the posted handout for an example Predctons (and Resduals) wth OLS Estmates 7 OLS coeffcent estmates eld predcted values and resduals: a Predcted values: For gven x, the predcted value gven the estmated coeffcents s: = ˆ β + ˆ β x (as above, we tpcall use hats for predcted or estmated values) ˆ b Resduals: And for the gven predcted value, the resdual, u ˆ, s as above the uˆ = ˆ = ˆ β + ˆ β x dfference between the actual and predcted values: ( ) 8 Sample Regresson Functon (SRF): The predcted values from the estmated equaton, ŷ = ˆ β + ˆ β x, s called the Sample Regresson Functon a SRFs wll depend on the actual sample used to estmate the slope and ntercept parameters: dfferent samples wll tpcall lead to dfferent parameter estmates and accordngl, dfferent SRFs ˆ β = ˆ β x assures that the SRF passes through ( x, ) snce the value of the SRF at x b ˆ β + ˆ β x = ˆ β x + ˆ β x = s ( ) 6
OLS: SLR Analss (Rev d v3) c The sample correlaton between the actuals and predcted values s the same as the sample correlaton between the actuals and the x s: ˆ ρ ˆ 3 = ρ 9 SRFs and elastctes measurng economc sgnfcance a We tpcall use dervatves and elastctes to estmate the responsveness of the predcted values ( s ˆ ' ) to changes n the explanator varable x b Dervatves wll be senstve to unts of measurement but elastctes are not That s wh t s not uncommon to use elastctes to measure economc sgnfcance: Is the estmated relatonshp szable or noteworth? or s t so small that t s of lttle consequence? c Usng the SRF to estmate relatonshps: Dervatves: The estmated average margnal relatonshp between x and : d ˆ = ˆ β dx x d (Pont) Elastct: ˆ ˆ x = β evaluated at (, ˆ) = ( x, ˆ β ˆ + βx), or somewhere ˆ dx ˆ on the SRF Where ou evaluate the elastct on the SRF s often arbtrar but be sure to evaluate the elastct at some pont on the SRF You wll tpcall get dfferent elastctes dependng on where along the SRF ou estmate the elastct We often evaluate the elastct at the means: ˆ β x a Recall that the mean of the predcted values wll be and that the SRF passes through ( x, ) = ( x, ˆ β ˆ + βx) ˆ x Propertes of OLS resduals a Recall that uˆ ˆ ( ˆ ˆ β βx) = = + b The average resdual s zero: u ( ˆ ˆ β βx) n ˆ = + = c The sample correlaton between the x ' s and the ˆ ' ˆ = u s s zero, snce u ( x x ) 3 S ρ = But snce S ˆ ˆ = βsx and snce ˆ ˆ ˆ SS ˆ S = ˆ β S, ˆ ρ ˆˆ 7 xx ˆ S ˆ ˆ βsx Sx = = = = ˆ ρx SS S ˆ β S SS ˆ x x
OLS: SLR Analss (Rev d v3) ( )( ) Proof: ( ˆ )( x x) = ˆ ˆ ( β + βx x x (( ˆ ) β = ( x x) )( x x) = ( )( ) ˆ x x β ( x x) = gven the defnton of ˆβ d The sample correlaton between the predcted values ( ˆ s) and the resduals ( uˆ s) s zero, whch s zero (see Proof: ( uˆ uˆ)( ˆ ˆ) = uˆ ( ˆ ) = ˆ β ( ˆ )( x x) prevous proof) e Decomposton: And so OLS essentall decomposes actuals nto two uncorrelated parts, predcteds and resduals: = ˆ + uˆ and ˆ ρ ˆˆ = Ths result wll prove useful later u Examples n Excel and Stata Let s frst do ths n Excel Open the bodfatxlsx fle n Excel Generate the x- scatterplot of Brozek v wgt, and add trendlne You should see somethng lke: Case wgt Brozek wgt-wbar Brozek-Bbar product 545 6 (467) (634) 564 Brozek = 67x - 9995 735 69 6 (567) (4) 683 3 54 46 (49) 566 (4) 4 8475 9 5 583 (84) (4683) 5 845 78 533 886 479 6 5 6 4 333 66 55 7 8 9 8 6 3 8 76 8 (9) (64) 795 3 9 9 5 8 (384) (67) 985 933 (694) (349) 865 75 733 (44) (8379) 6 85 378 (44) (387) 3 85 5 58 56 46 4 55 8 633 86 49 5 8775 7 883 76 437 5 5 5 3 35 4 6 675 5 (67) 56 (56) For Brozek and wgt, compute sample means, varances, standard devatons, as well as the covarance and correlaton, and appl the varous formulae for the OLS slope and ntercept estmates You should get somethng lke: 8
OLS: SLR Analss (Rev d v3) Sample Varances Sample Cov Sample Corr Slope estmates 8637 68 3967 63 S/Sxx 67 StDevs 939 775 corr*(s/s) 67 Sum Squares Sum Intercept estmate Means 7894 8938 6,7944 5,79 35,5755 Bbar-b*wbar (9995) Case wgt Brozek wgt-wbar Brozek-Bbar product 545 6 (467) (634) 564 735 69 (567) (4) 683 3 54 46 (49) 566 (4) 4 8475 9 583 (84) (4683) 5 845 78 533 886 479 So who knew? The Excel Trendlne s generated b OLS! Runnng regressons n Excel You can also run the OLS regresson n Excel usng Data/Data Analss/Regresson (ou ma have to load the Data Analss Tool-Pak (go to Optons/Add-Ins) SUMMARY OUTPUT Regresson Statstcs Multple R 636 R Square 37596 Adjusted R Square 37346 Standard Error 635 Observatons 5 ANOVA df SS MS F Sgnfcance F Regresson 5,669 5,669 56 595E-7 Resdual 5 9,499 3764 Total 5 5,79 Coeffcents Standard Error t Stat P-value Lower 95% Upper 95% Intercept (9995) 389 (48) 39776E-5 (474) (5899) wgt 67 3 7 595E-7 358 877 Same OLS slope and ntercept! 9
OLS: SLR Analss (Rev d v3) Now for Stata bcuse bodfat Contans data from http://fmwwwbcedu/ec-p/data/wooldrdge/bodfatdta obs: 5 vars: 4 6 Nov 4:4 sze: 4,68 -------------------------------------------------------------------------------------- storage dspla value varable name tpe format label varable label -------------------------------------------------------------------------------------- Case nt %g Case Brozek double %g 457/denst - 44 wgt double %g weght (lbs) -------------------------------------------------------------------------------------- Sorted b: reg Brozek wgt Source SS df MS Number of obs = 5 -------------+---------------------------------- F(, 5) = 56 Model 5669335 5669335 Prob > F = Resdual 949937 5 3763963 R-squared = 376 -------------+---------------------------------- Adj R-squared = 3735 Total 57966 5 6757635 Root MSE = 635 ------------------------------------------------------------------------------ Brozek Coef Std Err t P> t [95% Conf Interval] -------------+---------------------------------------------------------------- wgt 6788 3765 7 357578 876598 _cons -99955 38956-48 -4739-58998 ------------------------------------------------------------------------------ predct bhat scatter bhat Brozek wgt 3 4 5 5 5 3 35 weght (lbs) Ftted values 457/denst - 44 Use the summarze, correlaton and dspla commands to generate the OLS slope and ntercept estmates:
OLS: SLR Analss (Rev d v3) summ Brozek wgt Varable Obs Mean Std Dev Mn Max -------------+--------------------------------------------------------- Brozek 5 893849 775856 45 wgt 5 78944 93896 85 3635 corr Brozek wgt, covar Brozek wgt -------------+------------------ Brozek 6758 wgt 3967 86373 slope coeffcent : rato of sample covar to sample var d 3967 / 86373 6795 ntercept estmate: d 893849-6795 * 78944-999545 corr Brozek wgt Brozek wgt -------------+------------------ Brozek wgt 63 slope coeffcent : sample corr * rato of sample standard devatons d 63 * 775856 / 93896 6734 Verf that the correlaton of Brozek wth wgt s the same as the correlaton of Brozek wth bhat corr Brozek bhat wgt Brozek bhat wgt -------------+--------------------------- Brozek bhat 63 wgt 63 Capture the resduals and verf that the are uncorrelated wth the predcteds (bhats) and as well wth the explanator varable wgt predct resds, res corr bhat wgt resds bhat wgt resds -------------+--------------------------- bhat wgt resds - -
OLS: SLR Analss (Rev d v3) Evaluate the elastct assocated wth the estmated OLS coeffccents: d 6788*78944/ 893849 577696 Or just run the margns command rght after the reg command reg Brozek wgt margns, eex(_all) atmeans Condtonal margnal effects Number of obs = 5 Model VCE : OLS Expresson : Lnear predcton, predct() e/ex wrt : wgt at : wgt = 78944 (mean) ------------------------------------------------------------------------------ Delta-method e/ex Std Err t P> t [95% Conf Interval] -------------+---------------------------------------------------------------- wgt 57769 8333 9 75 7857 ------------------------------------------------------------------------------