SLR Models Estmaton Those OLS Estmates Estmators (e ante) v. estmates (e post) The Smple Lnear Regresson (SLR) Condtons -4 An Asde: The Populaton Regresson Functon B and B are Lnear Estmators (condtonal on the s) OLS estmators are unbased! (under SLR.-SLR.4) but B s not alone OLS estmators have a varance SLR.5 - Homoskedastcty MSE/RMSE (Goodness of Ft) and Standard Errors OLS estmators are BLUE! (under SLR.-SLR.5) (separate handout) Those OLS Estmates ) Recall the brth of OLS estmates: a) You have a dataset consstng of n observatons of (, ) y: { }, y =,,... n. b) You beleve that ecept for random nose n the data, there s a lnear relatonshp between the s and the y s: y β + β and want to estmate the unknown parameters β (ntercept parameter) and β (slope parameter). c) Adoptng OLS, you estmate β and β by mnmzng sum squared resduals (SSRs): mn SSR = ( y ( )) β + β wrt β and β d) For the gven sample, the OLS estmates of the unknown ntercept and slope parameters are: ) Slope: ˆ ( )( y y) ( ) y y y y β = = = w, where ( ( n ) S ( w = and w = so the w ' s ( n ) S sum to and are proportonal to the square of the -dstance from. y y ) Recall that s the slope of the lne connectng (, y ) to the sample means pont, (, y ). ) Intercept: ˆ β ˆ = y β.
SLR Models Estmaton Estmates (e post) v. Estmators (e ante) ) epost (actual; after the event): After the data set s generated, OLS provdes numerc estmates of the slope and ntercept parameters, for the gven dataset (estmates are numbers, not random varables). a) After we have drawn the sample {, y } ) Slope estmate: ˆ ( )( y y) β =, and ( ) Intercept estmate: ˆ β ˆ = y β., we have the OLS estmates: 3) eante (before the event): But pror to the generaton of the data, the s and y s are random varables, X s and Y s, and OLS provdes a rule for estmatng the OLS coeffcents (for any realzed data set). a) We call these rules estmators. Estmators wll take on dfferent values dependng on the actual drawn sample, the actual 's and y's. Accordngly, estmators are random varables. b) The X ' s and Y ' s are random varables, and OLS provdes us wth slope and ntercept estmators: ( X X)( Y Y) ( X X) Y () Slope estmator: B = = ( X X) ( X X) () Intercept estmator: B = Y BX j j 4) To revew notaton, we have: a) Random varables (upper case letters): X's and Y's b) Data (lower case letters): 's and y's c) True parameters: β and β d) Parameter estmators (random varables; upper case letters): B and B e) Parameter estmates (estmated coeffcents): ˆ β ˆ and β
SLR Models Estmaton The Smple Lnear Regresson (SLR) Condtons -4 5) SLR. Lnear model (DGM): Y = β + βx + U, where X, Y and U are random varables and β and β are unknown parameters to be estmated. a) Ths s sometmes referred to as the Data Generaton Mechansm (DGM), as t descrbes the process by whch the data are assumed to have been generated. b) Notce that X, U and Y are now random varables, reflectng the random nature of the DGM. c) U s the uneplaned (or unobserved) error term (or dsturbance), and captures other factors (ecluded from the model) that eplan Y: ) = β β U Y X ) Put dfferently: U s the part of Y not eplaned/generated by the lnear functon of X. 6) SLR. Random samplng: the sample {(, )} y s a random sample; the th elements n the sample, (, y ), s the realzaton of two random varables ( X, Y ), where Y = β + β X + U. 7) SLR.3 Sample varaton n the ndependent varable: the ' s are not all the same value 8) SLR.4 U has zero condtonal mean: EU ( X= ) = for all a) EU ( X= ) = for any mples that: ) EU ( ) = (U has mean zero) () If the epected value of U s condtonal on any value of, then the overall epected value of U, whch wll be a weghted average of the condtonal epectatons of U, wll also be. () Put dfferently: E( U ) = prob( X = ) E( U X = ) = prob( X = ) = ) Cov( X, U ) = (X and U are uncorrelated) () Cov( X, U) E( ( X µ X)( U µ U) ) E( ( X µ X) U) U = E ( XU ) µ E ( U ) = E ( XU ) = prob( X = ) E( U X = ) = = snce µ = X = prob( X = ) E( U X = ) = prob( X = ) = At tmes we wll consder the X values to be eogenously gven, n whch case the eplanatory varable s not random. Recall that f X and U are ndependent then Cov( X, U ) =. A covarance of, however, does not mply ndependence, but rather than X and U do not move together n much of a lnear way. 3
SLR Models Estmaton () Notce the connecton to Omtted Varable Bas, whch s drven by correlaton between U and X. An Asde: The Populaton Regresson Functon 9) Under these assumptons (specfcally SLR. and SLR.4): a) β β β β EY ( X= ) = + + EU ( ) = + +, so the epected value of Y gven s EY ( X= ) = β + β. ) Populaton Regresson Functon (PRF): EY ( X= ) = β + β a) Ths traces out the condtonal means (condtonal on the values of X, the partcular values of the RHS varable) of the dependent varable Y. B and B are Lnear Estmators (condtonal on the s) ) The OLS slope estmator, B, s lnear n the Y ' s (condtonal on the s), snce we can epress the estmator as: ( ) ( ) a) B = by, where b = = ( ) ( n ) S j b) Note that condtonal on the s means that we are takng the values as gven, and not as random varables wth values to be determned. ) And the OLS ntercept estmator s also lnear n the Y ' s (condtonal on the s) snce: a) B = Y by = b Y n n 3) So condtonal on the 's, B and B are lnear estmators. 4
SLR Models Estmaton OLS estmators are unbased! (under SLR.-SLR.4) Who saw ths comng? 4) Gven assumptons/condtons SLR.-SLR.4, B and B, the OLS estmators of the ntercept and slope parameters, are unbased estmators (so for each, the epected value of the estmator s the true parameter value). a) We'll be provng ths below. b) The trck n the proof s to assume a partcular set of sample ' s, and to show that for any such sample, the OLS estmators wll be unbased. And snce ths s true for any sample of ' s, t must be true n epectaton. ) Ths s sometmes referred to as the Law of Iterated Epectaton. We wll skp the proof of ths Law but the ntuton s pretty straghtforward, yes? and you saw t n acton above n the proof that EU ( ) = gven SLR.4. 5) Condtonal on the s, the OLS estmators (random varables note the captal B's below) are defned by: a) ( )( Y Y) ( )( Y Y) ( ) Y Y B = = =, ( ) ( n ) S ( n ) S ( Y Y) and so B = w, ( ) where w ( = ( n ) S are non-negatve weghts that sum to, w =. b) B = Y B 6) An nterestng result: a) Assume SLR.-SLR.4. ) Then Y = β + βx + U, and µ Y = β + βµ X (snce EU ( ) = ) ) β β β β Cov( X, Y ) = Cov( X, + X + U ) = Cov( X, X ) + Cov( X, U ) = Cov( X, X ) snce Cov( X, U ) =. Cov( X, Y ) Cov( X, Y ) ) But then β =. and β = µ βµ = µ µ. Cov( X, X ) Y X Y Cov( X, X ) X b) Notce the resemblance to the OLS estmators! 5
SLR Models Estmaton 7) The OLS slope estmator B s unbased! - E( B ) = β Y Y a) Step - E = β: To evaluate the epected value of B condtonal on the s, we Y Y need to determne E, for each. But E( Y ) = β + β + EU ( ) = β + β (gven SLR.4). And snce EY ( s ' ) = EY ( ) = ( β + β) n n = β + β, we have: Y Y ( β + β ) ( β + β) β( ) E = = = β. ( ) ( ) b) Step - ( ' ) E B s = β ( B n an unbased estmator of β, condtonal on the s): Y Y Y Y E B ' s = E w = we = wβ = β weghts sum to w = ). ) Snce ( ) (snce the c) Step 3 - E( B ) = β : And snce E( B s ' ) = β for all s, E( B ) = β. 3 8) The OLS ntercept estmator B s also unbased! - E( B ) a) Step - ( ' ) = β E B s = β ( B n an unbased estmator of β, condtonal on the s): ( ' ) ( ) ( ) E B s = E Y E B = β + β β = β. b) Step - E( B ) = β : And snce E( B s ' ) = β for all s, E( B ) = β. 9) OLS LUE (gven SLR.-SLR.4): So gven the SLR condton -4, the OLS slope and ntercept estmators are lnear unbased estmators (LUEs) of the unknown parameter values! 4 ) Who saw ths comng? Who ever thought that process of mnmzng SSRs would lead to unbased estmators? 3 Ths last step s an applcaton of the Law of Iterated Epectatons. 4 But remember that they are only lnear condtonal on the 's. 6
SLR Models Estmaton But B s not alone!!! there are an nfnte number of unbased slope estmators (gven SLR.-SLR.4) ) Any weghted average of the slopes of the lnes connectng the dataponts to the samples means wll also be a LUE of the slope parameter: a) Consder the estmator defned by Y Y α X X, where α =. b) Then condtonal on the s: E Y Y Y Y α = αe = αβ = β, snce α =. c) And snce ths s the case for all s, we have an unbased estmator of β. d) Snce we only requre that the α ' s sum to one, we have an nfnte number of unbased slope estmators (as we vary the α ' s ). ) Test your understandng! From before, you know that gven SLR.-SLR.4, E Y β β EY s ' = β β +. Use these results to show that each of the ( ) = + and ( ) followng wll be unbased slope estmators, condtonal on the 's. And accordngly, by the Law of Iterated Epectatons, unbased slope estmators overall: Y Y a) B = X X 's. Answer: E( B s ' ) 5 b).5 Y Y B.5 Y Y = + X X X5 X 5 c).9 Y Y B. Y Y = + X X X5 X 5 d).5 Y Y B.5 Y Y = X X X5 X Y Y B = 5 e) X X5 Y Y Y Y B =.5 +.5 5 3 7 f) X X5 X3 X7 ( ) ( ) β + β β + β β ( = = = β, all ( ) ( ) 3) And so gettng to BLUE (Best Lnear Unbased Estmators) wll be all about fndng the LUE(s?) (amongst the many) wth the mnmum varance. 7
SLR Models Estmaton OLS estmators (B and B ) have varances 4) Varances of the OLS estmators: Yes, because they are estmators, the OLS estmators, B and B, are random varables, wth a jont dstrbuton, means, varances and a covarance. The sample you are workng wth s just one of many possble samples that you could have drawn. 5) An eample: Y = +.5X + U, X Unform[,] and U N(,). a) The followng show the results from, samples, each wth observatons generated by the random process above wth one slope and ntercept estmate per sample. Dstrbuton of OLS ntercept and slope estmates. summ best best Varable Obs Mean Std. Dev. Mn Ma -------------+--------------------------------------------------------- best, -.77699.776-7.5879 5.9449 best,.548.948-5.437995 6.7959 Densty...3.4 - -5 5 b est. Densty...3.4-5 5 b est. 6) The means of the, estmates are qute close to the true parameters values but notce the large varaton n slope and ntercept estmates, drven by the random nature of the DGM and dependng on the partcular sample that you are workng wth. 7) Worth repeatng!!!: Because they are estmators, the OLS estmators B and B are random varables, wth a jont dstrbuton, means, varances, and a covarance. Dfferent samples wll generate dfferent ntercept and slope estmates. Who knows f your sample s representtve? your estmates could n fact be not at all close to the true parameter values. It all depends on your sample. 8
SLR Models Estmaton SLR.5 Homoskedastcty 8) SLR.5: Homoskedastcty (constant condtonal varance of the error term) a) To derve the varances of the estmators, we make one addtonal assumpton: SLR.5: Var( U X ) σ = = for all b) SLR.5 holds f U s ndependent of X, so that Var( U X = ) = Var( U ) = σ. c) Heteroskedastcty: the condtonal varances are not all the same. Eample: Newton real estate sales prces and lot szes (heteroscedastcty) Source SS df MS Number of obs = 84 -------------+---------------------------------- F(, 8) = 69.4 Model.374e+3.374e+3 Prob > F =. Resdual 5.4e+3 8.7873e+ R-squared =.97 -------------+---------------------------------- Adj R-squared =.943 Total 6.776e+3 83.8e+ Root MSE = 4.e+5 ------------------------------------------------------------------------------ prce Coef. Std. Err. t P> t [95% Conf. Interval] -------------+---------------------------------------------------------------- lotsze 4.99 5.7549 8.3. 3.393 5.98 _cons 37448.4 6384.9 6.. 5348.8 49578 ------------------------------------------------------------------------------.e+6.e+6 3.e+6 5 5 5 LotSze Prce Ftted values Resduals - 5 5 5 LotSze predcteds v. actuals resduals v. lot sze 9
SLR Models Estmaton A smple test for heteroskedastcty: Regress the squared resduals on the RHS varable. Are there more complcated test? Of course!. predct resd, res. gen resd=resd^. reg resd lotsze Source SS df MS Number of obs = 84 -------------+---------------------------------- F(, 8) = 43.3 Model 6.3833e+4 6.3833e+4 Prob > F =. Resdual 4.89e+5 8.4833e+3 R-squared =.34 -------------+---------------------------------- Adj R-squared =.93 Total 4.8e+5 83.736e+3 Root MSE = 3.9e+ ------------------------------------------------------------------------------ resd Coef. Std. Err. t P> t [95% Conf. Interval] -------------+---------------------------------------------------------------- lotsze 3.3e+7 46344 6.56..e+7 3.94e+7 _cons -.57e+ 5.59e+ -.8.5 -.67e+ -4.73e+ ------------------------------------------------------------------------------ Varance of the OLS Estmators (assumng SLR.-SLR.5) 9) If SLR.5 holds, n addton to SLR.-SLR.4, then we have the followng varances of the : OLS estmators, condtonal on the partcular sample of { } a) b) Var( B ) = Var( B ) σ ( j σ = n ( j and StdDev B sd B Var B ( ) = ( ) = ( ) = 3) Here s the proof for the varance of the slope estmator: σ ( ( a) From above we have B = ( Y = α Y, where α ( n ) S =. ( n ) S b) Snce the Y ' s are ndependent, the varance of the sum s the sum of the varances, and Var( B ) = Var( αy ) = αvar( Y ) = ασ = σ α c) But ( α = ( n ) S. and so
SLR Models Estmaton ( j [( n ) S ] [( n ) S ] ( ) ( n ) S α = = = = ( n ) S ( ) σ σ d) Therefore: Var( B ) = = ( ) ( n ) S j j ) Note that ths ncreases wth ncreases n the error varance, σ, and wth decreases n the varaton of the ndependent varable. Makes sense? ) Where does ths varance come from? The estmator s always just the OLS estmator, so all of the varaton s comng from the dfferent possble data samples generated by the DGM. (See the Ecel smulatons.) MSE/RMSE (Goodness of Ft) and Standard Errors 3) Mean Squared Error (MSE): Typcally, we don t know the actual value of the varance σ. SSR But we can estmate t wth the: ˆ σ = = MSE. n a) Recall that MSE was one of our Goodness of Ft metrcs n OLS/SLR Assessment. b) Under SLR.-SLR.5 and condtonal on the s, MSE = σˆ wll be an unbased estmator of the varance, σ, of the error term U (the homoskedastc error). 5 MSE 3) s an unbased estmator of Var( B ) ( n ) S a) Snce ˆ σ = MSE s an unbased estmator of σ and snce MSE MSE = n S ( ) ( ) s an unbased estmator of Var( B ). Var( B ) = σ ( 33) RMSE: The standard error of the regresson, sometme called the Root MSE (or RMSE), s SSR the square root of ths: ˆ σ = = MSE = RMSE. n a) If we have RMSE = ˆ σ, then we can calculate the standard error of B : ˆ σ RMSE RMSE ( ) = ( ) = = =. S n ) StdErr B se B ( ) ( ) b) Ths s an estmate of StdDev( B) = sd( B), and s useful for constructng confdence ntervals for, and testng hypotheses about, β, the true slope parameter n the DGM. And now you see where that std. err. formula came from n OLS/SLR Assessment!, 5 For the proof, see the tet.