Econometrics I Lecture 3: The Simple Linear Regression Model Mohammad Vesal Graduate School of Management and Economics Sharif University of Technology 44716 Fall 1397 1 / 32
Outline Introduction Estimating parameters Properties of OLS Functional form and interpretation Mean and variance of OLS estimators Reference: Wooldridge, Ch 2; Stock and Watson, Ch 4. 2 / 32
Motivation In much of economics we want to know the relationship between two (or more) variables. So far, we have learned the definitions of joint and conditional distributions, the conditional expectation function, how to test a hypothesis about a distribution parameter. In much of the remaining of this course we study linear regression as one method for investigating such relations. Today we start with the Simple Linear Regression Model 3 / 32
Definition In modeling the impact of x on y we must allow for other factors influencing y specify a functional form for the relationship and make sure we are capturing the causal effect! Simple regression model y = β 0 + β 1 x + u y: dependent variable x: independent (regressor, explanatory) variable u: error term - factors that affect y other than x β 0 and β 1 intercept and slope parameters β 1 = y x if u x =0 4 / 32
Example Wheat yield and guaranteed purchase price yield = β 0 + β 1 price + u u could include rainfall, land productivity, temperature, use of fertilizer, farmer s ability/knowledge linearity one unit change in price has the same effect on yield regardless of the starting price. Does this sound reasonable? Could the regression framework accommodate other alternatives? 5 / 32
Question of causality When can we be sure that the regression captures the causal effect of x on y? If when x changes, the other factors are held constant (i.e. u = 0) then we are fine! In other words we want x and u to be unrelated. Is it enough to assume Corr(x, u) = 0? rules out linear dependence but could be related non-linearly! What about E(u x) = E(u) (conditional mean independence)? Average value of u does not depend on x. We can assume E(u) = 0! Why? But this is close to impossible. example: when price increases, use of fertilizer would change! 6 / 32
Population regression function (PRF) Assuming E(u x) = 0 we can write E (y x) = β 0 + β 1 x E (y x) is the population regression function. in the population, average y changes linearly with x. this is the systematic part of y (the part we can explain!). 7 / 32
Outline Introduction Estimating parameters Properties of OLS Functional form and interpretation Mean and variance of OLS estimators 8 / 32
How to estimate intercept and slope? Consider a random sample {(x 1, y 1 ),..., (x n, y n )} Data comes from PRF y = β 0 + β 1 x + u; therefore we can write y i = β 0 + β 1 x i + u i How can we derive good estimators for β 0 and β 1? Method of moments Least squares 9 / 32
Method of moments Method of moments estimators: use moment conditions to estimate β 0 and β 1 The two assumptions we made give us such restrictions 1. E(u) = 0 E(y β 0 β 1 x) = 0 2. E(xu) = E(xE(u x)) = 0 E(x(y β 0 β 1 x)) = 0 Sample counterparts of restrictions 1 n 1 n n y i β 0 β 1 x i = 0 (1) i=1 n x i (y i β 0 β 1 x i ) = 0 (2) i=1 10 / 32
Method of moments - cont. Find the estimator for β 0 from (1) ˆβ 0 = 1 n n i=1 = ȳ ˆβ 1 x y i ˆβ 1 1 n n i=1 Use this in (2) to find the estimator for β 1 1 n ( x i y i (ȳ n 1 x) ˆβ ) 1 x i i=1 = 0 n n x i (y i ȳ) ˆβ 1 x i (x i x) = 0 i=1 i=1 x i ˆβ 1 = ˆβ 1 = n i=1 x i(y i ȳ) n i=1 x i(x i x) Cov(x, y) V ar(x) 11 / 32
Least squares Could derive the same estimators using the least squares method. define fitted values ŷ i = ˆβ 0 + ˆβ 1 x i residual û i = y i ŷ i Choose ˆβ 0 and ˆβ 1 so the sum of squared residuals is minimized n û 2 i = i=1 n (y i ˆβ 0 ˆβ ) 2 1 x i i=1 Intuitively fit the best line through the data points F.O.C.s give the same results as before! We often use the term Ordinary Least Squares (OLS) to refer to these estimates/estimators. 12 / 32
Graphical representation of OLS 13 / 32
Regression line ŷ = ˆβ 0 + ˆβ 1 x gives OLS regression line also called sample regression function (SRF). this is compared to PRF E (y x) = β 0 + β 1 x 14 / 32
Example - CEO salary CEO salary and return on equity Fitted regression Causality? salary = β 0 + β 1 roe + u ˆ salary = 963.2 + 18.5roe 15 / 32
Example - Causality 16 / 32
Outline Introduction Estimating parameters Properties of OLS Functional form and interpretation Mean and variance of OLS estimators 17 / 32
Algebraic properties Deviations from regression line sum up to zero n i=1 ûi = 0 Correlation between deviations and regressors is zero n i=1 x iû i = 0 Sample averages of y and x lie on the regression line ȳ = ˆβ 0 + ˆβ 1 x 18 / 32
Goodness of fit How well does the explanatory variable explain the dependent variable? Regression as a decomposition: y i = ŷ i + û i Decomposition of total variation SST = SSE + SSR Total sum of squares: SST = n i=1 (y i ȳ) 2 Explained sum of squares: SSE = n Residual sum of squares: SSR = n i=1 (ŷ i ȳ) 2 i=1 û2 i R-squared R 2 = SSE SST = 1 SSR SST This is NOT a measure of causality! percentage of sample variation in y that is explained by x. 19 / 32
Outline Introduction Estimating parameters Properties of OLS Functional form and interpretation Mean and variance of OLS estimators 20 / 32
Functional form and interpretation of coefficients Linear: In the simple model y = β 0 + β 1 x + u, 1 unit increase in x leads to β 1 unit increase in y. Semi-log (semi-elasticity): log y = β 0 + β 1 x + u 1 unit increase in x leads to 100β 1 percent increase in y. Log-log (elasticity): log y = β 0 + β 1 log x + u, 1 percent increase in x leads to β 1 percent increase in y. see table 2.3 in Wooldridge (2013). 21 / 32
Meaning of linear regression The regression equation is linear in parameters. Regressors could enter in any form. E.g. y = β 0 + β 1 x + β 2 x 2 + u Example of non-linear regression model Logit regression y = 1 1+exp( (β 0+β 1x)) 22 / 32
Outline Introduction Estimating parameters Properties of OLS Functional form and interpretation Mean and variance of OLS estimators 23 / 32
Properties of OLS estimators Are OLS estimators unbiased? What is their variance? ˆβ 0 = ȳ ˆβ 1 x ˆβ 1 = n i=1 (x i x)(y i ȳ) n i=1 (x i x) 2 ( ) ( ) ( ) ( ) E ˆβ1, E ˆβ0, V ar ˆβ1, V ar ˆβ0? 24 / 32
OLS assumptions A1: PRF is y = β 0 + β 1 x + u A2: Random sample {(x i, y i ) : i = 1,..., n} y i = β 0 + β 1 x i + u i A3: Sample variation in explanatory variable n i=1 (x i x) 2 > 0 A4: Conditional mean independence E (u x) = 0 25 / 32
Unbiasedness Under A1-A4 OLS estimators are unbiased! The mean of the distribution of ˆβ 1 and ˆβ 0 coincides with the( true ) value of the( parameters. ) E ˆβ1 = β 1 and E ˆβ0 = β 0. Proof: see Wooldridge (2013). 26 / 32
Variance of estimators How far can we expect our estimates to be away from the true population values on average? Add another assumption to simplify derivations A5: Homoskedasticity V ar(u x) = σ 2 Independence of u and x would give homoskedasticity but A4 doesn t. 27 / 32
Homoskedasticity 28 / 32
Heteroskedasticity 29 / 32
Variance of OLS estimators Under A1-A5 we have V ar( ˆβ 1 ) = V ar( ˆβ 0 ) = σ 2 n i=1 (x i x) 2 σ 2 n i=1 x2 i n n i=1 (x i x) 2 higher error variance increases the variance of estimators. higher variability of the explanatory variable reduces the variance of estimators. Proofs. 30 / 32
Errors vs. residuals We want to estimate V ar( ˆβ i ), i.e. σ 2. errors (unobserved): u i = y i β 0 β 1 x i residuals (calculated): û i = y i ˆβ 0 ˆβ 1 x i Under A1-A5, an unbiased estimator for σ 2 is ˆσ 2 = 1 n 2 n i=1 û 2 i = SSR n 2 standard error of ˆβ 1 is therefore estimated from se( ˆβ 1 ) = ˆσ n i=1 (x i x) 2 31 / 32
Summary In this topic we learned What the simple linear regression is. How we estimate the intercept and slope coefficients. Properties of the OLS estimators. Assumptions required for validity of OLS estimators. Next sessions extend the model to have several explanatory variables. What are the general properties of OLS estimators? 32 / 32