Econometrics I KS. Module 1: Bivariate Linear Regression. Alexander Ahammer. This version: March 12, 2018

Transcription

1 Econometrics I KS Module 1: Bivariate Linear Regression Alexander Ahammer Department of Economics Johannes Kepler University of Linz This version: March 12, 2018 Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 1 / 40

2 Getting started What do we need to do Econometrics? Econometrics tries to answer economic questions using statistical methods. What do we need to do econometrics? 1 An economic question How does education affect wages? Does reduced class size improve students test scores? What determines a country s GDP growth? 2 Data Cross-sectional data (y i, x i), i = 1,..., n Time-series data (y t, x t), t = 1,..., T Panel data (y it, x it), i = 1,..., n; t = 1,..., T 3 A functional relationship Linear relation: y i = β 0 + β 1x i + u i Nonlinear and additive: y i = g(x i, β 1) + u i Nonlinear and non-additive: h(y i, x i, u i, β 1) Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 2 / 40

3 Getting started What do we need to do Econometrics? 4 An object of interest Typically we are interested in the slope parameter β 1 For non-linear models there are other interesting effects 5 Assumptions Marginal effects at the mean Marginal effects at representative values Average marginal effects Are the u i independent? There can be spatial or time correlation. Are the u i identically distributed? Are they stationary? Is u i correlated with x i? Do we have an endogeneity problem? 6 An estimator In the linear case, the OLS estimator is most popular. In non-linear cases, usually Maximum Likelihood is used. For semi-parametric models: GMM and many more Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 3 / 40

4 Preliminaries The simple linear regression model In econometrics, we are often interested in describing a linear relationship between a dependent variable y i and k covariates x i = (x i1, x i2,..., x ik ). A simple bivariate (k = 1) regression model for the population reads where β 0 is the intercept, β 1 is the slope parameter, and u is an error term with E(u) = 0. y = β 0 + β 1 x + u, (1) For an i.i.d. sample (y i, x i ) of size n: y i = β 0 + β 1 x i + u i, i = 1,..., n (2) Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 4 / 40

5 Preliminaries The simple linear regression model Holding u fixed, β 1 = y/ x. The linearity of (1) implies that a one-unit change in x has the same effect on y, regardless of the initial value of x. Crucial assumption necessary for the ceteris paribus interpretation: the expected value of u does not depend on the value of x, i.e. E(u x) = E(u) = 0. (3) Then we can write the population regression function as E(y x) = β 0 + β 1 x (4) A one-unit increase in x changes the expected value of y by β 1. Implication: for any x the distribution of y is centered around E(y x). Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 5 / 40

6 Getting started A real world example 1 An economic question How does the share of foreign-born in the population affect GDP? 2 Data Data from the OECD 2011 cross-section Descriptive statistics: Variable Obs. Mean Std. dev. Min. Max. Share of foreign-born in % GDP per capita in $ Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 6 / 40

7 Getting started A real world example 100 GDP vs. share of foreigners LUX GDP per capita in $ POL 20 TUR MEX NOR USA DEN NED AUT GERSWE IRE FIN BEL ICE FRA ITA UK SPA CZE GREPOR SLO HUN EST CAN NZL ISR CHE AUS % of population foreign-born Source: own calculations based on OECD.Stat data Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 7 / 40

8 Getting started A real world example 3 A functional relationship We assume a linear relationship: GDP = β 0 + β 1share + u (5) where GDP is per capita GDP and share is the share of foreign-born in the total population. Can you think of other plausible relationships? Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 8 / 40

9 Getting started A real world example 100 GDP vs. share of foreigners LUX GDP per capita in $ POL 20 TUR MEX NOR USA DEN NED AUT GERSWE IRE FIN BEL ICE FRA ITA UK SPA CZE GREPOR SLO HUN EST CAN NZL ISR CHE AUS % of population foreign-born Source: own calculations based on OECD.Stat data Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 9 / 40

10 Getting started A real world example 100 GDP vs. share of foreigners LUX GDP per capita in $ POL 20 TUR MEX NOR USA DEN NED AUT GERSWE IRE FIN BEL ICE FRA ITA UK SPA CZE GREPOR SLO HUN EST CAN NZL ISR CHE AUS % of population foreign-born Source: own calculations based on OECD.Stat data Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 10 / 40

11 Getting started A real world example 4 An object of interest Our regression model reads: GDP = β 0 + β 1share + u (6) The intercept β 0 is precisely the (hypothetical) level of GPD when the share of foreign-born is zero. Is that interesting? Why do we need an intercept? We are rather interested in the slope parameter β 1. β 1 is the effect of a one percentage point increase in the share of foreign-born on GDP. What sign do you expect for β 1? How do we estimate β 1? Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 11 / 40

12 Getting started A real world example 5 Assumptions Assumption SLR.1 Linear in parameters The population model is linear: GDP = β 0 + β 1 share + u. Assumption SLR.2 Random sampling We have a random sample {(GDP i, share i ) i = 1,..., n} that follows the population model: GDP i = β 0 + β 1 share i + u i. Assumption SLR.3 Sample variation in the expl. variable The sample outcomes on share, namely {share i, i = 1,..., n}, are not all the same value Var(share i ) 0. Assumption SLR.4 Zero conditional mean The error u has an expected value of zero given any values of the explanatory variable, i.e., E(u i share i ) = 0 i = 1,..., n. Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 12 / 40

13 Getting started A real world example 6 An estimator Given that we have a linear model, we choose Ordinary Least Squares (OLS) to obtain an estimate for β 1 (and β 0). OLS picks ˆβ 0 and ˆβ 1 that minimize the sum of squared residuals: arg min ˆβ 0, ˆβ 1 n û 2 i (7) where is the residual for observation i. Hence, arg min ˆβ 0, ˆβ 1 û i = y i ˆβ 0 ˆβ 1x 1 (8) n (y i ˆβ 0 ˆβ 1x 1) 2 (9) 7 We will show later that whenever assumptions SLR.1 SLR.4 hold, OLS is unbiased. If errors are homoskedastic, OLS is efficient as well (BLUE). Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 13 / 40

14 Intermezzo Deriving the OLS estimate Define a fitted value for y when x = x i as ŷ i = ˆβ 0 + ˆβ 1 x i. (10) Define a residual for observation i as the difference between the actual (observed) y i and its fitted value, û i = y i ŷ i = y i ˆβ 0 ˆβ 1 x i. (11) Now choose ˆβ 0 and ˆβ 1 to make the sum of squared residuals, i.e. as small as possible, i.e. n û 2 i = arg min ˆβ 0, ˆβ 1 n (y i ˆβ 0 ˆβ 1 x i ) 2 (12) n (y i ˆβ 0 ˆβ 1 x 1 ) 2 (13) Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 14 / 40

15 Intermezzo Deriving the OLS estimate In order to solve this minimization problem, the partial derivatives of (13) with respect to ˆβ 0 and ˆβ 1 must be zero: n û2 i ˆβ 0 n û2 i ˆβ 1 Note that (14) can be written as = 2 = 2 n (y i ˆβ 0 ˆβ 1 x i ) = 0 (14) n (y i ˆβ 0 ˆβ 1 x i )x i = 0 (15) where ȳ = n 1 n y i and x = n 1 n x i. ȳ = ˆβ 0 + ˆβ 1 x, (16) Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 15 / 40

16 Intermezzo Deriving the OLS estimate Figure: Residuals and fitted values. Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 16 / 40

17 Intermezzo Deriving the OLS estimate Figure: OLS minimizes the sum of squared residuals. Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 17 / 40

18 Intermezzo Deriving the OLS estimate Figure: OLS minimizes the sum of squared residuals. Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 18 / 40

19 Intermezzo Deriving the OLS estimate From (16) we can see that once we have ˆβ 1, ˆβ 0 can easily be obtained: ˆβ 0 = ȳ ˆβ 1 x. (17) Plugging (17) into (15) yields n x i [y i (ȳ ˆβ 1 x) ˆβ 1 x i ] = 0 (18) which, upon rearrangement, gives n x i (y i ȳ) = ˆβ n 1 x i (x i x) (19) From basic properties of the summation operator [see (A.7) and (A.8) in the n textbook] we know x i(x i x) = n (x i x) 2 and n x i(y i ȳ) = n (x i x)(y i ȳ). Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 19 / 40

20 Intermezzo Deriving the OLS estimate Therefore, ˆβ 1 can be written as ˆβ 1 = n x i(y i ȳ) n (x i x) 2 (20) or as ˆβ 1 = n (x i x)(y i ȳ) n (x i x) 2 (21) Equation (21) is simply the sample covariance between x and y divided by the sample variance of x: Cov(x, y) ˆβ 1 = Var(x) (22) So, if the correlation between x and y is positive, ˆβ 1 will be positive as well. Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 20 / 40

21 Getting started A real world example, cont d Back to our data these are the first 15 countries in the dataset:. list in 1/15, sep(0) country foreign GDP 1. AUS AUT BEL CAN CZE DEN EST FIN FRA GER GRE HUN ICE IRE ISR Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 21 / 40

22 Getting started A real world example, cont d Now we perform the OLS regression:. reg GDP foreign Source SS df MS Number of obs = 30 F( 1, 28) = Model Prob > F = Residual R-squared = Adj R-squared = Total Root MSE = GDP Coef. Std. Err. t P> t [95% Conf. Interval] foreign _cons Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 22 / 40

23 Getting started A real world example, cont d 100 GDP vs. share of foreigners LUX GDP per capita in $ POL 20 TUR MEX NOR USA DEN NED AUT GERSWE IRE FIN BEL ICE FRA ITA UK SPA CZE GREPOR SLO HUN EST CAN NZL ISR CHE AUS % of population foreign-born Source: own calculations based on OECD.Stat data Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 23 / 40

24 Getting started A real world example, cont d. predict yhat, xb. predict uhat, r. list in 1/15, sep(0) country foreign GDP yhat uhat 1. AUS AUT BEL CAN CZE DEN EST FIN FRA GER GRE HUN ICE IRE ISR Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 24 / 40

25 Ordinary Least Squares Now we look at the OLS mechanics in a bit more depth. Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 25 / 40

26 Ordinary Least Squares Statistical properties Some important (algebraic) properties of OLS: Residuals and fitted values Each ŷ i lies on the OLS regression line. If û i < 0, then ŷ i overpredicts y i, if û i > 0, then ŷ i underpredicts y i. n Note that ûi = 0 by construction! This follows from the first-order condition in (14). n Analogously ûixi = 0, follows from the second f.o.c. (15). Goodness-of-fit We use the R 2 as a measure of how well the explanatory variable x explains the dependent variable y, R 2 = SSE SST = 1 SSR SST where SSE = n (ŷi ȳ)2, SSR = n û2 i, and SST = n (yi ȳ)2. (23) Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 26 / 40

27 Ordinary Least Squares Expected value and variance We will now study expected values and variances of the OLS estimates. Recall the four assumption we made earlier: Assumption SLR.1 Linear in parameters The population model is linear: y = β 0 + β 1x + u. Assumption SLR.2 Random sampling We have a random sample of n observations, {(y i, x i) i = 1,..., n}, that follows the population model: y i = β 0 + β 1x i + u i. Assumption SLR.3 Sample variation in the expl. variable The sample outcomes on x, namely {x i, i = 1,..., n}, are not all the same value Var(x i) 0. Assumption SLR.4 Zero conditional mean The error u has an expected value of zero given any values of the explanatory variable, i.e. E(u i x i) = 0 i = 1,..., n. Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 27 / 40

28 Ordinary Least Squares Expected values Theorem: Unbiasedness of OLS Under assumptions SLR.1 through SLR.4, E( ˆβ 0 ) = β 0, and E( ˆβ 1 ) = β 1 (24) for any values of β 0 and β 1. In other words, ˆβ 0 is an unbiased estimate for β 0, and ˆβ 1 is an unbiased estimate for β 1. Proof. The OLS estimator is given by ˆβ 1 = n (x i x)(y i ȳ) n (x i x) 2 = n (x i x)y i (x i x)ȳ n (x i x) 2 (25) Because n (x i x)ȳ = ȳ n (x i x) = ȳ(n x n x) = 0, we have ˆβ 1 = n (x i x)y i n (x i x) 2 (26) Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 28 / 40

29 Now, define φ i = x i x n (x i x) 2 which has the following properties: n n φ i = (x i x) n (x i x) = 0 2 (27) n φ i x i = n (x i x)x i n (x i x) 2 = 1 (28) Substituting y i = β 0 + β 1 x i + u i and φ i into equation (26) gives ˆβ 1 = = n φ i (β 0 + β 1 x i + u i ) (29) n φ i β 0 + n φ i β 1 x i + n φ i u i (30) Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 29 / 40

30 Due to (27) and (28), equation (30) becomes ˆβ 1 = 0 + β 1 + n φ i u i (31) Taking expectations on both sides gives ( n ) E( ˆβ 1 ) = E(β 1 ) + E φ i u i = E(β 1 ) + n E(φ i u i ) (32) Recall that the population parameter β 1 is deterministic, so E(β 1 ) = β 1. Furthermore, in SLR.4 we have assumed that E(u i ) = 0, thus E( ˆβ 1 ) = β 1 (33) This concludes the proof for β 1. The proof for β 0 is left as an exercise. Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 30 / 40

31 Ordinary Least Squares Expected variances In addition to knowing that the sampling distribution of ˆβ 1 is centered around β 1 (i.e., ˆβ 1 is unbiased), it is important to know how far we can expect ˆβ 1 to be away from β 1 on average. We are interested in the variance or the standard deviation of the OLS estimator. This helps to think about efficiency of estimators. If we assume that the unobservable term u has a constant (and finite) variance (homoskedasticity), it is easier to describe the variance of the OLS estimator. The opposite of homoskedasticity is heteroskedasticity. Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 31 / 40

32 Ordinary Least Squares Expected variances Homoskedastic error Heteroskedastic error, Var[u] = 0.1*inc 3 u 0 0 u Income Income Figure: Simulated error structures, with Var(u) being conditionally constant (left graph), and Var(u) being a function of one of the explanatory variables, Var(u) = 0.1 inc 3. Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 32 / 40

33 Ordinary Least Squares Expected variances Assumption SLR.1 Linear in parameters The population model is linear: y = β 0 + β 1x + u. Assumption SLR.2 Random sampling We have a random sample {(y i, x i) i = 1,..., n} that follows the population model: y i = β 0 + β 1x i + u i. Assumption SLR.3 Sample variation in the expl. variable The sample outcomes on x, namely {x i, i = 1,..., n}, are not all the same value Var(x i) 0. Assumption SLR.4 Zero conditional mean The error u has an expected value of zero given any values of the explanatory variable, i.e. E(u i x i) = 0 i = 1,..., n. Assumption SLR.5 Homoskedasticity The error u has the same variance given any value of the explanatory variable., i.e. Var(u i x i) = σ 2 i = 1,..., n. Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 33 / 40

34 Ordinary Least Squares Expected variances Assumption SLR.5: Homoskedasticity The error u has the same variance given any value of the explanatory variable. In other words Var(u x) = σ 2 (34) where σ 2 R is some constant. This assumption does not affect (un)biasedness of OLS. If Var(u x) depends on x, the error term is said to exhibit heteroskedasticity. See example 2.13 in Wooldridge (2013). Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 34 / 40

35 Ordinary Least Squares Expected variances Figure: What do we see here? [Source: Wooldridge (2013), Figure 2.8] Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 35 / 40

36 Ordinary Least Squares Expected variances Figure: What do we see here? [Source: Wooldridge (2013), Figure 2.9] Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 36 / 40

37 Ordinary Least Squares Expected variances Theorem: Sampling variances of the OLS estimators Under assumptions SLR.1 through SLR.5, ( Var( ˆβ σ 2 ) 1 ) = n (x 1 x) 2 (35) and Var( ˆβ 0 ) = ( σ 2 n 1 n ) x2 i n (x 1 x) 2 (36) where these are conditional on the sample values {x 1,..., x n } Proof. See Wooldridge (2013), pp These formulas are invalid in the case of heteroskedasticity. Note that the error variance σ 2 is unobserved. Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 37 / 40

38 Ordinary Least Squares Expected variances Goal: Obtain standard errors. We know that σ 2 = E(u 2 ), so n 1 n u2 i is an unbiased estimator of σ2. Replacing u i with the estimated û i, we have n 1 n It turns out that this estimator is biased. Bias diminishes as n. û2 i. Note that there are only n 2 degrees of freedom in the OLS residuals due to the two first-order conditions (14) and (15), so ˆσ 2 = 1 n 2 n û 2 i = SSR n 2 (37) is an unbiased estimator for σ 2. Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 38 / 40

39 Ordinary Least Squares Expected variances Theorem: Unbiased esimation of σ 2 Under assumptions SLR.1 through SLR.5, E(ˆσ 2 ) = σ 2 (38) where ˆσ 2 = 1 n k 1 n û 2 i = SSR n k 1 (39) Proof. See Wooldridge (2013), p. 54. The term (n k 1) is the degrees of freedom adjustment for the general OLS problem with n observations and k independent variables. ˆσ = ˆσ 2 is called the standard error of regression. Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 39 / 40

40 Literature Main reference: Wooldridge, J. M. (2015). Introductory Econometrics: A Modern Approach, 5th ed., South Western College Publishing. Thanks to Martin Halla for providing the basis for these slides. Alexander Ahammer (JKU) Module 1: Bivariate Linear Regression 40 / 40