Lecture 1: OLS derivations and inference

Transcription

1 Lecture 1: OLS derivations and inference Econometric Methods Warsaw School of Economics (1) OLS 1 / 43

2 Outline 1 Introduction Course information Econometrics: a reminder Preliminary data exploration 2 OLS: theoretical reminder Point estimation Measuring precision Model quality diagnostics under OLS 3 Multicollinearity (1) OLS 2 / 43

3 Outline 1 Introduction 2 OLS: theoretical reminder 3 Multicollinearity (1) OLS 3 / 43

4 Course information Course information lecturers: & Marcin Owczarczuk my website: (lecture slides, exercise les, literature, contact) nal grade: homework (50%) + written exam (50%); details: website (1) OLS 4 / 43

5 Course information Course information lecturers: & Marcin Owczarczuk my website: (lecture slides, exercise les, literature, contact) nal grade: homework (50%) + written exam (50%); details: website (1) OLS 4 / 43

6 Course information Course information lecturers: & Marcin Owczarczuk my website: (lecture slides, exercise les, literature, contact) nal grade: homework (50%) + written exam (50%); details: website (1) OLS 4 / 43

7 Econometrics: a reminder Why econometrics? investigation of relationships nding parameter values in economic models (eg elasticities) confronting economic theories with data forecasting simulating policy scenarios (1) OLS 5 / 43

8 Econometrics: a reminder Why econometrics? investigation of relationships nding parameter values in economic models (eg elasticities) confronting economic theories with data forecasting simulating policy scenarios (1) OLS 5 / 43

9 Econometrics: a reminder Why econometrics? investigation of relationships nding parameter values in economic models (eg elasticities) confronting economic theories with data forecasting simulating policy scenarios (1) OLS 5 / 43

10 Econometrics: a reminder Why econometrics? investigation of relationships nding parameter values in economic models (eg elasticities) confronting economic theories with data forecasting simulating policy scenarios (1) OLS 5 / 43

11 Econometrics: a reminder Why econometrics? investigation of relationships nding parameter values in economic models (eg elasticities) confronting economic theories with data forecasting simulating policy scenarios (1) OLS 5 / 43

12 Econometrics: a reminder Data structure 1 time series (industrial production, , monthly index) 2 cross section (exit polls with 1000 respondents) 3 longitudinal data (quarterly GDP dynamics in EU states, ) (1) OLS 6 / 43

13 Econometrics: a reminder Data structure 1 time series (industrial production, , monthly index) 2 cross section (exit polls with 1000 respondents) 3 longitudinal data (quarterly GDP dynamics in EU states, ) (1) OLS 6 / 43

14 Econometrics: a reminder Data structure 1 time series (industrial production, , monthly index) 2 cross section (exit polls with 1000 respondents) 3 longitudinal data (quarterly GDP dynamics in EU states, ) (1) OLS 6 / 43

15 Econometrics: a reminder Selection of explanatory variables theory, institutional setup, expert evaluation mechanical methods data mining correlation-matrix-based treatments from general to specic approach variable transformations (1) OLS 7 / 43

16 Econometrics: a reminder Selection of explanatory variables theory, institutional setup, expert evaluation mechanical methods data mining correlation-matrix-based treatments from general to specic approach variable transformations (1) OLS 7 / 43

17 Econometrics: a reminder Selection of explanatory variables theory, institutional setup, expert evaluation mechanical methods data mining correlation-matrix-based treatments from general to specic approach variable transformations (1) OLS 7 / 43

18 Econometrics: a reminder Selection of explanatory variables theory, institutional setup, expert evaluation mechanical methods data mining correlation-matrix-based treatments from general to specic approach variable transformations (1) OLS 7 / 43

19 Preliminary data exploration Preliminary exploration (1) distribution data errors, outliers? (1) OLS 8 / 43

20 Preliminary data exploration Preliminary exploration(2) time series graph trend? seasonality? volatility clustering? (1) OLS 9 / 43

21 Preliminary data exploration Preliminary exploration (3) scatterplot dependency, functional form, transformations? (1) OLS 10 / 43

22 Preliminary data exploration Preliminary exploration (4) correlations? (1) OLS 11 / 43

23 Preliminary data exploration Example (1/5) Student satisfaction survey Master students of Applied Econometrics at Warsaw School of Economics in Winter semester 2016/2017 were asked about their satisfaction from studying to be evaluated from 0 to 100 In addition, their average note from previous studies and their sex were registered 1 What kind of data is this? Cross-section, time series, panel? Frequency? Micro- or macroeconomic? 2 How can we quickly visualise a hypothesised causality from average note to satisfaction from studying? 3 Does such a relationship seem to be there? 4 How can sex of the respondent potentially aect the satisfaction from studies or the relationship in question? How can we visualise this? 5 Bottom line, what is the right specication of the linear regression model? (1) OLS 12 / 43

24 Outline 1 Introduction 2 OLS: theoretical reminder 3 Multicollinearity (1) OLS 13 / 43

25 Point estimation Linear regression model y i = β 0 + β 1 x 1,i + β 2 x 2,i + + β k x k,i + ε i = β 0 [ ] β 1 1 x1,i x 2,i x k,i β 2 + ε i = x i β + ε i β k Vector of parameters [ ] T β 0 β 1 β 2 β k is unknown Minimize the dispersion of ε i around zero, as measured eg by n ε 2 i t=1 (1) OLS 14 / 43

26 Point estimation Ordinary Least Squares (OLS) ε 2 i i=1 S = n = n i=1 (y i β 0 β 1 x 1,i β 2 x 2,i β k x k,i ) 2 min β 0,β 1, Denote: y = y 1 y 2 y n FOC: S β = 0 1 x 1,1 x 2,1 x k,1, X = 1 x 1,2 x 2,2 x k,2 1 x 1,n x 2,n x k,n and obtain:, β = β 0 β 1 β 2 β k β = ( X T X) 1 X T y (1) OLS 15 / 43

27 Point estimation Proof S = n ε 2 i = ε T ε = (y Xβ) T (y Xβ) = i=1 = y T y β T X T y y T Xβ + β T X T Xβ = = y T y 2y T Xβ + β T X T Xβ (2 and 3 component were transposed scalars, so they were equal) S β = 0 yt y β + βt X T Xβ β = 0 According to the rules of matrix calculus: 2y T X + β T ( 2X T ) X = 0 β T ( X T ) ( X = y T X X T ) X β = X T y β = ( X T X ) 1 X T y 2yT Xβ β (1) OLS 16 / 43

28 Point estimation Example (2/5) Student satisfaction survey 1 Run the regression model with an automated command in R 2 Write the equation and try to interpret the parameters Be careful it's tricky! (Why?) 3 Manually replicate the parameter estimates (1) OLS 17 / 43

29 Measuring precision Estimator as a random variable ˆβ is an estimator of the true parameter value β (function of the random sample choice) samples, and hence the values of ˆβ, can be dierent estimator as a (vector) random variable has its variance(-covariance matrix) ˆβ = ˆβ 0 ˆβ 1 ˆβ 2 ( ) Var ˆβ = ˆβ ( k ) var ˆβ 0 ( ) cov ˆβ 0, ˆβ 1 ( cov ˆβ0, ˆβ ) 2 ( ) cov ˆβ 0, ˆβ 1 ( ) var ˆβ 1 ( cov ˆβ1, ˆβ ) 2 ( ) cov ˆβ 0, ˆβ 2 ( ) cov ˆβ 1, ˆβ 2 ( ) var ˆβ2 ( ) var ˆβ k (1) OLS 18 / 43

30 Measuring precision Estimator as a random variable ˆβ is an estimator of the true parameter value β (function of the random sample choice) samples, and hence the values of ˆβ, can be dierent estimator as a (vector) random variable has its variance(-covariance matrix) ˆβ = ˆβ 0 ˆβ 1 ˆβ 2 ( ) Var ˆβ = ˆβ ( k ) var ˆβ 0 ( ) cov ˆβ 0, ˆβ 1 ( cov ˆβ0, ˆβ ) 2 ( ) cov ˆβ 0, ˆβ 1 ( ) var ˆβ 1 ( cov ˆβ1, ˆβ ) 2 ( ) cov ˆβ 0, ˆβ 2 ( ) cov ˆβ 1, ˆβ 2 ( ) var ˆβ2 ( ) var ˆβ k (1) OLS 18 / 43

31 Measuring precision Estimator as a random variable ˆβ is an estimator of the true parameter value β (function of the random sample choice) samples, and hence the values of ˆβ, can be dierent estimator as a (vector) random variable has its variance(-covariance matrix) ˆβ = ˆβ 0 ˆβ 1 ˆβ 2 ( ) Var ˆβ = ˆβ ( k ) var ˆβ 0 ( ) cov ˆβ 0, ˆβ 1 ( cov ˆβ0, ˆβ ) 2 ( ) cov ˆβ 0, ˆβ 1 ( ) var ˆβ 1 ( cov ˆβ1, ˆβ ) 2 ( ) cov ˆβ 0, ˆβ 2 ( ) cov ˆβ 1, ˆβ 2 ( ) var ˆβ2 ( ) var ˆβ k (1) OLS 18 / 43

32 Measuring precision Variance-covariance matrix of a random vector Denition: Var (β) = E {[β } E (β)] [β E (β)] T For a centered variable, ie E (ε) = 0, this denition simplies: Var (ε) = E ( εε T ) (1) OLS 19 / 43

33 Measuring precision OLS estimator: properties ˆβ = ( X T ) 1 X X T y is an estimator (function of the sample) of the true, unknown values β (population / data generating process) Under certain conditions (ia E(X T ε) = 0 E(εε T ) = σ 2 I ), the OLS estimator is: ) unbiased: E (ˆβ = β consistent: ˆβ converges to β with growing n ecient: least possible estimator variance (ie highest precision) (1) OLS 20 / 43

34 Measuring precision OLS estimator: properties ˆβ = ( X T ) 1 X X T y is an estimator (function of the sample) of the true, unknown values β (population / data generating process) Under certain conditions (ia E(X T ε) = 0 E(εε T ) = σ 2 I ), the OLS estimator is: ) unbiased: E (ˆβ = β consistent: ˆβ converges to β with growing n ecient: least possible estimator variance (ie highest precision) (1) OLS 20 / 43

35 Measuring precision OLS estimator: properties ˆβ = ( X T ) 1 X X T y is an estimator (function of the sample) of the true, unknown values β (population / data generating process) Under certain conditions (ia E(X T ε) = 0 E(εε T ) = σ 2 I ), the OLS estimator is: ) unbiased: E (ˆβ = β consistent: ˆβ converges to β with growing n ecient: least possible estimator variance (ie highest precision) (1) OLS 20 / 43

36 Measuring precision Variance of the error term (1) 1 Variance of the error term (scalar): ˆσ 2 = 1 n (k+1) Why such a formula if the general formula is n Var(X ) = 1 (x n 1 i x) 2? i=1 First of all note that ε = 0 (prove it on your own) n ε 2 i i=1 Second, we need to know why 1 turned into (k + 1) in the denominator (1) OLS 21 / 43

37 Measuring precision Variance of the error term (2) By Your intuition, what is the standard deviation in the following dataset of 3 observation? Without a correction in denominator: ] Var = [(3 2) 2 + (2 2) 2 + (1 2) 2 = (1) OLS 22 / 43

38 Measuring precision Variance of the error term (2) By Your intuition, what is the standard deviation in the following dataset of 3 observation? Without a correction in denominator: ] Var = [(3 2) 2 + (2 2) 2 + (1 2) 2 = (1) OLS 22 / 43

39 Measuring precision Variance of the error term (3) The intuition behind the standard deviation of 1 is build upon an implicit, graphical calibration of mean based on the data sample With an adequate correction for thatin denominator: ] Var = [(3 2) 2 + (2 2) 2 + (1 2) 2 = = 1 (1) OLS 23 / 43

40 Measuring precision Variance of the error term (3) The intuition behind the standard deviation of 1 is build upon an implicit, graphical calibration of mean based on the data sample With an adequate correction for thatin denominator: ] Var = [(3 2) 2 + (2 2) 2 + (1 2) 2 = = 1 (1) OLS 23 / 43

41 Measuring precision Variance of the error term (4) When X is directly observed, the terms like (x i x) consume one degree of freedom (there is one x estimated before) When ε is not observed, the terms ε i = y i ˆβ 0 ˆβ 1 x 1i ˆβ k x ki consume k + 1) degrees of freedom (there are k + 1 elements in vector ˆβ estimated before) (1) OLS 24 / 43

42 Measuring precision Variance-covariance matrix of the estimator ) [ ) ) ] T Var (ˆβ = E (ˆβ β (ˆβ β = { [( = E X X) T 1 ] [ T ( X y β X X) T 1 ] } T T X y β = { [( = E X X) T 1 ] [ ( T X (Xβ + ε) β X X) T 1 ] } T T X (Xβ + ε) β = [ ( ) = E X T 1 ( ) X T X ε ε T X X T 1 ] X = = ( X X) T 1 T X E ( εε ) ( T X X X) T 1 = = ( X X) T 1 ( T X σ 2 IX X X) T 1 = = σ ( ) 2 X T 1 ( T X X X X X) T 1 = = σ ( ) 2 X T 1 X ( ) Empirical counterpart: Var ˆβ = ˆσ ( 2 X X) T 1 [di,j ] (k+1) (k+1) (1) OLS 25 / 43

43 Measuring precision Standard errors of estimation Standard ( ) errors of estimation (vector for each parameter): s ˆβ 0 = ( ) d 1,1 s ˆβ 1 = ( ) d 2,2 s ˆβ 2 = d 3,3 Calculating SE 1 estimate parameters, 2 compute the empirical error terms, 3 estimate their variance, 4 compute the variance-covariance matrix of the OLS estimator, 5 compute the SE as a square root of its diagonal elements (1) OLS 26 / 43

44 Measuring precision Standard errors of estimation Standard ( ) errors of estimation (vector for each parameter): s ˆβ 0 = ( ) d 1,1 s ˆβ 1 = ( ) d 2,2 s ˆβ 2 = d 3,3 Calculating SE 1 estimate parameters, 2 compute the empirical error terms, 3 estimate their variance, 4 compute the variance-covariance matrix of the OLS estimator, 5 compute the SE as a square root of its diagonal elements (1) OLS 26 / 43

45 Measuring precision t-tests for variable signicance t-student test H 0 : β i = 0, ie i-th explanatory variable does not signicantly inuence y H 1 : β i 0, ie i-th explanatory variable does not signicantly inuence y Test statistic: t = ˆβ i s( ˆβ is distributed as t (n k 1) 1) p-value<α reject H 0 p-value>α do not reject H 0 (1) OLS 27 / 43

46 Measuring precision Example (3/5) Student satisfaction survey 1 Compute the tted values and the error terms 2 Use this result to estimate the variance of the error term 3 Estimate the variance-covariance matrix of the ˆβ estimates 4 Derive the standard errors from this matrix 5 Replicate and interpret the t statistics and the p-values (1) OLS 28 / 43

47 Model quality diagnostics under OLS R-squared (1) (1) OLS 29 / 43

48 Model quality diagnostics under OLS R-squared (2) (1) OLS 30 / 43

49 Model quality diagnostics under OLS R-squared (3) (1) OLS 31 / 43

50 Model quality diagnostics under OLS R-squared (4) (1) OLS 32 / 43

51 Model quality diagnostics under OLS R-squared (5) (1) OLS 33 / 43

52 Model quality diagnostics under OLS R-squared (6) (1) OLS 34 / 43

53 Model quality diagnostics under OLS R-squared (7) R 2 [0; 1] is a share of y t volatility explained by the model in total y t volatility: T (y t ȳ) 2 = T (ŷ t ȳ) 2 + T (y t ŷ t ) 2 R 2 = t=1 t=1 t=1 T (ŷ t ȳ) 2 t=1 T (y t ȳ) 2 t=1 Standard goodness-of-t measure in OLS regressions with a constant (1) OLS 35 / 43

54 Model quality diagnostics under OLS Wald test statistic Wald test H 0 : β 1 = β 2 = = β k = 0, ie no explanatory variable inuences y H 1 : i β i 0, at least 1 explanatory variable inuences y Test statistic: F = distributed as F (k, T k 1) R 2 /k (1 R 2 )/(T k 1) (1) OLS 36 / 43

55 Model quality diagnostics under OLS Adjusted R-squared R 2 = }{{} R 2 fit k ( ) 1 R 2 T (k + 1) }{{} penalty for overparametrization (1) OLS 37 / 43

56 Model quality diagnostics under OLS Example (4/5) Student satisfaction survey 1 Interpret the F-test result 2 Replicate the F statistic and its p-value manually 3 Interpret the R-squared 4 Replicate the R-squared and adjusted R-squared manually (1) OLS 38 / 43

57 Outline 1 Introduction 2 OLS: theoretical reminder 3 Multicollinearity (1) OLS 39 / 43

58 Multicollinearity Multicollinearity Regressors are not independent: some are a linear combination of others (extreme case), then X T X is a singular matrix and we cannot compute β = ( X T X) 1 X T y some are highly correlated (usual case), then X T X may not be singular, but its diagonal elements still close to zero then the diagonal elements of (and so the standard errors) ( X T X) 1 extremely high (1) OLS 40 / 43

59 Multicollinearity Multicollinearity diagnostics 1 correlation matrix only bilateral relationships no cut-o value above which the problem can be considered serious 2 ination variance factor (VIF) for regressor j VIF j = 1 1 R j 2 where Rj 2 is R-squared from the regression of variable j on the rest of the explanatory variables limit value: 10, above multicollinearity 3 condition index κ = λ max λ min where λ denotes eigenvalues of the matrix derived from X T X by division of every cell (i, j) by the product of diagonal elements (i, i) and (j, j) limit value: 20, above multicollinearity (1) OLS 41 / 43

60 Multicollinearity Multicollinearity solutions strengthen the precision of estimation by expanding sample size, removing a variable, imposing restrictions or calibrating the parameter manually increase the diagonal values in X T X (ridge regression) squeeze the common variance of the collinear variables into a lower number of new, independent ones (principal components) (1) OLS 42 / 43

61 Multicollinearity Multicollinearity solutions strengthen the precision of estimation by expanding sample size, removing a variable, imposing restrictions or calibrating the parameter manually increase the diagonal values in X T X (ridge regression) squeeze the common variance of the collinear variables into a lower number of new, independent ones (principal components) (1) OLS 42 / 43

62 Multicollinearity Multicollinearity solutions strengthen the precision of estimation by expanding sample size, removing a variable, imposing restrictions or calibrating the parameter manually increase the diagonal values in X T X (ridge regression) squeeze the common variance of the collinear variables into a lower number of new, independent ones (principal components) (1) OLS 42 / 43

63 Multicollinearity Example (5/5) Student satisfaction survey Investigate the issue of multicollinearity in the proposed model (1) OLS 43 / 43