Linear Regression with 1 Regressor. Introduction to Econometrics Spring 2012 Ken Simons

Size: px

Start display at page:

Download "Linear Regression with 1 Regressor. Introduction to Econometrics Spring 2012 Ken Simons"

Barry Cook
6 years ago
Views:

1 Linear Regression with 1 Regressor Introduction to Econometrics Spring 2012 Ken Simons

2 Linear Regression with 1 Regressor 1. The regression equation 2. Estimating the equation 3. Assumptions required for OLS 4. Sampling distribution of the OLS estimators 5. Confidence intervals for one coefficient 6. Hypothesis tests for one coefficient 7. If X is a 0-1 variable 8. R 2 and standard error of the regression 9. Heteroskedastic vs. homoskedastic errors

3 1. Regression Line Want to know average change in Y as X changes E.g., Average test scores rise how much if studentteacher ratio falls by 1? Simple way to show this relation: Y = β 0 + β 1 X where β 0 and β 1 are constants Then: ΔY = β 1 ΔX Or: dy dx =! 1

4 Capital Investment and Current Equipment True Relationship for SIC Industries in the US in 1995 Capital Investment, $1000 per Employee Equipment, $1000 per Employee

5 Capital Investment and Current Equipment True Relationship for SIC Industries in the US in 1995 Capital Investment, $1000 per Employee What are X and Y? β 0 and β 1? If equipment increases by $100,000 per employee, investment increases by how much? Equipment, $1000 per Employee

6 Regression Equation The regression line is: Y = β 0 + β 1 X where β 0 and β 1 are constants Complications: 1. Other influences, besides X, may affect Y Divide influences: Non-random part β 0 Random part (with mean zero) u 2. X, Y, and u are different for each individual i sampled Equation for each i, with other influences, is: Y i = β 0 + β 1 X i + u i So E(Y i X i ) = β 0 + β 1 X i and ΔE(Y i X i ) = β 1 ΔX i

7 Capital Investment and Current Equipment True Relationship and Data for SIC Industries in the US in 1995 Capital Investment, $1000 per Employee Equipment, $1000 per Employee

8 Capital Investment and Current Equipment True Relationship and Data for SIC Industries in the US in 1995 Capital Investment, $1000 per Employee What are i s? What are X i and Y i s? What is E(Y i X i )? What are u i s? Equipment, $1000 per Employee

9 2. Estimating the Equation Usually, don t know the true relationship (For the preceding example, I made it up) Estimate the true relationship What are the coefficients β 0 and β 1? How to estimate? A good way, ordinary least squares regression (OLS): Consider fitting errors, between estimated line and data Vertical errors from estimated line to data points Minimize sum of squared errors

10 Capital Investment and Current Equipment 459 SIC Industries in the US in 1995 Capital Investment, $1000 per Employee of the 459 errors are shown possible estimated line Equipment, $1000 per Employee

11 Minimizing Sum of Squared Errors For our estimates of! 0 and! 1, call them ˆ! 0 and ˆ! 1 Estimated regression line is Ŷ i = ˆ! 0 + ˆ! 1 X i Actual data are X i, Y i for each i Errors in fitting are: Y i " Ŷ i = Y i " ˆ! 0 " ˆ! 1 X i Sum of squared errors is: n # i=1 ( Y i " ˆ! 0 " ˆ! 1 X i ) 2 For this sum to be minimized, must have: ( ) 2 ( ) n $ Y i " ˆ! 0 " ˆ! n # 1 X i = "2# Y i " ˆ! 0 " ˆ! 1 X i = 0 $ˆ! 0 i=1 i=1 n $ ( Y i " ˆ! 0 " ˆ! 1 X i ) 2 n # = "2#( Y i " ˆ! 0 " ˆ! 1 X i )X i = 0 $ˆ! 1 i=1 i=1 Rearrange and divide by n to get: Y-ˆ! 0 " ˆ! 1 X = 0 and 1 n n # X i Y i " ˆ! 0 X " ˆ! n # X n i = 0 i=1 Solve these simultaneous equations to get: ˆ! 1 = n # i=1 i=1 (X i " X)(Y i " Y) n # i=1 ˆ! 0 = Y " ˆ! 1 X (X i " X) 2 If you know the method for 2- variable optimization, try checking the 2nd order conditions.

12 Capital Investment and Current Equipment 459 SIC Industries in the US in 1995 Capital Investment, $1000 per Employee estimated line Equipment, $1000 per Employee

13 Capital Investment and Current Equipment 459 SIC Industries in the US in 1995 Capital Investment, $1000 per Employee true line estimated line Goal is to pick estimated line very close to true line. Only by sheer luck would we be exactly right Equipment, $1000 per Employee

14 3. Assumptions OLS has very good properties, if the following assumptions are true: 1. E(u i X i )=0 I.e., the other influences on Y i not systematically related to X i 2. (X i,y i ) are independently & identically distributed Guaranteed for a random sample In experiments where you choose X i non-randomly, wise to randomly assign treatments X i to subjects i in a way to which the subjects & experimenters are blind (i.e., they can t tell which subjects received which treatment) 3. X i and u i have four moments 0<E(X i4 )<, 0<E(u i4 )< Guaranteed if the data can never exceed some maximum

15 4. Sampling Distribution of OLS Estimators Given the preceding assumptions, ˆ! 0 and ˆ! 1 have good properties. Using the law of large numbers and central limit theorem shows, like for a sample average Y estimating µ Y, 1. ˆ! 0 and ˆ! 1 are consistent estimators of! 0 and! 1 I.e., as n " #, we are certain to get the right answer 2. ˆ! 0 and ˆ! 1 have asymptotic normal distributions I.e., as n " #, they are normally distributed In particular: ˆ! 0 is N(! 0,Var( ˆ! 0 )) and ˆ! 1 is N(! 1,Var( ˆ! 1 )) where Var(ˆ! 1 )= 1 n Var( ˆ! 0 ) = 1 n var[(x i -µ X )u i ] [var(x i )] 2 var(h i u i ) [E(H i 2 )] 2, where H i = 1$ % ' & µ X ( E(X 2 * i )) X i

16 5. Confidence Intervals 95% confidence interval for β 1 : ˆ! 1 ±1.96 "SE(ˆ! 1 ) For other confidence levels, use a different number than 1.96 Note t=1.96 corresponds to p=.05 (.025 in each tail of normal distribution) Try looking this up, pp ; then look up appropriate number for other confidence levels (e.g. 90% has.05 in each tail; 99%; 99.5%) Inside back cover of book are the numbers for 90%, 95%, 99%

17 6. Hypothesis Tests If β 1 had a hypothetical value, β 1,0, what is chance of observing data as averse to the hypothesis as the actual data? That chance is a p-value Compute t and use normal distribution As we did for sample means But now use ˆ! t = 1 "! 1,0 SE( ˆ! see text eqns for SE( ˆ! 1 ) 1 ) p-value is 2 times probability to the right of t in standard normal distribution (pp in text; t=1.96 yields p=.05)

19 One-Sided Hypothesis Tests Rarely, only viable alternative to null hypothesis is β 1 >0 (or β 1 <0) instead of β 1 0 Then, use just 1 side of normal distribution to compute p-value 1 times probability to the right of t in standard normal distribution This is easily abused Scientists often do this when they hypothesize β 1 >0 Easier to reject null hypothesis But other scientists may hypothesize β 1 <0 Then for all scientists combined, using 1-sided tests, null hypotheses incorrectly rejected 10% of the time at level p<.05! When does this problem occur? When β 1 >0 (or β 1 <0) is not the only viable alternative! (How can we know anyway?)

20 7. If X is a 0-1 Variable A 0-1 variable is called a dummy variable or indicator variable Then, β 1 is not a slope (how could you draw a regression line?) Instead, β 1 tells how much larger is E(Y i ) when X i is 1 instead of 0 Example: Expected wage for people who did (X=1) or did not (X=0) earn a college degree Same as the difference of means you estimated in chapter 3

21 8. R 2 How much of the variance in Y does the regression explain? n TSS = "(Y i! Y) 2 total sum of squares (sample variance times n-1) i=1 n ESS = "(Ŷi! Y) 2 explained sum of squares n i=1 (explaining data differences from sample mean) 2 RSS= " û i = "(Y i! Ŷi )2 residual sum of squares (or SSR ) i=1 n i=1 TSS=ESS + RSS So, R 2 = ESS TSS is the fraction of the sample variance explained (0 # R 2 # 1)

22 Note on R 2 Compare R 2 values from different regressions only if they have the same Y i data, same i=1,,n Otherwise TSS is different, so comparison is nonsense A small R 2 value says you explain little of variance in Y i, but that may be ok For some applications, explaining a tiny bit is helpful How much can be explained depends on random variation in u i If Var(u i ) is large, little can be explained

23 Standard Error of the Regression Estimate of standard deviation of u i Formula used to compute: Estimated standard deviation of u i (across all i) : SER = 2, where s 2 u ˆ = 1 n " ˆ n! 2 u 2 = RSS i n! 2 s ˆ u Consistent estimator of the standard deviation of u i i =1

24 9. Heteroskedastic vs. Homoskedastic Errors The u i s are often called errors The assumptions made do not require any particular values for Var(u i X i ) The values can be different, called heteroskedastic errors Often are different in practice, e.g. different variances in wage for people who did/did not complete college If know Var(u i X i ) is same for all X i Called homoskedastic errors OLS estimators of β 0 and β 1 are then best (lowest-variance) among unbiased estimators with formulas linear in Y 1,,Y n ; best among all unbiased estimators if also each u i is normally distributed Can also use a more efficient formula to estimate If know values of heteroskedastic Var(u i X i ) SE(ˆ! 0 ) and SE(ˆ! 1 ) Can use weighted least squares method for more efficient estimates In practice, Var(u i X i ) often differs with X i, and we don t know the values

25 Using Regression Software Most software uses an outdated formula for Yields biased tests and confidence intervals unless errors happen to be homoskedastic The better (heteroskedastic-robust) methods are relatively new, so not the default in software To turn on these better methods, look for a robust standard errors option Robust standard errors are also called Heteroskedasticity-robust standard errors Eicker-Huber-White standard errors Or just Huber-White or White standard errors SE(ˆ! 0 ) and SE(ˆ! 1 ) Because they were independently proposed by Eicker (1967), Huber (1967), and White (1980) Don t forget to turn on this option in your software!

26 You have learned: 1. The regression equation 2. Estimating the equation 3. Assumptions required for OLS 4. Sampling distribution of the OLS estimators 5. Confidence intervals for one coefficient 6. Hypothesis tests for one coefficient 7. If X is a 0-1 variable 8. R 2 and standard error of the regression 9. Heteroskedastic vs. homoskedastic errors

27 What s Next Do assignment for chapters 4-5 ( assignment ch4-5 ) Due Thursday Feb. 16 Next we ll continue with empirical application of regression Reminder: chapter 17 and chapter 18 exercises can be done for extra credit (you now have the background to do chapter 17)

WISE MA/PhD Programs Econometrics Instructor: Brett Graham Spring Semester, Academic Year Exam Version: A

WISE MA/PhD Programs Econometrics Instructor: Brett Graham Spring Semester, 2016-17 Academic Year Exam Version: A INSTRUCTIONS TO STUDENTS 1 The time allowed for this examination paper is 2 hours. 2 This