Basic Linear Model. Chapters 4 and 4: Part II. Basic Linear Model

Size: px

Start display at page:

Download "Basic Linear Model. Chapters 4 and 4: Part II. Basic Linear Model"

Jane Bryant
5 years ago
Views:

1 Basic Linear Model Chapters 4 and 4: Part II Statistical Properties of Least Square Estimates Y i = α+βx i + ε I Want to chooses estimates for α and β that best fit the data Objective minimize the sum of squared errors Actual error ε I is unobserved For any estimates (a,b), we can estimate the error 1 Basic Linear Model e i =y i -a-bx i --- error for i-th observation Sum of squared errors SSE=Σ i e i = Σ i (y i -a-bx i ) b=ks y /s x = Σ I (y i -N)(x i -n) / Σ I (x i -n) a= N-bn Ordinary least squares or OLS Per capita packs/year Cigarette Consumption and Taxes Tax per pack (cents) 3 4 Cigarette Consumption and Taxes ln(weekly Wages) vs. Education Per capita packs/year Tax per pack (cents) ln(weekly Wages) Years of Education 5 6 1

2 ln(weekly Wages) vs. Education This section: ln(weekly Wages) Examine statistical properties of OLS estimators Is b and unbiased estimate of β? What is the var(b)? Hypothesis tests on b? Years of Education 7 8 Simplify the OLS model Without loss of generality, N= n=0 model: Y i = βx i + ε I b=ks y /s x = Σ I y i x i / Σ I x i In looking at the OLS estimate of b, notice that the estimate is in fact a linear combination of y a= N-bn=0 Let x i / Σ I x i = d i b=ks y /s x = Σ I y i x i / Σ I x i b= Σ i d i y i 9 10 OLS estimates are a class of linear estimators not because the underlying model (Y i = βx i + ε i) ) is linear but because the estimator is linear in the random variable y The random variable y, does however have its own source of variation Y i = βx i + ε I The true source of random variation in the model is the variation generated by ε I The properties of our estimator will be governed by the assumed variation of ε i 11 1

3 Assumptions concerning ε i 1: E[ε i ] = 0 (trivial arbitrary normalization) : Var[ε i ] = σ ε The variance is the same for all obs. 3: Cov[ε i, ε k ] = 0 Errors are not correlated across obs. 4: Cov[x i, ε i ]=0 KEY ASSUMPTION Cov[x i, ε i ]=0 Definition: E[(X i -µ x )(ε i -µ ε )] Does the realization that ε I is above or below average convey any information about x Remember -- ε i is the error in the endogenous variable in the system X is considered exogenous or fixed We want to know the impact of x on y Some detailed algebra The model has hypothesized that x causes y If Cov[x i, ε i ] 0, then as we will see in a moment, then the value y causes x and the assumptions of the model are violated The results is that the estimate b will contain not only information about the impact of x on y but the feedback of y on x We can write b as b= Σ I y i x i / Σ I x i = Σ i d i y i Where d i =x i / Σ I x i = We know that Y i = βx i + ε I Substitute this value into b b= Σ i d i [βx i + ε i ] Re-write equation by taking out the [ ] b= Σ i d i βx i + Σ i d i ε i Remember that β is fixed for all i so we can bring it outside the summation b= β Σ i d i x i + Σ i d i ε I Substitute in the definition for d i d i =x i / Σ I x i

4 Therefore b=β Σ I x i x i /Σ I x i + Σ i x i ε i / Σ I x i b=β Σ I x i /Σ I x i + Σ i x i ε i / Σ I x i b= β + Σ i x i ε i / Σ I x i Divide the numerator and denominator in the second term by (n-1) Σ i x i ε i / Σ I x i = [(1/(n-1)]Σ i x i ε i / [(1/(n-1)]Σ I x i Remember that we have assumed n=0 The first term is the sample correlation between x and ε [(1/(n-1)]Σ i x i ε i =J xε 19 0 b= β + J xε /s x The second term is the sample variance in x [(1/(n-1)]Σ I x i = [(1/(n-1)]Σ(x i -n) = s x Therefore b= β + Σ i x i ε i / Σ I x i b= β + J xε /s x Notice that the sample estimate b contains the true value of β, plus the sample covariance between x and ε If we expect J xε = 0, then b is an unbiased estimate E[b] = β If however, J xε 0, then the estimate of b does not provide accurate information about the true value of β 1 Why? Key question: We assumed the y is endogenous and x is exogenous or that x causes y. If J xε 0, then in some respects, y is causing x and the assumption of our model is incorrect To evaluate whether J xε 0, I like to ask the question Does the realization of ε convey any information about x? If it does, then J xε 0 Does the realization of ε convey any information about x? If it does, then b is not an unbiased estimate We can in theory sign the bias 3 4 4

5 Example 1: Spanking example from Washington Post Take the spanking example from the introduction to this chapter Researchers examines whether spanking (x) causes behavioral problems (y) Model: Y i = βx i + ε i Expect that β>0 Find b>0 kids who were spanked have more behavioral problems Example 1 Is this an accurate reflection of the impact of x on y? Does the realization of ε convey any information about x? Suppose ε>0 a kid has higher than average rate of behavioral problems If we think kids with greater than average behavioral problems (ε>0) are more likely to be spanked, then J xε >0 5 6 Example 1 If J xε >0 And b= β + J xε /s x It must be the case then that b> β The estimate for b reflects not only the impact of spanking on behavioral problems (β>0) but also the fact that kids with more behavioral problems are more likely to be spanked (J xε >0) Example : Human Capital Earnings Function Regression of log wages (y) of years of education We expect β>0 and estimates of b confirm this result Is this an accurate reflection of the impact of x on y? Does the realization of ε convey any information about x? 7 8 Example Example What does ε>0 mean? Someone with above average earnings given their characteristics Earnings could be greater than average for lots of reasons: drive, competitiveness, likeability, inherent intelligence, ability to get along with other people Suppose that people with these same traits are the people who are more likely to have aboveaverage levels of education In this case, that fact that someone has (ε>0) reveals that they are more likely to be a higher educated person and therefore J xε >0 9 If J xε >0 And b= β + J xε /s x It must be the case then that b> β The estimate for b reflects not only the impact of more education on earnings, but the fact that people who have unmeasured traits that are rewarded in the job market may also be the same people who are more likely to get lots of education 30 5

6 Example After 30 yrs of research cannot find too much evidence that J xε is very large b is an unbiased estimate Example 3 Impact of higher state cigarette taxes (x) on per capita cigarette consumption (y) Y i = βx i + ε i Expect that β<0 Find b<0 Is this an accurate estimate? Does the realization of ε convey any information about x? 31 3 Example 3 Suppose a state has lower than average consumption of cigarettes (ε<0) Why might this be? Tobacco producing state History of tobacco consumption Does this reflect anything about the likely values of taxes? Maybe tobacco producing states (VA, NC, KY, SC) have high consumption (because they have a cultural history linked to tobacco) May also be reflected in the fact that they are less likely to tax cigarettes Example 3 Example 3 J xε <0 (high consuming states less likely to tax cigarettes) And b= β + J xε /s x β <0 It must be the case then that b< β<0 -- b is too large of a negative number The large negative coefficient is picking up two factors higher taxes reduce consumption AND high consuming states are less likely to tax cigarettes b is over estimated in absolute value 35 Other research has shown that this simple model overstates by a factor of the impact of taxes on smoking 36 6

7 Example 4 Example 4 Project STAR in TN Wanted to examine the impact of smaller classes on student test scores Paid to reduce average kindergarten class sizes in some schools from 7 to 18 Did so randomly schools were asked whether they wanted to participate some classes were randomly selected to receive funds to take classes of 7 and make 3 classes of 18 (for example) Key schools receiving funds randomly assigned Data collected on treatment and control groups Example 4 Example 4 Basic model regress test scores (y) on teachers/pupil (x) Y i = βx i + ε i Expect that β>0 Find b>0 More teachers produced higher test scores Is this an accurate reflection of the impact of x on y? Does the realization of ε convey any information about x? In this case the answer is NO!! Suppose a class has higher than average performance on exams (ε>0) Does this change the class size we would expect to see? No because teachers/pupil (x) was randomly assigned Realization of ε cannot convey information abut x because x is randomly assigned What is the Var(b)? Given the linear model Y i = α+βx i + ε I The parameter b is an unbiased estimate of the population parameter β We showed above that b=ks y /s x = Σ I (y i -N)(x i -n) / Σ I (x i -n) The estimate b is a function of a random variable (y) so it is also a random variable We showed that under certain conditions, b is an unbiased estimate of the true population parameter β Now, we want to examine the properties of the second moment the variance the estimate b Definition if z is a random variable E[z] = µ z Var[z] = E[(z-E(z)) ] = E[(z - µ z ) ]

8 Var[b] = E[(b E[b]) ] Recall two facts from above We know that b= Σ I (y i -N)(x i -n) / Σ I (x i -n) Which can be re-written as b= β + Σ i x i ε i / Σ I x i We also know that E[b] = β Therefore b-e[b] = b- β = Σ i x i ε i / Σ I x i Therefore E[(b-E[b]) ] = E[(b- β) ] =E[(Σ i x i ε i / Σ I x i ) ] We do not have time to derive this expression but it can be show that Obtaining an Estimate for Var(ε I )=σ ε Var(b) = Var(ε I )/Σ i (x i -n) = σ ε /Σ i (x i -n) The denominator is easy to calculate Sample variation in x Σ i (x i -n) is the numerator in the s x Var(ε I )=σ ε is unknown However, we can obtain an estimate For any observation, we can obtain an estimate of ε I e i =y i -a-bx i We already know that Σ i e i = 0 (1/n) Σ i e i = 0 (sample mean of e is 0) A consistent and unbiased estimate of σ ε is Looking at printout s e=[1/(n-)] Σ i e i = SSE/(n-) Therefore Var(b) can be estimated by replacing Var(ε I )=σ ε with the sample analog s e And therefore Var(b) is estimated to be Var(b) = s e /Σ i (x i -n) SSE = N=918 N- = 916 s e=[1/(n-)] Σ i e i =SSE/(n-) s e = /916 = Root MSE (mean squared error) = Root MSE =.57 = s e

9 Var(b) from the Cigarette/tax example Looking at Printout In most cases, we do not report Var(b) but its square root se(b) or the standard error of b se(b)=s e /[Σ i (x i -n) ] 0.5 Note that s x =[(1/(n-1))Σ i (x i -n) ] 0.5 [Σ i (x i -n) ] 0.5 =s x (n-1) 0.5 =14.317(917) 0.5 = se(b)=s e /[Σ i (x i -n) ] 0.5 =.57/ = 0.05 Root MSE =.57 = s e Hypothesis testing in the OLS model We know that b is an unbiased estimate of β se(b)=s e /[Σ i (x i -n) ] 0.5 s e=[1/(n-)] Σ i e i =[1/(n-)] SSE Much like in chapters 8 and 9, we want to specify the likely values of β given the sampling variation of b and the distribution of the parameter Given that we had to use an estimate for Var(ε I ) -- s e -- rather than the true value, we cannot use the normal distributions to construct confidence intervals Confidence intervals will be based on the t-distribution 51 5 t-distribution Exploit the symmetry of the t-distribution. Define term α = 1 confidence level α = the probability the true value does not equal the values in the confidence interval t(dof) α/ is the value of the t-distribution where only α/ percent of the distribution lies above the value and α/ percent lies below - t(dof) α/ In this case, the appropriate degrees of freedom is n-k n=number of obs k=number of parameters () Therefore, a confidence interval for β is b± t(n-) 0.05 se(b)

10 Looking at printout Hypothesis testing for β b= se(b)=0.05 t(916) 0.05 =1.96 b± t(n-) 0.05 se(b) = ± 1.96(0.05) (-1.41, ) Fairly small interval Suppose we have a hypothesis about β How can we use the confidence interval to test the hypothesis The most basic hypothesis is whether the variable can be distinguished from zero Therefore, the first test is always H o : β= In this case, we can easily reject the null because 0 is not a likely value for β We can also use t-tests to test the hypothesis E[b]= β It can be shown that [b- β]/se(b) is distributed as a t with n- degrees of freedom t= [b- β]/se(b) If the null hypothesis is true, this tstatistic is centered on zero with 95% of the distribution between the values t(n-) 0.05 and t(n-) 0.05 If the estimated value for t= [b- β]/se(b) is between these values, then we cannot reject the null hypothesis because the null is a likely outcome Example 3 from Handout: Impact of weather on house prices If however t = [b- β]/se(b) > t(n-) 0.05 The t-statistic is not a likely outcome under the null hypothesis, and therefore, we reject the null In smoking example, if H o : β=0 t(b) = /0.05 = > t(n-) 0.05 Easily reject null hypothesis Model: Home Prices i = α + β(january Temps) i + ε I Sample of 4 cities DOF = n =

11 b=ks y /s x = (48.008/1.979) = 1.15 a=n -bn = (36.805) = R = SSM/SST = /94495 = Root MSE = S e = = (SSE/(n-)) 0.5 = (85334/40) 0.5 H o : β =0 b= se(b)= % confidence interval N- = 40 t(n-) 0.05 = area in right hand tail of distribution dof b± t(n-) 0.05 se(b) 1.15 ±.0(0.556) = (0.09, 3.143) Can reject null INF How to interpret the coefficient What about a 99% test? For every 1 degree increase in average daily high temperatures, prices go up by 1.15 (thousand) If you move from one city to another and January temperatures increase by 15 degrees, you can expect to pay (15)(1.15) = $17.3 (thousand) more H o : $ = 0 " = = 0.01 "/ = t(n-) = t(40) "/ =.70 t=[b - $]/se(b) = 1.15/ =.0 t < t(40) = cannot reject null

The Simple Linear Regression Model

The Simple Linear Regression Model Lesson 3 Ryan Safner 1 1 Department of Economics Hood College ECON 480 - Econometrics Fall 2017 Ryan Safner (Hood College) ECON 480 - Lesson 3 Fall 2017 1 / 77 Bivariate