Regression Analysis Tutorial 34 LECTURE / DISCUSSION. Statistical Properties of OLS

Size: px

Start display at page:

Download "Regression Analysis Tutorial 34 LECTURE / DISCUSSION. Statistical Properties of OLS"

Earl Wood
6 years ago
Views:

1 Regression Analysis Tutorial 34 LETURE / DISUSSION Statistical Properties of OLS

2 Regression Analysis Tutorial 35 Statistical Properties of OLS y = " + $x + g dependent included omitted variable explanatory variables variables Energy ε true line Income Elements of g : size of dwelling number of members insulation level cracks around windows and doors attitudes toward conservation L Some omitted variables could be included, and some cannot be measured completely and so cannot be included completely

3 Regression Analysis Tutorial 36 Entire population: E true line I true $ = 9 Usually we sample from the population Get a different estimated line for each sample Sample A Sample B Sample E OLS line OLS line OLS line I ˆ ' 5 ˆ ' 12 ˆ ' 95

4 Regression Analysis Tutorial 37 There is a distribution of ˆ's : ^ β Issues About Distribution of OLS ˆ Mean Variance Shape

5 Regression Analysis Tutorial 38 Unbiasedness An estimator is unbiased if the mean of its frequency distribution is the true value That is, if E(ˆ) ', then ˆ is unbiased => unbiased true β ^ = E( β) => biased true β E( β^)

6 Regression Analysis Tutorial 39 OLS ˆ is biased if omitted variables are correlated with included variables in population Example: E OLS line true line I Higher income households tend to have more members True line: Effect of extra income holding other factors, like number of members, constant Problem: OLS household size ˆ for income picks up the effects of

7 Regression Analysis Tutorial 40 Solutions: 1 Add the correlated omitted variables to the regression 2 If option 1 is not possible, use instrumental variables estimation

8 Regression Analysis Tutorial 41 Example AIDS rate estimated true Syphilis rate (AIDS rate) = " + $(syphilis rate) + g Actually: frequency of unprotected sex causes transmission of syphilis and HIV By omitting frequency of unprotected sex, OLS regression makes it look like syphilis causes AIDS

9 Regression Analysis Tutorial 42 Another Way of Seeing the Problem OLS finds the line for which the residuals are uncorrelated with the explanatory variables If residuals are in reality correlated with the explanatory variables, then OLS will give the wrong line

10 Regression Analysis Tutorial 43 OLS ˆ is unbiased if omitted variables are uncorrelated with included variables in population y true line = OLS line x

11 Regression Analysis Tutorial 44 Proof of Unbiasedness Assume: Y n = " + $X n + g n orr(g,x) = 0 in population Proof: Because " is included, the equation can be rewritten as deviations: y n ' x n % g n ˆ ' j y n x n j x 2 n ' j ( x n % g n )x n j x 2 n ' j x 2 n % j g n x n j x 2 n ' % j g n x n j x 2 n So: E(ˆ) ' % E j g n x n j x 2 n ' % ov(g n x n ) Var(x n ) '

12 Regression Analysis Tutorial 45 Now consider the intercept Suppose true " is not zero Omitting intercept in the regression makes the OLS ˆ biased y OLS line true line x Solution: Include an intercept

13 Regression Analysis Tutorial 46 Exception: Suppose true " is zero OLS ˆ is unbiased with or without intercept y true line = OLS line x

14 Regression Analysis Tutorial 47 Summary OLS ˆ is unbiased if 1 An intercept is included, or true intercept is zero and 2 Omitted variables are uncorrelated with included variables Formally: E(g*x) ' 0 which implies 1 E(g) = 0 2 orr (g,x) = 0

15 Regression Analysis Tutorial 48 Variance of Distribution of OLS ˆ Larger variance spread true β ^ β Smaller variance spread true β ^ β Want as small variance as possible

16 Regression Analysis Tutorial 49 Variance of ˆ is Lower When fewer variables are omitted, and more variables are included When sample size is larger When variance of included variables is larger

17 Regression Analysis Tutorial 50 Think of population as follows: For any value of X, there are different Y s because of g s E 10K 25K 40K I

18 Regression Analysis Tutorial 51 Variance of ˆ is Lower When Fewer Variables are Omitted Large influence from omitted variables Y true X Large change in ˆ from one sample to next

19 Regression Analysis Tutorial 52 Small influence from omitted variables Y true X Small change in ˆ from one sample to next Implication: Include as many causal variables as possible

20 Regression Analysis Tutorial 53 Variance of ˆ is lower with larger samples Population Y true X Sample size = 3 Y Y X X Large change in ˆ from one sample to next

21 Regression Analysis Tutorial 54 Smaller change in ˆ from one sample to next Sample size = 10 Y Y X X Implication: Use as large samples as possible

22 Regression Analysis Tutorial 55 Variance of ˆ is Lower When the Variance in the Included Variables is Higher Small spread in X: Population Y true X Samples Y Y X X Large change in ˆ from one sample to next

23 Regression Analysis Tutorial 56 Large spread in X: Population Y true X Samples Y Y X X Small change in ˆ from one sample to next Implication: Obtain as much variance as possible in explanatory variables

24 Regression Analysis Tutorial 57 Proof of Implications for Variance Assume y n = $x n + g n (intercept allows deviations) x n non-stochastic E(g n x n ) = 0 g n independent over n V(g n ) = F 2 (homoscedasticity) Recall ˆ ' % j g n x n j x 2 n V(ˆ) ' V ' ' ' j g n x n 1 j x 2 n 1 j x 2 n 1 j x 2 n j x 2 n ' 2 j x 2 n j x 2 n V j g n x n j x 2 n V(g n ) j x 2 n 2 2 ' 2 / j x 2 n

25 Regression Analysis Tutorial 58 V(ˆ) ' 2 / j x 2 n decreases when F 2 decreases decreases when sample size increases, since sum in denominator gets larger decreases when the variance of x increases, since the denominator is proportional to the variance

26 Regression Analysis Tutorial 59 Summary To get lower variance in ˆ 1 Include as many explanatory variables as possible 2 Increase sample size 3 Obtain as large a variance in explanatory variables as possible

27 Regression Analysis Tutorial 60 How to Measure Variance? Measure of variance of ˆ V(ˆ) ' 2 / j x 2 n where F 2 is variance in errors Estimate of F 2 s 2 ' 1 N & K j r 2 n s is called the "standard error of regression" Estimate of V(ˆ) V(ˆ) ' s 2 / j x 2 n V(ˆ) is called the "standard error of ˆ "

28 Regression Analysis Tutorial 61 Shape of Distribution of ˆ Normal distribution if 1 g n s are distributed normally or 2 Sample size is large Variance = 2 σ / Σ x 2 n Normal distribution true β β ^ ˆ - N(, 2 / j x 2 n )

29 Regression Analysis Tutorial 62 Summary 1 OLS ˆ is unbiased if intercept is included omitted variables are uncorrelated with included variables 2 Variance of ˆ is smaller when more variables are included sample size is larger explanatory variables have larger variance 3 ˆ has a normal distribution if errors have normal distribution or sample size is large 4 Standard error of ˆ is estimate of the standard deviation of the distribution of ˆ

Multiple Linear Regression CIVL 7012/8012

Multiple Linear Regression CIVL 7012/8012 2 Multiple Regression Analysis (MLR) Allows us to explicitly control for many factors those simultaneously affect the dependent variable This is important for