Ordinary Least Squares Regression

Size: px

Start display at page:

Download "Ordinary Least Squares Regression"

Maud Woods
5 years ago
Views:

1 Ordinary Least Squares Regression

2 Goals for this unit More on notation and terminology OLS scalar versus matrix derivation

3 Some Preliminaries In this class we will be learning to analyze Cross Section Data Panel Data (Longitudinal Data) Time Series Data is ignored in this class (take ECON 408)

4 Cross Section Data Information at a point in time for N individuals, firms, or economic units id year wage education experience : : : : : : : : : :

5 Balanced Panel Data Information at T points in time for each of N individuals, firms, or economic units id year wage education experience : : : : : : : : : : Here T = 3 (indexed by years) and N=100 (indexed by id)

6 Unbalanced Panel Data Information on up to T points in time for each of N individuals, firms, or economic units id year wage education experience Here T = 3 (indexed by years) and N=100 (indexed by id)

7 Time Series Data (Not Covered) id year wage education experience : : : : : : : : : : Here T = 20 (indexed by years) and N=1 (indexed by id)

8 Yardsticks Unbiased: E[b] = β Consistent: E[b] β, as N approaches

9 Yardsticks, cont Efficiency: The estimator uses all information at hand for the best estimate of β and the variance/covariance matrix An estimator is asymptotically efficient if it is consistent, asymptotically normally distributed, and has an asymptotic covariance that is not larger than the asymptotic covariance matrix of any other consistent, asymptotically distributed estimator

10 OLS Consider a situation where we want to test a hypothesis about economic behavior or some type of economic phenomena What are the returns to education? How do premiums impact choice of Obamacare? y i = β 0 + β 1 x i1 + β 2 x i2 + β 3 x i3 + ɛ i

11 OLS and linear algebra y 1 y i y N = 1 x 11 x 12 x 13 1 x i1 x i2 x i3 1 x N1 x N2 x N3 β 0 β 1 β 2 β 3 + ɛ 1 ɛ i ɛ N (1)

12 Exercise Check for conformability Show that any row in the above relationship can be written y i = β 0 + β 1 x i1 + β 2 x i2 + β 3 x i3 + ɛ i (2) Why is a column of ones needed in x?

13 OLS and linear algebra, cont y 1 y i y N N 1 = 1 x 11 x 12 x 13 1 x i1 x i2 x i3 1 x N1 x N2 x N3 N 4 β 0 β 1 β 2 β ɛ 1 ɛ i ɛ N N 1

14 Estimation Problem For most empirical research in the Social Sciences, interested in marginal effects or elasticities The ME are: Don t know β or ɛ y i x ik = β k However, given an estimate for β (b), we can construct estimates for ɛ i (e i ) as e i = y i (b 0 + b 1 x i1 + b 2 x i2 + b 3 x i3 ) = y i x i b ] where x i = [1 x i1 x i2 x i3

15 Visualizing OLS Note that for every point, y i = β 0 + β 1 x i1 + β 2 x i2 + β 3 x i3 + ɛ i = b 0 + b 1 x i1 + b 2 x i2 + b 3 x i3 + e i Or, for all points y = xβ + ɛ = xb + e

16 Choosing b One way to calculate b is to minimize the sum squared errors (SSE): n n (e i ) 2 = (y i (b 0 + b 1 x i1 + b 2 x i2 + b 3 x i3 )) 2 i=1 i=1 Exercise: show that n i=1 (e i ) 2 is equivalent to (y xb) (y xb) = e e

17 The Minimization Problem min b SSE(b) min b e e

18 The Minimization Problem min b SSE(b) min b e e Simplifying and using the inner product rule: min y y 2y xb + b x xb b

19 First order conditions: S b = 2x y + 2x xb = 0

20 First order conditions: S b = 2x y + 2x xb = 0 can be simplified to solve for the OLS estimator b b = (x x) 1 x y

21 What have we just solved for?

22 Implementation in Stata

23 Some Assumptions we will use for deriving model Recall the assumptions in the classic multiple regression model: 1 Linear in parameters 2 There is no linear relationships between the columns in the matrix x 3 Exogeneity of the independent variables: E[ɛ i x j1, x j2,, x j3 ] = 0, for all j This condition ensures that the independent variables for observation i, or for any other observation are not useful for predicting the disturbance terms in the model 4 Homoskedasticity and no autocorrelation: E[ɛ i ɛ j ] j n and share the same variance σ 2 5 Disturbance terms are normally distributed, and given condition (3) and (4): ɛ N(0, σ 2 I)

24 Properties: Unbiased Is E(b) = β?

25 Properties: Unbiased Is E(b) = β? E(b) = E[(x x) 1 x y] = E[(x x) 1 x (xβ + ɛ)] = E[(x x) 1 x xβ + (x x) 1 x ɛ] = β + E[(x x) 1 x ɛ] (3) Need E[x ɛ] = 0 Two equivalent ways to think about this: 1 Errors uncorrelated with independent variables 2 Independent variables have no useful information for predicting errors

26 Properties: For unbiased b need E[x ɛ] = 0 Implications: If errors correlated with independent variables we have endogeneity and the results are biased Exercise: investigate dimensions of E[x ɛ] = 0

27 Properties: Variance of the estimator The variance of any random variable (scalar) is defined as Var(a) = E[(a E(a)) 2 ] = E[(a µ a ) 2 ] We are interested in the variance/covariance relationship between elements of our estimated vector b in order to perform statistical inference on our parameter estimates The definition of a variance/covariance matrix for a random vector b is Var(b) =E[(b E[b])(b E[b]) ] =E[(b β)(b β) ] Exercise: what are dimensions?

28 Properties: Variance of the estimator, cont Var(b) = E[(b β)(b β) ] = E[((x x) 1 x y β)((x x) 1 x y β) ] substitute y = xβ + ɛ, check for conformability and simplify

29 Properties: Variance of the estimator, cont Var(b) = (x x) 1 x E[ɛɛ ]x((x x) 1 ) = (x x) 1 x E[ɛɛ ]x(x x) 1 To simplify further, we invoke the assumption above (ɛ N(0, σ 2 I)), so that E[ɛɛ ] = σ 2 I Exercise: simplify Var(b) further

30 Properties: Variance of the estimator, cont Finally, we have Var(b) = σ 2 (x x) 1 k k I refer to this as the Variance/Covariance Matrix of the Parameters

31 How to recover σ 2? σ 2 is the variance of the error Given assumption (3), errors are homoskedastic- shared by everyone in the population σ 2 is unobserved, but estimate it by using estimated errors for each observation and recover sample average s 2 = e e N K = (y xb) (y xb) N K

32 Estimated Variance/Covariance Matrix of Parameters var(b) = s 2 (x x) 1 Use diagonal elements of this matrix for the variance of each parameter estimated in the model: Var[b] = s 2 (x x) 1 = s So, the standard error from our regression for b 0 (assumed to be the first parameter) is s

33 Implement in Stata

34 Heteroskedasticity

35 Testing for Heterskedasticity The symptoms of Heteroskedasticity are embodied in the relationship between the linear predictor xb and the estimated error Idea: see if the variances of the estimated model errors (e) are a function of the independent variables (x)

36 Testing for Heterskedasticity: What stata does (estat hettest) Step 1: Recover model residuals (e = y xb) and predictions (^y = xb) Step 2: Using the residuals, calculate diagonal(ee ) r = 1 N (y xb) (y xb) Step 3: Run the regression r = δ 0 + δ 1^y + v, and recover the Model Sum of Squares (MSS): (^r ^r) (^r ^r) Step 4: Using the MSS 2 (distributed χ 2 (1)), perform the following hypothesis test: H 0 : No Heteroskedasticity: σ 2 i = σ 2 i N H1 : Heteroskedasticity: σ 2 i σ 2 i N

37 The Fix -Remember back to our proof Var(b) = (x x) 1 x E[ɛɛ ]x(x x) 1 -OLS assumes we can estimate Var(ɛ) as s s var(e) = s 2 I N N = 0 0 s s 2

38 The Fix, cont For robust standard errors, use this where s s var(e) = ˆV = 0 0 s sn 2 s i = e i = y i x i b = y i ŷ i

39 Robust Standard Errors The robust standard error variance covariance matrix is then: Var(b) = (x x) 1 x ˆVx(x x) 1 Notice we use information in the estimated error to fix the Heteroskedasticity problem Things to reflect on: 1 Should we always use robust standard errors? 2 We are able to leverage estimated errors in this way because b is unbiased in the presence of heteroskedasticity Therefore, our estimates for ɛ i are unbiased

Fixed Effects Models for Panel Data. December 1, 2014

Fixed Effects Models for Panel Data December 1, 2014 Notation Use the same setup as before, with the linear model Y it = X it β + c i + ɛ it (1) where X it is a 1 K + 1 vector of independent variables.