Simple Linear Regression Estimation and Properties

Size: px

Start display at page:

Download "Simple Linear Regression Estimation and Properties"

Abel Martin
6 years ago
Views:

1 Simple Linear Regression Estimation and Properties

2 Outline Review of the Reading Estimate parameters using OLS Other features of OLS Numerical Properties of OLS Assumptions of OLS Goodness of Fit

3 Checking Understanding What is the best estimate of E(Y)? How would we find E(Y Xi)? Y = B 1 + B 2 X + u What is B 1? What is B 2? What is u?

4 Checking Understanding What is a z-score? z(x) = x What is the mean of z(x)? What is the standard deviation of z(x)? x x

5 Checking Understanding What is a z-score? z(x) = x x x Correlation: r = P zx z y n 1

6 Checking Understanding Correlation: r = P zx z y n 1 The regression line in z-scores: z y = mz x

7 Checking Understanding Correlation: r = P zx z y n 1 The regression line in z-scores: Can also be written as: ẑ y = mz x z y = mz x Can also be written as: ẑ y = rz x

8 Checking Understanding Correlation: r = P zx z y n 1 The regression line in z-scores: Can also be written as: Can also be written as: ẑ y = mz x ẑ y = rz x z y = mz x Remember: m = cov(x, Y ) var(x)

9 And What is Covariance? xy = cov(x, Y )=E[(X µ x )(Y µ y )] xy = cov(x, Y )=E[(X x)(y ȳ)] Cov(X,Y) = E[(X-E[X])(Y-E[Y])] Cov(X,Y) = E[XY]-E[X]E[Y] Covariance is positive if x and y are both below their mean or both above their mean. It is negative if x is above its mean while y is below its mean or vice versa.

10 And What is Covariance? xy = cov(x, Y )=E[(X µ x )(Y µ y )] xy = cov(x, Y )=E[(X x)(y ȳ)] Cov(X,Y) = E[ ( X - E[X] ) ( Y - E[Y] ) ] Cov(X,Y) = E[XY] - E[X] E[Y] Covariance is positive if x and y are both below their mean or both above their mean. It is negative if x is above its mean while y is below its mean or vice versa. But it has units. It is easy to interpret the sign, but hard to interpret the number

11 Total Population of Money Spent and the Number of Votes Effect of Money on Votes Number of Votes Amount Spent- in millions

12 What we can see from the graph We can see the average value of Y for each value of X These are the conditional expected values E(Y X) If we join the conditional values of Y given each value of X we get the Population Regression Line

13 Population Regression Function and the Linear Model E(Y X i )=f(x i ) The expected value of the distribution of Y, given X i is functionally related to X i E(Y X i )=B 1 +B 2 X i

14 Two interpretations of linearity Linear in Variables Which of the following is linear in variables and why?: E(Y X i )=B 1 +B 2 X i 2 E(Y X i )=B 1 +B 2 X i Linear in Parameters Which of the following is linear in parameters and why? E(Y X i )=B 1 +B 2 X i 2 E(Y X i )=B 1 +B 2 2X i Why Should We Care? Linear Regression Requires linearity in parameters only

15 Straight Line Y=B 1 +B 2 X i

16 Quadratic Y=B 1 +B 2 X+B 3 X 2

17 Adding in the Stochastic Term Y i =E(Y X i ) + u i Systematic Component: E(Y X i ) Stochastic Disturbance: U

18 The Sample Regression Function (SRF) Because of sampling fluctuation, any sample will only approximate our true Population Regression Function Stochastic form of the SRF:

19 Primary Goal in Regression Analysis We want to estimate the PRF Y i =B 1 +B 2 X i +u i On the basis of the SRF

20 One method Choose the Sample Regression Function such that the sum of the residuals is as small as possible

21 Illustration and Problem Y u1=10 u3=2 u2=-2 u4=-10 X

22 Alternative Method Ordinary Least Squares (OLS) is a method of finding the linear model which minimizes the sum of the squared errors. Example: (10) 2 + (-2) 2 + (2) 2 + (-10) 2 = 208 This method is the best, linear unbiased estimator

23 Good Spot for a break

24 Minimizing the Sum of Squares Our goal is to minimize the sum of the squared errors. Since we have two unknowns, B 1 and B 2, we need to take the partial derivatives for the following equation:

25 Partial Derivatives for B s We start with our original equation: Now we take the partial derivatives First equation is the partial derivative with respect to B 1, Second equation is with respect to B 2

26 Set Equal to Zero Last set of equations: Next:

27 The Normal Equations Last: Divide both equations by 2 Multiply through Separate summation terms and rearrange:

28 Rewriting the Equation Last Equation: We can rewrite

29 Solving Equation We have two equations with two unknowns, for which we can use algebra Multiply first equation by sum of X i and second by n End up with

30 Subtract first equation from second and rearranging

31 Last step Last equation Multiply numerator and denominator by 1/n recall that End up with

32 We can now solve for B 1 If we go back to the first normal equation:

33 What Does B 2 Mean? Equation for B 2 may not seem to make intuitive sense at first But if we break it down into pieces we can begin to see the logic

34 In sum If the changes in X are EQUAL to the changes in y, then B 2 = 1 If the changes in Y are LARGER than the changes in X, then B 2 > 1 If the changes in Y are SMALLER than the changes in X, then B 2 < 1

35 Let s Do An Example!

36 Calculating a and b Mean of X is 4 Mean of Y is

37 Calculating B1 and B2

38 Which Looks Like This! 30 Regression of Y on X

39 Practice Problem We have a sample of the amount of money a each candidate spent in a state (in millions) and the percentage of the vote they received. Calculate the regression line and interpret.

40 Data State % vote Money spent CA FL GA 15 4 MO 20 6 OH VT 25 8

41 Numerical Properties of OLS Those properties that result from the method of OLS Expressed from observable quantities of X and Y Point Estimator for B s Sample regression line passes through sample means of Y and X Sum of residuals is zero Residuals are uncorrelated with the predicted Y i Residuals uncorrelated with X i

42 Assumptions of Classical Linear Regression A1: Linear Regression Model-Linear in parameters A2: X values are fixed in repeated sampling. A3: Zero mean value of the disturbance term u i A4: Homoskedasticity or Equal Variance of u i.

43 More Assumptions A5: No autocorrelation between disturbances A6: Zero covariance between u i and X i A7: Number of observations n is greater than the number of parameters to be estimated A8: Variability in X values

44 More Assumptions A9: Regression model is correctly specified. The correct variables are included We have the correct functional form Correct assumptions about the probability distributions of Y i, X i and u i. A10: With multiple regression, we add the assumption of no perfect multicollinearity

45 How good does it fit? To measure reduction in errors we need a benchmark for comparison. The mean of the dependent variable is a relevant and tractable benchmark for comparing predictions. The mean of Y represents our best guess at the value of Y i absent other information.

46 Sums of Squares This gives us the following 'sum-of-squares' measures: Total Variation = Explained Variation + Unexplained Variation

47 How well does our model R squared statistic = TSS-USS/TSS =ESS/TSS perform? Bounded between 0 and 1 Higher values indicate a better fit Lower values more unexplained than explained variance

Business Economics BUSINESS ECONOMICS. PAPER No. : 8, FUNDAMENTALS OF ECONOMETRICS MODULE No. : 3, GAUSS MARKOV THEOREM

Business Economics BUSINESS ECONOMICS. PAPER No. : 8, FUNDAMENTALS OF ECONOMETRICS MODULE No. : 3, GAUSS MARKOV THEOREM Subject Business Economics Paper No and Title Module No and Title Module Tag 8, Fundamentals of Econometrics 3, The gauss Markov theorem BSE_P8_M3 1 TABLE OF CONTENTS 1. INTRODUCTION 2. ASSUMPTIONS OF