Chapter 2 The Simple Linear Regression Model: Specification and Estimation

Transcription

1 Chapter The Simple Linear Regression Model: Specification and Estimation Page 1

2 Chapter Contents.1 An Economic Model. An Econometric Model.3 Estimating the Regression Parameters.4 Assessing the Least Squares Estimators.5 The Gauss-Markov Theorem.6 The Probability Distributions of the Least Squares Estimators.7 Estimating the Variance of the Error Term s Page

3 .1 An Economic Model Page 3

4 .1 An Economic Model As economists we are usually more interested in studying relationships between variables Economic theory tells us that expenditure on economic goods depends on income Consequently we call y the dependent variable and x the independent or explanatory variable In econometrics, we recognize that real-world expenditures are random variables, and we want to use data to learn about the relationship Page 4

5 .1 An Economic Model The pdf is a conditional probability density function since it is conditional upon an x The conditional mean, or expected value, of y is E(y x) The expected value of a random variable is called its mean value, which is really a contraction of population mean, the center of the probability distribution of the random variable This is not the same as the sample mean, which is the arithmetic average of numerical values Page 5

6 .1 An Economic Model Figure.1a Probability distribution of food expenditure y given income x = $1000 Page 6

7 .1 An Economic Model The conditional variance of y is σ which measures the dispersion of y about its mean μ y x The parameters μ y x and σ, if they were known, would give us some valuable information about the population we are considering Page 7

8 .1 An Economic Model Figure.1b Probability distributions of food expenditures y given incomes x = $1000 and x = $000 Page 8

9 .1 An Economic Model In order to investigate the relationship between expenditure and income we must build an economic model and then a corresponding econometric model that forms the basis for a quantitative or empirical economic analysis This econometric model is also called a regression model Page 9

10 .1 An Economic Model The simple regression function is written as E E( y x) μy β 1β x ( y x) x Eq..1 y Eq..1 where β 1 is the intercept and β is the slope Page 10

11 .1 An Economic Model E y 1 ( y x) x It is called simple regression not because it is easy, but because there is only one explanatory variable on the right-hand side of the equation Page 11

12 .1 An Economic Model Figure. The economic model: a linear relationship between average per person food expenditure and income Page 1

13 .1 An Economic Model The slope of the regression line can be written as: Eq.. β E( y x) de( y x) x dx E ( y x) de( y x) where Δ denotes change x in dxand de(y x)/dx denotes the derivative of the expected value of y given an x value Eq.. Δ denotes change in and de(y x)/dx denotes the derivative of the expected value of y given an x value Page 13

14 . An Econometric Model Page 14

15 . An Econometric Model Figure.3 The probability density function for y at two levels of income Page 15

16 . An Econometric Model There are several key assumptions underlying the simple linear regression More will be added later Page 16

17 . An Econometric Model ASSUMPTIONS OF THE SIMPLE LINEAR REGRESSION MODEL - I Assumption 1: The mean value of y, for each value of x, is given by the linear regression E( y x) β β 1 x Page 17

18 . An Econometric Model ASSUMPTIONS OF THE SIMPLE LINEAR REGRESSION MODEL - I Assumption : For each value of x, the values of y are distributed about their mean value, following probability distributions that all have the same variance var( y x) σ Page 18

19 . An Econometric Model ASSUMPTIONS OF THE SIMPLE LINEAR REGRESSION MODEL - I Assumption 3: The sample values of y are all uncorrelated, and have zero covariance, implying that there is no linear association among them cov( y i, y j ) 0 This assumption can be made stronger by assuming that the values of y are all statistically independent Page 19

20 . An Econometric Model ASSUMPTIONS OF THE SIMPLE LINEAR REGRESSION MODEL - I Assumption 4: The variable x is not random, and must take at least two different values Page 0

21 . An Econometric Model ASSUMPTIONS OF THE SIMPLE LINEAR REGRESSION MODEL - I Assumption 5: (optional) The values of y are normally distributed about their mean for each value of x y N x ~ (β1 β,σ ) Page 1

22 . An Econometric Model..1 Introducing the Error Term The random error term is defined as Eq..3 e ye y x yβ β x 1 Rearranging gives Eq..4 y β β x e 1 where y is the dependent variable and x is the independent variable Page

23 . An Econometric Model..1 Introducing the Error Term The expected value of the error term, given x, is Ee ( x) Ey ( x) β β x 0 1 The mean value of the error term, given x, is zero Page 3

24 . An Econometric Model Figure.4 Probability density functions for e and y..1 Introducing the Error Term Page 4

25 . An Econometric Model ASSUMPTIONS OF THE SIMPLE LINEAR REGRESSION MODEL - II..1 Introducing the Error Term Assumption SR1: The value of y, for each value of x, is: y β β xe 1 Page 5

26 . An Econometric Model ASSUMPTIONS OF THE SIMPLE LINEAR REGRESSION MODEL - II..1 Introducing the Error Term Assumption SR: The expected value of the random error e is: E ( e) 0 This is equivalent to assuming that E( y) β β 1 x Page 6

27 . An Econometric Model ASSUMPTIONS OF THE SIMPLE LINEAR REGRESSION MODEL - II..1 Introducing the Error Term Assumption SR3: The variance of the random error e is: var( e) σ var( y) The random variables y and e have the same variance because they differ only by a constant. Page 7

28 . An Econometric Model ASSUMPTIONS OF THE SIMPLE LINEAR REGRESSION MODEL - II..1 Introducing the Error Term Assumption SR4: The covariance between any pair of random errors, e i and e j is: cov( e i, e j ) cov( y i, y j ) 0 The stronger version of this assumption is that the random errors e are statistically independent, in which case the values of the dependent variable y are also statistically independent Page 8

29 . An Econometric Model ASSUMPTIONS OF THE SIMPLE LINEAR REGRESSION MODEL - II..1 Introducing the Error Term Assumption SR5: The variable x is not random, and must take at least two different values Page 9

30 . An Econometric Model ASSUMPTIONS OF THE SIMPLE LINEAR REGRESSION MODEL - II..1 Introducing the Error Term Assumption SR6: (optional) The values of e are normally distributed about their mean if the values of y are normally distributed, and vice versa e ~ N(0,σ ) Page 30

31 . An Econometric Model Figure.5 The relationship among y, e and the true regression line..1 Introducing the Error Term Page 31

32 .3 Estimating the Regression Parameters Page 3

33 .3 Estimating the Regression Parameters Table.1 Food Expenditure and Income Data Page 33

34 .3 Estimating the Regression Parameters Figure.6 Data for food expenditure example Page 34

35 .3 Estimating the Regression Parameters.3.1 The Least Squares Principle The fitted regression line is: Eq..5 ˆ i y b b x 1 i The least squares residual is: Eq..6 ˆ ˆ ei yi yi yi b1 b x i Page 35

36 .3 Estimating the Regression Parameters Figure.7 The relationship among y, ê and the fitted regression line.3.1 The Least Squares Principle Page 36

37 .3 Estimating the Regression Parameters.3.1 The Least Squares Principle Suppose we have another fitted line: * * * ˆ i 1 y b b x The least squares line has the smaller sum of squared residuals: if SSE N i1 eˆ i and SSE * N i1 eˆ i * i then SSE SSE * Page 37

38 .3 Estimating the Regression Parameters.3.1 The Least Squares Principle Least squares estimates for the unknown parameters β 1 and β are obtained my minimizing the sum of squares function: N S(β,β ) ( y β β xi) 1 i 1 i1 Page 38

39 .3 Estimating the Regression Parameters THE LEAST SQUARES ESTIMATORS.3.1 The Least Squares Principle Eq..7 b ( x x )( y i i ( xi x ) y ) Eq..8 b 1 y b x Page 39

40 .3 Estimating the Regression Parameters.3. Estimates for the Food Expenditure Function b ( x x)( y y) i i ( xi x) b1 y b x (10.096)( ) A convenient way to report the values for b 1 and b is to write out the estimated or fitted regression line: yˆ i x i Page 40

41 .3 Estimating the Regression Parameters Figure.8 The fitted regression line.3. Estimates for the Food Expenditure Function Page 41

42 .3 Estimating the Regression Parameters.3.3 Interpreting the Estimates The value b = 10.1 is an estimate of, the amount by which weekly expenditure on food per household increases when household weekly income increases by $100. Thus, we estimate that if income goes up by $100, expected weekly expenditure on food will increase by approximately $10.1 Strictly speaking, the intercept estimate b 1 = 83.4 is an estimate of the weekly food expenditure on food for a household with zero income Page 4

43 .3 Estimating the Regression Parameters.3.3a Elasticities Income elasticity is a useful way to characterize the responsiveness of consumer expenditure to changes in income. The elasticity of a variable y with respect to another variable x is: percentage change in percentage change in y x y x x y In the linear economic model given by Eq..1 we have shown that β E( y) x Page 43

44 .3 Estimating the Regression Parameters.3.3a Elasticities Eq..9 The elasticity of mean expenditure with respect to income is: E ( y ) E ( y ) ( ) E y x β x xx x Ey ( ) E( y) A frequently used alternative is to calculate the elasticity at the point of the means because it is a representative point on the regression line. x ˆ b 10.1 y Page 44

45 .3 Estimating the Regression Parameters.3.3b Prediction Suppose that we wanted to predict weekly food expenditure for a household with a weekly income of $000. This prediction is carried out by substituting x = 0 into our estimated equation to obtain: yˆ x i (0) We predict that a household with a weekly income of $000 will spend $87.61 per week on food Page 45

46 .3 Estimating the Regression Parameters Figure.9 EViews Regression Output.3.3c Computer Output Page 46

47 .3 Estimating the Regression Parameters.3.4 Other Economic Models The simple regression model can be applied to estimate the parameters of many relationships in economics, business, and the social sciences The applications of regression analysis are fascinating and useful Page 47

48 .4 Assessing the Least Squares Fit Page 48

49 .4 Assessing the Least Squares Fit We call b 1 and b the least squares estimators. We can investigate the properties of the estimators b 1 and b, which are called their sampling properties, and deal with the following important questions: 1. If the least squares estimators are random variables, then what are their expected values, variances, covariances, and probability distributions?. How do the least squares estimators compare with other procedures that might be used, and how can we compare alternative estimators? Page 49

50 .4 Assessing the Least Squares Fit.4.1 The Estimator b Eq..10 The estimator b can be rewritten as: N b i1 w i y i Eq..11 where w i xi x ( x x) i It could also be write as: Eq..1 b β we i i Page 50

51 .4 Assessing the Least Squares Fit.4. The Expected Values of b 1 and b Eq..13 We will show that if our model assumptions hold, then E(b ) = β, which means that the estimator is unbiased. We can find the expected value of b using the fact that the expected value of a sum is the sum of the expected values: E( b) E( b we i i) E(β we 1 1we... wnen) E(β ) E( we 1 1) E( we)... E( wnen) E(β ) E( we ) e i using E( ) 0 and β we i ( ei) β i i E( wiei ) wi E( ei ) Page 51

52 .4 Assessing the Least Squares Fit.4. The Expected Values of b 1 and b The property of unbiasedness is about the average values of b 1 and b if many samples of the same size are drawn from the same population If we took the averages of estimates from many samples, these averages would approach the true parameter values b 1 and b Unbiasedness does not say that an estimate from any one sample is close to the true parameter value, and thus we cannot say that an estimate is unbiased We can say that the least squares estimation procedure (or the least squares estimator) is unbiased Page 5

53 .4 Assessing the Least Squares Fit Table. Estimates from 10 Samples.4.3 Repeated Sampling Page 53

54 .4 Assessing the Least Squares Fit Figure.10 Two possible probability density functions for b.4.3 Repeated Sampling The variance of b is defined asvar( b ) E[ b E( b )] Page 54

55 .4 Assessing the Least Squares Fit.4.4 The Variances and Covariances of b 1 and b Eq..14 Eq..15 Eq..16 If the regression model assumptions SR1-SR5 are correct (assumption SR6 is not required), then the variances and covariance of b 1 and b are: var( b ) x i 1 σ N xi x var( b ) cov( b, b ) σ σ xi x x 1 xi x Page 55

56 .4 Assessing the Least Squares Fit.4.4 The Variances and Covariances of b 1 and b MAJOR POINTS ABOUT THE VARIANCES AND COVARIANCES OF b 1 AND b 1. The larger the variance term σ, the greater the uncertainty there is in the statistical model, and the larger the variances and covariance of the least squares estimators.. The larger the sum of squares, x i x, the smaller the variances of the least squares estimators and the more precisely we can estimate the unknown parameters. 3. The larger the sample size N, the smaller the variances and covariance of the least squares estimators. i 4. The larger the term x, the larger the variance of the least squares estimator b The absolute magnitude of the covariance increases the larger in magnitude is the sample mean x, and the covariance has a sign opposite to that of x. Page 56

57 .4 Assessing the Least Squares Fit.4.4 The Variances and Covariances of b 1 and b Figure.11 The influence of variation in the explanatory variable x on precision of estimation (a) Low x variation, low precision (b) High x variation, high precision The variance of b is defined as var( b ) Eb E( b ) Page 57

58 .5 The Gauss-Markov Theorem Page 58

59 .5 The Gauss-Markov Theorem GAUSS-MARKOV THEOREM Under the assumptions SR1-SR5 of the linear regression model, the estimators b 1 and b have the smallest variance of all linear and unbiased estimators of b 1 and b. They are the Best Linear Unbiased Estimators (BLUE) of b 1 and b Page 59

60 .5 The Gauss-Markov Theorem MAJOR POINTS ABOUT THE GAUSS-MARKOV THEOREM 1. The estimators b 1 and b are best when compared to similar estimators, those which are linear and unbiased. The Theorem does not say that b 1 and b are the best of all possible estimators.. The estimators b 1 and b are best within their class because they have the minimum variance. When comparing two linear and unbiased estimators, we always want to use the one with the smaller variance, since that estimation rule gives us the higher probability of obtaining an estimate that is close to the true parameter value. 3. In order for the Gauss-Markov Theorem to hold, assumptions SR1- SR5 must be true. If any of these assumptions are not true, then b 1 and b are not the best linear unbiased estimators of β 1 and β. Page 60

61 .5 The Gauss-Markov Theorem MAJOR POINTS ABOUT THE GAUSS-MARKOV THEOREM 4. The Gauss-Markov Theorem does not depend on the assumption of normality (assumption SR6). 5. In the simple linear regression model, if we want to use a linear and unbiased estimator, then we have to do no more searching. The estimators b 1 and b are the ones to use. This explains why we are studying these estimators and why they are so widely used in research, not only in economics but in all social and physical sciences as well. 6. The Gauss-Markov theorem applies to the least squares estimators. It does not apply to the least squares estimates from a single sample. Page 61

62 .6 The Probability Distributions of the Least Squares Estimators Page 6

63 .6 The Probability Distributions of the Least Squares Estimators If we make the normality assumption (assumption SR6 about the error term) then the least squares estimators are normally distributed: Eq..17 b σ i 1 ~ Nβ 1, i x N x x Eq..18 b ~ N β, σ x i x Page 63

64 .6 The Probability Distributions of the Least Squares Estimators A CENTRAL LIMIT THEOREM If assumptions SR1-SR5 hold, and if the sample size N is sufficiently large, then the least squares estimators have a distribution that approximates the normal distributions shown in Eq..17 and Eq..18 Page 64

65 .7 Estimating the Variance of the Error Term Page 65

66 .7 Estimating the Variance of the Error Term The variance of the random error e i is: var( e ) σ E[ e E( e )] E( e) i i i i if the assumption E(e i ) = 0 is correct. Since the expectation is an average value we might consider estimating σ as the average of the squared errors: ˆσ e i N where the error terms are e y β1βx i i i Page 66

67 .7 Estimating the Variance of the Error Term Eq..19 The least squares residuals are obtained by replacing the unknown parameters by their least squares estimates: eˆ y yˆ y b b x eˆ i σ N There is a simple modification that produces an unbiased estimator, and that is: so that: i i i i 1 i e i ˆ N E ˆσ σ Page 67

68 .7 Estimating the Variance of the Error Term.7.1 Estimating the Variance and Covariance of the Least Squares Estimators Eq..0 Eq..1 Eq.. Replace the unknown error variance σ in Eq..14 Eq..16 by to obtain: ˆ x var( b ) Nxi x ˆσ var( b ) x x i ˆ 1 σ cov( b, b ) σˆ 1 i x x i x Page 68

69 .7 Estimating the Variance of the Error Term.7.1 Estimating the Variance and Covariance of the Least Squares Estimators The square roots of the estimated variances are the standard errors of b 1 and b : Eq..3 se( b) var( b) 1 1 Eq..4 se( b ) var( b ) Page 69

70 .7 Estimating the Variance of the Error Term Table.3 Least Squares Residuals.7. Calculations for the Food Expenditure Data e i ˆ ˆσ N 38 Page 70

71 .7 Estimating the Variance of the Error Term.7. Calculations for the Food Expenditure Data The estimated variances and covariances for a regression are arrayed in a rectangular array, or matrix, with variances on the diagonal and covariances in the off-diagonal positions. var( b1) cov( b1, b) cov( b1, b) var( b) Page 71

72 .7 Estimating the Variance of the Error Term.7. Calculations for the Food Expenditure Data For the food expenditure data the estimated covariance matrix is: C Income C Income Page 7

73 .7 Estimating the Variance of the Error Term.7.3 Interpreting the Standard Errors The standard errors of b 1 and b are measures of the sampling variability of the least squares estimates b 1 and b in repeated samples. The estimators are random variables. As such, they have probability distributions, means, and variances. In particular, if assumption SR6 holds, and the random error terms e i are normally distributed, then: ~ β,var( ) σ i b N b x x Page 73

74 .7 Estimating the Variance of the Error Term.7.3 Interpreting the Standard Errors The estimator variance, var(b ), or its square root, σ which we might call the true b var b standard deviation of b, measures the sampling variation of the estimates b The bigger b is the more variation in the least squares estimates b we see from sample to sample. If b is large then the estimates might change a great deal from sample to sample If b is small relative to the parameter b,we know that the least squares estimate will fall near b with high probability Page 74

75 .7 Estimating the Variance of the Error Term.7.3 Interpreting the Standard Errors The question we address with the standard error is How much variation about their means do the estimates exhibit from sample to sample? Page 75

76 .7 Estimating the Variance of the Error Term.7.3 Interpreting the Standard Errors b We estimate σ, and then estimate using: se( b ) var( b ) x ˆσ i x The standard error of b is thus an estimate of what the standard deviation of many estimates b would be in a very large number of samples, and is an indicator of the width of the pdf of b shown in Figure.1 Page 76

77 .7 Estimating the Variance of the Error Term Figure.1 The probability density function of the least squares estimator b..7.3 Interpreting the Standard Errors Page 77

78 Key Words Page 105

79 Keywords Page 106