MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE. ASVABC + u

Transcription

1 MULTIPLE REGRESSION

2 MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE EARNINGS α + HGC + ASVABC + α EARNINGS ASVABC HGC This seqence provides a geometrical interpretation of a mltiple regression model with two eplanatory variales.

3 MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE EARNINGS α + HGC + ASVABC + α EARNINGS ASVABC HGC Specifically, we will look at an earnings fnction model where horly earnings, EARNINGS, depend on highest grade completed, HGC, and a measre of aility, ASVABC.

4 MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE EARNINGS α + HGC + ASVABC + α EARNINGS ASVABC HGC The model has three dimensions, one each for EARNINGS, HGC, and ASVABC. The starting point for investigating the determination of EARNINGS is the intercept, α. 3

5 MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE EARNINGS α + HGC + ASVABC + α EARNINGS ASVABC HGC Literally the intercept gives EARNINGS for those respondents who have no schooling and who scored zero on the aility test. However, the aility score is scaled in sch a way as to make it impossile to score zero. Hence a literal interpretation of α wold e nwise. 4

6 MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE EARNINGS α + HGC + ASVABC + α EARNINGS pre HGC effect α + HGC ASVABC HGC The net term on the right side of the eqation gives the effect of variations in HGC. A one year increase in HGC cases EARNINGS to increase y dollars, holding ASVABC constant. 5

7 MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE EARNINGS α + HGC + ASVABC + α + ASVABC pre ASVABC effect α EARNINGS ASVABC HGC Similarly, the third term gives the effect of variations in ASVABC. A one point increase in ASVABC cases earnings to increase y dollars, holding HGC constant. 6

8 MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE EARNINGS α + HGC + ASVABC + α + ASVABC pre ASVABC effect α + HGC + ASVABC comined effect of HGC and ASVABC α EARNINGS pre HGC effect α + HGC ASVABC HGC Different cominations of HGC and ASVABC give rise to vales of EARNINGS which lie on the plane shown in the diagram, defined y the eqation EARNINGS α + HGC + ASVABC. This is the nonstochastic (nonrandom component of the model. 7

9 MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE EARNINGS α + HGC + ASVABC + α + ASVABC pre ASVABC effect α + HGC + ASVABC + α + HGC + ASVABC comined effect of HGC and ASVABC α EARNINGS pre HGC effect α + HGC ASVABC HGC The final element of the model is the distrance term,. This cases the actal vales of EARNINGS to deviate from the plane. In this oservation, happens to have a positive vale. 8

10 MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE EARNINGS α + HGC + ASVABC + α + ASVABC pre ASVABC effect α + HGC + ASVABC + α + HGC + ASVABC comined effect of HGC and ASVABC α EARNINGS pre HGC effect α + HGC ASVABC HGC A sample consists of a nmer of oservations generated in this way. Note that the interpretation of the model does not depend on whether HGC and ASVABC are correlated or not. 9

11 MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE EARNINGS α + HGC + ASVABC + α + ASVABC pre ASVABC effect α + HGC + ASVABC + α + HGC + ASVABC comined effect of HGC and ASVABC α EARNINGS pre HGC effect α + HGC ASVABC HGC However we do assme that the effects of HGC and ASVABC on EARNINGS are additive. The impact of a difference in HGC on EARNINGS is not affected y the vale of ASVABC, or vice versa. 0

12 MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE y + i α+ i + i i ˆ + yi a+ i i The regression coefficients are derived sing the same least sqares principle sed in simple regression analysis. The fitted vale of y in oservation i depends on or choice of a, and.

13 i i i i y α i i i a y ˆ + + i i i i i i a y y y e ˆ MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE The residal e i in oservation i is the difference etween the actal and fitted vales of y.

14 MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE S e i ( yi a i i We define S, the sm of the sqares of the residals, and choose a,, and so as to minimize it. 3

15 ( i i i i a y e S ( i i i i i i i i i i i i a a y y ay a y i i i i i i i i i i i i a a y y y a na y a S 0 S 0 S MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE First we epand S as shown, and then we se the first order conditions for minimizing it. 4

16 MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE a y Cov(,yVar( Var( Var( - Cov(, ycov( [ Cov(, ], Cov(,yVar( Var( Var( - Cov(, ycov( [ Cov(, ], We ths otain three eqations in three nknowns. Solving for a,, and, we otain the epressions shown aove. 5

17 MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE a y Cov(,yVar( Var( Var( - Cov(, ycov( [ Cov(, ], Cov(,yVar( Var( Var( - Cov(, ycov( [ Cov(, ], The epression for a is a straightforward etension of the epression for it in simple regression analysis. 6

18 MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE a y Cov(,yVar( Var( Var( - Cov(, ycov( [ Cov(, ], Cov(,yVar( Var( Var( - Cov(, ycov( [ Cov(, ], However, the epressions for the slope coefficients are consideraly more comple than that for the slope coefficient in simple regression analysis. 7

19 MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE a y Cov(,yVar( Var( Var( - Cov(, ycov( [ Cov(, ], Cov(,yVar( Var( Var( - Cov(, ycov( [ Cov(, ], For the general case when there are many eplanatory variales, ordinary algera is inadeqate. It is necessary to switch to matri algera. 8

20 MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE. reg earnings hgc asvac Sorce SS df MS Nmer of os F(, Model Pro > F Residal R-sqared Adj R-sqared 0.05 Total Root MSE earnings Coef. Std. Err. t P> t [95% Conf. Interval] hgc asvac _cons EARNINGS ˆ HGC+ 0. 6ASVABC Here is the regression otpt for the earnings fnction sing Data Set. 9

21 MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE. reg earnings hgc asvac Sorce SS df MS Nmer of os F(, Model Pro > F Residal R-sqared Adj R-sqared 0.05 Total Root MSE earnings Coef. Std. Err. t P> t [95% Conf. Interval] hgc asvac _cons EARNINGS ˆ HGC+ 0. 5ASVABC It indicates that earnings increase y $0.74 for every etra year of schooling and y $0.5 for every etra point increase in ASVABC. 0

22 MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE. reg earnings hgc asvac Sorce SS df MS Nmer of os F(, Model Pro > F Residal R-sqared Adj R-sqared 0.05 Total Root MSE earnings Coef. Std. Err. t P> t [95% Conf. Interval] hgc asvac _cons EARNINGS ˆ HGC+ 0. 6ASVABC Literally, the intercept indicates that an individal who had no schooling and an ASVABC score of zero wold have horly earnings of -$4.6.

23 MULTIPLE REGRESSION WITH TWO EXPLANATORY VARIABLES: EXAMPLE. reg earnings hgc asvac Sorce SS df MS Nmer of os F(, Model Pro > F Residal R-sqared Adj R-sqared 0.05 Total Root MSE earnings Coef. Std. Err. t P> t [95% Conf. Interval] hgc asvac _cons EARNINGS ˆ HGC+ 0. 6ASVABC Oviosly, this is impossile. The lowest vale of HGC in the sample was 6, and the lowest ASVABC score was. We have otained a nonsense estimate ecase we have etrapolated too far from the data range.

24 GRAPHING A RELATIONSHIP IN A MULTIPLE REGRESSION MODEL

25 GRAPHING A RELATIONSHIP IN A MULTIPLE REGRESSION MODEL. reg earnings hgc asvac Sorce SS df MS Nmer of os F(, Model Pro > F Residal R-sqared Adj R-sqared 0.05 Total Root MSE earnings Coef. Std. Err. t P> t [95% Conf. Interval] hgc asvac _cons EARNINGS ˆ HGC+ 0. 5ASVABC The otpt aove shows the reslt of regressing EARNINGS, horly earnings in dollars, on HGC, highest grade completed, and ASVABC, the aility score.

26 GRAPHING A RELATIONSHIP IN A MULTIPLE REGRESSION MODEL Horly earnings ($ Highest grade completed Sppose that yo were particlarly interested in the relationship etween EARNINGS and HGC and wished to represent it graphically, sing the sample data.

27 GRAPHING A RELATIONSHIP IN A MULTIPLE REGRESSION MODEL Horly earnings ($ Highest grade completed A simple plot, like the one aove, wold e misleading. 3

28 GRAPHING A RELATIONSHIP IN A MULTIPLE REGRESSION MODEL Horly earnings ($ cor hgc asvac (os570 hgc asvac hgc.0000 asvac Highest grade completed There appears to e a strong positive relationship, t it is distorted y the fact that HGC is positively correlated with ASVABC, which also has a positive effect on EARNINGS. 4

29 GRAPHING A RELATIONSHIP IN A MULTIPLE REGRESSION MODEL Horly earnings ($ cor hgc asvac (os570 hgc asvac hgc.0000 asvac Highest grade completed We will investigate the distortion mathematically when we come to omitted variale ias. 5

30 GRAPHING A RELATIONSHIP IN A MULTIPLE REGRESSION MODEL. reg earnings asvac Sorce SS df MS Nmer of os F(, Model Pro > F Residal R-sqared Adj R-sqared Total Root MSE earnings Coef. Std. Err. t P> t [95% Conf. Interval] asvac _cons To eliminate the distortion, yo prge oth EARNINGS and HGC of their components related to ASVABC and then draw a scatter diagram sing the prged variales. 6

31 GRAPHING A RELATIONSHIP IN A MULTIPLE REGRESSION MODEL. reg earnings asvac Sorce SS df MS Nmer of os F(, Model Pro > F Residal R-sqared Adj R-sqared Total Root MSE earnings Coef. Std. Err. t P> t [95% Conf. Interval] asvac _cons predict eearn, resid We start y regressing EARNINGS on ASVABC, as shown aove. The residals are the part of EARNINGS which is not related to ASVABC. The "predict" command is the Stata command for saving the residals from the most recent regression. We name them EEARN. 7

32 GRAPHING A RELATIONSHIP IN A MULTIPLE REGRESSION MODEL. reg hgc asvac Sorce SS df MS Nmer of os F(, Model Pro > F Residal R-sqared Adj R-sqared Total Root MSE.05 hgc Coef. Std. Err. t P> t [95% Conf. Interval] asvac _cons predict ehgc, resid We do the same with HGC. We regress it on ASVABC and save the residals as EHGC. 8

33 GRAPHING A RELATIONSHIP IN A MULTIPLE REGRESSION MODEL EEARN (EARNINGS residals EHGC (HGC residals Now we plot EEARN on EHGC and the scatter is a faithfl representation of the relationship, oth in terms of the slope of the trend line (the lack line and in terms of the variation aot that line. 9

34 GRAPHING A RELATIONSHIP IN A MULTIPLE REGRESSION MODEL EEARN (EARNINGS residals EHGC (HGC residals As yo wold epect, the trend line is flatter that in scatter diagram which did not control for ASVABC (reprodced here as the gray line. 0

35 GRAPHING A RELATIONSHIP IN A MULTIPLE REGRESSION MODEL. reg eearn ehgc Sorce SS df MS Nmer of os F(, 568. Model Pro > F Residal R-sqared Adj R-sqared Total Root MSE eearn Coef. Std. Err. t P> t [95% Conf. Interval] ehgc _cons -5.99e Here is the regression of EEARN on EHGC.

36 GRAPHING A RELATIONSHIP IN A MULTIPLE REGRESSION MODEL. reg eearn ehgc Sorce SS df MS Nmer of os F(, 568. Model Pro > F Residal R-sqared Adj R-sqared Total Root MSE eearn Coef. Std. Err. t P> t [95% Conf. Interval] ehgc _cons -5.99e From mltiple regression: earnings Coef. Std. Err. t P> t [95% Conf. Interval] hgc asvac _cons A mathematical proof that the techniqe works reqires matri algera. We will content orselves y verifying that the estimate of the slope coefficient, and eqally importantly, its standard error and t statistic, are the same as in the mltiple regression.

37 PROPERTIES OF THE MULTIPLE REGRESSION COEFFICIENTS

38 PROPERTIES OF THE MULTIPLE REGRESSION COEFFICIENTS y α+ + + y ˆ a+ + Cov(,yVar( Var( Var( - Cov(, ycov( [ Cov(, ], Cov(, [ α+ + + ]Var( - Cov(,[ α+ + + ]Cov(, [ Var( + Cov(, + Cov(, ]Var( -[ Cov(, + Var( + Cov(, ]Cov(, Provided that the model is correctly specified and that the Gass-Markov conditions are satisfied, the OLS estimators in the mltiple regression model are niased, efficient, and consistent, as in the simple regression model.

39 PROPERTIES OF THE MULTIPLE REGRESSION COEFFICIENTS y α+ + + y ˆ a+ + Cov(,yVar( Var( Var( - Cov(, ycov( [ Cov(, ], Cov(, [ α+ + + ]Var( - Cov(,[ α+ + + ]Cov(, [ Var( + Cov(, + Cov(, ]Var( -[ Cov(, + Var( + Cov(, ]Cov(, We will not attempt to prove efficiency or consistency. We will however give a proof of niasedness. The mathematical details of the proof are nimportant, t yo shold nderstand the general principle.

40 PROPERTIES OF THE MULTIPLE REGRESSION COEFFICIENTS y α+ + + y ˆ a+ + Cov(,yVar( Var( Var( - Cov(, ycov( [ Cov(, ], Cov(, [ α+ + + ]Var( - Cov(,[ α+ + + ]Cov(, [ Var( + Cov(, + Cov(, ]Var( -[ Cov(, + Var( + Cov(, ]Cov(, The first step, as always, is to sstitte for y from the tre relationship. To save space, we will write the denominator as. 3

41 PROPERTIES OF THE MULTIPLE REGRESSION COEFFICIENTS y α+ + + y ˆ a+ + Cov(,yVar( Var( Var( - Cov(, ycov( [ Cov(, ], Cov(, [ α+ + + ]Var( - Cov(,[ α+ + + ]Cov(, [ Var( + Cov(, + Cov(, ]Var( -[ Cov(, + Var( + Cov(, ]Cov(, When we decompose the covariance epressions, the terms involving α disappear ecase Cov(, α and Cov(, α are oth zero. 4

42 PROPERTIES OF THE MULTIPLE REGRESSION COEFFICIENTS y α+ + + y ˆ a+ + [ Var( + Cov(, + Cov(, ]Var( -[ Cov(, + Var( + Cov(, ]Cov(, Var( Var( -[Cov(, ] + Cov(, Var( - Var( Cov( + Cov(, Var( Cov(, Cov(,, + Cov(, Var( Cov(, Cov( Var( Var( [Cov(, ], This is the last line of the previos slide. 5

43 PROPERTIES OF THE MULTIPLE REGRESSION COEFFICIENTS y α+ + + y ˆ a+ + [ Var( + Cov(, + Cov(, ]Var( -[ Cov(, + Var( + Cov(, ]Cov(, Var( Var( -[Cov(, ] + Cov(, Var( - Var( Cov( + Cov(, Var( Cov(, Cov(,, + Cov(, Var( Cov(, Cov( Var( Var( [Cov(, ], The slide shows where the terms involving came from. 6

44 PROPERTIES OF THE MULTIPLE REGRESSION COEFFICIENTS y α+ + + y ˆ a+ + [ Var( + Cov(, + Cov(, ]Var( -[ Cov(, + Var( + Cov(, ]Cov(, Var( Var( -[Cov(, ] + Cov(, Var( - Var( Cov( + Cov(, Var( Cov(, Cov(,, + Cov(, Var( Cov(, Cov( Var( Var( [Cov(, ], Similarly with the terms. Notice that they are going to cancel. 7

45 PROPERTIES OF THE MULTIPLE REGRESSION COEFFICIENTS y α+ + + y ˆ a+ + [ Var( + Cov(, + Cov(, ]Var( -[ Cov(, + Var( + Cov(, ]Cov(, Var( Var( -[Cov(, ] + Cov(, Var( - Var( Cov( + Cov(, Var( Cov(, Cov(,, + Cov(, Var( Cov(, Cov( Var( Var( [Cov(, ], The coefficient of is as defined aove. This cancels with the otside the parentheses. Hence we have decomposed into the tre vale and a complicated error term. 8

46 PROPERTIES OF THE MULTIPLE REGRESSION COEFFICIENTS y α+ + + y ˆ a+ + + Cov(, Var( Cov(, Cov( Var( Var( [Cov(, ], Var( Cov(, E( + E ( Cov(, E( Cov(, We now eamine the epected vale of to determine whether it is niased. Assming that and are nonstochastic, we can write the epected vale epression as shown. 9

47 PROPERTIES OF THE MULTIPLE REGRESSION COEFFICIENTS y α+ + + y ˆ a+ + + Cov(, Var( Cov(, Cov( Var( Var( [Cov(, ], Var( Cov(, E( + E ( Cov(, E( Cov(, E[Cov(, ] and E[Cov(, ] are oth zero for the same reasons that E[Cov(, ] was zero in the simple regression model (see the second seqence in Chapter 3. 0

48 PROPERTIES OF THE MULTIPLE REGRESSION COEFFICIENTS y α+ + + y ˆ a+ + + Cov(, Var( Cov(, Cov( Var( Var( [Cov(, ], Var( Cov(, E( + E ( Cov(, E( Cov(, Hence E( is eqal to and so is an niased estimator. Similarly for.

49 PROPERTIES OF THE MULTIPLE REGRESSION COEFFICIENTS Finally we will show that a is an niased estimator of α. This is qite simple, so yo shold attempt to do this yorself, efore looking at the rest of this seqence. y α ˆ a y + + ( y a α

50 3 PROPERTIES OF THE MULTIPLE REGRESSION COEFFICIENTS First sstitte for the sample mean of y. y α ˆ a y + + ( y a α

51 4 PROPERTIES OF THE MULTIPLE REGRESSION COEFFICIENTS Now take epectations. y α ˆ a y + + ( y a α α α α ( ( ( ( E E E a E

52 5 PROPERTIES OF THE MULTIPLE REGRESSION COEFFICIENTS The epected vale of the mean of the distrance term is zero since E( is zero in each oservation. We have jst shown that E( is eqal to and that E( is eqal to. y α ˆ a y + + ( y a α α α α ( ( ( ( E E E a E

53 6 PROPERTIES OF THE MULTIPLE REGRESSION COEFFICIENTS Hence a is an niased estimator of α. y α ˆ a y + + ( y a α α α α ( ( ( ( E E E a E

54 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS

55 y α i i i a y ˆ + +, Var( pop.var( r n σ PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS This seqence investigates the poplation variances and standard errors of the slope coefficients in a model with two eplanatory variales.

56 y α i i i a y ˆ + +, Var( pop.var( r n σ PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS The epression for the poplation variance of is shown aove. The epression for is the same, with the sscripts and interchanged.

57 y α i i i a y ˆ + +, Var( pop.var( r n σ 3 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS The first factor in the epression is identical to that for the poplation variance of the slope coefficient in a simple regression model.

58 y α i i i a y ˆ + +, Var( pop.var( r n σ 4 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS The poplation variance of depends on the poplation variance of the distrance term, the nmer of oservations, and the variance of for eactly the same reasons as in a simple regression model.

59 y α i i i a y ˆ + +, Var( pop.var( r n σ 5 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS The difference is that in mltiple regression analysis the epression is mltiplied y a factor which depends on the correlation etween and.

60 y α i i i a y ˆ + +, Var( pop.var( r n σ 6 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS The higher is the correlation etween the eplanatory variales, the greater will e the poplation variance.

61 y α i i i a y ˆ + +, Var( pop.var( r n σ 7 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS This is easy to nderstand intitively. The greater the correlation, the harder it is to discriminate etween the effects of the eplanatory variales on y, and the less accrate will e the regression estimates.

62 y α i i i a y ˆ + +, Var( pop.var( r n σ 8 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS Note that the poplation variance epression aove is valid only for a model with two eplanatory variales. When there are more than two, the epression ecomes mch more comple and it is sensile to switch to matri algera.

63 y α i i i a y ˆ + +, Var( pop.var( r n σ 9 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS, Var( standard deviation of r n σ The standard deviation of the distrition of is of corse given y the sqare root of the poplation variance.

64 y α i i i a y ˆ + +, Var( pop.var( r n σ 0 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS { } ( Var n k n e E σ, Var( standard deviation of r n σ With the eception of the poplation variance of, we can calclate the components of the standard deviation from the sample data.

65 y α i i i a y ˆ + +, Var( pop.var( r n σ PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS { } ( Var n k n e E σ, Var( standard deviation of r n σ The poplation variance of has to e estimated. The sample variance of the residals provides a consistent estimator, t it is iased downwards y a factor (n-k-/n in a finite sample, where k is the nmer of eplanatory variales.

66 y α i i i a y ˆ + +, Var( pop.var( r n σ PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS { } ( Var n k n e E σ Var( e k n n s, Var( standard deviation of r n σ Oviosly we can otain an niased estimator y mltiplying the sample variance of the residals y a factor n/(n-k-. We denote this niased estimator s.

67 y α i i i a y ˆ + +, Var( pop.var( r n σ 3 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS { } ( Var n k n e E σ Var( e k n n s, Var( standard deviation of r n σ, Var( ( s.e. r n s Ths the estimate of the standard deviation of the proaility distrition of, known as the standard error of for short, is given y the epression aove.

68 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS. reg earnings hgc asvac if collarg0 Sorce SS df MS Nmer of os F(, Model Pro > F Residal R-sqared Adj R-sqared Total Root MSE earnings Coef. Std. Err. t P> t [95% Conf. Interval] hgc asvac _cons We will se this epression to analyze why the standard error of HGC in an earnings fnction regression is smaller for the non-nion ssample than for the nion ssample in an earnings fnction sing Data Set. 4

69 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS. reg earnings hgc asvac if collarg0 Sorce SS df MS Nmer of os F(, Model Pro > F Residal R-sqared Adj R-sqared Total Root MSE earnings Coef. Std. Err. t P> t [95% Conf. Interval] hgc asvac _cons To select a ssample in Stata, yo add an "if" statement to a command. The COLLBARG variale is eqal to for respondents whose rates of pay are determined y collective argaining, and it is 0 for the others. 5

70 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS. reg earnings hgc asvac if collarg0 Sorce SS df MS Nmer of os F(, Model Pro > F Residal R-sqared Adj R-sqared Total Root MSE earnings Coef. Std. Err. t P> t [95% Conf. Interval] hgc asvac _cons Note that "eqals" in Stata is rendered as a dole sign, for some arcane reason. 6

71 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS. reg earnings hgc asvac if collarg0 Sorce SS df MS Nmer of os F(, Model Pro > F Residal R-sqared Adj R-sqared Total Root MSE earnings Coef. Std. Err. t P> t [95% Conf. Interval] hgc asvac _cons In the case of the non-nion ssample, the standard error of HGC is

72 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS. reg earnings hgc asvac if collarg Sorce SS df MS Nmer of os F(, Model Pro > F Residal R-sqared Adj R-sqared Total Root MSE 5.79 earnings Coef. Std. Err. t P> t [95% Conf. Interval] hgc asvac _cons In the case of the nion ssample, the standard error of HGC is , twice as large. 8

73 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS s.e.( s n Var( r, Decomposition of the standard error of HGC Component s n Var(HGC r HGC, ASVABC s.e. Non-nion Union Factor prodct Non-nion Union We will eplain the difference y looking at the components of the standard error. 9

74 , Var( s.e.( r n s 0 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS We will start with s. RSS k n e k n e e n k n n e k n n s i i ( Var(

75 , Var( s.e.( r n s PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS We have replaced Var(e y the mathematical epression for a variance. RSS k n e k n e e n k n n e k n n s i i ( Var(

76 , Var( s.e.( r n s PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS The mean of the residals in an OLS regression mst e zero. RSS k n e k n e e n k n n e k n n s i i ( Var(

77 , Var( s.e.( r n s 3 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS Hence or estimator of the poplation variance of the distrance term can e written as the residal sm of sqares, RSS, divided y n-k-. RSS k n e k n e e n k n n e k n n s i i ( Var(

78 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS. reg earnings hgc asvac if collarg0 Sorce SS df MS Nmer of os F(, Model Pro > F Residal R-sqared Adj R-sqared Total Root MSE earnings Coef. Std. Err. t P> t [95% Conf. Interval] hgc asvac _cons Here is RSS for the non-nion ssample. 4

79 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS. reg earnings hgc asvac if collarg0 Sorce SS df MS Nmer of os F(, Model Pro > F Residal R-sqared Adj R-sqared Total Root MSE earnings Coef. Std. Err. t P> t [95% Conf. Interval] hgc asvac _cons There are 507 oservations in the non-nion ssample. k is eqal to. Ths n-k- is eqal to

80 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS. reg earnings hgc asvac if collarg0 Sorce SS df MS Nmer of os F(, Model Pro > F Residal R-sqared Adj R-sqared Total Root MSE earnings Coef. Std. Err. t P> t [95% Conf. Interval] hgc asvac _cons Ths RSS/(n-k- is eqal to To otain s, we take the sqare root. This is

81 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS s.e.( s n Var( r, Decomposition of the standard error of HGC Component s n Var(HGC r HGC, ASVABC s.e. Non-nion Union Factor prodct Non-nion Union We place this in the tale, along with the nmer of oservations. 7

82 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS. reg earnings hgc asvac if collarg Sorce SS df MS Nmer of os F(, Model Pro > F Residal R-sqared Adj R-sqared Total Root MSE 5.79 earnings Coef. Std. Err. t P> t [95% Conf. Interval] hgc asvac _cons Similarly, in the case of the nion ssample, s is the sqare root of , which is We also note that the nmer of oservations in that ssample is 63. 8

83 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS s.e.( s n Var( r, Decomposition of the standard error of HGC Component s n Var(HGC r HGC, ASVABC s.e. Non-nion Union Factor prodct Non-nion Union We place these in the tale. 9

84 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS s.e.( s n Var( r, Decomposition of the standard error of HGC Component s n Var(HGC r HGC, ASVABC s.e. Non-nion Union Factor prodct Non-nion Union We calclate the sample variance of HGC for the two ssamples from the sample data. 30

85 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS. cor hgc asvac if collarg0 (os507 hgc asvac hgc.0000 asvac cor hgc asvac if collarg (os63 hgc asvac hgc.0000 asvac The correlation coefficients for HGC and ASVABC are and for the non-nion and nion ssamples, respectively. (Note that "cor" is the Stata command for compting correlations. 3

86 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS s.e.( s n Var( r, Decomposition of the standard error of HGC Component s n Var(HGC r HGC, ASVABC s.e. Non-nion Union Factor prodct Non-nion Union These entries complete the top half of the tale. We will now look at the impact of each item on the standard error, sing the mathematical epression at the top. 3

87 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS s.e.( s n Var( r, Decomposition of the standard error of HGC Component s n Var(HGC r HGC, ASVABC s.e. Non-nion Union Factor prodct Non-nion Union The s components need no modification. It is relatively large for the non- nion ssample, having an adverse effect on the standard error. 33

88 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS s.e.( s n Var( r, Decomposition of the standard error of HGC Component s n Var(HGC r HGC, ASVABC s.e. Non-nion Union Factor prodct Non-nion Union The nmer of oservations is mch larger for the non-nion ssample, so the second factor is mch smaller than that for the nion ssample. 34

89 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS s.e.( s n Var( r, Decomposition of the standard error of HGC Component s n Var(HGC r HGC, ASVABC s.e. Non-nion Union Factor prodct Non-nion Union Perhaps a little srprisingly, the variance in schooling is aot the same for oth ssamples. 35

90 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS s.e.( s n Var( r, Decomposition of the standard error of HGC Component s n Var(HGC r HGC, ASVABC s.e. Non-nion Union Factor prodct Non-nion Union The correlation etween schooling and ASVABC is greater for the non-nion ssample, and this has an adverse effect on its standard error. 36

91 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS s.e.( s n Var( r, Decomposition of the standard error of HGC Component s n Var(HGC r HGC, ASVABC s.e. Non-nion Union Factor prodct Non-nion Union Mltiplying the for factors together, we otain the standard errors. (The discrepancies in the last digit have een cased y ronding error. 37

92 PRECISION OF THE MULTIPLE REGRESSION COEFFICIENTS s.e.( s n Var( r, Decomposition of the standard error of HGC Component s n Var(HGC r HGC, ASVABC s.e. Non-nion Union Factor prodct Non-nion Union We see that the reason that the standard error is smaller for the non-nion ssample is that there are far more oservations than in the non-nion ssample. Otherwise the standard error wold have een slightly greater. 38

93 MATRIX APPROACH n k 0 kn n n k k n k k X y.. y y Y X Y... y (X X Var(ˆ X Y (X X ˆ σ OLS ESTIMATES

94 MATRIX APPROACH k k k k k k k n X X y. y y X Y k (X X Var(ˆ X Y (X X ˆ σ OLS ESTIMATES

95 STANDARDIZED OLS REGRESSION COEFFICENTS Standardized variales X X Z (X-µ/σ r r3... rk r r3... r r3 r3... r... rk rk rk3... k 3k X Y r r. r y y k y STANDARDIZED OLS ESTIMATES ˆ (X X X Y Var(ˆ σ (X X

96 MULTICOLLINEARITY

97 MULTICOLLINEARITY y Change Change Change y in in in y Sppose that y and that -. There is no distrance term in the eqation for y, t that is not important. Sppose that we have the si oservations shown.

98 MULTICOLLINEARITY y The three variales are plotted as line graphs aove. Looking at the data, it is impossile to tell whether the changes in y are cased y changes in, y changes in, or jointly y changes in oth and.

99 MULTICOLLINEARITY y Change Change Change y in in in y Nmerically, y increases y 5 in oservation. changes y. 3

100 MULTICOLLINEARITY y Hence the tre relationship cold have een y

101 MULTICOLLINEARITY y Change Change Change y in in in y However, it can also e seen that increases y in each oservation. 5

102 MULTICOLLINEARITY y Hence the tre relationship cold have een y

103 MULTICOLLINEARITY y These two possiilities are special cases of y p + 5p +.5(-p, which wold fit the relationship for any vale of p. 7

104 MULTICOLLINEARITY y There is no way that regression analysis, or any other techniqe, cold determine the tre relationship from this infinite set of possiilities, given the sample data. 8

105 MULTICOLLINEARITY y α+ + + λ+ µ What wold happen if yo tried to rn a regression when there is an eact linear relationship among the eplanatory variales? 9

106 MULTICOLLINEARITY y α+ + + λ+ µ We will investigate, sing the model with two eplanatory variales shown aove. [Note: A distrance term has now een inclded in the tre model, t it makes no difference to the analysis.] 0

107 MULTICOLLINEARITY y α+ + + λ+ µ Cov(,yVar( Var( Var( - Cov(, ycov( [ Cov(, ], Cov( Cov(,yVar( λ+ µ Var( Var(,yVar( µ Var( µ Var( λ+ µ - Cov([ λ+ µ - Cov( µ [ Cov(,[ λ+ µ ]] [ Cov(, µ ], ycov( ], ycov(, µ,[ λ+ µ ] The mltiple regression coefficient is calclated as shown.

108 MULTICOLLINEARITY y α+ + + λ+ µ Cov(,yVar( Var( Var( - Cov(, ycov( [ Cov(, ], Cov( Cov(,yVar( λ+ µ Var( Var(,yVar( µ Var( µ Var( λ+ µ - Cov([ λ+ µ - Cov( µ [ Cov(,[ λ+ µ ]] [ Cov(, µ ], ycov( ], ycov(, µ,[ λ+ µ ] Sstitte for wherever it appears.

109 MULTICOLLINEARITY y α+ + + λ+ µ Cov(,yVar( Var( Var( - Cov(, ycov( [ Cov(, ], Cov( Cov(,yVar( λ+ µ Var( Var(,yVar( µ Var( µ Var( λ+ µ - Cov([ λ+ µ - Cov( µ [ Cov(,[ λ+ µ ]] [ Cov(, µ ], ycov( ], ycov(, µ,[ λ+ µ ] Using Variance Rle 4, we can drop the additive λ in the variance terms. 3

110 MULTICOLLINEARITY Covy ([ λ α+ µ ], y+ Cov( + λ, y + Cov( µ, λ+ Cov( µ y µ, y Cov(,yVar( Var( Var( - Cov(, ycov( [ Cov(, ], Cov( Cov(,yVar( λ+ µ Var( Var(,yVar( µ Var( µ Var( λ+ µ - Cov([ λ+ µ - Cov( µ [ Cov(,[ λ+ µ ]] [ Cov(, µ ], ycov( ], ycov(, µ,[ λ+ µ ] Likewise, sing Covariance Rles and 3, we can drop the additive λ in the covariance terms. 4

111 MULTICOLLINEARITY Cov( y,[ α ] + Cov( +, Cov(, λ+ Cov( µ λ + µ λ + µ, µ Cov(,yVar( Var( Var( - Cov(, ycov( [ Cov(, ], Cov( Cov(,yVar( λ+ µ Var( Var(,yVar( µ Var( µ Var( λ+ µ - Cov([ λ+ µ - Cov( µ [ Cov(,[ λ+ µ ]] [ Cov(, µ ], ycov( ], ycov(, µ,[ λ+ µ ] Likewise, sing Covariance Rles and 3, we can drop the additive λ in the covariance terms. 5

112 MULTICOLLINEARITY y α+ + + λ+ µ Cov(,yVar( µ Var( Var( µ - Cov( µ, ycov( [ Cov(, µ ], µ Cov(,y µ Var( Var( µ Var( - Cov( µ, y µ Cov( [ µ Cov(, ], µ Cov(,yVar( µ Var( Var( -µ Cov(, yvar( 0 [ µ Var( ] 0 This is the last line from the previos slide. 6

113 MULTICOLLINEARITY y α+ + + λ+ µ Cov(,yVar( µ Var( Var( µ - Cov( µ, ycov( [ Cov(, µ ], µ Cov(,y µ Var( Var( µ Var( -µ Cov(, y µ Cov( [ µ Cov(, ], µ Cov(,yVar( µ Var( Var( -µ Cov(, yvar( 0 [ µ Var( ] 0 We can take µ ot of the variance terms, sqaring it as we do. 7

114 MULTICOLLINEARITY y α+ + + λ+ µ Cov(,yVar( µ Var( Var( µ - Cov( µ, ycov( [ Cov(, µ ], µ Cov(,y µ Var( Var( µ Var( -µ Cov(, y µ Cov( [ µ Cov(, ], µ Cov(,yVar( µ Var( Var( -µ Cov(, yvar( 0 [ µ Var( ] 0 Likewise we can take µ ot of the covariance terms. 8

115 MULTICOLLINEARITY y α+ + + λ+ µ Cov(,yVar( µ Var( Var( µ - Cov( µ, ycov( [ Cov(, µ ], µ Cov(,y µ Var( Var( µ Var( -µ Cov(, y µ Cov( [ µ Cov(, ], µ Cov(,yVar( µ Var( Var( -µ Cov(, yvar( 0 [ µ Var( ] 0 Cov(, is the same as Var(. 9

116 MULTICOLLINEARITY y α+ + + λ+ µ Cov(,yVar( µ Var( Var( µ - Cov( µ, ycov( [ Cov(, µ ], µ Cov(,y µ Var( Var( µ Var( -µ Cov(, y µ Cov( [ µ Cov(, ], µ Cov(,yVar( µ Var( Var( -µ Cov(, yvar( 0 [ µ Var( ] 0 It trns ot that oth the nmerator and the denominator are eqal to zero. The regression coefficient is not defined. 0

117 MULTICOLLINEARITY y α+ + + λ+ µ Cov(,yVar( µ Var( Var( µ - Cov( µ, ycov( [ Cov(, µ ], µ Cov(,y µ Var( Var( µ Var( -µ Cov(, y µ Cov( [ µ Cov(, ], µ Cov(,yVar( µ Var( Var( -µ Cov(, yvar( 0 [ µ Var( ] 0 It is nsal for there to e an eact relationship among the eplanatory variales in a regression. When this occrs, it s typically ecase there is a logical error in the specification.

118 MULTICOLLINEARITY. reg earnings hgc asvac asva5 Sorce SS df MS Nmer of os F( 3, Model Pro > F Residal R-sqared Adj R-sqared 0.3 Total Root MSE earnings Coef. Std. Err. t P> t [95% Conf. Interval] hgc asvac asva _cons However, if often happens that there is an approimate relationships. Here is a regression of EARNINGS on HGC, ASVABC, and ASVAB5. ASVAB5 is the score on a speed test of the aility to perform very simple arithmetical comptations.

119 MULTICOLLINEARITY. reg earnings hgc asvac asva5 Sorce SS df MS Nmer of os F( 3, Model Pro > F Residal R-sqared Adj R-sqared 0.3 Total Root MSE earnings Coef. Std. Err. t P> t [95% Conf. Interval] hgc asvac asva _cons Like ASVABC, the raw scores on this test were scaled so that they had mean 50 and standard deviation 0. 3

120 MULTICOLLINEARITY. reg earnings hgc asvac asva5 Sorce SS df MS Nmer of os F( 3, Model Pro > F Residal R-sqared Adj R-sqared 0.3 Total Root MSE earnings Coef. Std. Err. t P> t [95% Conf. Interval] hgc asvac asva _cons The regression reslt indicates that an etra year of schooling increases horly earnings y $0.7. 4

121 MULTICOLLINEARITY. reg earnings hgc asvac asva5 Sorce SS df MS Nmer of os F( 3, Model Pro > F Residal R-sqared Adj R-sqared 0.3 Total Root MSE earnings Coef. Std. Err. t P> t [95% Conf. Interval] hgc asvac asva _cons An etra point on ASVABC increases horly earnings y $0.. Someone with a score one standard deviation aove the mean wold therefore tend to earn an etra $.0 per hor, compared with someone at the mean. 5

122 MULTICOLLINEARITY. reg earnings hgc asvac asva5 Sorce SS df MS Nmer of os F( 3, Model Pro > F Residal R-sqared Adj R-sqared 0.3 Total Root MSE earnings Coef. Std. Err. t P> t [95% Conf. Interval] hgc asvac asva _cons An etra point on the nmerical comptation speed test increases horly earnings y $

123 MULTICOLLINEARITY. reg earnings hgc asvac asva5 Sorce SS df MS Nmer of os F( 3, Model Pro > F Residal R-sqared Adj R-sqared 0.3 Total Root MSE earnings Coef. Std. Err. t P> t [95% Conf. Interval] hgc asvac asva _cons Does ASVAB5 elong in the earnings fnction? If we perform a t test, we find that its coefficient is jst significantly different from zero at the 5% level, sing a one-tailed test. (Jstification: it is nlikely that a good score on this test wold adversely affect earnings. 7

124 MULTICOLLINEARITY. reg earnings hgc asvac asva5 Sorce SS df MS Nmer of os F( 3, Model Pro > F Residal R-sqared Adj R-sqared 0.3 Total Root MSE earnings Coef. Std. Err. t P> t [95% Conf. Interval] hgc asvac asva _cons We note that in this regression, the coefficient of ASVABC is significant only at the 5% level. 8

125 MULTICOLLINEARITY. reg earnings hgc asvac Sorce SS df MS Nmer of os F(, Model Pro > F Residal R-sqared Adj R-sqared 0.05 Total Root MSE earnings Coef. Std. Err. t P> t [95% Conf. Interval] hgc asvac _cons In the regression withot ASVAB5, its t statistic was 3.60, making it significantly different from zero at the 0.% level. 9

126 MULTICOLLINEARITY. reg earnings hgc asvac asva5 Sorce SS df MS Nmer of os F( 3, Model Pro > F Residal R-sqared Adj R-sqared 0.3 Total Root MSE cor asvac asva5 earnings Coef. Std. Err. t P> t [95% Conf. Interval] (os hgc asvac asva5 asvac asva asvac.0000 _cons asva The reason for the redction in its t ratio is that it is highly correlated with ASVAB5. 30

127 MULTICOLLINEARITY. reg earnings hgc asvac asva5 Sorce SS df MS Nmer of os F( 3, Model Pro > F Residal R-sqared Adj R-sqared 0.3 Total Root MSE cor asvac asva5 earnings Coef. Std. Err. t P> t [95% Conf. Interval] (os hgc asvac asva5 asvac asva asvac.0000 _cons asva This makes it difficlt to pinpoint the individal effects of ASVABC and ASVAB5. As a conseqence the regression estimates tend to e erratic. 3

128 MULTICOLLINEARITY. reg earnings hgc asvac asva5 earnings Coef. Std. Err. t P> t [95% Conf. Interval] hgc asvac asva _cons reg earnings hgc asvac earnings Coef. Std. Err. t P> t [95% Conf. Interval] hgc asvac _cons The high correlation cases the standard errors to e larger than they wold have een if ASVABC and ASVAB5 had een less highly correlated, warning s that the point estimates are nreliale. 3

129 MULTICOLLINEARITY. reg earnings hgc asvac asva5 earnings Coef. Std. Err. t P> t [95% Conf. Interval] hgc asvac asva _cons reg earnings hgc asvac earnings Coef. Std. Err. t P> t [95% Conf. Interval] hgc asvac _cons When high correlations among the eplanatory variales lead to erratic point estimates of the coefficients, large standard errors and nsatisfactorily low t statistics, the regression is said to said to e sffering from mlticollinearity. 33