17: INFERENCE FOR MULTIPLE REGRESSION. Inference for Individual Regression Coefficients

Transcription

1 17: INFERENCE FOR MULTIPLE REGRESSION Inference for Individual Regression Coefficients The results of this section require the assumption that the errors u are normally distributed. Let c i ij denote the (i,j ) entry of (X X ). Then b and 1 j s u j j are independently distributed as N (β, σ c ), and σ u n p 1 χ /(n p 1), respectively. An unbiased estimate of var (b ) is given by s c. j jj j W j e can test H : β =b * using the test statistic t = * b b s c jj j. If H is true, then t = N (, σ c ) u jj cjj σu χ n p 1 /(n p 1)

2 = -- N (, 1). χ n p 1 /(n p 1) Therefore, t has a t -distribution with n p 1 degrees of freedom under H. This fact can be used to show that a level 1 α confidence interval for β j is given by b ± t j α/ ; (n p 1) s c jj. Simultaneous Inference for All Regression Coefficients If a confidence region is desired for the entire ( p +1) 1 vector β of regression coefficients, it would not be appropriate to use the (p +1) indivi- confidence intervals. dual (Why?)

3 -3- Definition: If χ and χ are two independent χ ν N ν D random variables with degrees of freedom ν N and ν D respectively, then the ratio random variable F = χ / ν χ / ν N ν ν D N D is said to have an F-Distribution with ν N and ν D degrees of freedom. The mean of this distribu- D D tion is ν /(ν ). Table 4 of Jobson gives critical values for the F distribution. It is possible to construct an invertible and sym- metric matrix C of dimension (p +1) (p +1) such that (X X ) 1= C. Consider the (p +1) dimensional 1 vector w = C (b β). Then w is multivariate

4 -4- normal with mean zero, and covariance matrix Cov (w ) = E [w w ] = C 1E[(b β)(b β) ]C = C Cov (b ) C = C 1σ (X X ) 1C 1 u =σu C 1 C C 1 =σu I p +1, so that the entries of w are iid N (, σ ). u p + 1 u Conse- quently, w w is distributed as σ χ. Since w w is independent of s, we find that w w /{(p +1)s } = (b β) X X (b β)/{(p +1)s } has an F -distribution with p +1 and n p 1 degrees of freedom. Therefore, we can test the hypothesis H * : β=β versus H : β β using the test statistic 1 F = (b β *) X X (b β *)/{(p +1)s }. *

5 -5- The test which rejects H whenever F > F α ; p +1,n p 1 will have level α. (Why?) Suppose we want a level 1 α confidence region for β. Consider the set of all parameter vectors β * such that F F α ; p +1,n p 1, i.e., such that (b β *) X X (b β *) (p +1)sF α ; p +1,n p 1.(1) The region described by (1) consists of the β values for which the null hypothesis H : β=β * * rior of a (p +1)-dimensional ellipsoid. Clearly, the would not be rejected by our level-α test. Geometr- ically, the region consists of the boundary and interegion will contain the true parameter vector β if and only if

6 -6- (b β) X X (b β) (p +1)sF α ; p +1,n p 1. () Under repeated sampling, the event described by () has probability 1 α. Therefore, the ellipsoid (1) will contain β (1 α) 1% of the time, in the long run. We call the region (1) a level 1 α confidence ellipsoid for β. Inference For the Model It is always possible that our "explanatory vari- ables" are completely useless for describing the behavior of E [y X ]. Therefore, we may want to test the hypothesis H : β =β =... =β =. 1 p

7 -7- Under H, the true regression function does not depend at all on our explanatory variables, although our estimator b will almost certainly be different from zero, due to natural variability, i.e., "noise", as opposed to "signal". We can test H using the overall goodness of fit test, using the statistic SSR / p F = SSE /(n p 1) MSR =. (3) MSE Under H, the test statistic has an F -distribution with p and n p 1 degrees of freedom. To prove this, recall the orthogonal matrix V and dout. the random vector z = V y from the previous han- It is possible to arrange so that the first column of V is v = (1/ n,...,1/ n ). Define y

8 -8- as the n 1 vector, with all entries equal to 1 n y. n Σ i i = 1 n Σ 1 Since y = zv, j j each entry of y is equal to j = 1 n z V. But since v,...,v are all n Σ 1 Σ n j ij 1 n 1 j = i =1 n ij i =1 orthogonal to v, it follows that Σ V = ifj >. Therefore, y = z v. We have Σ p j j Σ p ŷ y = zv z v = zv j j j = j =1, and n 1 y ŷ = Σ zv j j. j =p +1 Under H, z 1,...,zn 1 u are iid N (, σ ). Therefore, SSR = ŷ y and SSE = y ŷ are

9 -9- independent, and distributed as σ χ and σ u χ n p 1 u p, respectively. So F defined by (3) does have the indicated F -distribution. From a practical point of view, there are some problems with the goodness of fit test. If the F statistic exceeds the critical value, then we have some indication that at least one of the β is i nonzero. However, the test gives us no clue as to which of the β is (are) nonzero. Unfortunately, i this is precisely what we will typically want to know in practice. Going back to the individual t - statistics for each parameter and picking those which are significant does not solve the problem. ( Why?)

10 -1- It is tempting to conclude that if H is rejected, the model, with all p variables, must be "good". But this is not necessarily true. For example, if β, but β =... =β =, then we will typically 1 p reject H, but most of the variables in the model are useless. A better model (e.g., for prediction), would be the one with just the first variable. So even if H is rejected, we still need to worry about the possibility that we have too many variables in our model.