Hypothesis testing Goodness of fit Multicollinearity Prediction. Applied Statistics. Lecturer: Serena Arima

Transcription

1 Applied Statistics Lecturer: Serena Arima

2 Hypothesis testing for the linear model Under the Gauss-Markov assumptions and the normality of the error terms, we saw that β N(β, σ 2 (X X ) 1 ) and hence s 2 = 1 n k e2 i χ n k V (β) = s 2 (X X ) 1 χ n k These results can be used to construct test statistics and confidence intervals.

3 Basic ideas of statistical test Starting from a given hypothesis, the null hypothesis, a test statistic is computed. This statistic has a known distribution under the assumption that the null hypothesis is valid. Then we compute the value of the test statistic according to our data and decide whether the computed value of the test statistic is unlikely to come from this distribution, which indicates that the null hypothesis is unlikely to hold.

4 Suppose we have A simple t-test H 0 : β k = β 0 k H 1 : β k β 0 k where βk 0 is a specific value chosen by the researcher. If this hypothesis is true, we know that the test statistic is t k = β k β 0 k se( β k ) T n k If the test statistic realizes a value that is very unlikely under the null distribution, we reject the null hypothesis.

5 A simple t-test To be precise, one rejects the null hypothesis if P( t k > t n k,α/2 ) α where t n k,α/2 is the critical value and its the α/2 quantile of a T distribution with n k degrees of freedom. We remind that α is the significance level. This test is defined as two sided test. For one-sided test such as H 1 = β k > β 0 k or H 1 = β k < β 0 k the null hypothesis is rejected when P(t k > t n k,α ) α

6 A simple t-test To be precise, one rejects the null hypothesis if P( t k > t n k,α/2 ) α where t n k,α/2 is the critical value and its the α/2 quantile of a T distribution with n k degrees of freedom. We remind that α is the significance level. This test is defined as two sided test. For one-sided test such as H 1 = β k > β 0 k or H 1 = β k < β 0 k the null hypothesis is rejected when P(t k > t n k,α ) α

7 A simple t-test A confidence interval can be defined as the interval of all values βk 0 for which the null hypothesis H 0 : β k = βk 0 is not rejected by the t-test. It is derived from the fact that t n k,α/2 < β k β k se( β k ) < t n k,α/2 and hence the confidence interval is β k t n k,α/2 se( β k ) < β k < β k + t n k,α/2 se( β k ) It means that in repeated sampling, 95% of these intervals will contain the true value β k.

8 A simple t-test A confidence interval can be defined as the interval of all values βk 0 for which the null hypothesis H 0 : β k = βk 0 is not rejected by the t-test. It is derived from the fact that t n k,α/2 < β k β k se( β k ) < t n k,α/2 and hence the confidence interval is β k t n k,α/2 se( β k ) < β k < β k + t n k,α/2 se( β k ) It means that in repeated sampling, 95% of these intervals will contain the true value β k.

9 A simple t-test In the regression context, we are interest in evaluating whether the variable X k has a significant influence on the response variable. Hence we usually consider βk 0 = 0 that is and the test statistic is H 0 : β k = 0 H 1 : β k 0 t = β k se( β k ) If H 0 is rejected, it is said that β k significantly differs from 0 or that the corresponding variable X k has a significant impact on y i ; If H 0 is not rejected, it is said that the variable X k does not have a significant impact on y i.

10 Example: a simple t-test Consider the Individual Wages example: the model is If we want to test y i = x i Parameter Estimate Standard Error β β H 0 : β 1 = 0 H 1 : β 1 0 the statistic is T = β 1 se( β 1 ) T n 2

11 Example: a simple t-test We can compute the p-value: ( P value = P(T > t) = P T > ) = P(T > ) In the same way, we can compute the confidence interval: β 1 ± t n 1,1 α/2 se( β 1 ) ± [0.9461; ] This means that with 95% confidence we can say that over the entire population the expected wage differential males and females is between $ and $ per hour.

12 Example: a simple t-test We can compute the p-value: ( P value = P(T > t) = P T > ) = P(T > ) In the same way, we can compute the confidence interval: β 1 ± t n 1,1 α/2 se( β 1 ) ± [0.9461; ] This means that with 95% confidence we can say that over the entire population the expected wage differential males and females is between $ and $ per hour.

13 A joint test on regression coefficients Suppose we want to test the null hypothesis that J of the K coefficients are equal to zero (J < K): H 0 : β K J+1 =... = β K The alternative hypothesis in this case is that H 0 is not true, that is at least one of these J coefficients is not equal to zero. Full model M 1 : y = β 0 + β 1 X β K X k + ɛ Restricted model M 0 : y = β 0 + β 1 X β J X j + ɛ

14 A joint test on regression coefficients Suppose we want to test the null hypothesis that J of the K coefficients are equal to zero (J < K): H 0 : β K J+1 =... = β K The alternative hypothesis in this case is that H 0 is not true, that is at least one of these J coefficients is not equal to zero. Full model M 1 : y = β 0 + β 1 X β K X k + ɛ Restricted model M 0 : y = β 0 + β 1 X β J X j + ɛ

15 A joint test on regression coefficients Let S 1 : residual sum of squares of the full model; S 0 : residual sum of squares of the restricted model. Under the null hypothesis we have F = (S 0 S 1 )/J S 1 /(n k) F J,N K This test is refereed to as F - test.

16 A joint test on regression coefficients In several statistical software, the F-test aims at testing H 0 : β 0 = β 1 =... = β K = 0 The alternative hypothesis in this case is that H 0 is not true, that is at least one of these coefficients is not equal to zero. What does it mean when we reject the null hypothesis?

17 A joint test on regression coefficients In several statistical software, the F-test aims at testing H 0 : β 0 = β 1 =... = β K = 0 The alternative hypothesis in this case is that H 0 is not true, that is at least one of these coefficients is not equal to zero. What does it mean when we reject the null hypothesis?

18 F test vs T test In most applications the different estimators β 1,..., β K will be correlated, which means that the explanatory powers of the explanatory variables overlap; Consequently, the marginal contribution of each explanatory variable may be quite small; Hence it is possible that all t-tests on each variable s coefficient to be insignificant, while F-test for a number of these coefficients is highly significant.

19 F test vs T test That is, it is possible that the null hypothesis H 0 : β 1 = 0 is not rejected and that the null β 2 = 0 is not rejected, but the joint null β 1 = β 2 = 0 is quite unlikely to be true. As a consequence, in general, t-test on each restriction separately may not reject while a joint F-test does.

20 The General case The most general linear null hypothesis is a combination of the previous cases and comprises a set of J linear restrictions on the coefficients. We can formulate these restrictions as Rβ = q where R is a J K matrix and q is a J-dimensional vector (K number of variables and J number of constraints).

21 The General case An example of this is the ret of restrictions β 2 + β β K = 1 β 2 = β 3 in which case ( R = ) q = ( 1 0 )

22 The General case The statistic we use is F = (Rβ q) (R(X X ) 1 R ) 1 (Rβ q) Js 2 F J,N K As before, large values of F lead to rejection of the null. When we are testing one linear restriction (J = 1) it can be shown that it is the square of the corresponding t statistic (check on R examples) and the two tests are equivalent.

23 The goodness of fit Having estimated a particular linear model, a goodness of fit measure is R 2 defined as R 2 = V (ŷ i ) V (y i ) 1 1/(N 1) N i=1 = (ŷ i ȳ) 2 1/(N 1) N i=1 (y i ȳ) = 2 1/(N 1) N i=1 e2 i 1/(N 1) N i=1 (y i ȳ) 2 It indicates which proportion of the sample variation in y i is explained by the model

24 The goodness of fit It can be shown that 0 R 2 1 R 2 = 1 e i = 0 for all i; R 2 = 0 the model does not explain anything in addition to the sample mean of y i, that is the model with intercept only. For the simple regression model, it can be shown R 2 = corr(y i, ŷ i ) 2

25 The goodness of fit An important drawback of R 2 is that it will never decrease if the number of regressors is increased, even if the additional variables have no real explanatory power. A common way to solve this is to correct the index as follows R 2 = 1 1/(N K) N i=1 e2 i 1/(N 1) N i=1 (y i ȳ) 2 defined as adjusted R 2. This goodness of fit measure has some punishment for the inclusion of additional explanatory variables in the model and therefore it does not automatically increase when regressors are added to the model.

26 The goodness of fit An important drawback of R 2 is that it will never decrease if the number of regressors is increased, even if the additional variables have no real explanatory power. A common way to solve this is to correct the index as follows R 2 = 1 1/(N K) N i=1 e2 i 1/(N 1) N i=1 (y i ȳ) 2 defined as adjusted R 2. This goodness of fit measure has some punishment for the inclusion of additional explanatory variables in the model and therefore it does not automatically increase when regressors are added to the model.

27 The OLS estimator for β is β = (X X ) 1 X y Multicollinearity In general the term multicollinearity is used to describe the problem when an approximate linear relationship among the explanatory variables leads to unreliable regression estimates. when X s are perfectly correlated X X cannot be inverted; when X s are strongly correlated unreliable estimates with high standard errors and of unexpected sign or magnitude; when X s are very correlated difficult interpretation of the parameters.

28 The OLS estimator for β is β = (X X ) 1 X y Multicollinearity In general the term multicollinearity is used to describe the problem when an approximate linear relationship among the explanatory variables leads to unreliable regression estimates. when X s are perfectly correlated X X cannot be inverted; when X s are strongly correlated unreliable estimates with high standard errors and of unexpected sign or magnitude; when X s are very correlated difficult interpretation of the parameters.

29 Multicollinearity The last situation is the most common one. Consider the wage data: if age and experience are highly correlated it may be hard for the model to identify the individual impact of these two variables. In this situation it may happen that we have a large number of observations with sufficient variation in both age and experience so that we can somehow quantify the effect of the two variables; if this is not the case, we can only conclude that there is insufficient information to identify their effects (not significant t-tests).

30 Multicollinearity The variance inflation factor is used to detect multicollinearity. It is given by 1 VIF ( β k ) = 1 Rk 2 where Rk 2 denotes squared the multiple correlation coefficient between x ik and the other explanatory variables (i.e. the R 2 from regressing x ik upon the other explanatory variables).

31 Multicollinearity VIF indicates the factor by which the variance of β k is inflated compared with the hypothetical situation when there is no correlation between the variables. Large values of VIF (VIF> 10) are problematic: an inspection of VIFs may be helpful if estimation results are unsatisfactory and suspected to be affected by multicollinearity.

32 Prediction Suppose we want to predict, using the regression model, the value for the dependent variable at a given value of the explanatory variables x 0. The obvious predictor is that can be estimated as y 0 = x 0β + ɛ 0 ŷ 0 = x 0 β As E[ β] = β it is easily verified that E[ŷ 0 y 0 ] = 0 that ŷ 0 is an unbiased predictor.

33 Prediction Under the Gauss-Markov assumptions, the variance of the predictor is V (ŷ 0 ) = V (x 0 β) = x 0V ( β)x 0 = σ 2 x 0(X X ) 1 x 0 This variance, however, is only an indication of the variation in the predictor if different samples were drawn. In order to evaluate the accuracy of the predictor, we need the variance of the prediction error defined as ŷ 0 y 0 whose variance is V (ŷ 0 y 0 ) = σ 2 + σ 2 x 0(X X ) 1 x 0

34 Prediction interval Hence, the accuracy of the prediction is reflected in a so called prediction interval. A 95% prediction interval for y 0 is [x 0 β 1.96s 1 + x 0 (X X ) 1 x 0 ; x 0 β s 1 + x 0 (X X ) 1 x 0 ] where s is the estimate of σ and 1.96 is the critical value of a standard normal distribution. With probability 95% this interval contains the true unobserved value of y 0.

35 Prediction in econometrics Econometric predictions are useful in different ways. 1 They can be employed to determine the expected value of y for a unit that is not included in the sample: for example, we can determine the expected sales price of a house give its characteristics; 2 We can predict the value of y under alternative values of x; for example, we could try to predict the reduction in cigarette consumption if the sales tax on cigarettes would be increased by 50 cents per package; 3 We can samply try to predict a future outcome of y given currently observed values of x, using a time series model; for example, we can try to predict next month s stock market returns given historical returns and other variables.

36 Prediction in econometrics Econometric predictions are useful in different ways. 1 They can be employed to determine the expected value of y for a unit that is not included in the sample: for example, we can determine the expected sales price of a house give its characteristics; 2 We can predict the value of y under alternative values of x; for example, we could try to predict the reduction in cigarette consumption if the sales tax on cigarettes would be increased by 50 cents per package; 3 We can samply try to predict a future outcome of y given currently observed values of x, using a time series model; for example, we can try to predict next month s stock market returns given historical returns and other variables.

37 Prediction in econometrics Econometric predictions are useful in different ways. 1 They can be employed to determine the expected value of y for a unit that is not included in the sample: for example, we can determine the expected sales price of a house give its characteristics; 2 We can predict the value of y under alternative values of x; for example, we could try to predict the reduction in cigarette consumption if the sales tax on cigarettes would be increased by 50 cents per package; 3 We can samply try to predict a future outcome of y given currently observed values of x, using a time series model; for example, we can try to predict next month s stock market returns given historical returns and other variables.