Multiple Linear Regression Analysis

Transcription

1 LINEAR REGREION ANALYSIS MODULE III Lecture 3 Multiple Linear Regsion Analysis Dr Shalabh Department of Mathematics and Statistics Indian Institute of Technology Kanpur

2 Likelihood ratio test for H : Rβ r The same logic and reasons used in the development of likelihood ratio test for the likelihood ratio test for Let β H : Rβ r as follows: {( β, ) : βi,, i,,, k} {(, ): i, R r, } Ω < < > ω β < β< β > ( X ' X) X ' y H : β β can be extended to develop Then ER ( β) Rβ V( R β) E R( β β)( β β)' R' RV ( β ) R ' RX ( ' X) R' Since β~ N β, ( X ' X) so R β~ N Rβ, RX ( ' X) R' R β r R β Rβ R β β N RX X R ( ) ~, ( ' ) '

3 3 There exists a matrix Q such that and then Therefore under ( ' ) ' ' R X X R QQ ξ QR b β N I n ( ) (, ) H : Rβ r, ξξ ' ( R β r)' QQ'( R β r) ( R β r)' R( X ' X) R' ( R ) β r ( β β)' R' RX ( ' X) R' R( β β) ε' X( X ' X) R' R( X ' X) R' R( X ' X) X ' ε ~ χ ( J ) which is obtained as X( X ' X) R' R( X ' X) R' R( X ' X) X ' associated degrees of freedom with the pective quadratic form Also, irpective of whether H is true or not, is an idempotent matrix and its trace is J which is the e ' e ( yxβ)'( yxβ) y' Hy ( nk) ˆ χ ~ ( n k)

4 4 Moreover, the product of quadratic form matrices of and ( β β)' R' RX ( ' X) R' R( β β) is zero implying that both the quadratic forms are independent ee ' So in terms of likelihood ratio test statistic λ β ( R r)' R( X ' X) R' ( R r) J ( R β r)' R( X ' X) R' ( R β r) ) J ˆ ~ FJn (, k) under H ( n k) ˆ n k β So the decision rule is to reject H whenever λ Fα ( Jn, k) where F( Jn, k) α is the upper critical points on the central F distribution with J and (n k) degrees of freedom

5 Test of significance of regsion (Analysis of variance) If we set R [ I ], r, then the hypothesis H : Rβ r reduces to the following null hypothesis: k H β β3 β k : which is tested against the alternative hypothesis H : β j for at least one j, 3,, k 5 This hypothesis determines if there is a linear relationship between y and any set of the explanatory variables Notice that X corponds to the intercept term in the model and hence x for all i,,, n i X, X3,, X k This is an overall or global test of model adequacy Rejection of the null hypothesis indicates that at least one of the explanatory variables among X, X3,, Xk contributes significantly to the model This is called as analysis of variance Since ε ~ N(, I), so y N XβI ~ (, ) b X X X y N β X X ( ' ) ' ~, ( ' ) Also ˆ n k ( y yˆ)'( y yˆ) n k y' I X( X ' X) X ' y y ' Hy y ' y b ' X ' y nk nk nk Since X X X H so b and - ( ' ) ', ˆ are independently distributed

6 6 Since y ' Hy ε' Hε and is an idempotent matrix, so H ~ χ ( nk), ie, central χ distribution with (n - k) degrees of freedom Partition X [ X, X ] where the submatrix X contains the explanatory variables X, X3,, Xk and partition β [ β, β ] where the subvector contains the regsion coefficients β, β,, β k β 3 Now partition the total sum of squa due to y s as where T y ' Ay + reg b ' X ' b reg is the sum of squa due to regsion and b is the OLSE of β The sum of squa due to iduals is given by ( y Xb)'( y Xb) y ' Hy T re g

7 Further 7 and the mean square due to error is Then re g β ' X ' β β ' X ' β ~ χ k, ie, non-central χ distribution with non centrality parameter, T β ' X ' β β ' X ' β ~ χ n, ie, non-central χ distribution with n on centrality parameter Since X H, so and are independently distributed The mean squa due to regsion is reg MS MS re g re g k n k re s MSreg β ' X ' β ~ Fk, nk MS which is a non-central F -distribution with (k )(n k) degrees of freedom and non-centrality parameter β ' X ' β Under H : β β β k, 3 MS F MS reg ~ F k, nk The decision rule is to reject at F F ( k, nk) α α level of significance whenever

8 8 The calculation of F-statistic can be summarized in the form of an analysis of variance (ANOVA) table given as follows: Source of variation Sum of squa Degrees of freedom Mean squa F Regsion Error re g k n k MSreg r e g / k MS r e s /( n k) F Total n T Rejection of H indicates that it is likely that atleast one βi ( i,,, k)

9 9 Test of hypothesis on individual regsion coefficients In case the test in analysis of variance is rejected, then another question arises is that which of the regsion coefficients is/are ponsible for the rejection of null hypothesis The explanatory variables corponding to such regsion coefficients are important for the model Adding such explanatory variables also increases the variance of fitted values, so one needs to be cautious that only those regsors are added which are really important in explaining the ponse Adding unimportant explanatory variables may increase the idual mean square which may decrease the usefulness of the model ŷ H : β j To test the null hypothesis versus the alternative hypothesis H : β j has already been discussed is the case of simple linear regsion model In pent case, if H is accepted, it implies that the explanatory variable X j can be deleted from the model The corponding test statistic is bj t ~ tn ( k) under H se( b ) j where the standard error of OLSE bj of β j is where C jj denotes the j th se( b j ) ˆ C jj diagonal element of ( X ' X) corponding to b j The decision rule is to reject H at level of significance if α t > t α, n k Note that this is only a partial or marginal test because that are in the model This is a test of the contribution of b depends on all the other explanatory variables X ( i j) j X j given the other explanatory variables in the model i