Introduction to F-testing in linear regression models

Size: px

Start display at page:

Download "Introduction to F-testing in linear regression models"

Ernest Allen
6 years ago
Views:

1 ECON 43 Harald Goldste, revsed Nov. 4 Itroducto to F-testg lear regso s (Lecture ote to lecture Frday 4..4) Itroducto A F-test usually s a test where several parameters are volved at oce the ull hypothess cotrast to a T-test that cocers oly oe parameter. The F-test ca ofte be cosde a refemet of the more geeral lelhood rato test (LR) cosde as a large sample ch-square test. The F-test ca (e.g.) be used the specal case that the error term a regso s ormally dstrbuted. Ths s the same way as the T-test for a sgle parameter a wth ormally dstrbuted data s a refemet of a more geeral large sample Z-test. The F-test (as the T-test) ca be used also for small data sets cotrast to the large sample ch-square tests (ad large sample Z-tests), but requre addtoal assumptos of ormally dstrbuted data (or error terms). Note also that, f the ull-hypothess cossts of oly oe parameter, the the F ad T test statstcs satsfy F T exactly, so that a two-sded T-test wth d degrees of freedom s equvalet to a F-test wth ad d degrees of freedom. Example from o-semar exercse wee 39 (Hog Kog cosumer data). Y Cosumpto (me): housg, cludg fuel ad lght. X Icome (.e., we use total expedture as a proxy).,,, where cosumers. Lower c. (< 5) Hgher c. (> 5) Y =cos. X=c. Y=cos. X=c

2 Exp. Commodty group Males Household expedtu me XM Testg of structural brea as a example of F-testg Ths s a typcal F-test type of problem a regso. Full (cludg the possblty of a structural brea betwee lower ad hgher comes) Suppose ( X, Y ),( X, Y ),,( X, Y ) are d pars as ( X, Y) ~ f ( x, y) f ( y x) f X ( x) (where f ( x, y ) deotes the ot populato pdf of ( XY, ). As dscussed before, whe all parameters of tet are cotaed the codtoal pdf f ( y x ), we do ot eed to say aythg about the margal pdf f ( x ), ad we ca cosder all X as fxed equal to ther observed values, x. Let D be a dummy for hgher come, Note that D s a fucto of X. f X 5 D f X 5 For usg the F-test we eed to postulate a ormal ad homoscedastc pdf for f ( y x ),.e., ( Y X x) ~ N E( Y x),, where X E( Y x) x d dx 3 ( ) ( 3) x f d,.e., for x 5 x f d,.e., for x 5 dcatg a structural brea f at least oe of, 3 s dfferet from zero. Cosderg the observed X s as fxed, we may exps the smpler as

3 3 Y x d d x e where e, e,, e ~ d wth () 3 e N. ~ (, ) We wat to test the ull hypothess of o structural brea as expsed by the Reduced () Y x e where e, e,, e ~ d wth e N. ~ (, ) whch s the same as testg H : ad 3 agast H : At least oe of, 3 (.e.) the. We see that H here cotas two trctos o the betas so a F-test s proper here.. The F-test has a smple recpe, but to uderstad ths we eed to defe the F-dstrbuto ad 5 smple facts about the multple (homoscedastc) regso wth d ad ormally dstrbuted error terms. Frst the F-dstrbuto: Itroducto to the F-dstrbuto (see Rce, secto 6.) Defto. If Z, Z are depedet ad ch-square dstrbuted wth r, r degrees of freedom (df) pectvely ( short Z r F Z r Z ~ r,, ), the has a dstrbuto called the F-dstrbuto wth freedom ( short F ~ F( r, r ) ). r ad r degrees of [ Pdf (optoal readg): ( ) r r r r r r r r r r ff ( x) x ( r r ) x r ( ff ( x) for x ) Expectato: r for x for r ]

4 ..4 y Two F-destes (both wth expectatos 6/4 =.4) F(,6) F(6,6) x Notes The F-dstrbuto s a oe-topped o-symmetrc dstrbuto o the postve axs cocetrated aroud (ote that, sce E( Z ) df r, the E Z r ). If F ~ F( r, r ), the F ~ F( r, r ) (follows drectly from defto). Table 5 the bac of Rce gves oly upper percetles for varous F-dstrbutos. If you eed lower percetles, use the prevous property (a lower percetle of F s a upper percetle of F ). The basc tool for performg a F-test s the Source table a Stata-output, whch summarzes varous measu of varato relevat to the aalyss. Full Y x d 3dx e where e, e,, e ~ d wth e N ~ (, ) Stata output Source df MS (=/df) Number of obs = F( 3, 6) = 68.9 Model Prob > F =. Resdual R-squa = Ad R-squa =.947 Total Root MSE = 67.6 Y Coef. Std. Err. t P> t [95% Cof. Iterval] D DX X _cos Other programs call ths Aova table. Aova stads for aalyss of varace.

5 5 Recpe for the F-test of the uced agast the Ru two regsos, oe for the ad oe for the uced. Pc out the dual sums of squa (.e., dual that we call ad pectvely) from the two source tables. Pc out the dual degrees of freedom (.e., df dual that we call df ad df pectvely) from the two source tables ad calculate the umber of trctos to be tested, s df df. Calculate the F statstc, ( ) / s F, ad reect H f F s larger tha the / df upper percetle the F( s, df ) dstrbuto (corpodg to the level of sgfcace, ). Or calculate the p-value, P ( ) H F F obs (usg e.g., the F.DIST fucto Excel or a smlar fucto Stata). [Example: The F-test reported ( ) s test for all the regso coeffcets frot of explaatory varables,.e., H : 3 agast some ' s. Ths s a stadard F- test all OLS-outputs. No-reecto of ths test dcates that there s o evdece the data that the explaatory varables have ay explaatory power at all thus dcatg that further aalyss may be futle. ] The source tables of the two regso rus are all that we eed for performg a F-test. 3 Some basc facts about the regso ad the source table Frst a summary of OLS Model. () Y x x e,,, where the { x ;,,, ad,,, } are cosde fxed umbers ad repet observatos of explaatory varables, X, X,, X (see ustfcato the appedx of the lecture ote o pcto). For the error terms we assume, e, e,, e are d ad ormally dstrbuted, e N. ~ (, ) The error terms (beg o observable sce the beta s are uow) ca be wrtte () e Y x x Y E( Y ) The OLS estmators (equal to the mle estmators ths ) are determed as mmzg

6 6 (3) Q( ) Y x x e wth pect to (,,, ). The soluto to ths mmzato problem (whch s always uque uless there s a exact lear relatoshp the data betwee some of the X- varables) are the OLS estmators, ˆ ˆ ˆ,,,, satsfyg the so called ormal equatos : (4) Q( ˆ ),,,,, We defe the pcted Y s ad duals as pectvely Yˆ ˆ ˆ x ˆ x, ad eˆ Y Yˆ,,,, The ormal equatos (4) ca be expsed terms of the duals as (defg, for coveece, a costat term varable, x ), (5) eˆ x for,,,, I partcular, the frst ormal equato (5) shows that eˆ ˆ e x, ad, therefore that the mea of the Y s must be equal to the mea of the pcted Y s, ) (6) Y Yˆ. (Notce Yˆ Yˆ ( Y eˆ ) Y Y We ow troduce the relevat sums of squa ( s) whch satsfy the same (fudametal) relatoshp (fact ) as the smple regso wth oe explaatory varable: Defe Total sum of squa, tot Y Y Resdual sum of squa, ˆ eˆ Y Y Q( ˆ ) Model sum of squa, ˆ ˆ Y ˆ Y Y Y (6) Wrtg Y Y Y Yˆ Yˆ Y, squarg, ad usg a lttle bt of smple (matrx) OLS algebra, we get the fudametal (ad bass for the Source table) Wheever the regso fucto has a costat term,, ad oly the.

7 7 Fact : tot Y Y = Y ˆ ˆ Y or + Y ˆ Y where Yˆ ˆ ˆ ˆ ˆ ˆ x x (explaed), ad e Y Y (uexplaed),,,, Ofte s terpreted as measurg the varato of the explaed part ( Y ˆ ) of the pose Y, ad Itroducg R tot as the varato of the uexplaed part of Y. we get the so called coeffcet of determato terpreted as the percetage (.e., R ) of the total varato of Y explaed by the regsors, X, X,, X, the data. It ca also be show that, defg R as the sample correlato betwee, Y ad Y ˆ (called the (sample) multple correlato betwee Y ad X, X,, X ), the exactly equal to the defto gve. I the Stata output the Source table. R beg a correlato coeffcet mples that R s R s reported to the rght of R. To do ferece we also eed to ow the dstrbutoal propertes of the s. Frst of all, they ca be used to estmate the error varace,, uder varous crcumstaces. Notce frst (see secto 6 below) that e N e N e ~ (, ) ~ (,) ~ (as show Rce as a example). Sce a sum of depedet ch-square varables s tself ch-square wth degrees of freedom equal to the sum of degrees of freedom for each varable (recall also that the expected value of ch-square varable s equal to the degree of freedom), we have e ~ E e E e Hece, f we could observe the e s, we could use The e as a ubased estmator of. e s beg o observable, we use the duals, e ˆ s, stead. The ormal equatos (5) show that the duals must satsfy trctos, eˆ x for,,,,, so oly duals ca vary freely. Hece the term degree of freedom, beg df for the duals.

8 8 Fact If the regso fucto cotas free parameters, (,,, ), the df (o. of free parameters the regso fucto). Now the matrx OLS algebra (detals omtted) gves us fact 3 showg that s chsquare dstrbuted wth degrees of freedom, Fact 3 eˆ ~ df E ( df ) E df Hece, defg the mea sum of squa duals as MS df ( ), we have obtaed a ubased estmator of, (7) MS ( ˆ df Q ) df (Note cotrast that the mle estmator s ˆ (show the appedx).) Fact 4 () ad are depedet rv s. are, the E () If all,,, Otherwse, f some E, ~ All the formato facts,,,5 s summarzed the Source table 3 costructed as follows, (8) The Source table Source df MS=/df Model Resdual Total df MS df MS ( ) tot Y Y MS tot The Source table for the () the example - together wth the dagostc formato to the rght - became 3 Ths source table repet a regso wth a costat term ( ). If the regso fucto cotas X s oly wthout a costat term, the source table s slghtly dfferet. The ( ) tot Y, p df, df, ad df. Otherwse, the same. p tot

9 9 (9) The Source table for the () Source df MS Number of obs = F( 3, 6) = 68.9 Model Prob > F =. Resdual R-squa = Ad R-squa =.947 Total Root MSE = 67.6 Accordg to ths, the estmate of the error varace,, s The square root of ths (67.6) s the estmate of ad s gve as Root MSE to the rght. base The F-test for the H (cosstg of 3 trctos) s at the rght ad has a p-value., dcatg that the (3) explaatory varables have explaatory power, so t maes sese to cotue the aalyss. R-squa s smply tot ad shows that 9.8% of the varato the data of Y s explaed by the 3 varables the 4 (all determed by our sgle X). Also the adusted R-square 5 s a dagostc tool. If the dfferece betwee the two R- squa s substatal, ths s a sg that too may explaatory varables have bee cluded the relato to the umber of observatos (). (I the extreme case, for example, that we clude X s the, we get all ˆ Y Y all eˆ ad, therefore, R. I ths case the regso aalyss collapses completely,.e., there s o formato at all the data for such a.) I the pet example there s o dager of such a possblty sce both values are qute close. 4 The recpe for F-testg of regso coeffcets The Model s as () () Y x x e,,, where the { x ;,,, ad,,, } are cosde fxed umbers ad repet observatos of explaatory varables, X, X,, X (see ustfcato the appedx of the lecture ote o pcto). For the error terms we assume, e, e,, e are d ad ormally dstrbuted, e N. ~ (, ) 4 I.e., ths case all 3 varables the regso fucto (usually called regsor varables) are actually determed by a sgle X. Ths s o, however, as log as the three ultg varable are ot exactly learly depedet. If they had bee exactly learly depedet, the becomes o detfable ad OLS braes dow. 5 For the curous oes: We have The formula for R. R s, ad R tot tot DEF df ( R ) df ad tot tot

10 The uced Model We wat to test a ull hypothess cosstg of s (lear) trctos o,,,. Whe the trctos are lear, the uder H ca be expsed as a regso (called the uced ) wth p regsor varables some of whch may be dfferet from the X s (see the extra exercse the semar wee 47 for a example) ad p regso parameters, ' (,,, p ), (wth a costat term f pet), where p. [For example: Suppose the s Y X X 3X3 e, ad we wat to test H : (call the commo value, say). The the uced Y X X X e ( X X ) X e. The becomes, ' (,, ) (,, ), ad p = ad s. 3 The aalyss s OLS regso of Y o X, X, X 3 (wth df df 3 ). The uced aalyss s acheved by OLS of Y o two varables, ( X X ) ad X3 (wth df df ) ] Let, deote the dual sum of squa ( ) for the ad the uced pectvely ad the corpodg degrees of freedom ( the case that a costat occurs both the ad the uced otherwse, see footote 3), df - - ad df p. The lelhood rato prcple tells us (see the appedx) that we should compare ad to test the uced agast the. Ths s exactly what the F-test does. The matrx OLS algebra (detals omtted) gves us what we eed for the F-test fact 5: Fact 5 () The rv s ad are depedet. () If H (the uced ) s true, the ( ) s ch-square dstrbuted wth degree of freedom (equal to the expected value) equal to s df df (vald geeral wth or wthout costat terms the two s). () If H s false, the ( ) commo the s dstrbuto teds to get larger values tha what s

11 Hece, ( ) s s a ubased estmator of f H s true, ad, as ca be prove, has expectato of freedom statstc f H s wrog. Sce, ay case, df, ad, hece, s ch-square wth degree df ubased (ad cosstet),we get our F test ( ) / s ( ) / ( s) Z s / df / ( df ) Z df F, where Z, Z are depedet ad, uder H, ch-square wth s ad df degrees of freedom pectvely. The, accordg to the costructo secto, F s F-dstrbuted wth s df df ad df degrees of freedom f H s true. If H s wrog, the F teds to get larger, so we reect H f F s suffcetly large. ( ) / s Note also that F, where s a ubased ad cosstet estmator of, o matter f H s true or false. I other words, the recpe of the F-test s as follows: () Recpe for the F-test of the uced agast the Ru two regsos, oe for the ad oe for the uced. Pc out the dual sums of squa ( ad ) from the two source tables. Pc out the dual degrees of freedom ( df ad df ) from the two source tables ad calculate the umber of trctos to be tested, s df df. Calculate the F statstc, ( ) / s F, ad reect H f F s larger tha the / df upper percetle the F( s, df ) dstrbuto (corpodg to the level of sgfcace, ). Or calculate the p-value, P ( ) H F F obs (usg e.g., the F.DIST fucto Excel or a smlar fucto Stata). Example of testg structural brea descrbed the troducto. Full Y x d 3dx e where e, e,, e ~ d wth e ~ N(, )

12 Stata output Source df MS Number of obs = F( 3, 6) = 68.9 Model Prob > F =. Resdual R-squa = Ad R-squa =.947 Total Root MSE = 67.6 M Coef. Std. Err. t P> t [95% Cof. Iterval] D DX XM _cos Reduced ( H ) Y x e where e, e,, e ~ d wth H : 3 e N ~ (, ) Stata output uced Source df MS Number of obs = F(, 8) = Model Prob > F =. Resdual R-squa = Ad R-squa =.7553 Total Root MSE = 83.3 M Coef. Std. Err. t P> t [95% Cof. Iterval] D _cos The relevat quattes are df df 8 No. of trctos uder H : s df df ( ) / s ( ) / F 7.8 / df /6 F ~ F (,6) uder H. P-value (usg F.Dst Excel): P F F P F 5 H ( ) ( 7.8) obs H so the evdece for a structural brea as defed s strog,.e., the uced s reected.

13 3 5. Specfcato test of same varace the two come groups The F-test secto 4 assumes costat error varace,, both groups. If ths assumpto s wrog, the F-test secto 4 s valdated. It s therefore atural to as f there s ay evdece the data for doubtg the costat varace assumpto. For ths purpose we ca use aother F test whch ofte ca be used to compare the varaces two depedet groups. Let, be the error term varaces for the d group ad d group pectvely. We wat to test H : agast H : The F test s well suted for ths: Ru two regsos, oe for each group. Pc out the two MS, called MS ad MS pectvely, from the two rus ad form / the F statstc, F MS df, where df MS / df, df are the dual degrees of freedom the two groups. Note that MS ad MS must be depedet sce they come from two depedet groups. Sce / ( df ) F V, where V ~ F( df, df ), t follows that / ( df) F ~ F( df, df ) f H s true. The problem s two-sded, so we reect H f F c or F c, where the crtcal values, c, c for level of sgfcace, are determed by P ( F c ) ad P ( F c ). H H Or calculate the p-value: the smallest of P H ( F Fobs ) ad P H ( F Fobs ). Stata output for the example Group D = Source df MS Number of obs = F(, ) = 4.56 Model Prob > F =. Resdual R-squa = Ad R-squa =.757 Total Root MSE = 56.8 M Coef. Std. Err. t P> t [95% Cof. Iterval] XM

14 4 _cos Group D = Source df MS Number of obs = F(, 4) =. Model Prob > F =.994 Resdual R-squa = Ad R-squa = -.5 Total Root MSE = M Coef. Std. Err. t P> t [95% Cof. Iterval] XM _cos MS Test: F ~ F(, 4) uder H. ( F ~ F(4,) uder H) MS The crtcal values at the 5% level from table 5 bac Rce : P ( F c ).5 P ( F c ).975 c 8.75 H H PH ( F c ).5 PH PH F c F c 4. c.4 c 4. so we reect H f F.4 or F MS Observed: Fobs.64 MS Cocluso: Do t reect H. I other words: Our () secto 4 passed the specfcato test, whch creases ts cblty. 6 Some useful facts about ch-square- ad T-dstrbutos () () () (, ) dstrbutos. r r Z E Z r Z r ~ r ( ), var( ) X ~ N(,) Z X ~ (v) Z, Z,, Z depedet ad ~ r, ~, where r Z Z Z r r r r (v) Costructo of T: X (.e., t- Zr If X, Z are depedet, X ~ N(, ), ad Z ~ r, the T ~ tr dstrbuted wth r degrees of freedom (see Rce Chap. 6 (optoal readg)). (v) From () ad secto above, we coclude that, f T ~ r t, the F T ~ F(, r).

15 5 (v) Testg a dvdual coeffcet, H : agast H :, we would use a ˆ t-test wth r degrees of freedom ad test-statstc T ~ t SE( ˆ ) uder H. Ths test s equvalet wth a F (, ) - test, sce F T F H ~ (, ) uder. 7 Appedx The F-test as a lelhood rato test (optoal readg) Cosder the () () Y E( Y ) e x x e,,,, where e, e,, e are d ad e N. Ths mples that Y, Y,, Y are depedet ad ~ (, ) Y N E Y. ~ ( ( ), ) for,,, The lelhood s (wrtg (,,, ) ) ( ( )) ( ) ye Y Q L(, ) f ( y, y,, y ;, ) e e ( ) ( ) Sce h( x) e x s a decreasg fucto, the, whatever the value of, the maxmum of L over s obtaed by mmzg Q( ),.e., whe s equal to the OLS ˆ. Hece the mle ˆ s equal to the OLS estmator. We the fd the mle of by maxmzg ˆ l L(, ) l( ) l Q( ˆ ) wth pect to. ˆ l (, ) ( ˆ L Q ) gves the mle ˆ Q( ˆ ) 3. Substtutg ths the lelhood, we get the maxmum value (3) Q( ˆ ) ( ˆ Q ) ˆ ˆ L(, ˆ) e e e ( ) ( ) ( ) Q( ˆ ) Q( ˆ ) Now let deote the parameter set, ( ),, uder the (), ad the parameter set, ( ),, uder the uced secto 4. Let over ad pectvely. The lelhood rato (LR) the becomes L ad L be the maxmum lelhoods

16 6 ( ) e ˆ L Q( ˆ ) Q( ) L ( ˆ) Q e ( ) ˆ Q( ) The LR test tells us to reect the uced ( H ) f W l l suffcetly large, whch s the same as sayg that H should be reected f suffcetly large (sce the l-fucto s creasg), or f s suffcetly large. Ths s equvalet to reectg H f the F statstc, F s suffcetly large. The dstrbuto of F s ow exactly (as a s F-dstrbuto) uder H - o matter sample sze - cotrast to the geeral LR test whch s oly approxmately a Ch-square test (wth degree of freedom s) for large samples. s s

UNIVERSITY OF OSLO DEPARTMENT OF ECONOMICS

UNIVERSITY OF OSLO DEPARTMENT OF ECONOMICS Exam: ECON430 Statstcs Date of exam: Frday, December 8, 07 Grades are gve: Jauary 4, 08 Tme for exam: 0900 am 00 oo The problem set covers 5 pages Resources allowed: