Hypothesis Testing, Model Selection, and Prediction in Least Squares and Maximum Likelihood Estimation

Size: px

Start display at page:

Download "Hypothesis Testing, Model Selection, and Prediction in Least Squares and Maximum Likelihood Estimation"

Bryce Campbell
5 years ago
Views:

1 Hyothesis Testig, Model Selectio, ad Predictio i Least Squares ad Maximum Likelihood Estimatio Greee Ch.5, 4; Keedy Ch. 4 R scrit mod3sa, mod3sb, mod3sc Testig for Restrictios i a LS Model Liear Restrictios Assume you have secified a CLRM of the usual form y = Xβ+ ε. Let s call this the geeral or urestricted model. You may hyothesize that some elemets of β are liearly related or costraied to take a secific value. Such costraits are called liear costraits or liear restrictios. Here are a few examles: Sigle value restrictio: β 3 Multile value restrictios: β3 β = 5 Sigle restrictio o liear relatioshis: β β = 3 0 Multile restrictios o liear relatioshis: β β3 β + β + 4β = Mixed liear restrictios: β β4 β5 = β + β = 3 3 I geeral, such hyotheses o restrictios arise from cometig uderlyig theoretical models. For examle, for our wage regressio (see R scrit mod3sa), you might hyothesize that a male worker ears $3 more tha a female worker, ceteris aribus. This would traslate ito the followig hyothesis: H0: β = 3 () As a geeral rule, you should always exress a liear costrait such that there is oly a umerical value left o the right had side of the equality sig. I geeral we ca write a set of J liear costraits as

2 r β + r β + + r β = q k k r β + r β + + r β = q k k r β + r β + + r β = q J J Jk k J () or, more comactly as Rβ= q where r r r k q r r r k q R =, q = Jxk Jx r r r q J J Jk J (3) I ricile you ca test ay umber of liear restrictios, but you must always obey the followig rules:. J k, i.e. you ca t imose more restrictios tha there are coefficiets.. The rows of R must be liearly ideedet, i.e. you ca t have ay redudat or coflictig restrictios. Here are a few examles for R ad q usig the wage data. Recall the elemet of X: % cotets of X %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % costat term % female = worker = female %3 o-white = worker = o-white %4 uio = worker = uioized %5 educatio years of educatio %6 exeriece years of work exeriece So k=6. Examle : H 0: Female workers ear $3 less tha male workers, ceteris aribus H0 : β = 3 J = R q= 3 [ ] Examle : H 0: additioal year of educatio is worth 8 additioal years of exeriece H0 : β5 8β6 J = R q= 0 [ ] Examle 3: H 0: Female workers ear $3 less tha male workers, ceteris aribus AND additioal year of educatio is worth 8 additioal years of exeriece

3 3 H : β = 3 0 β 8β J = R = = q 0 Examle 4: H 0: Female workers ear $3 less tha male workers, ceteris aribus AND additioal year of educatio is worth 8 additioal years of exeriece AND a uioized worker ears $ more tha a o-uioized worker, cet. ar. H : β = 3 0 β 8β 5 6 β = J = R = q= Oce R ad q are exlicitly defied, you ca comute a F-statistic to imlemet the test as follows: F( ) ( ) ˆ J, k V = Rb q ( ) ( ) J R b R Rb q = ( Rb q) R s ( X X) R ( Rb q) J (4) A radom variable that follows the F-distributio has arameters, Degrees of Freedom for the umerator (DoF) ad Degrees of Freedom for the deomiator (DoFd). The DoF are always equal to the umber of restrictios (J), ad the DoFd are equal to the samle size mius the umber of arameters (-k). F-Tables with critical values for differet DoFs ad levels of sigificace ca be readily obtaied from the iteret (see also. 096 i Greee's 6 th editio). We ll use the oe for α.05. I that Table, = J, ad = -k. The table shows critical values. For examle, if J = 4 ad (-k) >00, the critical F value is.37. If your comuted F exceeds this value, you reject H 0. Otherwise do t reject. More coveietly, you ca directly comute the -value for your test statistic i R. The comare the -value to your level of sigificace ad draw your coclusio. F-test vs. t-test For a sigle restrictio t = F, so you ca either do a F-test (usig the F-table) or a t-test (usig the t- table). Both will always arrive at the same test decisio. Recall that both t- ad F-statistics require the ormality assumtio for ε i the CLRM. Testig for the iequality of the etire β -vector over two grous: The Chow Test Ofte times your data set ca be cocetually slit ito two grous, such as male / female, white / owhite for the wage data, or si casters / fly fishers for the aglig data, etc. You may the 3

4 4 hyothesize that the margial effect of all or some of the remaiig regressors differs over the two grous. Let s use the wage data ad white / owhite as a examle. Usig the full set of remaiig regressors as examle, your ull ad alterative hyothesis could be stated as: H0 : βcostat,w βcostat,w βgeder,w βgeder,w βuio,w βuio,w (5) βeducatio,w βeducatio,w β β exeriece,w exeriece,w H a: At least oe of these equalities does t hold. Where w stads for white ad w stads for owhite. I ricile, you thus have J=5 restrictios ad could use a F-test. The tricky art for a Chow test is i desigig the ucostraied model, i.e. a model that estimates searate coefficiets for these 5 regressors (icl. the costat) for the two race grous. You eed to re-costruct your y ad X as follows:. Pick the variables of iterest from the origial X ad grou them ito a ew X (let s call it X it for ow). Usually these will be all variables i the origial X mius the colum of oes ad the variable that origially defied the two grous of iterest (here race ).. Create idicator (dummy variables) for each grou. 3. Create iteractio terms betwee this dummy ad X it. 4. Estimate a CLRM of y agaist the first idicator dummy, its iteractios, the oosite idicator dummy, ad its iteractios. The test for the stated iequalities usig a F-test. Cocetually, the ucostraied model takes the followig form (assume that observatios have bee sorted by grou this is ot actually eeded for estimatio): y = Xβ+ ε where yw Xw 0 βw y =, X=, β = yw 0 Xw βw (6) Clearly, if your H a holds for the etire set of restrictios, this would be equivalet to ruig two searate regressio models, oe usig oly data for owhite workers, ad oe usig oly the sub-samle of white workers. See mod3sa for the imlemetatio of this examle. Noliear restrictios To test for oliear restrictios o β i the CLRM we eed to ivoke asymtotic results. Secifically, we eed to aly the Delta Method to derive the estimated asymtotic variace of the restricted coefficiet or set of coefficiets. Assume we imose e set of J oliear restrictios, each of which ivolves oe or more elemets of β. Usig Greee s (. 4) otatio, we ca comactly exress the set of restrictios as 4

5 5 c( β) = q c( β) = c ( ) c ( ) c β β ( β) (7), where J Jx ad q is a J by vector of umerical values, (as for the liear restrictio case). As a first ste, we eed to estimate the asymtotic variace of c( β ). By the Delta method, we have ( ) ˆ( ( )) ˆ c V c b = CV( b) C' = C s ( XX ) C where C =. (8) Jxk b Next, we comute a Wald test statistic essetially the asymtotic versio of a F-test. As a geeral rule, we do t use degrees of freedom based o samle size i asymtotic test statistics. Thus: ( b) ( ( ) ) a ( ) ( b) = s ( XX ) ( ( ) ) ( ) ( ) χ ( ) W = c b q C Vˆ b C c b q ~ J where Vˆ a (9) The last term idicates that W follows a chi-squared distributio with J degrees of freedom. The table of critical values for this distributio is give o. 095 of Greee's 6 th editio, or you ca easily fid it olie. I Greee, the colum labeled 0.95 corresods to the 5% level of sigificace (i.e. α.05 ). For examle, you ca see that the critical value for J= equals If c ( β) = q cotais oly a sigle restrictio, you ca alteratively use a z-test as show o. 98. The z- W relatioshi is aalogous to the t-f case, i.e. for a sigle restrictio we have z = W. A examle of a Wald test ivolvig oliear restrictios is give i scrit mod3sa. I theory, the Wald test could also be used for liear restrictios, i which case we have c( β) q = Rβ q as before. However, i the LS case where the fiite roerties of b are kow the F test usually yields more accurate results, esecially uder small samle sizes. 5

6 6 Testig for Restrictios i a MLE Model There exist three asymtotically equivalet test statistics for MLE estimatio. They differ i the umber ad tye of models you eed to estimate before you ca ru a secific test. Here is a summary: Test Abbreviatio Required estimated models Likelihood- LR Costraied & Ratio ucostraied Lagrage Multilier LM Costraied Wald W Ucostraied Distributio of test Stregths / Limitatios statistic χ Costraied model ( J ) ofte difficult to estimate; robably most reliable test of the three χ Costraied model ( J ) ofte difficult to estimate χ Probably most ( J ) straightforward to imlemet LR test The LR test requires the estimatio of both the ucostraied ad costraied models. The test statistic is the derived as LR = ( l L( ˆ ) ( ˆ u l L c) ) ~ χ ( J ) where l L( ˆ u ) ad l L( ˆ c ) θ θ (0) θ is the value of the log-likelihood fuctio at the solutio for the ucostraied model, θ is the aalogous value for the costraied model. As before, J idicates the total umber of joit restrictios to be tested. The ituitio for this test is that if the imosed costrait is correct, the value of ll for the costraied model should be close to that for the ucostraied model (it ca ever be better, i.e. less egative tha the ll for the ucostraied model). A roouced differece betwee the two values would suggest that the costrait (or set of costraits) does ot hold. As before, the test decisio is based o a comariso of the comuted LR to the alicable critical value i a chi-squared table. A examle of a LR test is give i mod3sb. Wald Test The Wald test requires estimatio results from the ucostraied model. It takes the same form as give i (9) with the OLS estimator b relaced by the MLE estimator (say ˆβ ), ad the estimated asymtotic variace of ˆβ give by ay of the usual aroximatios, i.e. either the iverted egative Hessia or the outer roduct of gradiets. Thus: 6

7 7 βˆ q ( C( βˆ ) C ) ( ( βˆ) q) χ ( ) ˆ ( ) a( ) ( ) ( ( ) ) a ( ) ( ) ( ) or W = c Vˆ c ~ J where Vˆ βˆ = H βˆ V βˆ = G'G a If the restrictios are liear c( ˆ ) β q takes the form of Rβ ˆ q. () The ituitio for the Wald test is that if the hyothesized restrictios are valid c( β ˆ ) q should be close to zero ad W will be small. A large value for the Wald statistic would raise doubts as to the validity of the costrait. The Lagrage Multilier (LM) test The LM test (also kow as score test) is based o the costraied model ad the ituitio that if the imosed restrictios are correct, the samle gradiet (or score fuctio) should be close to zero at covergece. Sice, by the iformatio matrix idetity, the variace of the gradiet is equal to the iformatio matrix, the test statistic is comuted as follows: LM = g βˆ ' I βˆ g β ˆ () ( ) ( ) ( ) ( ) where I ( β ˆ ) is agai estimated by the egative Hessia or the outer roduct of gradiets. A imlemetatio of the three tests is give i scrit mod3sb. As you ca see, the results for the three tests ca vary substatially uder small to moderate samle sizes. This fact ad the sesitivity of test V β ˆ is also illustrated i Greee s examle (. 53). statistics to the aroximatio used for ˆa ( ) So which test should be used i ractice? I may cases the costraied model is difficult to secify & estimate, which makes the LM ad LR tests somewhat less oular tha the Wald. The Wald test is also coveiet whe you have estimated the urestricted model, ad you wat to test a whole series of hyotheses o differet costraits. Usig the LR, you would have to secify & estimate a searate restricted model for every set of restrictios. I ractice, if both ucostraied ad costraied models are straightforward to estimate, the LR test is robably the most trusted aroach as the value of the log-lh fuctio at covergece is somewhat less V β ˆ tha the other two test statistics. sesitive to the choice of ˆa ( ) Ideally, you would erform all three tests & hoe that they all oit to the same decisio rule. Good luck! 7

8 8 Model Selectio based o Predictio As metioed i Poirier s Itermediate Statistics ad Ecoometrics,. 405, the toic of redictio is somewhat eglected i most ecoometric textbooks, which focus more o estimatio ad tests o arameters. However, the redictio of yet uobserved outcomes is of cetral imortace whe ecoometric aalysis is suosed to roduce olicy recommedatios, for examle i the cotext of beefit-cost aalysis. We will use the term redictio i the geeral sese of combiig some vector of exlaatory values, yˆ = f x, θ ˆ where f may be a liear or x, with estimated arameters, ˆθ, to geerate a oit estimate ( ) oliear fuctio. (I time series aalysis, redictio is usually referred to as forecastig ). Withi-Samle Predictio ad Predictive Accuracy The strategy of comutig a redicted outcome, y ˆi, based o combiig a observed vector x i with estimated arameters is called withi samle redictio. I cotrast, combiig estimated arameters with a ew set of values for the regressors, say x, is called out-of-samle redictio. Hyothesis tests are oe aroach to select betwee cometig models. Aother criterio relates to the redictive accuracy of cometig models, i.e. how well the redicted deedet observatios ( fitted values i the CLRM cotext) agree with the actually observed y i s. Oe such measure for the CLRM is R ad its adjusted couterart. Aother, more geerally alicable, criterio is the redictive Mea Squared Error, defied as i= ( ˆ ) i i (3) MSE = y y Sice this umber ca grow quite large, its square root is geerally used istead, leadig to the Root Mea Squared Error(RMSE). A alterative criterio is the Mea Absolute Error, give as i= MAE = yi yˆ i (4) A smaller value for these measures idicates a better fit with the actual data. A (mior) roblem with the RMSE ad MAE is that they are ot ivariat to scalig of y if y is multilied by a factor, say α, the the RMSE ad MAE will also be scaled by α. If a scale-free measure of redictive accuracy is desired, the Theil U-statistic ca be used: U = i= ( y yˆ ) i i= y i i (5) 8

9 9 Note that a good fit based o these withi-samle redictive statistics does ot ecessarily imly that the chose model will geerate accurate out of samle redictios as well. However, a oor fit with actual data would certaily raise serious doubts as to the model s out-of-samle redictive abilities, esecially if the values i x are ot too differet from other values observed i the samle (e.g. redictig the rice of a home with 5 bedrooms whe the samle icludes oly homes with,3,4, ad 6 bedrooms). Out-of-Samle Predictio ad Predictive Efficiecy Whe we redict out-of-samle we ca o loger comare our redicted outcome to a actually observed value. However, we ca still comute a stadard error ad cofidece iterval for the redicted value, ad comare models based o the width of this iterval. If two or more cometig models roduce similar oit redictios, we would choose the oe that geerates the tightest cofidece iterval aroud its oit estimate. Liear redictios Assume you are iterested i the out-of-samle-redicted value y = x βˆ. Thus, y is a liear combiatio of regressor values ad estimated coefficiets. A 95% cofidece iterval ca be quickly costructed as 95% = { x ˆ ( ˆ) } ( ˆ) ˆ ( ˆ β± xβ xβ = xβ ) ( x ˆ) = V ( ˆ β x ˆa β) x C. I..96 se, where se V ad Vˆ (6) Noliear redictios Now assume your redictive costruct of iterest is a oliear fuctio of regressor values ad estimated y f x, β ˆ. The derivatio of a 95% cofidece iterval is as i (6), with x β ˆ coefficiets, i.e. = ( ) relaced by f ( x ˆ, β ), ad Vˆ (, ˆ a f ) ( ) x β derived via the Delta Method. Alteratively, the etire cofidece iterval ca be derived via simulatio by drawig multile sets of ˆβ from its asymtotic f x ˆ, β for each draw, ad usig the.5 th ad 97 th ercetiles as cofidece distributio, comutig ( ) bouds. See scrit mod3sc for a examle. 9

To make comparisons for two populations, consider whether the samples are independent or dependent.

To make comparisons for two populations, consider whether the samples are independent or dependent. Sociology 54 Testig for differeces betwee two samle meas Cocetually, comarig meas from two differet samles is the same as what we ve doe i oe-samle tests, ecet that ow the hyotheses focus o the arameters