Three Classical Tests; Wald, LM(core), ad LR tests uose that we hae the desity y; of a model with the ull hyothesis of the form H ; =. Let L be the log-likelihood fuctio of the model ad be the MLE of. Wald test is based o the ery ituitie idea that we are willig to accet the ull hyothesis whe is close to. The distace betwee ad is the basis of costructig the test statistic. O the other had, cosider the followig costraied maximizatio roblem, max 5B L s.t. = If the costrait is ot bidig (the ull hyothesi is true), the Lagragia multilier associated with the costrait is zero. We ca costruct a test measurig how far the Lagragia multilier is from zero. - LM test. Fially, aother way to check the alidity of ull hyothesis is to check the distace betwee two alues of maximum likelihood fuctio like L? L = log y; y; If the ull hyothesis is true, the aboe statistic should ot be far away from zero, agai. Asymtotic Distributios of the Three Tests Assume that the obsered ariables ca be artitioed ito the edogeous ariables X ad exogeous ariables Y. To simlify the resetatio, we assume that the obseratios Y i,x i are i.i.d. ad we ca obtai coditioal distributio of edogeous ariables gie the exogeous ariables as f y i P x i ; with 5 B R. The coditioal desity is kow u to ukow arameter ector. By i.i.d. assumtio, we ca write dow the log-likelihood fuctio of obseratios of Y i,x i as L = logf y i P x i ; i= We assume all the regularity coditios for existece, cosistecy ad asymtotic ormality of MLE ad deote MLE as. The hyotheses of iterest are gie as where g 6 ;R R r ad the rak of /g is r. Wald test Proositio Y W = g H ;g = H A ;g fi /g /g I g i e r uder H. where I = E X E? / logf YPX; ad I is the ierse of I ealuated at =. From the asymtotic characteristics of MLE, we kow that The first order Taylor series exasio of g? d N,I aroud the true alue of, we hae g = g + /g? + o g? g = /g? + o Hece, combiig () ad () gies Uder the ull hyothesis, we hae g d /g g? g N, I /g =. Therefore, 3 d /g g N, I /g By formig the quadratic form of the ormal radom ariables, we ca coclude that g /g I /g g i e r uder H. 5 The statistic i (5) is useless sice it deeds o the ukow arameter. Howeer, we ca cosistetly aroximate the 4
terms i ierse bracket by ealuatig at MLE,. Therefore, Y W = g /g /g I g i e r uder H. t A asymtotic test which rejects the ull hyothesis with robability oe whe the alteratie hyothesis is true is called a cosistet test. Namely, a cosistet test has asymtotic ower of. t The Wald test we discussed aboe is a cosistet test. A heuristic argumet is that if the alteratie hyothesis is true istead of the ull hyothesis, g g fi. Therefore, g /g I /g g is coergig to a costat istead of zero. By multilyig a costat by, Y W K as K, which imlies that we always reject the ull hyothesis whe the alteratie is true. t Aother form of the Wald test statistic is gie by - cautio: this is quite cofusig - Y W = g /g I /g g i e r uder H. where I = E X E? / L / = E X E? log y ipx i ; i= ad I is the ierse of I ealuated at =. Note that I = I. t A quite commo form of the ull hyothesis is the zero restrictio o a subset of arameters, i.e., H ; = H A ; fi where is a q subector ofwith q <. The, the Wald statistic is gie by Y W = I i e q uder H. where I is the uer left block of the ierse iformatio matrix, I = I I I I the, I = I? I I by the formula for artitioed ierse. I is I ealuated at MLE. LM test (core test) If we hae a riori reaso or eidece to beliee that the arameter ector satisfies some restrictios i the form of g =, icororatig the iformatio ito the maximizatio of the likelihood fuctio through costraied otimizatio will imroe the efficiecy of estimator comared to MLE from ucostraied maximizatio. We sole the followig roblem; FOC s are gie by maxl s.t.g = /L + /g V = g = where is the solutio of costraied maximizatio roblem called costraied MLE adv is the ector of Lagrage multilier. The LM test is based o the idea that roerly scaled V has a asymtotically ormal distributio. Proositio Y = /L /L I = V/g /g I V i e r uder H. First order Taylor exasios of g ad g aroud gies, igorig o terms, 6 7 Note that g g = g + /g? g = g + /g? = from (7) ad substractig (9) from (8), we hae g = /g? 8 9
O the other had, takig first order Taylor series exasios of /L /L = /L ad /L + / L? aroud gies, igorig o terms, /L = /L + / L? ote that? / L =? i= /L / log y ipx i; = /L? I? I by the law of large umbers. imilarly, /L = /L? I? Cosiderig the fact that /L = by FOC of the ucostraied maximizatio roblem, we take the differece betwee () ad (). The, /L =?I? = I? 3 Hece, From () ad (4), we obtai Usig (6), we deduce sice hece g g. Therefore,? = I /L g = /g I /L g =? /g /g I? /g I /g V =? /g I /g d /g From (4), uder the ull hyothesis, g N, I /g V d N, /g I /g Agai, formig the quadratic form of the ormal radom ariables, we obtai /g V I /g V V g. Cosequetly, we hae V i e r uder H. Alteratiely, usig (6), aother form of the test statistic is gie by /L /L I i e r uder H Note that (8) ad (9) are useless sice they deed o the ukow arameter alue. We ca ealuate the terms ioled i at the costraied MLE, to get a usable statistic. t Agai, aother form of LM test isy = /L I /L t We ca aroximate I with either? i= aroximatio, the LM test statistic becomes / log y ipx i ; = V /g I /g or i= / log y ipx i ; V. / log y ipx i ; 4 5 6 7 8 9. If we choose the secod
Y = /L i= = i= = i= /log y i P x i ; /log y i P x i ; i= /log y i P x i ; i= /log y i P x i ; /log y i P x i ; /log y i P x i ; /L /log y i P x i ; /log y i P x i ; i= i= /log y i P x i ; /log y i P x i ; this exressio seems quite familiar to us - looks like a rojectio matrix -. The ituitio is correct. The ucetered R u from the regressio of o where Hece, X = / log y ipx i; is gie by / log y Px ; / log y Px ; 666 R u = i= R u = X X X X X X X / log y Px ; / log y ipx i; / log y ipx i ; i= Y = R u ad = X X X X = 666 / log y ipx i ;. The, / log y ipx i ; i= This is quite a iterestig result sice the comutatio of LM statistic is othig but a OL regressio. We regress o the scores ealuated at costraied MLE ad comute ucetered R ad the multily it with the umber of obseratios to get LM statistic. Oe thig to be cautious is that most software will automatically try to rit out cetered R, which is imossible i this case sice the deomiator of cetered R is simly zero. t LM test is also a asymtotically cosistet test. t From (6) ad (8), Likelihood ratio(lr) test Proositio Y W = g /g /g I g /g I /g g g = Y Y R = L? L i e r uder H. We cosider the secod order Taylor exasios of L ad L aroud. Uder the ull hyothesis, igorig stochastically domiated terms, L = L + /L? +? / L? = L + /L? +? / L? L = L + /L? +? / L? = L + Takig differeces ad multilyig by, we obtai /L? +? / L?
L? L = /L? +? / L? sice /L = I? from () ad??? / L?? I??? I? +? I? / L I. Cotiuig the deriatio, L? L =? I??? I? +? +? I? +? =? I??? I? +? I? +? I? +? I? +? I? =? I? +? I??? I??? I? =? I? ote that? I? =? I?. Now, from (3) ad (), we hae L? L =? I? = /L /L I I I = /L /L I = Y i e r uder H. t Calculatig LR test statistic requires two maximizatios of likelihood fuctio oe with ad the other without costrait. t LR test is also a asymtotically cosistet test. t As show aboe, Wald, LM ad LR test are asymtotically equialet withe r. Examles of tests i the liear regressio model uose the regressio model such as The hyotheses are gie by H ; R r y i = K x i + P i P i i i.i..,a K = L The log-likelihood fuctio is gie by L K u? loga? a The, the ucostraied MLE is gie by K = X X X y H ;RKfi L y? XK y? XK Iformatio matrix is gie by The Wald statistic is, from Proositio, a = I = y? XK y? XK X X a a 4
Y W = RK?L R I R RK? L = RK?L R I R RK? L = RK?L R a X X R RK? L = a RK?L R X X R RK? L i e r uder H. Deote the costraied MLE ask ada, resectiely. The, a?a = y? XK y? XK? y? XK y? XK = XK? XK XK? XK = K?K X X K? K = RK? L R X X R RK? L sicek = K + X X R R X X R RK? L. Therefore, Y W = a?a a = RK? L R X X R RK? L y? XK y? XK = RK?L R X X R RK? L /r y? XK y? XK /? K r? K = r? K F O the other had, the Lagrage multilier of the costraied maximizatio roblem is V =? a Uder H, the distributio of the Lagrage multilier is V i N, 4 a R X X R L? RK R X X R sice L? RK i N,a R X X R. The, the LM test statistic is Y = a 4 V R X X R V = a To obtai LR test statistic, ote that O the other had, = a?a a RK? L R X X R =? + a a?a L =? log^? loga? a =? log^? loga? a =? log^? loga? a =? log^? loga? = + a RK? L a?a y? XK a y? XK = + +K rf y? XK y? XK
L =? log^? loga? a =? log^? loga? a =? log^? loga? a y? XK a y? XK y? XK y? XK =? log^? loga? Hece, Y R = L? L =? loga + loga = log a a = log? + a a = log + rf? K A iterestig result ca be obtaied usig the followig iequalities, x log + x x-x + x Let x = rf ad alyig the aboe iequalities,we obtai?k Y Y R Y W = log + a? a a