FINITE-SAMPLE PROPERTIES OF THE MAXIMUM LIKELIHOOD ESTIMATOR FOR THE BINARY LOGIT MODEL WITH RANDOM COVARIATES

Transcription

1 conometrcs Workng Paper WP0906 ISSN Department of conomcs FINIT-SAMPL PROPRTIS OF TH MAIMUM LIKLIHOOD STIMATOR FOR TH BINARY LOGIT MODL WITH RANDOM COVARIATS Qan Chen School of Publc Fnance an Publc Polcy, Central Unversty of Fnance an conomcs, Bejng, People s Republc of Chna & Dav. Gles Department of conomcs, Unversty of Vctora, Vctora, B.C., Canaa Ths verson: August 009 Abstract: We examne the fnte sample propertes of the maxmum lkelhoo estmator for the bnary logt moel wth ranom covarates. Analytc expressons for the frst-orer bas an secon-orer mean square error functon for the maxmum lkelhoo estmator n ths moel are erve, an we unertake some numercal evaluatons to analyze an llustrate these analytc results for the sngle covarate case. For varous ata strbutons, the bas of the estmator s sgne the same as the covarate s coeffcent, an both the absolute bas an the mean square errors ncrease symmetrcally wth the absolute value of that parameter. The behavour of a bas-ajuste maxmum lkelhoo estmator, constructe by subtractng the maxmum lkelhoo estmator of the frstorer bas from the orgnal estmator, s examne n a Monte Carlo experment. Ths bascorrecton s effectve n all of the cases consere, an s recommene when the logt moel s estmate by maxmum lkelhoo wth small samples. Keywors: MSC Coes: Logt moel; bas; mean square error; bas correcton; ranom covarates 6F0; 6J; 6P0 Contact Author: Dav. Gles, Dept. of conomcs, Unversty of Vctora, P.O. Box 700, STN CSC, Vctora, B.C., Canaa V8W Y; e-mal: gles@uvc.ca; Voce: ; FA:

2 FINIT-SAMPL PROPRTISS OF TH MAIMUM LIKLIHOOD STIMATOR FOR TH BINARY LOGIT MODL WITH RANDOM COVARIATS. Introucton Qualtatve response QR moels are very wely use n varous fels, nclung boassay, mecne, transportaton research, economcs, an other socal scences. These moels have the characterstc that the epenent varable s qualtatve, rather than quanttatve. To make the moel estmable, these qualtatve attrbutes are coe numercally. The bnary choce moel, wth the epenent varable coe as zero or unty wthout loss of generalty, s the most wely use of the QR moels. In ths case, t s well known that conventonal lnear regresson methos are napproprate: the precte probabltes can be negatve, or excee unty; the error must be heteroskeastc; an the error term clearly cannot follow a normal strbuton. These problems can be overcome by makng the probablty of occurrence for one of the attrbutes a non-lnear, rather than a lnear, functon of the covarates. In partcular, f ths functon s taken to be a cumulatve strbuton functon, t wll be monotoncally non-ecreasng, an boune between zero an unty. Choosng the stanar normal strbuton for ths functon gves rse to the probt moel, whle the logstc strbuton results n the logt or logstc regresson moel. Of course, other choces are possble, but the logt an probt moels are the two that are encountere most frequently n practce, an they generally yel smlar scale estmates. The appeal of the logt specfcaton s that the logstc strbuton functon can be expresse n close form, an ths has certan computatonal avantages when the moel s extene to the multnomal case nvolvng more than two characterstcs. In ths paper we focus on the logt, rather than probt, moel. The maxmum lkelhoo estmator ML s the usual choce for QR moels. For both the logt an probt moels the lkelhoo functon s strctly concave, so t has a unque maxmum, but the lkelhoo equatons are non-lnear n the parameters, an must be solve numercally. If the covarates are non-ranom, the lkelhoo functons for QR moels satsfy the usual regularty contons an so the maxmum lkelhoo estmators MLs are weakly consstent an best asymptotcally normal, an the strong consstency of the ML for the logt moel has been establshe by Goureroux an Montfort 98. Taylor 95 showe that the ML estmator an the mnmum ch square estmator MCS propose by Berkson 944, an efene

3 vgorously by that author, have the same asymptotc normal strbuton for ths moel. Other estmators that have been suggeste for the logt moel nclue the mnmum φ -vergence estmator Paro et al. 005, Menénez et al. 009., whch s a generalzaton of ML an s also consstent an asymptotcally normal. Turnng to the case of ranom covarates, Fahrmer an Kaufmann 986 nvestgate the asymptotc propertes of varous screte an qualtatve response moels nclung the logt moel, an prove contons on the behavour of the covarates uner whch the ML has ts usual asymptotc propertes. Wle 008 scusse the nconsstency of the generalze metho of moments estmator for QR moels wth enogenous ranom regressors, an suggeste a sutable mofcaton n the case of the probt, but not logt, moel. A number of results relatng to the fnte-sample propertes of the ML an some other estmators for the logt moel have also been establshe. However, all of these relate to the case of non-stochastc covarates, an t s ths last assumpton that we relax n ths paper. Usng the approach of Cox an Snell 968, Corero an McCullagh 99 prove analytcal expressons for the On - bas of the ML n the famly of generalze lnear moels. Ths famly nclues logstc regresson, of course. Several authors have nvestgate the propertes of the ML for the logt moel n the context of a two-stage samplng scheme nvolvng groupe ata of a type that arses frequently n the bologcal scences. Ths nclues smple ranom samplng as a specal case. Berkson 955 evaluate the fnte-sample bas an MS of the ML an the MCS estmator for some smple examples of ths moel, an foun the MCS to be preferre to the ML n terms of MS n the cases that he consere. Amemya 980 erve analytc expressons for the On - MSs of the ML an the MCS for the chotomous bnary logt moel an prove some numercal results for the relatve qualty of these two estmators. Several other stues have extene Berkson s an Amemya s results. Ghosh an Snha 98 prove necessary an suffcent contons for mprovng the MS of the ML, an apple these to Berkson s moels an ata. They also showe the relatve MS rankng of the ML an the MCS s moel-specfc. Davs 984 foun some examples n whch the ML has smaller MS than the MCS estmator, an Hughes an Savn 994 prove further results ncatng that the choce between these two estmators s not straghtforwar. Another somewhat relate stuy s that of MacKnnon an Smth 998. Those authors scusse methos for reucng the bas of consstent estmators that are base n fnte samples, an apple these methos to the ML for the lnear AR moel an the stanar logt moel base

4 on smple ranom samplng wth fxe regressors. Fnally, L 005 use a Monte Carlo experment to examne some of the small sample propertes of the ML for three fferent moels - the probt moel, the logt moel an the bnary choce moel where the unerlyng strbuton s the extreme value strbuton. She also consere the case where the unerlyng strbutonal process s ms-specfe, an foun that ths ncreases the MS for each of the estmators. The assumpton that the covarates n the logt moel are non-ranom or fxe n repeate samples s obvously unsatsfactory n many stuatons. One example s when survey ata are use, as s very common wth applcatons n economcs an the other socal scences. So, n ths paper, we use results ue to Rlstone et al. RSU 996, as correcte by Rlstone an Ullah 005, to erve analytc expressons for the bas an MS functons for the ML n the logt moel base on smple ranom samplng wth stochastc covarates. Base on the analytc bas expresson we can erve a bas-correcte ML an the stanar error assocate wth ths bascorrecte estmator. We also prove some numercal evaluatons base on these analytc results. The approach that we aopt was also use by Rlstone an Ullah 00 n the context of Heckman s sample selecton estmator, an coul also be use to exten our results to other QR moels. The next secton ntrouces the logt moel. In secton we summarze the requre results of RSU 996 an use them to erve analytc expressons for the bas an mean square error of the ML n the bnary logt moel. Some numercal evaluatons an Monte Carlo results follow n secton 4; an the fnal secton proves our conclusons.. The Logt Moel an the Maxmum Lkelhoo stmator A bnary choce moel s structure as follows: where vector, y * β ε, y y ; f β ε a 0 ; f β ε a < * y s the latent epenent varable to ncorporate the effects of covarates; an the row, represents the th observaton on all of the covarates. As s well known, prove that an ntercept s nclue among the covarates, the threshol value, a, may be assgne to zero wthout affectng the results. We make ths assgnment n what follows. 4

5 The basc moel can be structure as: P Pr y F β P Pr y 0 F β. The form of the cumulatve strbuton functon, F β, wll etermne whch partcular moel s use. As note above, we focus on the logt moel, so: where P Pr y F β exp β exp β s the c..f. for the logstc strbuton. The ML for the parameter vector n can be erve as the soluton of the followng loglkelhoo equatons: log L β n y 0. 4 The ML cannot be wrtten as a close-form expresson, an ths s what substantally complcates the task of evaluatng the characterstcs of ts fnte-sample samplng strbuton, whether the covarates are ranom or not.. Analytc Results Before ervng the analytc results for the bas an MS of the ML n the bnary logt moel, we frst ntrouce the results of RSU 996. The class of estmators consere by RSU nclues those whch can only be expresse mplctly as a functon of the ata. Suppose we have a regresson moel of the form y f ; β 0 ε. 5 The regressor vector can nclue any enogenous or exogenous varables. Let Z y, an assume Z, Z, Z, K be a sequence of m mensonal... ranom vectors. θ 0 represents the true parameter vector, whch coul nclue only β 0, or any other parameters of nterest. The estmator θˆ can be wrtten n the form: n ψ ˆ θ ˆ θ 0, 6 n g n 5

6 where g θ g z, θ s a k vector nvolvng the known varables an the parameters, an g θ 0 only for the true value θ 0. Some assumptons about the functon g θ are neee for the ervaton of the Lemmas below. See Ullah 004:. Assumpton The s th orer ervatves of g θ exst n a neghborhoo of θ 0 an s g θ 0 <, where A trace A / s A enotes the usual norm of the matrx A; an Aθ s the matrx of s th orer partal ervatons of the matrx A θ wth respect to θ, obtane recursvely. Assumpton For some neghborhoo of θ 0, ψ θ. n O p Assumpton s s g θ g θ 0 θ θ 0 M for some neghborhoo of θ 0, where M satsfes the conton M C <,,, K As the log-lkelhoo functon of the bnary logt moel s strctly concave, these three assumptons are satsfe. In the notaton that follows, for smplcty we wll suppress the argument for any functon where ths can be one wthout confuson. So, g θ wll be wrtten as 0 g. Then, RSU erve the followng lemmas, correcte here accorng to the corrgenum n Rlstone an Ullah 005. Lemma Proposton., RSU, 996; Ullah 004: Let assumptons - hol for some s. Then the bas of θˆ to O n s B ˆ θ Q V H, 7 n where H j j g Qg, Q g, V g g, an. A bar over a functon ncates ts expectaton, so that g. g 6

7 7 Lemma Proposton.4, RSU, 996; Ullah 004: If Assumptons - hol for some s, then the MS of θˆ to n O s ˆ 4 4 n n n MS θ 8 where { } H V Q { }Q V V V V V V Q { } Q H H Q 4 { } Q H V V V Q { }Q V V V H Q { } 4 V QV V QV V QV Q { } V V V Q { } W W W Q { } QV QV QV H Q { } 4 { } QV QV QV H Q { } 4 { } 6 H Q 9 where g g W. To apply the above lemmas to erve the bas an MS for ML n the bnary logt moel, we assume that both the epenent an nepenent varables are ranom an... Comparng 4 an 6, we can see that for the logt moel we shoul set y g. We know that g 0, so by the law of terate expectatons, g 0. We then have the followng results: g ; g H

8 g ; H g g ; H g Q g ; Qg y V g g W g g, 0 where s s the s th orer ervatve of wth respect to the argument of exp β exp β exp β exp β exp β β an exp β { 4exp β exp β }. 4 exp β Then we can erve the followng theorems an corollares. Theorem If assumptons - hol for some s, then the bas of the ML n the logt moel, to O n s Bas ˆ β vecq. n Theorem If Assumptons - hol for some s, then the MS of ML n the logt moel to O n s MS ˆ β 4 4, n n n where Q 8

9 Q{ V Q Q H { vec Q Q Q } { V } Q QV Q { vecq vecq Q 4 Q { vec vec } Q Q } H Q Q Q Q 4 Q VQVQ { 4 Q Q H vec Q Q vec Q Q vec Q Q }Q { 4 Q H vec Q Q Q vec Q Q Q vec Q Q Q }Q { 6 vec Q Q Q vec Q Q Q vec Q Q Q }Q 4 Now we conser the logt moel wth only one regressor, whch mples that the coeffcent of the ntercept term n the latent regresson moel equals the true threshol a n. For ths smple moel, we have the followng corollares. Corollary If assumptons - hol for some s. Then the O n moel wth only one regressor, s Corollary bas of the ML of β n the logt Bas ˆ β. 5 n If Assumptons - hol for some s, then the O n MS of the ML of β n the logt moel wth only one regressor, s MS ˆ β 4 4 n n n, 6 where 4 9

10 4 4 The proofs of the Theorems an Corollares are gven n the Appenx.. 7 Base on these results, we can obtan two bas-ajuste estmators, βˆ BC an ~ β BC, efne as follows: ˆ β ~ β BC BC ˆ β B ˆ β, 8 ˆ β Bˆ ˆ β, 9 where B βˆ s the bas base on 5 an the true parameter β, an B ˆ ˆ β s the estmate bas base on 5 an the ML, βˆ. In practce, of course, βˆ BC s an nfeasble estmator as t nvolves the unknown true parameter. It can be shown that both of these bas-ajuste estmators are unbase to On -. Fnally, n the sngle covarate case the true stanar evaton, s.. βˆ, an the stanar error, s.e. βˆ, can be obtane as: s.. βˆ MS ˆ β B ˆ β, 0 s.e.βˆ Mˆ S ˆ β Bˆ ˆ β. where MS βˆ s base on 6 an the true parameter β, an M ˆS ˆ β s the estmate MS base on 6 an the ML, βˆ. 4. Numercal valuatons Gven ther complexty, t s ffcult to nterpret the above analytcal results. Here we prove some numercal evaluatons an Monte Carlo smulaton results obtane usng coe for the R 008 statstcal package. For the one-regressor moel we frst conser how the true value of the coeffcent, β 0, an the strbuton of the regressor affect the fnte sample propertes of the ML. Secon, we conuct a small Monte Carlo experment to evaluate the performance of the 0

11 feasble bas-correcte estmator of β relatve to that of the nfeasble bas-correcte estmator an the basc ML. In that experment we also evaluate s.e. βˆ as an estmator of s.. βˆ. In both the numercal evaluatons an the Monte Carlos smulaton, three fferent strbutons are use to generate the ranom regressor normal, ch-square an Stuent-t. Ths enables us to allow for fferent egrees of varablty, skewness an kurtoss n covarate ata by varyng the moments of each strbuton. The value of β 0 s chosen for each case to control the sgnal-tonose ratos to sensble levels. Sample szes of n 5, 50 an 00 are consere. The numercal evaluatons of βˆ are summarze n Tables to where, to conserve space, the results for n 00 are omtte, but they corroborate the tabulate results. We note the followng. The sgn of the bas of the ML s the same as the sgn of β 0, an both the absolute bas an the MS ncrease symmetrcally wth the absolute value of β 0. For each of the strbutons consere for the regressor, as n ncreases, the absolute bas an MS both ecrease for all β 0 values, reflectng the mean-square consstency of the ML. Whatever strbuton the covarate follows, as the moments of ths strbuton change, the bas an MS of the ML change n a manner that epens on the absolute value of β 0. In percentage terms, the absolute bases of the ML can be substantal. For example, when n 5 50, an for the range of values of β 0 that are tabulate, the percentage bases assocate wth a normally strbute covarate range up to 48% 4%. The corresponng fgures for a ch-square strbute covarate are 40% 0%; an those for a Stuent-t strbute covarate are 5% %. Percentage MSs range n value up to 7% 4%, 9% % an 4% 8% n Tables, an respectvely, agan for the values of β0 that are consere n these evaluatons. In aton, a small Monte Carlo experment has been unertaken to check the fnte-sample performance of the feasble bas-correcte estmator, compare wth the ML an the nfeasble bas-correcte estmator. The same sample szes an strbutons for the ranom regressor are consere, but only a lmte selecton of parameter values for these strbutons, an a small number of values for β 0 are consere. Table 4 reports results average over,000 Monte Carlo replcatons. Several features of these are noteworthy. Frst, an as expecte, both the feasble an nfeasble bas correcton substantally reuces the bas of the ML. For nstance, when the

12 covarate s ch-square strbute, n 5 an β 0-0., the absolute percentage bases of the nfeasble an feasble bas-ajuste MLs are.5% an 5.0% respectvely, as compare wth a corresponng bas of 4.4% for βˆ tself. In aton, t s noteworthy that the feasble bascorrecte estmator ~ β BC often out-performng the nfeasble estmator βˆ BC. For example, for the normally strbute covarate, when n 5 an β 0.5, the absolute percentage bases of βˆ, βˆ BC an ~ β BC are 7.6%, 5.0% an.% respectvely; an the corresponng fgures for the Stuent-t strbute covarate, when n 5 an β 0-0.7, are.9%, 8.% an.%. Fnally, n each part of Table 4 we see that the average values of both the true stanar evaton an the stanar error of βˆ ecrease monotoncally as n ncreases, for a gven value of β 0. The frst of these results reflects the mean-square consstency of the estmator note above. In all but one of the cases consere, s.e. βˆ s an upwar-base estmator of s.. βˆ the excepton s for the ch-square strbute covarate when β For the stuatons tabulate, the largest bas for ths stanar error s.%, whch occurs for the Stuent-t strbute covarate when n 5 an β Ths maxmum bas reuces to 5% when n 00. In each part of Table 4, s.e. βˆ converges to s.. βˆ as n ncreases, as expecte. 5. Conclung remarks In ths paper we apply results from Rlstone et al. 996 an Rlstone an Ullah 005 to erve analytc expressons for the frst two moments of the maxmum lkelhoo estmator for the bnary logt regresson moel wth ranom covarates. Our analyss extens the lmte lterature on ths topc, notably by allowng for ranom covarates. The analytc expressons that we erve are complex, but some smple numercal evaluatons prove some clear messages. The bas an mean square error of ths estmator for the logt moel are etermne by both the value of the true parameter an the ata generatng process of the regressor. For the one-regressor case, the bas takes the sgn of the coeffcent of the regressor. The absolute bas an the mean square error ncrease wth the absolute true value of ths coeffcent, an of course ecrease as the sample sze ncreases. We also fn that a feasble bas-correcte estmator, constructe by subtractng the estmate bas from the maxmum lkelhoo estmator, substantally reuces the bas n all of the stuatons

13 examne n a lmte Monte Carlo experment. We recommen the use of ths bas correcton when the logt moel s estmate from samples of 00 observatons or less. The Monte Carlo experment also ncates that the stanar error assocate wth the maxmum lkelhoo estmator s qute a relable estmator of the true stanar evaton of that estmator. Although the stanar error s generally upwar-base for the cases consere, t converges raply to the true stanar evaton as the sample sze ncreases. The technques that are use n ths paper can be apple realy to etermne analytc expressons for the frst two moments of other maxmum lkelhoo estmators that are efne only mplctly because the lkelhoo equatons cannot be solve analytcally. For example, work n progress eals wth such estmators for moels for count ata.

14 Table : On - Bas an On - MS of ML. Normal μ, σ Regressors Normal 0, Normal, Normal 5, Normal 0, Normal 0, 4 n 5 β 0 Bas MS Bas MS Bas MS Bas MS Bas MS n 50 β 0 Bas MS Bas MS Bas MS Bas MS Bas MS

15 Table : On - Bas an On - MS of ML. Ch-square.f. regressors Ch-square Ch-square 4 Ch-square 5 n 5 β 0 Bas MS Bas MS Bas MS n 50 β 0 Bas MS Bas MS Bas MS

16 Table : On - Bas an On - MS of ML. Stuent-t.f. regressors t 5 t 0 t 5 n 5 β 0 Bas MS Bas MS Bas MS n 50 β 0 Bas MS Bas MS Bas MS

17 Table 4: Average of,000 Monte Carlo smulaton replcatons Normal 0, Ch-square β 0 n βˆ ~ β BC βˆ BC s.e. ˆ β s.. ˆ β β 0 n βˆ ~ β BC βˆ s.e. ˆ β s.. ˆ β BC β 0 n βˆ ~ β BC Stuent-t 5 βˆ BC s.e. ˆ β s.. ˆ β β 0 n βˆ ~ β BC βˆ s.e. ˆ β s.. ˆ β BC

18 Appenx: Proofs of Theorems an Corollares Proofs of Theorems an For the logt moel n, j y. By applyng 0 an the law of terate expectatons, we can erve the followng results. The terms, whch are n Lemmas to erve the bas an MS, but not specfe below, are all equal to zero. vecq V VQ Q { vec Q Q Q} V V V Q Q V vecq vecq Q Q VQV VQV Q { vec vec } Q Q Q Q Q Q H vec Q Q Q Q Q H vec Q Q Q Q Q H vec Q Q Q H vec Q Q Q Q Q H vec Q Q Q Q Q 8

19 9 Q Q Q Q vec H Q Q Q Q Q vec Q Q Q Q vec Q Q Q Q vec Therefore, base on Lemmas, an, Theorems an are prove. Proofs of Corollares an When the logt moel only nclues just one regressor, reuces to V V V QV V H Q

20 4 Base on Lemmas, an 4, Corollares an are prove. 0

21 References Amemya T 980 The n - -orer mean square errors of the maxmum lkelhoo an the mnmum logt ch-square estmator. Annals of Statstcs 8: Berkson J 944 Applcaton of the logstc functon to bo-assay. Journal of the Amercan Statstcal Assocaton 9: Berkson J 955 Maxmum lkelhoo an mnmum χ estmates of the logstc functon. Journal of the Amercan Statstcal Assocaton 50: 0-6. Corero GM, McCullagh P 99 Bas correcton n generalze lnear moels. Journal of the Royal Statstcal Socety, B 5: Cox DR, Snell J 968 A general efnton of resuals. Journal of the Royal Statstcal Socety, B 0: Davs L 984 Comments on a paper by T. Amemya on estmaton n a chotomous logt regresson moel. Annals of Statstcs : fron B 978 Regresson an ANOVA wth zero-one ata: measures of resual varaton. Journal of the Amercan Statstcal Assocaton 7: -. Fahrmer L, Kaufmann H 986 Asymptotc nference n screte response moels. Statstcal Papers 7: Ghosh JK, Snha BK 98 A necessary an suffcent conton for secon orer amssblty wth applcatons to Berkson s boassay problem. Annals of Statstcs 9: 4-8. Goureroux C, Montfort A 98 Asymptotc propertes of the maxmum lkelhoo estmator n chotomous logt moels. Journal of conometrcs 7: Hughes GA, Savn N 994 Is the mnmum ch-square estmator the wnner n logt regresson? Journal of conometrc 6: L J 005 Small sample propertes of screte choce moel estmators base on symmetrc an asymmetrc cumulatve strbuton functons. M.A. xtene ssay, Department of conomcs, Unversty of Vctora. MacKnnon JG, Smth AA 998 Approxmate bas correcton n econometrcs. Journal of conometrcs 85: Menénez ML, Paro, L, Paro MC 009 Prelmnary ph-vergence test estmators for lnear restrctons n a logstc regresson moel. Statstcal Papers 50: Paro JA, Paro L, Paro MC 005 Mnmum φ -vergence estmator n logstc regresson moels. Statstcal Papers 47: R 008 The R Project for Statstcal Computng,

22 Rlstone P, Srvatsava VK, Ullah A 996 The secon orer bas an MS of nonlnear estmators. Journal of conometrcs 75: Rlstone P, Ullah A 00 Samplng bas n Heckman s sample selecton estmator. In: Chaubey YP e Recent avances n statstcal methos. Worl Scentfc, Hackensack NJ. Rlstone P, Ullah A 005 Corrgenum to: The secon orer bas an mean square error of non-lnear estmators. Journal of conometrcs 4: Taylor WF 95 Dstance functons an regular best asymptotcally normal estmates. Annals of Mathematcal Statstcs 4: Ullah A 004 Fnte sample econometrcs. Oxfor Unversty Press, Oxfor. Wle J 008 A note on GMM estmaton of probt moels wth enogenous regressors. Statstcal Papers 49: