ASYMPTOTIC PROPERTIES OF ESTIMATES FOR THE PARAMETERS IN THE LOGISTIC REGRESSION MODEL

Transcription

1 Asymptotc Asan-Afrcan Propertes Journal of Estmates Economcs for and the Econometrcs, Parameters n Vol. the Logstc, No., Regresson 20: Model 65 ASYMPTOTIC PROPERTIES OF ESTIMATES FOR THE PARAMETERS IN THE LOGISTIC REGRESSION MODEL B. Munswamy * and Shbru Temesgen Wakweya ** ABSTRACT The logstc regresson model s one of the popular mathematcal models for the analyss of bnary data wth applcatons n physcal, bomedcal, and behavoral scences, among others. The feature of ths model s to quantfy the effects of several explanatory varables on one dchotomous outcome varable. Normally, the asymptotc propertes of the maxmum lkelhood estmates n the model parameters are used for statstcal nference. In ths paper, a smulaton study was carred usng R-software to see a clear pcture of the asymptotc propertes of maxmum lkelhood estmators for the logstc regresson models. The result of the smulaton studes performs well n showng the consstency and normalty of the maxmum lkelhood estmators for dfferent sample szes. Key words: Logstc regresson, maxmum lkelhood estmator, consstency and normalty.. INTRODUCTION In ths paper, the logstc regresson models, as well as the maxmum lkelhood procedure for the estmaton of ther parameters, are to be ntroduced n detal. Under very general condtons, maxmum lkelhood estmates are consstent, asymptotcally effcent, and asymptotcallynormally dstrbuted. Notce that ths normalty allows one to compute the confdence nterval and perform statstcal tests n a manner analogous to the analyss of lnear multple regresson models, provded the sample sze s large. However, asymptotc propertes of the maxmum lkelhood (ML) estmator n logstc models had been studed earler, see, for example, A. C. Davson, et al. (2006), Goureroux and Monfort (98) and Amemya (985), and dfferent results have been establshed. For example, dfferent proofs of consstency can be found n the lterature such as Beer (200), Goureroux and Monfort (98), and Amemya (985). All of them are based upon the fact that the probablty of the exstence of the estmators approaches one as sample sze tends to nfnty. Furthermore, they proceed on the assumpton that the number of explanatory varables s fxed. Another result presented by Beer (200) enables us to relax the former condton. It allows for any number of varables, but depends on sample sze, and examnes the relatonshp between the number of varables and sample sze that s necessary to preserve the consstency of the estmators. Inference for a generalzed lnear model s generally performed usng asymptotc approxmatons for the bas and the covarance matrx of the parameter estmators. For small * Department of Statstcs, Andhra Unversty, Vsakhapatnam, , Andhra Pradesh, Inda, E-mal: munstats.au@gmal.com, shbru@gmal.com

2 66 B. Munswamy & Shbru Temesgen Wakweya experments, these approxmatons can be poor and result n estmators wth consderable bas (K. G. Russell et.al, 2009). As ndcated by Sujuan Gao and Janzhao Shen(2007) Maxmum lkelhood estmates n logstc regresson may encounter serous bas or even non-exstence wth many covarates or wth hghly correlated covarates and therefore, due attenton should be gven for ts exstence and asymptotc propertes. Ths paper rely on a completely dfferent approach to nvestgate the asymptotc propertes of maxmum lkelhood estmators for logstc regresson models. More precsely, we wll show that the maxmum lkelhood estmators converge under certan condtons to the real value of the parameters f the number of observatons tends to nfnty. To show ths, we follow the theorem descrbed by Lehman and Casella (998) n whch the asymptotc propertes of maxmum lkelhood estmators hold f certan regularty condtons are satsfed. It needs to be ponted out that none of the authors cted above verfed ther work va the Monte Carlo smulaton study. Goureroux and Monfort (98) note, t should be stressed that all these asymptotc results gve lttle ndcaton on the propertes of the estmators n fnte sample, and t would be nterestng to clarfy ths pont by means of Monte Carlo studes. Therefore, ths paper s ntended to provde an extensve standard Monte Carlo smulaton study to show the consstency and asymptotc normalty of the ML estmators of the logstc regresson models. 2. LOGISTIC REGRESSION MODEL A lmtaton of ordnary lnear models s the requrement that the dependent varable s numercal rather than categorcal. But many nterestng varables are categorcal - patents may lve or de, people may be employed or not, student may pass or fal ther exams, an tem may be defectve or not and so on. A range of technques have been developed for analyzng data wth categorcal dependent varables. There are a lot of multvarate statstcal technques such as multple regressons Analyss, Dscrmnant Analyss and Logstc Regresson Analyss that can be used to predct bnary dependent varables from knowledge of a set of ndependent varables. All these technques have ther own statstcal assumptons. One of the major assumptons s the one about the dstrbuton of the response varable under study. Lnear regresson analyss s wdely applcable to predct the value of dependent varable. However, when the dependant varable assumes only two values, an event occurrng or not, present or absent, and success or falure, the use of multple lnear regresson brngs some dffcultes. We also need the assumpton of normal errors and well defned varance-covarance matrx. Moreover, when the dependent varable take only two values statstcal nferences are mpossble. [Dobson, A.J. (990); Chrstensen, R. (997); Nelder, J. A., Wedderburn, R. W. M. (972)]. Multvarate Dscrmnant Analyss does allow drect predcton of group membershp. But, the problem s t requres the assumpton of multvarate normalty of the ndependent varables as well as equal varance-covarance matrces n the two groups for the predcton rule to be optmum. [Anderson, T.W. (984)]. Logstc regresson s a specal type of generalzed lnear model wth many nterestng propertes. Loosely speakng logstc regresson analyss does not requre strct assumptons about the dstrbuton of the response varable, although, t s clear that the response has a

3 Asymptotc Propertes of Estmates for the Parameters n the Logstc Regresson Model 67 bnary outcomes. Ths means mplctly that the Bernoull/Bnomal dstrbutons are the natural choces. Therefore, the assumpton on dstrbuton of the response s qute evdent. Thus, logstc regresson model s approprate to predct the bnary dependent varable. [ Efron, B. (975)]. In logstc regresson, a sngle outcome varable y follows a Bernoull probablty functon that takes on the value wth probablty π(x) and 0 wth probablty π(x). Then π(x) vares over the observatons as an nverse logstc functon of a vector x, whch ncludes a constant and p explanatory varable: The specfc form of the logstc regresson model wth unknown parameters β 0, β, , β p s β 0 +β x + β px p e π () x = + e β 0 +β x + β px p At tmes, t s convenent to change the notaton slghtly by wrtng x o =, thus the above model becomes Where X = (x 0, x,.... x p ) T and β = (β 0, β,..., β p ) T T X β e π () x = T () X β + e A transformaton of π(x) s called the logt transformaton, and s gven by π() X log() t π X = In () π X Under the above transformaton, we can wrte the regresson model () as (2) logt π(x) = X T β (3) The nonlnear relatonshp between π(x) and x are often monotonc wth π(x) ncreasng contnuously as x ncreases, or π(x) decreases contnuously as x ncreases. The S-shaped curves are often shapes for the relatonshps. [Agrest, A.. (996); Hosmer and Lemeshow. (2000)] 2.. Maxmum Lkelhood (ML) Estmaton of the Parameters Suppose we have a sample of n ndependent observatons (y, x ), =,2,,n. Where y denotes the value of a dchotomous outcome varable and x s the value of the explanatory varables for the th subject. Assume. Y ~ Bernoull (, π(x )) To fnd the ML of β = (β 0, β,..., β p ) T n (), we defne the lkelhood functon as follows: n = ( ) L ()() β = () π x π x y y π() x = (()) π () π x y x

4 68 B. Munswamy & Shbru Temesgen Wakweya = T y x β n e T (4) = x β + e Takng the natural logarthm of both sdes yelds the followng expresson for log lkelhood functon: = = T x β ( ) n T y x n β (5) log() L β = e In + e It can be verfed that the frst three partal dervatves of the log lkelhood functon exsts and are gven as follows: log() L β = n () y, µ x j β =, where µ j = E(y ) = π 2 log() L β = n π () π β β = j k x x j k 3 log() L β = n π ()(2) π π β β β = j k r x x x j k r Hence, through maxmzaton of equaton (5) or (4) we can theoretcally estmate the parameter vector β. But the equaton s nonlnear n β and the estmates do not have a closed form expresson. Therefore, β wll be obtaned by maxmzng (5) usng a numercal teratve method. [Agrest, A. (996)]. Newton Raphson method s used to obtan the MLE Asymptotc Propertes of the ML Estmators Under certan regularty condtons, the MLE exhbts several characterstcs that can be nterpreted to mean that t s asymptotcally optmal. Lehmann and Casella (998) provded the followng results n theorem of the MLE under some regularty condtons. These condtons are: (a) The dstrbutons P θ have common support whch does not depend on θ (b) The parameter space Ω s an open nterval of R (c) There exsts an open subset ω of Ω contanng the true parameter pont θ 0 such that for almost all x the densty f(x, θ), 2 3 log f log f log f,, 2 3 θ θ θ exsts for all ω Ω (d) The frst and second dervatves of log f satsfy the equatons

5 Asymptotc Propertes of Estmates for the Parameters n the Logstc Regresson Model 69 E θ log f = 0 θ and 2 2 log f log f Eθ = Eθ I() 2 = θ θ θ (e) 3 log f θ 3 M() x where E θ (M(x)) < for all θ ω Theorem: let f(x, θ) be a famly of densty whch satsfes a-e of the above regularty condtons. Then, wth probablty tendng to as n there exst solutons ˆθ of the lkelhood equatons log() L θ = 0, j =,2,3,... p θ such that j () ˆθ s consstent (2) n() θˆ θ s asymptotcally normal wth mean vector zero and covarance matrx (3) At ˆθ, there s a relatve maxmum of the lkelhood (4) Consstent soluton ˆθ of the lkelhood equaton s essentally unque We verfed all the regularty condtons under the logstc regresson model dscussed earler and then we appled the above Theorem to show the asymptotc propertes of ML estmators for the logstc regresson model and found that the logstc regresson model satsfes all the regularty condtons (a)-(e), therefore, ˆβ satsfes () (4) of above Theorem. 3. A SIMULATION STUDY 3.. Consstency of the ML Estmators The fnte sample performance of consstency of the maxmum lkelhood estmators of the logstc regresson model was assessed usng the standard Monte Carlo smulaton. In our smulaton study, we consdered fve explanatory varables, x, x 2, x 3, x 4, and x 5 whch are fxed and the bnary response varable y, whch s treated as a random varable n the logstc model. For fxed values of the ntercept parameter β 0 and fve other parameters β, β 2, β 3, β 4, and β 5 the objectve s to compare the performance of the values of parameters and ther standard errors when sample sze ncreases. For fxed values of β 0 = 0.8, β =.2, β 2 =.3, β 3 = 0.05, β 4 = 0.6, and β 5 = 0.5 2,000 ndependent sets of random samples for each dfferent sample sze were generated and the followng table gves the results of the smulaton study. As seen from the above table, when the sample sze ncreases the estmates of the parameters are close to the true value of the parameter and the standard errors of the estmates are notceably smaller. Fgure 3. and 3.2 below shows the behavor of the estmate ˆβ as the sample sze ncreases.

6 E s t m a t e d P a r a m e t e r V a l u e s a n d t h e r S t a n d a r d E r r o r s u s n g t h e Logstc 70 B. Munswamy & Shbru Temesgen Wakweya Table Dfferent Sample Szes Regresson Model for Sample Sze n = 200 n = 400 n = 600 n = 800 n = 000 Parameter Mean SE Mean SE Mean SE Mean SE Mean SE ˆβ ˆβ ˆβ ˆβ ˆβ ˆβ Fgure 3.: Dstance of ˆβ from ts Actual Values versus Sample Sze Ignorng Sgn

7 Asymptotc Propertes of Estmates for the Parameters n the Logstc Regresson Model 7 Fgure3.2: Smulated Standard Error of versus Sample Sze 3.2. Normalty of ML the Estmators A quantle-normal graph plots the quantles of the data set aganst the theoretcal quantles of the standard normal dstrbuton to see the normalty of the estmates. If the data set appears to be a sample from a normal populaton, then the ponts wll fall roughly along a lne. Fgure below s the quantle normal graph of ˆβ for dfferent sample szes Applcaton of the Logstc Regresson Model n a Real Data Set Here, we concentrate on the applcaton of the logstc regresson model n the area of educaton to dentfy mportant factors assocated wth the students satsfacton n the course Statstcs for Socologsts. It s assumed that there are many factors whch seem to affect student satsfacton such us: nsttutonal factors, extracurrcular factors, student expectatons, Mathematcs background, prevous nformaton about the course and student demographcs. The data for ths study s taken from Socology Undergraduate students (batch 2007/08) of Adds Ababa Unversty, Ethopa. In ths study the dependent varable s the satsfacton status of the student whch s dchotomzed as Satsfed or not satsfed. The explanatory varables ncluded n the study are: Accessblty to Faculty Admnstraton, Communcaton wthn the nstructor n the classroom, Communcaton wth the nstructor outsde the classroom, Qualty of educaton, Student-Faculty Relatons, Mathematcs background, Gender, Age, and Prevous nformaton

8 72 B. Munswamy & Shbru Temesgen Wakweya Fgure 3: Monte Carlo Smulaton of Fnte Sample Behavor for Normalty of the Estmate (Smulaton Sze = 2,000) about the course among others factor affectng satsfacton. The fnal (optmal) logstc regresson model ncludes only sgnfcant varables and the results of the ftted model can be summarzed n followng tabular form. The above table ndcates that Mathematcs background, age, prevous nformaton about the course and qualty of educatons are some of the factors assocated wth these Socology Undergraduate students. The odds rato are also gven due attenton and nterpreted as the odds of beng satsfed when comparng a gven category of a varable wth the reference category of the same varable. For nstance, the odds rato for varable qualty of educaton wth category Very Good s whch can be nterpreted as for a student who has responded to very good to the qualty of educaton, the odds of beng satsfed s about 49 tmes than a student who responded poor to the qualty of educaton mplyng that the level of satsfacton ncreases wth the qualty of educaton consderng the effect of other varables neglgble..

9 Asymptotc Propertes of Estmates for the Parameters n the Logstc Regresson Model 73 Table 2 Estmates for the Fnal Logstc Regresson Model Covarates Sub groups ˆβ SE. Sg. Odds rato Mathematcs background poor(ref) Good < Very good < Excellent < Age < Prevous nformaton about the course No(Ref) <0.05 Yes < Qualty of educaton poor(ref) Good < Very good < Excellent < Constant < E5 *ref= reference category for comparsons 4. CONCLUSION In ths paper, a Monte Carlo smulaton was used to gve a clear pcture of asymptotc propertes of the logstc regresson model usng R-software. The results of the smulaton studes were provded through the tabular form and graphcal dsplay to get a clear pcture of the consstency and normalty of the MLE for dfferent sample szes. When the sample sze ncreases the estmates of the parameters are close to the true value of the parameter and the standard errors of the estmates are notceably smaller. Ths ndcates that ths smulaton study performs well n showng the consstency of the maxmum lkelhood estmators for parameters of the logstc regresson model. The computaton results also ndcate that the dstrbuton of parameters approxmates normal dstrbuton as sample sze, n, ncreases. References A. C. Davson, D. A. S. Fraser, N. Red (2006), Improved Lkelhood Inference for Dscrete Data, Journal of the Royal Statstcal Socety. Seres B (Statstcal Methodology), Vol. 68, No. 3, pp Agrest, A. (996), An Introducton to Categorcal Data Analyss. John Wley and Sons, Inc., New York. Amemya, T. (985), Advanced Econometrcs. Cambrdge, Harvard Unversty Press. Anderson, T. W. (984), An Introducton to Multvarate Statstcal Analyss, Second Edton. New York: John Wley and Sons. Beer, M. (200), Asymptotc Propertes of the Maxmum Lkelhood Estmator n dchotomous Logstc Regresson Models. Dploma Thess, Unversty of Frbourg Swtzerland. Chrstensen, R. (997), Log lnear Models and Logstc Regresson. Chapman and Hall, London. Dobson, A. J. (990), An Introducton to Generalzed Lnear Models. Chapman and Hall, London.

10 74 B. Munswamy & Shbru Temesgen Wakweya Efron, B. (975), The Effcency of Logstc Regresson Compared to Normal Dscrmnant Analyss. Journal of the Amercan Statstcal Assocaton, 70, Goureroux, C. and Monfort, A. (98), Asymptotc Propertes of the Maxmum Lkelhood Estmator n Dchotomous Logt Models. Journal of Econometrcs, 7, Hosmer, W. and Lemeshow, S. (2000), Appled Logstc Regresson (2nd Ed.). New York, John Wley and Sons, Inc., New York. K. G. Russell; J. A. Eccleston; S. M. Lews; D. C. Woods (2009), Desgn Consderatons for Small Experments and Smple Logstc Regresson, Journal of Statstcal Computaton and Smulaton, Volume 79, Issue, January 2009, pp Lehman, E. L. and Casella, G. (998), Theory of Pont Estmaton. Sprnger, New York. Nelder, J. A., Wedderburn, R. W. M. (972), Generalzed Lnear Models, Journal of the Royal Statstcal Socety. Seres A (General), 35, 2, Sujuan Gao, Janzhao Shen (2007), Asymptotc Propertes of a Double Penalzed MaxmumLkelhood Estmator n Logstc Regresson,, Statstcs & Probablty Letters, Vol. 77, Issue 9, pp