Bayesian Inference for Logit-Model using Informative and Non-informative Priors

Size: px

Start display at page:

Download "Bayesian Inference for Logit-Model using Informative and Non-informative Priors"

Jordan Atkinson
6 years ago
Views:

1 ISSN Joural of Statstcs Volume, 4. pp Abstract Bayesa Iferece for Logt-Model usg Iformatve ad No-formatve Prors Tahr Abbas Malk ad Muhammad Aslam I the feld of ecoometrcs aalyss of bary data s wdely doe. Whe the sample s small Bayesa approach provdes more approprate results o classcal approach (MLE). I Bayesa approach the results ca be mproved by usg dfferet Prors. It s kow that the bary data ca be modeled by usg Logstc, Probt or Tobt Lks. I our study, we use Logstc Lk. Whe modelg bary data, the shape of the dstrbuto of Regresso coeffcets s o more ormal. To fd the coeffcet of Skewess of Regresso coeffcets we use dfferet Prors. It s observed that Haldae Pror provdes better results tha Jeffreys Pror, whle Iformatve Pror performs better tha the other No-formatve Prors for the data set uder cosderato. Keywords Polychotomous Respose, Bary Logt model, No-formatve Pror, Bayesa aalyss, Skewess. Itroducto If the depedet varable a data set s categorcal, t s ot possble to use Lear Regresso Models to estmate the ukow Regresso coeffcets. So, a Geeralzed Lear Model (GLM) s a alteratve techque to estmate the ukow parameters. Departmet of Statstcs, Govermet College Uversty, Lahore, Paksta. Emal: malk83qau@yahoo.com Departmet of Statstcs, Quad--Azam Uversty, Islamabad 44, Paksta. Emal: aslamsdqau@yahoo.com

2 Bayesa Iferece for Logt Model usg Iformatve ad No-formatve Prors Sce we kow that whle usg Maxmum Lkelhood method for the estmato of Regresso coeffcets t may mslead whe we have small sample data sets, because MLE are usually based o Asymptotc Theory. Grffths et al. (987) foud that MLE have sgfcat Bas for small samples. Ths problem ca be hadled by usg Bayesa Techque whle estmatg Regresso parameters. Let us cosder here that the respose varable y s categorcal ature wth bary optos coded as [ or ]. It s obvous that y follows a Beroull Dstrbuto where y = wth probablty p ad y = wth probablty -p. Thus E( y ) p ad Var( y ) p ( p ). Let y ( y, y,..., y) be a sample of observatos. The for a sample of observatos the Lkelhood Fucto s, y y L data parameter p ( p) (.) Now f p H x the the Jot Lkelhood Fucto could be wrtte as: y y ( ) ( ( )) L data H x H x (.) Here s the vector of Regresso coeffcets ad x the set of explaatory varables. Whle H s the Lk Fucto that s Logstc our case ad we wll use ths Lk throughout our study to obta the Posteror estmates. The ukow s are cosdered depedet here, so the Pot Posteror Dstrbuto ca also be wrtte as: y y p data H ( x ) ( H( x )) p (.3) The data set for our study s take from Cegz et al. (). The data set cotas the sample observatos of 3 dvduals. Ths research was actually made by the Isttute of Medcal Research, Kuala Lumpur, Malaysa. They used Erythrocyte Sedmetato Rate (ESR) related to two plasma protes, fbroge ad Y- globul, both measured gm / l, for a sample of thrty-two dvduals. We have classfed our varables as follows: y The Erythrocyte Sedmetato Rate (ESR) x f xg The amout of prote plasma fbroge The amout of prote plasma Y-globul 63

3 64 Tahr Abbas Malk ad Muhammad Aslam Now, we wll modfy the above geeral form wth respect to the varables ad we wll cosder the Bary Logstc Regresso Model wth two explaatory varables as: p Logt( p ) log x f x p g (.4) Here s the tercept whle ad are the slope coeffcets for the depedet explaatory varables fbroge ad Y-globul, respectvely. The above Logstc Regresso Model ca also be represeted as: p pr ( y ) (.5) exp x x f The the Jot Posteror Dstrbuto of the parameters, ad are defed as: p,, data L data,, p,, (.6) Let,, g the the above Jot Posteror Dstrbuto ca be modfed as: p data L data p (.7) The ESR s a o-specfc marker of lless. ESR s the rate at whch the red blood cells settle out of suspeso a blood plasma, whe measured uder stadard codtos. The orgal data were preseted by Collett ad Jema (985) ad were reproduced by Collett (996), who classfed the ESR as bary ( or ). Sce the ESR for a healthy dvdual should be less tha mm/h ad the absolute value of ESR s relatvely umportat, a respose of zero sgfes a healthy dvdual (ESR < ) whle a respose of uty refers to a uhealthy dvdual (ESR ). The am of study s to determe the ukow Regresso parameters by Bayesa approach usg Iformatve ad No-formatve Prors; all the ukow parameters are assumed to be depedet our study. As may wrters used these data.e. Cegz et al. (), Collett (996) ad Collett ad Jema (985), but say othg about the shape of the Dstrbuto of the Regresso parameters. We cosder ths case here ad aalyze possble chage the coeffcet of Skewess by usg dfferet Prors. We have also determed the probablty of rejectg the ull hypothess whch s very smple case whle we are stumble upo wth Bayesa approach. I proceedg sectos we obta the Posteror modes,

4 Bayesa Iferece for Logt Model usg Iformatve ad No-formatve Prors Posteror meas, stadard error ad Odds Rato to check mpact o the Regresso estmates. Logstc Regresso s uversally used whe the depedet varables clude both umercal ad omal measures ad the outcome varable s bary, havg two values. It requres o assumptos about the Dstrbuto of the depedet varables. Aother advatage s that, the Regresso coeffcet ca be terpreted term of Odds Rato. Bayesa Aalyss for Logstc Regresso Models s broadly used moder days. Bayesa Aalyss codtos o the data ad tegrates over the parameters to evaluate the probabltes rather tha estmates. Here, we preset dfferet Prors whch are sutable for gve data set. The stadard choce over the recet years has bee the varat Pror atcpated by Jeffreys (96) that s Jeffrey s Pror. We have also preseted Uform ad Improper (Haldae) Pror whch are descrbed by Berardo ad Smth (994). For Iformatve Pror we have assumed Normal Pror by suggestg sutable values of hyper-parameters. The evaluatos of tegrals rema very dffcult these stuatos ad hard to study t aalytcally but umercal methods ca be used to overcome ths dffculty.. Selecto of Prors A Pror may be declared as a Achlles heel (Weakess) of Bayesa Statstcs, as Bayesa Statstcs parameters are assumed radom. The Prors carry certa Pror formato about the ukow parameter(s) that s coheretly corporated to the Iferece va the Bayes Theorem. Choce of the Pror Dstrbuto depeds upo the ature ad the rage of the parameter(s) beg studed through the Bayesa Aalyss. Sometmes, t happes that the Pror elctato becomes dffcult, or a lttle Pror formato s avalable, the t s covetoal to choose Prors whch may reflect lttle Pror formato. Such Prors are termed as the No-formatve Prors, dfferet, gorat, ad vague or referece Prors. e.g. Berardo (979), Cegz (), Ghosh ad Mukerjee (99), Jeffreys (96), Kass ad Wasserma (996) ad Tbshra (989). Here our study we have suggested three No-formatve Prors ad a Iformatve Pror. No-formatve Pror, by whch we mea a Pror that usually, cotas o Pror formato about a parameter. The smplest stuato s to assg each elemet Uform probablty, the Uform Pror. p( ) (.) 65

5 66 Tahr Abbas Malk ad Muhammad Aslam Aother very mportat Improper Pror s Haldae Pror. Haldae Pror s the Improper Drchlet Dstrbuto havg all hyper-parameters to be zero. It was frst suggested by Rub (98). So, t ca be sad that f Beta s used as a Pror Dstrbuto wth both the parameters equal to zero the the Beta Pror wll be Haldae Pror. It ca also be derved from Beroull Dstrbuto f the Respose varables have oly two categores as yes or o. So, the Haldae Pror wll be as: p ( ) p ( p ) (.) H Ths Pror ca also be derved by partally dfferetatg the Log-lkelhood Fucto of Samplg Dstrbuto of depedet varable. As we have already defed the Logstc Model the value of p as: p exp( x x ) (.3) f g Jeffrey s (946, 96) proposes a No-formatve Pror Dstrbuto. Ths Pror ca also be obtaed whe we the values of parameter as (/, /) Beta Dstrbuto. These Prors are extesvely used lterature as far as the study of Logstc Aalyss s cocer as they are the Cojugate Pror for Samplg Dstrbuto of Respose varable. The Jeffrey s Pror ca be defed as: / / p ( ) p ( p ) (.4) J I Bayesa ecoometrcs, the Normal Pror for ukow parameter of the Regresso Model s extesvely used, but whle studyg Logstc Regresso Models the use of Normal Pror s ot as smple as Smple Regresso Models due to the complcated form of Posteror Dstrbuto as t s ot a Cojugate Pror for ths Model, usually t s used for scale parameter but our case we have cosder the mpact of both the parameters o the Posteror parameters by cosderg depedet Prors for each parameter.e. j j p j; j, j exp (.5) j j j 3. Posteror Aalyss May appled ecoometrc aalyses are cocered wth Modelg dscrete varables such as yes or o, or out of labor force, marred or umarred etc. The

6 Bayesa Iferece for Logt Model usg Iformatve ad No-formatve Prors basc Model ths area s the Dchotomous Respose Model, very smlar to Boassay Dosage Respose Models. We have the respose varable y. 3. Posteror Dstrbuto usg Uform Pror: The Uform Pror s very useful Pror whe there s lake of Pror formato about the parameters to be estmated, lterature the use of Uform Pror s very wde.e. Berardo (979), Ghosh ad Mukerjee (99), Jeffreys (96), Kass ad Wasserma (996) ad Tbshra (989), propose the Bayesa Aalyss of ukow parameters usg oe of the most wdely used No-formatve Prors, that s, a Uform (possbly Improper) Pror that routely used by Laplace (8). The Jot Posteror Dstrbuto usg Jot Uform Pror s gve as: y y exp( xf x ) g p data (3..) k exp( x f xg ) exp( x f xg) 3. Posteror Dstrbuto usg Jeffrey s Pror: Jeffrey s (96) proposes a No-formatve Pror. Berger (985) argues that Bayesa Aalyss usg Noformatve Pror s the sgle most powerful method of statstcal aalyss. The ma feature of Jeffrey s Pror s that t s a Uform measure Iformato Metrc, whch ca be regarded as the atural Metrc for statstcal ferece. Whe there s o formato avalable for the Regresso parameters the usually Jeffrey s Pror works as the approprate choce for the Bayesa Aalyss. We have cosdered the Jot Jeffrey s Pror for the parameters to obta the Jot Posteror Dstrbuto as: y / / y exp( xf x ) g p data (3..) k exp( x f xg ) exp( x f xg ) 3.3 Posteror Dstrbuto usg Haldae Pror: Aother sutable Pror suggested lterature for GLM s Haldae Pror kow Improper Pror because for GLM the Beta Pror works a Cojugate Pror for Beroull tral ad t sometme obta whe Beta has both the parameters equal to zero. Berardo ad Smth (994) use ths Improper Pror for GLM. Here, the Jot Haldae Pror s cosdered for the Jot Posteror Dstrbuto of the Regresso parameters. The Jot Posteror Dstrbuto s gve as: 67

7 68 Tahr Abbas Malk ad Muhammad Aslam exp( xf x ) g k exp( x f xg ) exp( x f xg) p data y y (3.3.) 3.4 Posteror Dstrbuto usg Iformatve Pror: Normal Prors for GLM have bee dscussed the lterature wrte by Dellaportas ad Smth (993). Atchso ad Dusmore (975) use Normal Prors for mult-parameter cases. Normal Dstrbuto has strog grouds to be used as Pror Bayesa Ecoometrc Models. We have cosdered the depedet Normal Prors for all the ukow parameters of the Bary Regresso Model, whch has three ukow parameters. So, the Jot Posteror Dstrbuto for all the parameters s gve below: y y exp( xf x ) g p data k exp( x f xg ) exp( x f xg) j j exp (3.4.) j j j We have Jot Posteror Dstrbutos for all the Jot Prors, to proceed further for Jot Iformatve Pror whch s Normal Pror, the selecto of approprate values for the hyper-parameters s very mportat, whle usg Bayesa Techque the selecto of hyper-parameters usually doe wth elctato usg Predctve Dstrbutos but some tme the Predctve Dstrbutos are very complcated ad to obta hyper-parameters usg these Predctve Dstrbuto are very dffcult so, the ext approprate techque s to select the values of hyper-parameters usg the expert opo keepg vew the possble varato the value of Posteror estmates wth these hyper-parameters. We have suggested the values of hyper-parameters whch are approprate for ths data aalyss. Sce we kow that the Pror Dstrbutos of parameters, ad are as follows, ~ N( a, b ), ~ N( a, b) ad ~ N( a, b ). A rage of values of hyper-parameters suggested ad the Posteror estmates are obtaed, the the approprate values of hyper-parameters that are used for further Bayesa Aalyss are suggested as, ~ N (8.95,5.5), ~ N(4.75,3.5) ad ~ N (3.5,.75). We have calculated a rage of values of hyper-parameters ad selected that values whch has mmum stadard error.

8 Bayesa Iferece for Logt Model usg Iformatve ad No-formatve Prors 4. Numercal Results I ths secto, the umercal results are obtaed for all the Posteror Dstrbutos that are obtaed the prevous sectos; we have used Loglkelhood Fucto to obta the Posteror modes, whle Quadrature method s used to obta Posteror mea ad stadard error. The results are gve Table, ths Table provdes us the Posteror modes, Posteror meas, stadard errors, Odds Rato ad Karl-Pearso coeffcet of Skewess. It ca be observed that the Posteror mode for s greater tha the Posteror mea of whch dcates that the Dstrbuto of ths Posteror estmate s egatvely Skewed how much t s Skewed, t ca be observed from the value of coeffcet of Skewess. We have also observed that the Posteror mea of s greater tha the Posteror mode of whch shows that the Dstrbuto of Posteror estmate s postvely Skewed, also same patter s observed for that also shows a postvely Skewed patter but coeffcet of Skewess s much hgher tha. The values of Odds Rato are computed to measure the stregth of assocato betwee ESR ad the avalable quattes of protes. As we kow that Odds Rato vares from zero to fty. So f Odds Rato s equal to oe t meas there s o assocato betwee the two varables ad f t s greater tha oe t meas the two varables are assocated ad ts stregth of assocato s measured wth the possble value of Odds Rato. If the value of Odds Rato s less tha oe tha they are less assocated wth each other. Here, gve Table, the values of Odds Rato are greater tha oe ad for they are hgher as compared to, so t ca be sad that the stregth of assocato betwee ESR ad fbroge s much hgher tha the assocato of ESR ad Y- globul. It s also observed that the Karl-Pearso coeffcet of Skewess approaches to zero for both parameters wth dfferet Prors,.e. t s hgher for Uform Pror ad least for Normal Pror whch dcates that the shape of the parameters approaches to Symmetry for Normal Pror. Ths type of patter ca also be observed graphcal represetato of the parameters wth dfferet Prors, these graphs are gve Fgures -. The graphs show the shape of the margal Dstrbutos of the Regresso coeffcets. I secto (a) the graphs are for the Uform Pror ad t ca be see 69

9 7 Tahr Abbas Malk ad Muhammad Aslam that ther shape s ot symmetrcal whle the graph of see a() show a patter for egatvely Skewed Dstrbuto whle the slope coeffcets see a() ad see a(3) show a patter of postvely Skewed Dstrbuto. Ths patter ca also be observed for secto (b) where we have preseted the graphs for tercept ad slope coeffcets usg Jeffreys Pror, the same ca be see secto (c) for Haldae Pror ad secto (d) we have the graphs usg Iformatve (Normal) Pror. The shape of Posteror Dstrbutos of the tercepts ad slope coeffcets are smlar but ther level of Symmetry vares wth dfferet Prors. 4. Hypothess Testg: Hypothess testg Bayesa s very smple; here we oly fd the Posteror probablty of acceptg the ull hypothess ad commet upo the possble sgfcace of the coeffcet by tegratg the Jot Posteror Dstrbuto upo the parameters.e. we test the hypothess here. H: Versus H: ad H : Versus H : The Posteror probablty for H whle testg s: p p,, data d d d Whle the Posteror probablty for H whle testg s: p p,, data d d d If we use the Posteror Dstrbuto whle usg the Normal Pror the Posteror probablty of acceptg the ull hypothess of H: s very small that s.368, so we wll coclude favor of alteratve hypothess ad say that t has a very hgh Odd Rato to play a sgfcat role effectg ESR. Whle the probablty of acceptg the ull hypothess of H : s lttle hgh as compare to above that s.38. So, the Posteror probablty dcate that uder Bayesa hypothess crtero there s 3% chace to accept H ad we coclude that Y-globul has a postve effect o ESR but caot be cosdered as sgfcat for ESR as the fbroge s. The No-formatve Pror also provdes the smlar cocluso wth lttle devato Posteror probabltes.

10 Bayesa Iferece for Logt Model usg Iformatve ad No-formatve Prors 5. Cocluso The am of ths artcle s to use Bayesa approach to estmate the Regresso parameters of Logt Model ad the coeffcet of Skewess by usg dfferet Prors. The results are more approprate by usg Iformatve Prors but case of No-formatve Prors t s observed that Haldae Pror provdes better results tha Jeffreys Pror for ths partcular data set. The Odds Rato provdes the evdece for all Prors that fbroge effect more o ESR as compare to globul as ts effect s hghly sgfcat. It ca also be see from graphs that whle usg Normal Pror the shape of the graphs are close to Symmetry as compare to others No-formatve Prors. So t ca be cocluded that the results obtaed through Normal Pror are more approprate tha the results of No-formatve Prors. 7 Table : Results usg Dfferet Prors Coeffcet No formatve Pror Iformatve Uform Pror Jeffrey s Pror Haldae Pror Pror ˆ ˆ ˆ Posteror Mode Posteror Mea Stadard Error SK P Posteror Mode Posteror Mea Odds Rato Stadard Error SK P Posteror Mode Posteror Mea Odds Rato Stadard Error SK P

11 7 Tahr Abbas Malk ad Muhammad Aslam a() a() a(3) Fgure -3: Graphs Posteror Dstrbuto usg Uform Pror b() b() b(3) Fgure 4-6: Graphs Posteror Dstrbuto usg Jeffrey s Pror c() c() c(3) Fgure 7-9: Graphs Posteror Dstrbutos usg Haldae Pror

12 Bayesa Iferece for Logt Model usg Iformatve 73 ad No-formatve Prors d() d() d(3) Fgure -: Graphs Posteror Dstrbutos usg Normal Pror Refereces. Atchso, J. ad I. R. Dusmore, I. R. (975). Statstcal Predcto Aalyss. Cambrdge Uversty Press, U.K.. Berger, J. O. (985). Statstcal Decso Theory ad Bayesa Aalyss, (Secod Edto). Sprger Verlag, New York. 3. Berardo, J. M. (979). Referece Posteror Dstrbutos for Bayesa Iferece. Joural of the Royal Statstcal Socety (B), 4, Berardo, J. M. ad Smth, A. F. M. (994). Bayesa Theory. Joh Wley ad Sos, Eglad. 5. Cegz A. M., Bek Y. ad Ylmaz R. (). Bayesa Iferece of Bary Logstc Regresso Model for Assessg Erythrocyte Sedmetato Rate. Paksta Joural of Bologcal Sceces, 4(9), Collett, D. (996). Modelg Bary Data. Chapma ad Hall, Lodo. 7. Collett, D. ad Jema, A. A. (985). Resduals, outlers ad fluetal observatos Regresso Aalyss. Sas Malaysaa, 4, Dellaportas, P. ad Smth, A. F. M. (993). Bayesa Iferece for Geeralzed Lear ad Proportoal Hazards Models va Gbbs Samplg. Joural of Appled Statstcs, 4, Ghosh, J. K. ad Mukerjee, R. (99). Bayesa Statstcs (Forth edto). Oxford Uversty Press, Eaglad.

13 74 Tahr Abbas Malk ad Muhammad Aslam. Grffths, W. E., Hll, R. C. ad Pope, P. J. (987). Small sample propertes of Probt Model estmators. Joural of Amerca Statstcal Assocato, 8, Jeffreys, H. (96). Theory of Probablty. Oxford Uversty Press, Oxford, Eglad.. Kass, R. E. ad Wasserma, L. (996). The selecto of Pror Dstrbutos by formal rules. Joural of the Amerca Statstcal Assocato, 9, Rub, D. B. (98). The Bayesa Bootstrap. The Aals of Statstcs, 9(), Tbshra, R. (989). No-formatve Prors for oe parameter of may. Bometrka, 76(3),

Chapter 14 Logistic Regression Models

Chapter 14 Logistic Regression Models Chapter 4 Logstc Regresso Models I the lear regresso model X β + ε, there are two types of varables explaatory varables X, X,, X k ad study varable y These varables ca be measured o a cotuous scale as