Multinomial logit regression

Transcription

1 07/0/6 Multnomal logt regresson Introducton We now turn our attenton to regresson models for the analyss of categorcal dependent varables wth more than two response categores: Y car owned (many possble cars,.e. Skoda, Fat, Ctroen, etc.), Y socoeconomc status (good, bad, average), Y moble phone provder (.e. Vrgn, Orange, T-Moble) Dependent varable can be ordered (.e. socoeconomc status) or not unordered (.e. car owned, moble phone provder) Several of the models that we wll study may be consdered generalzatons of logstc regresson analyss to polycho-tomous data.

2 07/0/6 Introducton The multnomal logt model assumes that data are case specfc; that s, each ndependent varable has a sngle value for each case. The multnomal logt model also assumes that the dependent varable cannot be perfectly predcted from the ndependent varables for any case. If the multnomal logt s used to model choces, t reles on the assumpton of ndependence of rrelevant alternatves (IIA), whch s not always desrable. Ths assumpton states that the odds of preferrng one class over another do not depend on the presence or absence of other "rrelevant" alternatves. For eample, the relatve probabltes of takng a car or bus to work do not change f a bcycle s added as an addtonal possblty. Introducton Multnomal logt regresson models, the multclass etenson of bnary logstc regresson, have long been used n econometrcs n the contet of modelng dscrete choce (McFadden 974; Bhat 995; Tran 2003) and n machne learnng as a lnear classcaton technque (Haste, Tbshran, and Fredman 2009) for tasks such as tet classcaton (Ngam, Laerty, and McCallum 999). 2

3 07/0/6 Generalzed odds rato (OR) Agrest Table 2. page 37 Myocardal nfarcton (MI) Fatal attack Nonfatal attack No attack Placebo Asprn For the 2 2 table, a a sngle measure can summarze the assocaton For the general I J case, a sngle measure cannot summarze the assocaton wthout loss of nformaton We want to estmate the assocaton of asprn use on myocardal nfarcton (heart attack) Generalzed odds rato (OR) We could collapse fatal attack and nonfatal attack categores together to get nformaton: Myocardal nfarcton Fatal attack or nonfatal attack No attack Placebo Asprn Then, the odds rato of havng myocardal nfarcton s: OR MI 89* * Thus, the odds of a MI are.83 tmes hgher when takng placebo when compared to asprn. 3

4 07/0/6 Generalzed odds rato (OR) For the general I J case: There are There are I pars of rows 2 J pars of columns 2 Ths can produce I J estmates of the odds rato 2 2 We are gong to consder three cases of the generalzed odds rato Generalzed odds rato (OR) For rows a and b, and columns c and d, the odds rato: ( π acπ cd s π bc most π ad ) loosely defned set of generalzed odds ratos. There are I of J ths type. For our heart 2 attack 2 eample, lets compare fatal heart attack to no heart attack: OR 8*0933 5*0845 fatal vs no MI 3.63 That s the odds of havng fatal heart attack vs no heart attack are 3.63 Tmes hgher for the placebo group when compared to the group takng asprn 4

5 07/0/6 Generalzed odds rato (OR) The local odds ratons are obtaned by comparng adjacent rows and columns. That s: For our heart attack eample:. Fatal heart attack vs non fatal heart attack: OR 2. Non fatal heart attack vs no heart attack: OR ( ) ( ).74 There are OR j ( 8 99) ( 5 7) 2.08 π π +, j ( I )( Jlocal ) odds rato j π +, j+ π, j+ Generalzed odds rato (OR) For the I J table wth I representng last row and J representng last column, then: π jπ IJ αj,,2,, I, j,2,, J π π J Ij represents odds rato obtaned by referencng the last column and last row. For our heart attack eample: α α ( 8*0933) ( 5*0933) ( 7*0933) ( 99*0845)

6 07/0/6 Generalzed odds rato (OR) Summary of the generalzed methods:. We have focused on an arbtrary I J table 2. Just as logstc regresson etended the odds rato for a bnary outcome wth several predctors 3. Multnomal logstc regresson wll etend the OR estmaton for the three cases presented prevously to multple predctors Multnomal regresson In general, suppose the response for ndvdual s dscrete wth J levels: Y f wth prob p 2 f wth prob p! J f wth prob p Let be the covarates for ndvdual. If Y s bnary J 2, we usually use logstc regresson model. 2 J 6

7 07/0/6 Multnomal regresson When J 2, we form J, non-redundant logts When J > 2, we often use poytomus (or multnomal) logstc regresson, formng J non-redundant logts: P Y log P Y P Y log P Y! P Y log [,, K [ J,, K [ 2,, K [ J,, [ J,, K P[ Y J,, K K β β 0 20 β + β J 0 + β 2 + β + + β + + β J K 2K K + + β 2K JK βʹ JK βʹ 2 βʹ J Multnomal regresson Each one of these logts can have dfferent set of parameters β j. Bascally, we can thnk of the j-th logt: [ P Y j,, K log j j jk jk j P[ Y as a usual J logstc β 0 + β + + β βʹ,, regresson K model when restrctng yourself to categores j and J Here we have formulated the last column (reference) defnton of the generalzed odds rato 7

8 07/0/6 Multnomal regresson Lets consder probabltes when J > 2: p j J + j ep ep [ βʹj [ βʹ j when j < J and p j J + j ep [ βʹ j when j J We know, that: j J p j Multnomal regresson Proof: J j J ep j J j J j [ βʹ ep ep ep J [ βʹ + ep[ βʹ [ βʹ [ βʹ j + ep J J J p + j j p j j pj j J + j [ βʹ + ep[ J βʹ + ep[ βʹ j 8

9 07/0/6 Multnomal regresson Log-odds for category j versus J for covarates, s:, pj log β j + β j + + β jk k + + β jk K p 0 J We want to know the nterpretaton of β jk s: ( ) K β jk s the log-odds rato for response j versus J for a one unt ncrease n covarate k Multnomal regresson For now we have looked at response j vesrus J Usng our prevous heart attack eample β would be the logodds of havng fatal heart attack versus no heart attack for subjects on placebo when comparng to subjects on asprn Smlarly β 2 s the log-odds of havng non-fatal heart attack nstead of a fatal heart attack Prevously, we stated that ths model suffcently descrbes all possble (( I ) ( odd J ) ratos ) Therefore, we should be able to estmate the odd rato for an arbtrary response j versus j. 9

10 07/0/6 Multnomal regresson Suppose we want log-odds rato for response j versus j for a one unt ncrease n covarate k : Then: p log p ʹjʹ jʹ pʹ j p log pj p ʹjʹ jʹ p p ʹJ J [ β β jʹk p log p jk ʹj j p p ʹJ J [ β β jʹk jk Is the log-odds rato for response j versus j for a one unt ncrease n covarate k Mamum lkehood usng the multnomal To wrte down the multnomal lkehood, we form J ndcator random varables (J of whch are non-redundant): Y Mamum lkehood can be used to estmate the parameters of these models,.e. mamze: as the functon of: j L f Y j 0 f otherwse n J ( β ) β j [ β ʹ, β2, ʹ, βʹ ʹ J p yj j 0

11 07/0/6 Eample For the followng eample a fcttous data set wll be used. The data ncludes a sngle categorcal dependent varable wth three categores. The data also ncludes three contnuous predctors. The data contans 600 cases. Frst, we wll mport the data usng the foregn package and we shall get a summary. Eample Net, we need to dentfy the outcome varable as a factor (.e. categorcal): mdata<-mdata mdata$y<-as.factor(mdata$y) prnt(summary(mdata))

12 07/0/6 Eample Net, we need to load the mglogt package (Crossant, 20) whch contans the functons for conductng the multnomal logstc regresson. Note, the mlogt packages requres s other packages. Net, we need to modfy the data so that the multnomal logstc regresson functon can process t. To do ths, we need to epand the outcome varable (y) much lke we would for dummy codng a categorcal varable for ncluson n standard multple regresson. Eample mdata2<-mlogt.data(mdata, varyngnull, choce"y", shape"wde") prnt(head(mdata2)) 2

13 07/0/6 Eample Now we can proceed wth the multnomal logstc regresson analyss usng the mlogt functon and the ubqutous summary functon of the results. Note that the reference category s specfed as. model.<-mlogt(y~ +2+3, datamdata2, reflevel"") prnt(summary(model.)) 3

14 07/0/6 Eample The results show the logstc coeffcent (B) for each predctor varable for each alternatve category of the outcome varable; alternatve category meanng, not the reference category. The logstc coeffcent s the epected amount of change n the logt for each one unt change n the predctor. The logt s what s beng predcted; t s the odds of membershp n the category of the outcome varable whch has been specfed (here the frst value: was specfed, rather than the alternatve values 2 or 3). The closer a logstc coeffcent s to zero, the less nfluence the predctor has n predctng the logt. Eample The table also dsplays the standard error, t stastc, and the p- value. The t test for each coeffcent s used to determne f the coeffcent s sgnfcantly dfferent from zero. The Pseudo R-Square (McFadden s R 2 ) s treated as a measure of effect sze, smlar to how R 2 s treated n standard multple regresson. However, these types of metrcs do not represent the amount of varance n the outcome varable accounted for by the predctor varables. Hgher values ndcate better ft, but they should be nterpreted wth cauton. 4

15 07/0/6 Eample The Lkelhood Rato ch-square test s alternatve test of goodness-of-ft. As wth most ch-square based tests however, t s prone to nflaton as sample sze ncreases. Here, we see model ft s sgnfcant χ² 29.40, p <.00, whch ndcates our full model predcts sgnfcantly better, or more accurately, than the null model. To be clear, you want the p-value to be less than your establshed cutoff (generally 0.05) to ndcate good ft. To get the epected B values, we can use the ep functon appled to the coeffcents. Eample prnt(ep(coef(model.))) The Ep(B) s the odds rato assocated wth each predctor. We epect predctors whch ncrease the logt to dsplay Ep(B) greater than.0, those predctors whch do not have an effect on the logt wll dsplay an Ep(B) of.0 and predctors whch decease the logt wll have Ep(B) values less than.0. Keep n mnd, the frst two lsted (alt2, alt3) are for the ntercepts. 5