Logistic Regression. John Fox. York Summer Programme in Data Analysis. Department of Sociology McMaster University May 2005.

Transcription

1 Lgistic Regressin Yrk Summer Prgramme in Data Analysis Jhn Fx Department f Scilgy McMaster University May by Jhn Fx

2 Lgistic Regressin 1 1. Gals: T shw hw mdels similar t linear mdels can be develped fr qualitative/categrical respnse variables. T intrduce lgit (and prbit) mdels fr dichtmus respnse variables. T intrduce similar statistical mdels fr plytmus respnse variables, including rdered categries. T describe hw lgit mdels can be applied t cntingency tables. c 2005 by Jhn Fx Yrk SPIDA Lgistic Regressin 2 2. Mdels fr Dichtmus Data T understand why special mdels fr qualitative data are required, let us begin by examining a representative prblem, attempting t apply linear regressin t it: In September f 1988, 15 years after the cup f 1973, the peple f Chile vted in a plebiscite t decide the future f the military gvernment. A yes vte wuld represent eight mre years f military rule; a n vte wuld return the cuntry t civilian gvernment. The n side wn the plebiscite, by a clear if nt verwhelming margin. Six mnths befre the plebiscite, FLACSO/Chile cnducted a natinal survey f 2,700 randmly selected Chilean vters. Of these individuals, 868 said that they were planning t vte yes, and 889 said that they were planning t vte n. Of the remainder, 558 said that they were undecided, 187 said that they planned t abstain, and 168 did nt answer the questin. c 2005 by Jhn Fx Yrk SPIDA Lgistic Regressin 3 I will lk nly at thse wh expressed a preference. Figure 1 plts vting intentin against a measure f supprt fr the status qu. Vting intentin appears as a dummy variable, cded 1 fr yes, 0 fr n. Supprt fr the status qu is a scale frmed frm a number f questins abut plitical, scial, and ecnmic plicies: High scres represent general supprt fr the plicies f the miliary regime. Des it make sense t think f regressin as a cnditinal average when the respnse variable is dichtmus? An average between 0 and 1 represents a scre fr the dummy respnse variable that cannt be realized by any individual. c 2005 by Jhn Fx Yrk SPIDA Lgistic Regressin Supprt fr the Status Qu Vting Intentin Figure 1. The Chilean plebiscite data: The slid straight line is a linear least-squares fit; the slid curved line is a lgistic-regressin fit; and the brken line is a nnparametric-regressin fit.the individual bservatins are all at 0 r 1 and are vertically jittered. c 2005 by Jhn Fx Yrk SPIDA

3 Lgistic Regressin 5 In the ppulatin, the cnditinal average E(Y x i ) is the prprtin f 1 s amng thse individuals wh share the value x i fr the explanatry variable the cnditinal prbability π i f sampling a yes in this grup: π i Pr(Y i ) Pr(Y =1 X = x i ) and thus, E(Y x i )=π i (1) + (1 π i )(0) = π i If X is discrete, then in a sample we can calculate the cnditinal prprtin fr Y at each value f X. The cllectin f these cnditinal prprtins represents the sample nnparametric regressin f the dichtmus Y n X. In the present example, X is cntinuus, but we can nevertheless resrt t strategies such as lcal averaging r lcal regressin, as illustrated in the figure. Lgistic Regressin The Linear-Prbability Mdel Althugh nn-parametric regressin wrks here, it wuld be useful t capture the dependency f Y n X as a simple functin, particularly when there are several explanatry variables. Let us first try linear regressin with the usual assumptins: Y i = α + βx i + ε i where ε i N(0,σ 2 ε), and ε i and ε j are independent fr i 6= j. If X is randm, then we assume that it is independent f ε. Under this mdel, E(Y i )=α + βx i,ands π i = α + βx i Fr this reasn, the linear-regressin mdel applied t a dummy respnsevariableiscalledthelinear prbability mdel. This mdel is untenable, but its failure pints the way twards mre adequate specificatins: c 2005 by Jhn Fx Yrk SPIDA c 2005 by Jhn Fx Yrk SPIDA Lgistic Regressin 7 Nn-nrmality: Because Y i can take n nly the values f 0 and 1, the errr ε i is dichtmus as well nt nrmally distributed: If Y i =1, which ccurs with prbability π i,then ε i =1 E(Y i ) =1 (α + βx i ) =1 π i Alternatively, if Y i =0, which ccurs with prbability 1 π i, then ε i =0 E(Y i ) =0 (α + βx i ) =0 π i = π i Because f the central-limit therem, hwever, the assumptin f nrmality is nt critical t least-squares estimatin f the nrmalprbability mdel. Lgistic Regressin 8 Nn-cnstant errr variance: If the assumptin f linearity hlds ver the range f the data, then E(ε i )=0. Using the relatins just nted, V (ε i )=π i (1 π i ) 2 +(1 π i )( π i ) 2 = π i (1 π i ) The heterscedasticity f the errrs bdes ill fr rdinary-leastsquares estimatin f the linear prbability mdel, but nly if the prbabilities π i getclset0r1. Nnlinearity: Mst seriusly, the assumptin that E(ε i )=0 that is, the assumptin f linearity is nly tenable ver a limited range f X-values. If the range f the X s is sufficiently brad, then the linear specificatin cannt cnfine π t the unit interval [0, 1]. It makes n sense, f curse, t interpret a number utside f the unit interval as a prbability. c 2005 by Jhn Fx Yrk SPIDA c 2005 by Jhn Fx Yrk SPIDA

4 Lgistic Regressin 9 This difficulty is illustrated in the plt f the Chilean plebiscite data, in which the least-squares line prduces fitted prbabilities belw 0 at lw levels and abve 1 at high levels f supprt fr the status-qu. Dummy regressr variables d nt cause cmparable difficulties because the general linear mdel makes n distributinal assumptins abut the regressrs. Nevertheless, fr values f π nt t clse t 0 r 1, the linear-prbability mdel estimated by least-squares frequently prvides results similar t thse prduced by mre generally adequate methds. Lgistic Regressin Transfrmatins f π: Lgit and Prbit Mdels T insure that π stays between 0 and 1, we require a psitive mntne (i.e., nn-decreasing) functin that maps the linear predictr η = α+βx int the unit interval. A transfrmatin f this type will retain the fundamentally linear structure f the mdel while aviding prbabilities belw 0 r abve 1. Any cumulative prbability distributin functin meets this requirement: π i = P (η i )=P(α + βx i ) where the CDF P ( ) is selected in advance, and α and β are then parameters t be estimated. An apririreasnable P ( ) shuld be bth smth and symmetric, and shuld apprach π =0and π =1as asympttes. c 2005 by Jhn Fx Yrk SPIDA c 2005 by Jhn Fx Yrk SPIDA Lgistic Regressin 11 Mrever, it is advantageus if P ( ) is strictly increasing, permitting us t rewrite the mdel as P 1 (π i )=η i = α + βx i where P 1 ( ) is the inverse f the CDF P ( ). Thus, we have a linear mdel fr a transfrmatin f π, r equivalently a nnlinear mdel fr π itself. The transfrmatin P ( ) is ften chsen as the CDF f the unit-nrmal distributin Φ(z) = 1 Z z e 1 2 Z2 dz 2π r, even mre cmmnly, f the lgistic distributin 1 Λ(z) = 1+e z where π ' and e ' are the familiar cnstants. Lgistic Regressin 12 Using the nrmal distributin Φ( ) yields the linear prbit mdel: π i = Φ(α + βx i ) = 1 Z α+βxi e 1 2 Z2 dz 2π Using the lgistic distributin Λ( ) prduces the linear lgisticregressin r linear lgit mdel: π i = Λ(α + βx i ) = 1 1+e (α+βx i) Once their variances are equated, the lgit and prbit transfrmatins are s similar that it is nt pssible in practice t distinguish between them, as is apparent in Figure 2. Bth functins are nearly linear between abut π =.2 and π =.8. This is why the linear prbability mdel prduces results similar t the lgit and prbit mdels, except fr extreme values f π i. c 2005 by Jhn Fx Yrk SPIDA c 2005 by Jhn Fx Yrk SPIDA

5 Lgistic Regressin 13 B Lgistic Nrmal " + $ X Figure 2. The nrmal and lgistic cumulative distributin functins (as a functin f the linear predictr and with variances equated). c 2005 by Jhn Fx Yrk SPIDA Lgistic Regressin 14 Despite their similarity, there are tw practical advantages f the lgit mdel: 1. Simplicity: The equatin f the lgistic CDF is very simple, while the nrmal CDF invlves an unevaluated integral. This difference is trivial fr dichtmus data, but fr plytmus data, where we will require the multivariate lgistic r nrmal distributin, the disadvantage f the prbit mdel is much mre acute. 2. Interpretability: The inverse linearizing transfrmatin fr the lgit mdel, Λ 1 (π), is directly interpretable as a lg-dds, while the inverse transfrmatin Φ 1 (π) des nt have a direct interpretatin. Rearranging the equatin fr the lgit mdel, π i = e α+βx i 1 π i The rati π i /(1 π i ) is the dds that Y i =1, an expressin f relative chances familiar t gamblers. c 2005 by Jhn Fx Yrk SPIDA Lgistic Regressin 15 Taking the lg f bth sides f this equatin, π i lg e = α + βx i 1 π i The inverse transfrmatin Λ 1 (π) =lg e [π/(1 π)], called the lgit f π, is therefre the lg f the dds that Y is 1 rather than 0. The lgit is symmetric arund 0, and unbunded bth abve and belw, making the lgit a gd candidate fr the respnse-variable side f a linear mdel: Lgistic Regressin 16 Prbability Odds Lgit π π π lg 1 π e 1 π.01 1/99 = /95 = /9 = /7 = /5 = /3 = /1 = /5 = /1 = c 2005 by Jhn Fx Yrk SPIDA c 2005 by Jhn Fx Yrk SPIDA

6 Lgistic Regressin 17 The lgit mdel is als a multiplicative mdel fr the dds: π i = e α+βx i = e α e βx i 1 π i = e α e β X i S, increasing X by 1 changes the lgit by β and multiplies the dds by e β. Fr example, if β =2, then increasing X by 1 increases the dds by afactrfe 2 ' = Still anther way f understanding the parameter β in the lgit mdel is t cnsider the slpe f the relatinship between π and X. Since this relatinship is nnlinear, the slpe is nt cnstant; the slpe is βπ(1 π), and hence is at a maximum when π =1/2, where the slpe is β/4: Lgistic Regressin 18 π βπ(1 π).01 β β β β β β β β β.0099 The slpe des nt change very much between π =.2 and π =.8, reflecting the near linearity f the lgistic curve in this range. The least-squares line fit t the Chilean plebescite data has the equatin bπ yes = Status-Qu This line is a pr summary f the data. c 2005 by Jhn Fx Yrk SPIDA c 2005 by Jhn Fx Yrk SPIDA Lgistic Regressin 19 The lgistic-regressin mdel, fit by the methd f maximum-likelihd, has the equatin bπ yes lg e = Status-Qu bπ n The lgit mdel prduces a much mre adequate summary f the data, ne that is very clse t the nnparametric regressin. Increasing supprt fr the status-qu by ne unit multiplies the dds f vting yes by e 3.21 =24.8. Put alternatively, the slpe f the relatinship between the fitted prbability f vting yes and supprt fr the status-qu at bπ yes =.5 is 3.21/4 =0.80. Lgistic Regressin An Unbserved-Variable Frmulatin An alternative derivatin psits an underlying regressin fr a cntinuus but unbservable respnse variable ξ (representing, e.g., the prpensity t vte yes), scaled s that ½ 0 when ξi 0 Y i = 1 when ξ i > 0 That is, when ξ crsses 0, the bserved discrete respnse Y changes frm n t yes. The latent variable ξ isassumedtbealinearfunctinfthe explanatry variable X and the unbservable errr variable ε: ξ i = α + βx i + ε i We want t estimate α and β, but cannt prceed by least-squares regressin f ξ n X because the latent respnse variable is nt directly bserved. c 2005 by Jhn Fx Yrk SPIDA c 2005 by Jhn Fx Yrk SPIDA

7 Lgistic Regressin 21 Using these equatins, π i Pr(Y i =1)=Pr(ξ i > 0) = Pr(α + βx i + ε i > 0) = Pr(ε i <α+ βx i ) If the errrs are independently distributed accrdingttheunit-nrmal distributin, ε i N(0, 1), then π i =Pr(ε i <α+ βx i )=Φ(α + βx i ) which is the prbit mdel. Alternatively, if the ε i fllw the similar lgistic distributin, then we get the lgit mdel π i =Pr(ε i <α+ βx i )=Λ(α + βx i ) We will return t the unbserved-variable frmulatin when we cnsider mdels fr rdinal categrical data. c 2005 by Jhn Fx Yrk SPIDA Lgistic Regressin Lgit and Prbit Mdels fr Multiple Regressin T generalize the lgit and prbit mdels t several explanatry variables we require a linear predictr that is a functin f several regressrs. Fr the lgit mdel, π i = Λ(η i )=Λ(α + β 1 X i1 + β 2 X i2 + + β k X ik ) 1 = 1+e (α+β 1X i1 +β 2 X i2 + +β k X ik ) r, equivalently, π i lg e = α + β 1 π 1 X i1 + β 2 X i2 + + β k X ik i Fr the prbit mdel, π i = Φ(η i )=Φ(α + β 1 X i1 + β 2 X i2 + + β k X ik ) The X s can be as general as in the general linear mdel, including, fr example: quantitative explanatry variables; transfrmatins f quantitative explanatry variables; c 2005 by Jhn Fx Yrk SPIDA Lgistic Regressin 23 plynmial regressrs frmed frm quantitative explanatry variables; dummy regressrs representing qualitative explanatry variables; and interactin regressrs. Interpretatin f the partial regressin cefficients in the general lgit mdel is similar t the interpretatin f the slpe in the lgit simple-regressin mdel, with the additinal prvisin f hlding ther explanatry variables in the mdel cnstant. Expressing the mdel in terms f dds, π i = e (α+β 1X i1+ +β k X ik) 1 π i = e α e β Xi1 1 e β Xik k Thus, e β j is the multiplicative effect n the dds f increasing Xj by 1, hlding the ther X s cnstant. Similarly, β j /4 is the slpe f the lgistic regressin surface in the directin f X j at π =.5. Lgistic Regressin 24 The general linear lgit and prbit mdels can be fit tdatabythe methd f maximum likelihd. Hypthesis tests and cnfidence intervals fllw frm general prcedures fr statistical inference in maximum-likelihd estimatin. Fr an individual cefficient, it is mst cnvenient t test the hypthesis H 0 : β j = β (0) j by calculating the Wald statistic Z 0 = B j β (0) j [ASE(B j ) where [ ASE(Bj ) is the estimated asympttic standard errr f B j. The test statistic Z 0 fllws an asympttic unit-nrmal distributin under the null hypthesis. c 2005 by Jhn Fx Yrk SPIDA c 2005 by Jhn Fx Yrk SPIDA

8 Lgistic Regressin 25 Similarly, an asympttic 100(1 a)-percent cnfidence interval fr β j is given by β j = B j ± z a/2 ASE(Bj [ ) where z a/2 is the value frm Z N(0, 1) with a prbability f a/2 t the right. Wald tests fr several cefficientscanbefrmulatedfrmthe estimated asympttic variances and cvariances f the cefficients. Wald tests in lgistic regressin can smetimes be far ff the mark, and s likelihd-rati tests (and mre cmplicated cnfidence intervals based n them) shuld generally be preferred. Lgistic Regressin 26 It is als pssible t frmulate a likelihd-rati test fr the hypthesis that several cefficients are simultaneusly zer, H 0 : β 1 = = β q =0. We prceed, as in least-squares regressin, by fitting tw mdels t the data: The full mdel (mdel 1) lgit(π) =α + β 1 X β q X q +β q+1 X q β k X k and the null mdel (mdel 0) lgit(π) =α +0X X q +β q+1 X q β k X k = α + β q+1 X q β k X k Each mdel prduces a maximized likelihd: L 1 fr the full mdel, L 0 fr the null mdel. c 2005 by Jhn Fx Yrk SPIDA c 2005 by Jhn Fx Yrk SPIDA Lgistic Regressin 27 Because the null mdel is a specializatin f the full mdel, L 1 L 0. The generalized likelihd-rati test statistic fr the null hypthesis is G 2 0 =2(lg e L 1 lg e L 0 ) Under the null hypthesis, this test statistic has an asympttic chisquare distributin with q degrees f freedm. A test f the mnibus null hypthesis H 0 : β 1 = = β k =0is btained by specifying a null mdel that includes nly the cnstant, lgit(π) =α. Lgistic Regressin 28 An analg t the multiple-crrelatin cefficient can als be btained frm the lg-likelihd. By cmparing lg e L 0 fr the mdel cntaining nly the cnstant with lg e L 1 fr the full mdel, we can measure the degree t which using the explanatry variables imprves the predictability f Y. The quantity G 2 2lg e L, called the deviance under the mdel, is a generalizatin f the residual sum f squares fr a linear mdel. Thus, R 2 =1 G2 1 G 2 0 =1 lg e L 1 lg e L 0 is analgus t R 2 fr a linear mdel. c 2005 by Jhn Fx Yrk SPIDA c 2005 by Jhn Fx Yrk SPIDA

9 Lgistic Regressin 29 Using data frm a sample survey f the Canadian ppulatin cnducted in 1977, I will examine hw the labr-frce participatin f yung married wmen (21 t 30 years f age) is related t the presence f children in their husehlds, t their husbands incme, and t the regin f the cuntry in which they reside. Labr-frce participatin is treated dichtmusly: Wrking versus nt wrking utside f the hme. Presence f children and husband s incme are expected t be negatively related t wrking utside f the hme. Husband s annual incme (I), measured in thusands f dllars, was determined by subtracting each wman s incme frm her reprted family incme. Presence f children is represented by a dummy variable (K) cded 1 if minr children are present in the husehld and 0 therwise. Lgistic Regressin 30 Since husband s incme might well have a greater effect amng wmen withut children, an interactin regressr (I K) is included in the mdel. Finally, fur dummy variables (R 1 R 4 ) are emplyed t represent the five regins f Canada (the Atlantic prvinces, Quebec, Ontari, the prairie prvinces, and British Clumbia, with British Clumbia as the baseline). Several mdels were fit t the wmen s labr-frce data t prvide likelihd-rati tests fr the terms in the lgit mdel; the deviance ( 2 lg-likelihd) and the degrees f freedm fr each f these mdels are shwn in the fllwing table: c 2005 by Jhn Fx Yrk SPIDA c 2005 by Jhn Fx Yrk SPIDA Lgistic Regressin 31 Mdel Terms k +1 Deviance 0 C C, I, K, R, I K C, I, K, R C, I, K, I K C, I, R C, K, R An analysis-f-deviance table shwing the several likelihd-rati tests appears belw. The first rw f this table reprts an mnibus test fr all f the terms in the mdel. Lgistic Regressin 32 Surce Mdels Cntrasted df G2 0 p I,K,R,I K <.0001 I K I (Hus. Inc.) K (Children) R (Regin) Deleting the nn-significant interactin and regin effects, the fitted lgit mdel is as fllws, with estimated asympttic standard errrs shwn beneath the cefficients: lgit(p W ) = I K (0.384) ( ) (0.292) R = = where P W is the fitted prbability f wrking utside f the hme, and the R 2 is calculated frm the deviance fr the mdel, c 2005 by Jhn Fx Yrk SPIDA c 2005 by Jhn Fx Yrk SPIDA

10 Lgistic Regressin 33 This fit is graphed in Figure 3, alng with the fit f the linearprbability mdel estimated by least-squares: P W = I K In this case, the tw mdels prduce reasnably similar descriptins f the data. The incme slpe fr the linear prbability mdel is clse t the incme slpe f the fittedlgisticsurfacenearp W =.5, since /4 = Likewise, the cefficients fr presence f children in the tw mdels arealssimilar: 1.576/4 = Lgistic Regressin 34 P[Wrking] 1 - Children Absent c 2005 by Jhn Fx Yrk SPIDA Children Present Husband's Incme Figure 3. Fit f lgit (slid lines) and linear-prbability (brken lines) mdels t the wmen s labur-frce partipcatin data. c 2005 by Jhn Fx Yrk SPIDA Lgistic Regressin Mdels fr Plytmus Data I will describe three general appraches t mdeling plytmus data: 1. mdeling the plytmy directly as a set f unrdered categries, using a generalizatin f the dichtmus lgit mdel; 2. cnstructing a set f nested dichtmies frm the plytmy, fitting an independent lgit r prbit mdel t each dichtmy; and 3. extending the unbserved-variable interpretatin f the dichtmus lgit and prbit mdels t rdered plytmies. Lgistic Regressin The Plytmus Lgit Mdel The dichtmus lgit mdel can be extended t a plytmy by emplying the multivariate-lgistic distributin. This apprach has the advantage f treating the categries f the plytmy in a nn-arbitrary, symmetric manner. The respnse variable Y cantakenanyfm qualitative values, which, fr cnvenience, we number 1, 2,..., m (using the numbers nly as categry labels). Fr example, a married wman can (1) wrk full-time, (2) wrk part-time, r (3) nt wrk utside f the hme. Let π ij dente the prbability that the ith bservatin falls in the jth categry f the respnse variable; that is, π ij Pr(Y i = j) fr j =1,...,m. We have k regressrs, X 1,..., X k,nwhichtheπ ij depend. c 2005 by Jhn Fx Yrk SPIDA c 2005 by Jhn Fx Yrk SPIDA

11 Lgistic Regressin 37 Mre specifically, suppse that this dependence can be mdeled using the multivariate lgistic distributin: π ij = e γ 0j+γ 1j X i1 + +γ kj X ik P 1+ m 1 e γ 0l+γ 1l X i1 + +γ kl X ik l=1 fr j =1,..., m 1 m 1 X π im =1 l=1 π ij There is ne set f parameters, γ 0j,γ 1j,...,γ kj, fr each respnsevariable categry but the last; categry m functins as a type f baseline. The use f a baseline categry is ne way f aviding redundant parameters because f the restrictin that P m j=1 π ij =1. Lgistic Regressin 38 Sme algebraic manipulatin f the mdel prduces π ij lg e = γ π 0j + γ 1j X i1 + + γ kj X ik im fr j =1,..., m 1 Theregressincefficients affect the lg-dds f membership in categry j versus the baseline categry. It is als pssible t frm the lg-dds f membership in any pair f categries j and j 0 : lg e π ij π ij 0 µ πij =lg e π im π ij =lg e Á πij 0 π ij0 lg π e im π im =(γ 0j γ 0j 0)+(γ 1j γ 1j 0)X i1 + +(γ kj γ kj 0)X ik The regressin cefficients fr the lgit between any pair f categries are the differences between crrespnding cefficients. π im c 2005 by Jhn Fx Yrk SPIDA c 2005 by Jhn Fx Yrk SPIDA Lgistic Regressin 39 Nw suppse that the mdel is specialized t a dichtmus respnse variable. Then, m =2,and π i1 π i1 lg e =lg π e i2 1 π i1 = γ 01 + γ 11 X i1 + + γ k1 X ik Applied t a dichtmy, the plytmus lgit mdel is identical t the dichtmus lgit mdel. The maximum-likelihd fit f the plytmus lgit mdel t the wmen s labr-frce-participatin data is as fllws, treating nt wrking utside f the hme as the baseline categry: lg e (P FT /P NW )= Incme (0.484) ( ) Lgistic Regressin 40 lg e (P PT /P NW )= Incme (0.592) ( ) Children (0.4690) The fitted prbabilities under this mdel are graphed in Figure Children (0.362) c 2005 by Jhn Fx Yrk SPIDA c 2005 by Jhn Fx Yrk SPIDA

12 Lgistic Regressin Nested Dichtmies Perhaps the simplest apprach t plytmus data is t fit separate mdels t each f a set f dichtmies derived frm the plytmy. These dichtmies are nested, making the mdels statistically independent. Lgit mdels fit t a set f nested dichtmies cnstitute a mdel fr the plytmy, but are nt equivalent t the plytmus lgit mdel previusly described. A nested set f m 1 dichtmies is prduced frm an m-categry plytmy by successive binary partitins f the categries f the plytmy. Twexamplesfrafur-categryvariableareshwninFigure5. In part (a), the dichtmies are {12, 34}, {1, 2}, and {3, 4}. In part (b), the nested dichtmies are {1, 234}, {2, 34}, and {3, 4}. Lgistic Regressin 42 Fitted Prbability Children Present Full-Time Nt Wrking Part-Time Husband's Incme ($1000s) Fitted Prbability 1 - Children Absent Full-Time Nt Wrking Part-Time Husband's Incme ($1000s) Figure 4. Fit f the plytmus lgit mdel t the wmen s labur-frce participatin data. c 2005 by Jhn Fx Yrk SPIDA c 2005 by Jhn Fx Yrk SPIDA Lgistic Regressin (a) (b) Figure 5. Alternative sets f nested dichtmies fr a fur-categry respnse. c 2005 by Jhn Fx Yrk SPIDA Lgistic Regressin 44 Because the results f the analysis and their interpretatin depend upn the set f nested dichtmies that is selected, this apprach t plytmus data is reasnable nly when a particular chice f dichtmies is substantively cmpelling. Nested dichtmies are attractive when the categries f the plytmy represent rdered prgress thrugh the stages f a prcess. Imagine that the categries in (b) represent adults attained level f educatin: (1) less than high schl; (2) high-schl graduate; (3) sme pst-secndary; (4) pst-secndary degree. Since individuals nrmally prgress thrugh these categries in sequence, the dichtmy {1, 234) represents the cmpletin f high schl; {2, 34} the cntinuatin t pst-secndary educatin, cnditinal n high-schl graduatin; and {3, 4} the cmpletin f a degree cnditinal n undertaking a pst-secndary educatin. c 2005 by Jhn Fx Yrk SPIDA

13 Lgistic Regressin 45 Fr the wmen s labr-frce data, the fllwing system f nested dichtmies appears reasnable: 1. Nt wrking utside the hme versus wrking: {Nt Wrking, (Full-Time, Part-Time)}. 2. Amng thse wrking utside the hme, wrking part-time versus full-time: {Part-Time, Full-Time}. We previusly fit the fllwing mdel t the first f these dichtmies, after deleting the small and nn-significant effects f regin and the interactin between husband s incme and presence f children: lg e [(P PT + P FT ) /P NW ] = I (0.384) ( ) K (0.292) R 2 =.102 Lgistic Regressin 46 Similar results are btained fr the full-time versus part-time dichtmy, where regin effects and the incme by children interactin are als negligible: lg e (P FT /P PT ) = I (0.767) (0.0391) K (0.541) R 2 =.276 c 2005 by Jhn Fx Yrk SPIDA c 2005 by Jhn Fx Yrk SPIDA Lgistic Regressin 47 Because the nested dichtmies are independent, we can pl Wald r likelihd-rati test statistics acrss the tw mdels: Fr example, the likelihd-rati test statistic fr the effect f incme n the wrking versus nt wrking dichtmy is G 2 0 =5.14, n ne degree f freedm. The likelihd-rati test statistic fr the effect f incme n the full-time versus part-time dichtmy is G 2 0 =12.98, als n ne degree f freedm. Cnsequently, G 2 0 = = n tw degrees f freedm (p =.0015) tests the effect f incme n the three-categry respnse variable. c 2005 by Jhn Fx Yrk SPIDA Lgistic Regressin Ordered Lgit and Prbit Mdels Imagine that there is a latent variable ξ that is a linear functin f the X s plus a randm errr: ξ i = α + β 1 X i1 + + β k X ik + ε i Suppse that instead f dividing the range f ξ int tw regins t prduce a dichtmus respnse, the range f ξ is dissected by m 1 bundaries int m regins. Denting the bundaries by α 1 <α 2 < <α m 1, and the resulting respnse by Y, we bserve Y i = 1 if ξ i α 1 2 if α 1 <ξ i α 2 m 1 if α m 2 <ξ i α m 1 m if α m 1 <ξ i c 2005 by Jhn Fx Yrk SPIDA

14 Lgistic Regressin 49 The bundaries, regins, and crrespnding values f ξ and Y are represented graphically in Figure 6. Using the mdel fr the latent variable, alng with categry bundaries, we can determine the cumulative prbability distributin f Y : Pr(Y i j) =Pr(ξ i α j ) =Pr(α + β 1 X i1 + + β k X ik + ε i α j ) =Pr(ε i α j α β 1 X i1 β k X ik ) If the errrs ε i are independently distributed accrding t the standard nrmal distributin, then we btain the rdered prbit mdel. If the errrs fllw the similar lgistic distributin, then we get the rdered lgit mdel: Pr(Y i j) lgit[pr(y i j)] = lg e Pr(Y i >j) = α j α β 1 X i1 β k X ik Lgistic Regressin m-1 m Y " "... " " 1 2 m-2 m-1 Figure 6. A latent respnse ξ divided int m rdered categries (Y =1,..., m) by cut-pints α 1,..., α m 1. > c 2005 by Jhn Fx Yrk SPIDA c 2005 by Jhn Fx Yrk SPIDA Lgistic Regressin 51 Equivalently, Pr(Y i >j) lgit[pr(y i >j)] = lg e Pr(Y i j) =(α α j )+β 1 X i1 + + β k X ik fr j =1, 2,..., m 1. The lgits in this mdel are fr cumulative categries at each pint cntrasting categries abve categry j with categry j and belw. The slpes fr each f these regressin equatins are identical; the equatins differ nly in their intercepts. The lgistic regressin surfaces are therefre hrizntally parallel t each ther, as illustrated in Figure 7 fr m =4respnse categries andasinglex. Fr a fixed set f X s, any tw different cumulative lg-dds say, at categries j and j 0 differ nly by the cnstant (α j α j 0). Lgistic Regressin 52 The dds, therefre, are prprtinal t ne-anther, and fr this reasn, the rdered lgit mdel is called the prprtinal-dds mdel. There are (k +1)+(m 1) = k + m parameters t estimate in the prprtinal-dds mdel, including the regressin cefficients α, β 1,..., β k and the categry bundaries α 1,..., α m 1. There is an extra parameter in the regressin equatins, since each equatin has its wn cnstant, α j, alng with the cmmn cnstant α. Asimpleslutinistsetα =0(and t absrb the negative sign in α j ), prducing lgit[pr(y i >j)] = α j + β 1 X i1 + + β k X ik Figure 8 illustrates the prprtinal-dds mdel fr m =4respnse categries and a single X. The cnditinal distributin f the latent variable ξ is shwn fr tw representative values f the explanatry variable, x 1 and x 2. c 2005 by Jhn Fx Yrk SPIDA c 2005 by Jhn Fx Yrk SPIDA

15 Lgistic Regressin Lgistic Regressin 54 Pr(Y=4 x 1 ) Pr(Y > j) Pr(Y > 1) > Pr(Y=4 x 2 ) Y 4 " Pr(Y > 2) Pr(Y > 3) " 2 " 1 E( > )=" + $ X 2 1 Figure 7. The rdered lgit mdel fr a fur-categry respnse and a single X. c 2005 by Jhn Fx Yrk SPIDA X x 1 x 2 X Figure 8. The prprtinal-dds mdel fr a fur-categry respnse and a single X. c 2005 by Jhn Fx Yrk SPIDA Lgistic Regressin 55 Applying the prprtinal-dds mdel t the wmen s labr-frce data prduces the fllwing results: P PT + P FT lg e = Incme Children P NW (0.377) (0.0194) (0.280) lg e P FT P NW + P PT = Incme Children (0.3619) (0.0194) (0.280) The prprtinal-dds mdel is nt reasnable fr these data: A scre test f the assumptin yields a chi-square statistic f 18.6 with tw degrees f freedm, fr which p Lgistic Regressin Cmparisn f the Three Appraches The three appraches t mdeling plytmus data the plytmus lgit mdel, lgit mdels fr nested dichtmies, and the prprtinaldds mdel address different sets f lg-dds, crrespnding t different dichtmies cnstructed frm the plytmy. Cnsider, fr example, the rdered plytmy {1, 2, 3, 4}: Treating categry 1 as the baseline, the cefficients f the plytmus lgit mdel apply directly t the dichtmies {1, 2}, {1, 3}, and {1,4}, and indirectly t any pair f categries. Frming cntinuatin dichtmies (ne f several pssibilities), the nested-dichtmies apprach mdels {1, 234}, {2, 34}, and {3, 4}. The prprtinal-dds mdel applies t the dichtmies {1, 234}, {12, 34}, and {123, 4}, impsing the restrictin that nly the intercepts f the three regressin equatins differ. c 2005 by Jhn Fx Yrk SPIDA c 2005 by Jhn Fx Yrk SPIDA

16 Lgistic Regressin 57 Which f these mdels is mst apprpriate depends partly n the structure f the data and partly upn ur interest in them. Lgistic Regressin Discrete Explanatry Variables and Cntingency Tables When the explanatry variables as well as the respnse variable are discrete, the jint sample distributin f the variables defines a cntingency table f cunts. An example, drawn frm TheAmericanVter(Cnverse et al., 1960), appears belw. This table, based n data frm a sample survey cnducted after the 1956 U.S. presidential electin, relates vting turnut in the electin t strength f partisan preference, and perceived clseness f the electin: c 2005 by Jhn Fx Yrk SPIDA c 2005 by Jhn Fx Yrk SPIDA Lgistic Regressin 59 Turnut Perceived Intensity f Did Nt Vted Clseness Preference Vte One-Sided Weak Medium Strng Clse Weak Medium Strng Lgistic Regressin 60 The fllwing table gives the empirical lgit fr the respnse variable, prprtin vting lg e prprtin nt vting fr each f the six cmbinatins f categries f the explanatry variables: Perceived Intensity f Clseness Preference lg Vted e Did Nt Vte One-Sided Weak Medium Strng Clse Weak Medium Strng c 2005 by Jhn Fx Yrk SPIDA c 2005 by Jhn Fx Yrk SPIDA

17 Lgistic Regressin 61 Fr example, lgit(vted ne-sided, weak preference) 91/130 =lg e 39/ =lg e 39 =0.847 Because the cnditinal prprtins vting and nt vting share the same denminatr, the empirical lgit can als be written as number vting lg e number nt vting The empirical lgits are graphed in Figure 9, much in the manner f prfiles f cell means fr a tw-way analysis f variance. Lgit mdels are fully apprpriate fr tabular data. When, as in the example, the explanatry variables are qualitative r rdinal, it is natural t use lgit r prbit mdels that are analgus t analysis-f-variance mdels. c 2005 by Jhn Fx Yrk SPIDA Lgistic Regressin 62 Turnut: lg(vted/did nt vte) * * Clse One-Sided Weak Medium Strng Intensity f Preference Figure 9. Empirical lgits fr the American Vter data. c 2005 by Jhn Fx Yrk SPIDA * Lgistic Regressin 63 Treating perceived clseness f the electin as the rw explanatry variable and intensity f partisan preference as the clumn explanatry variable, fr example, yields the mdel lgit π jk = µ + α j + β k + γ jk where π jk is the cnditinal prbability f vting in cmbinatin f categries j f perceived clseness and k f preference; µ is the general level f turnut in the ppulatin; α j is the main effect n turnut f membership in the jth categry f perceived clseness; β k is the main effect n turnut f membership in the kth categry f preference; and γ jk is the interactin effect n turnut f simultaneus membership in categries j f perceived clseness and k f preference. Lgistic Regressin 64 Under the usual sigma cnstraints, this mdel leads t deviatin-cded regressrs, as in the analysis f variance. Adapting the SS( ) ntatin, likelihd-rati tests fr main-effects and interactins can be cnstructed in clse analgy t the incremental F -tests fr the tw-way ANOVA mdel. Deviances under several mdels fr the American-Vter data and the analysis-f-deviance table fr these data are as fllws: Mdel k +1 Deviance G 2 α, β, γ α, β α, γ β,γ α β c 2005 by Jhn Fx Yrk SPIDA c 2005 by Jhn Fx Yrk SPIDA

18 Lgistic Regressin 65 Surce df G 2 0 p Perceived Clseness 1 α β α β,γ Intensity f Preference 2 β α <.0001 β α, γ Clseness Preference 2 γ α, β Lgistic Regressin 66 The lg-likelihd-rati statistic fr testing H 0 :allγ jk =0 fr example, is G 2 0(γ α, β) =G 2 (α, β) G 2 (α, β, γ) = =7.118 with 6 4=2degrees f freedm, fr which p =.03. c 2005 by Jhn Fx Yrk SPIDA c 2005 by Jhn Fx Yrk SPIDA Lgistic Regressin Summary It is prblematic t apply least-squares linear regressin t a dichtmus respnse variable: The errrs cannt be nrmally distributed and cannt have cnstant variance. Even mre fundamentally, the linear specificatin des nt cnfine the prbability fr the respnse t the unit interval. Mre adequate specificatins transfrm the linear predictr η i = α + β 1 X i1 + + β k X ik smthly t the unit interval, using a cumulative prbability distributin functin P ( ). Tw such specificatins are the prbit and the lgit mdels, which use the nrmal and lgistic CDFs, respectively. Lgistic Regressin 68 Althugh these mdels are very similar, the lgit mdel is simpler t interpret, since it can be written as a linear mdel fr the lg-dds: π i lg e = α + β 1 π 1 X i1 + + β k X ik i The dichtmus lgit mdel can be fit t data by the methd f maximum likelihd. Wald tests and likelihd-rati tests fr the cefficients f the mdel parallel t-tests and F -tests fr the general linear mdel. The deviance fr the mdel, defined as G 2 = 2 the maximized lg-likelihd, is analgus t the residual sum f squares fr a linear mdel. c 2005 by Jhn Fx Yrk SPIDA c 2005 by Jhn Fx Yrk SPIDA

19 Lgistic Regressin 69 Several appraches can be taken t mdeling plytmus data, including: (a) mdeling the plytmy directly using a lgit mdel based n the multivariate lgistic distributin; (b) cnstructing a set f m 1 nested dichtmies t represent the m categries f the plytmy; and (c) fitting the prprtinal-dds mdel t a plytmus respnse variable with rdered categries. When all f the variables explanatry as well as respnse are discrete, their jint distributin defines a cntingency table f frequency cunts. It is natural t emply lgit mdels that are analgus t analysis-fvariance mdels t analyze cntingency tables. c 2005 by Jhn Fx Yrk SPIDA