The Geometry of Logit and Probit

Transcription

1 The Geometry of Logt and Probt Ths short note s meant as a supplement to Chapters and 3 of Spatal Models of Parlamentary Votng and the notaton and reference to fgures n the text below s to those two chapters. To recap, the normal vector s denoted as N j and ts reflecton as -N j. The normal vector s perpendcular to the cuttng plane. The cuttng plane n two dmensons s defned by the equaton N j (W Z j ) + N j (W Z j ) = 0 () where N j and N j are the components of the normal vector and (Z j, Z j ) s the mdpont of the roll call outcomes (see Fgure.). Any pont, (W, W ), that satsfes the equaton above les on the cuttng plane. For example, f the normal vector s (3, -) and the roll call mdpont s (, 0), then ths produces the equaton: 3(w ) (w 0) = 3w 3 w = 3w w 3 = 0 or 3w w = 3 so that (0, -3/), (/3, -), (, 3/), etc., all le on the plane. In three dmensons the cuttng plane s defned by the equaton N j (W Z j ) + N j (W Z j ) + N j3 (W 3 Z j3 ) = 0 () Where, as above, any pont, (W, W, W 3 ), that satsfes the equaton above les on the cuttng plane. For example, f the normal vector s (, -, ) and the roll call mdpont s (,, ), ths produces the equaton: (w ) (w ) + (w 3 ) = w w + w 3 = 0 or w w + w 3 = so that (0, 0, ), (, 0, -), (0,, 3), etc., all le on the plane.

2 For s dmensons the cuttng plane s defned by the vector equaton N j (W Z j ) = 0 (book,.8) (3) where N j s the s by normal vector such that N j N j =, W and Z j are s by vectors and 0 s an s by vector of zeroes. The normal vector s constraned to be of unt length for roll call votng work but t s a general vector n other applcatons. In general, f W A and W B are both ponts n the plane, N j W A = N j W B = c j, where c j s a scalar constant. Geometrcally, every pont n the plane projects onto the same pont on the lne defned by the normal vector, N j and ts reflecton -N j (see Fgure.). Ths projecton pont for the general case (N j not necessarly normalzed to one; that s, N j N j = ): M j = c j s N k= N j jk (book,.9) (4) To see ths consder the example of the 3-dmensonal plane above: w w = w w + w3 = w 3 [ ] (5) We need to fnd the pont on the normal vector [ - ] that satsfes equaton (5); that s, the pont where the plane passes through the normal vector tself. Let k be a scalar constant. The soluton s: k k = 4k + 4k + k =, so that k= 9 k [ ]

3 And the pont s whch s gven n equaton (4). Note that, by constructon, N j M j = c j. In addton, n the roll call context, because the mdpont of the Yea and Nay polcy ponts, Z j, s on the cuttng plane, t also projects to the pont M j. The cuttng plane passes through the lne formed by the normal vector and ts reflecton (see Fgure.) at the pont M j. In the case of a smple probt analyss, the cuttng plane conssts of all possble legslator deal ponts such that the probablty of the correspondng legslator votng Yea or votng Nay s exactly.5; namely: P(legslator votes Yea) = P(legslator votes Nay) = ss Φ s = ss Φ s = Φ ( 0) =.5 Where Φ(.) s the dstrbuton functon for the normal and,,, s are legslator s coordnates on the s dmensons. Because the s and s cannot be separately dentfed, the usual assumpton s to set s =. The equaton above reduces to: s s = 0 or s s = = - 0 (6) where s the s-length vector of legslator coordnates: = 3 s 3

4 and s an s-length vector of the coeffcents,, 3,, s ; that s: = 3 s Note that the expresson = - 0 s exactly the same as N j W = c j, whch was used above. Namely, set N j = and every pont n the plane projects onto the pont: M j = - 0 s k= k. (book,.0) (7) In other words, n a regular probt context the coeffcents on the ndependent varables form a normal vector to a plane that passes through the pont - 0 s k= k. Note that ths pont s fxed wth regard to s. In partcular, ss 0 = + s s s So that - = - s s 0 0 s s k k k= s k= (8) When there s complete separaton that s, no error, ths plays an mportant role below. are: The smple logt case s dentcal to probt when s =. The logt probabltes 4

5 ( s s ) e = + e + e ( ) ( ) 0 s s 0 s s =.5 Cancelng out the denomnator and takng the natural log of both sdes yelds the same equaton as probt: s s = 0 In both probt and logt the coeffcents on the ndependent varables form a normal vector to a plane that passes through the pont - 0 s k= k. The cosne of the angle between the normal vectors from probt and logt should be very close to one. That s: ' P L cosθ = P L (9) where θ s the angle between the two normal vectors, and. s the correspondng norm of the normal vector. Computng cosθ s a useful check on the two estmaton technques. Several examples of ths are shown n the Appendx. The Geometry of Complete Separaton (Perfect Votng) To smplfy the notaton below let the normal probablty densty functon be 0 + σ 0 + and the dstrbuton functon be Φ σ whch I wll smplfy to just and Φ, respectvely. I leave s n the expressons because t s the source of the dentfcaton problem. Namely, as a practcal matter, the cuttng plane s dentfed (up to a slght wggle, dependng upon the number of ponts) but, because there s no error, 5

6 6 s 0 and the observed coeffcent vector explodes to entres of + or - because of the mplct dvson of the coeffcents by s. In a standard presentaton the th row of the matrx of ndependent varables would be: 3 s, and the coeffcent vector would be 0 3 s. But for clarty of presentaton I want to separate the ntercept term, 0, from the other s coeffcents. In the general context my use of Yea and Nay below corresponds to the bnary dependent varable beng and 0, respectvely. The frst dervatves for the Probt Log-lkelhood functon are: ln L = Yea Nay Yea Nay Yea Nay s s Yea Nay Φ Φ Φ Φ (0) Let a and b be ndces from to s. Treatng 0 separately, the second dervatves are

7 + + ( ) 0 0 +( Φ) ln L σ σ = 0 Yea Φ Nay ( ) + + ( ) 0 0 +( Φ) ln L σ σ = a 0 a Yea Φ Nay ( ) a (A) (B) And the remanng are: b + +b ( ) 0 0 +( Φ) ln L σ σ = a b ab ba bb Yea Φ Nay ( ) (C) Before turnng the pure case of complete separaton, consder the ntermedate but vexng case where one of the ndependent varables s an ndcator varable that separates wth respect to the bnary dependent varable. That s, whenever the ndcator s the correspondng value of the dependent varable s. Let the ndcator varable be. The frst partal dervatve s: ln L = + = = 0 Φ Φ Φ Φ () Yea,x= Yea,x= 0 Nay,x= 0 Yea,x= Because multplcaton by zero cancels the nd and 3 rd terms and the case Nay, x =, by defnton, does not occur. Hence, for equaton () to hold t must be the case that because =, 0 + σ + K σ = 0, where + Φ Φ + K σ σ Yea,x= Yea,x= 0 K = s s s. 7

8 The K can be treated as constants so that for equaton () to hold t must be the case that +. Note that ths means that the normal vector tself s not dentfed. Wth the K as constants so that s s a constant, then there wll be a dfferent normal vector wth every change n the value of so that the cuttng plane s not dentfed. The case of complete separaton has a dfferent geometry. By defnton, there exsts a plane that perfectly dvdes the Yeas ( s) from the Nays ( 0 s). Let N j be the s by normal vector to ths plane such that N j N j = (I do not need the j subscrpt here but I retan t for notatonal consstency). Hence, as dscussed above, f A and B are both ponts n the plane, then N j A = N j B = c j, where c j s a scalar constant. For complete separaton ether we have: f Yea, then N > c ; and f Nay, then N < c (3) j j j j or f Yea, then N < c ; and f Nay, then N > c j j j j Wthout loss of generalty I wll assume that equaton (3) s the correct polarty. Usng equaton (6) above, ths s equvalent to: 0 0 f Yea, then ; and f Nay, then > < (4) s s s s Or more smply: f Yea then > and f Nay, then < s s 0 s s 0 s s, 0; 0 However, f ths s true then the lkelhood functon forces the followng: f Yea, then Φ ; and f Nay, then Φ 0 σ σ 8

9 because σ 0. Agan, however, note that the cuttng plane s dentfed up to a slght wggle because the s are fxed constants and, by defnton, the plane perfectly dvdes the Yeas ( s) from the Nays ( 0 s). Ths plane passes through the s-dmensonal space of the ndependent varables and separates those cases correspondng to Yeas ( s) from those correspondng to Nays ( 0 s). In actual estmaton, the lkelhood functon for the lnear probt (and logt) problem s globally convex the nverse of the Hessan (equaton system above) s negatve defnte. Hence, the gradent vector wll quckly clmb up the surface of the lkelhood functon to the global maxmum. In the case of perfect separaton what ths means n practce s that the gradent vector explodes as t approaches the global maxmum of zero; that s, the vector becomes nfntely long. Therefore, a straghtforward soluton to ths problem s to apply the constrant = at each teraton and then adjustng the standard devaton term, σ, to compensate. When the standard devaton term begns to vansh, stop. It s then a smple matter to take the normal vector and fnd c j wth the Jance algorthm. Note that ths produces the coeffcent vector correspondng to a perfect specfcaton for ths partcular set of ndependent varables,, and the bnary dependent varable, y. However, snce there s no error, there s no probablty, and no standard errors. 9

10 Appendx Here are two examples from the NES 000 electon study. The varables are whether or not the respondent voted (0=not voted, > 0 voted), and the ndependent varables are party (0 6), ncome ( categores), race (0 = whte; = black), sex (0 = male, = female), south (0=north, =south), educaton (=hgh school, =some college, 3=college). Frst Example: Smple Probt and Smple Logt. probt voted party ncome race sex south educaton age Iteraton 0: log lkelhood = Iteraton : log lkelhood = Iteraton : log lkelhood = Iteraton 3: log lkelhood = Iteraton 4: log lkelhood = Probt regresson Number of obs = 06 LR ch(7) = Prob > ch = Log lkelhood = Pseudo R = voted Coef. Std. Err. z P> z [95% Conf. Interval] party ncome race sex -3.3e south educaton age _cons logt voted party ncome race sex south educaton age Iteraton 0: log lkelhood = Iteraton : log lkelhood = Iteraton : log lkelhood = Iteraton 3: log lkelhood = Iteraton 4: log lkelhood = Logstc regresson Number of obs = 06 LR ch(7) = Prob > ch = Log lkelhood = Pseudo R = voted Coef. Std. Err. z P> z [95% Conf. Interval] party ncome race sex south educaton age _cons

11 Probt Normalzed (Normal Vector) Logt Normalzed (Normal Vector) Party Income Race Sex South Educaton Age The correlaton (Cosne) between the two normal vectors =

12 Second Example: Ordered Probt and Ordered Logt. oprobt party voted ncome race sex south educaton age Iteraton 0: log lkelhood = Iteraton : log lkelhood = Iteraton : log lkelhood = Iteraton 3: log lkelhood = Ordered probt regresson Number of obs = 06 LR ch(7) = 0.73 Prob > ch = Log lkelhood = Pseudo R = party Coef. Std. Err. z P> z [95% Conf. Interval] voted ncome race sex south educaton age /cut /cut /cut /cut /cut /cut ologt party voted ncome race sex south educaton age Iteraton 0: log lkelhood = Iteraton : log lkelhood = Iteraton : log lkelhood = Iteraton 3: log lkelhood = Ordered logstc regresson Number of obs = 06 LR ch(7) = 96.3 Prob > ch = Log lkelhood = Pseudo R = party Coef. Std. Err. z P> z [95% Conf. Interval] voted ncome race sex south educaton age /cut /cut /cut /cut /cut /cut

13 Probt Normalzed (Normal Vector) Logt Normalzed (Normal Vector) voted Income Race Sex South Educaton Age The correlaton (Cosne) between the two normal vectors =