Maximum Likelihood Estimation of Binary Dependent Variables Models: Probit and Logit. 1. General Formulation of Binary Dependent Variables Models

Size: px

Start display at page:

Download "Maximum Likelihood Estimation of Binary Dependent Variables Models: Probit and Logit. 1. General Formulation of Binary Dependent Variables Models"

Garry Ezra Berry
5 years ago
Views:

1 ECO OE 4: Probt and Logt Models ECO OE 4 Maxmum Lkelhood Estmaton of Bnary Dependent Varables Models: Probt and Logt hs note demonstrates how to formulate bnary dependent varables models for maxmum lkelhood estmaton, and how to estmate by maxmum lkelhood the two most common formulatons of such models, namely probt and logt models.. General Formulaton of Bnary Dependent Varables Models A conventonal formulaton of bnary dependent varables models relates the observed bnary outcome varable Y to an unobserved (or latent) dependent varable Y. he unobserved (or latent) dependent varable Y s assumed to be generated by a classcal lnear regresson model of the form Y x + u () where: Y a contnuous real-valued ndex varable for observaton that s unobservable, or latent; x X X L X ), a K row vector of regressor values for observaton ; ( 2 k 0 2 L k ) (, a K column vector of regresson coeffcents; u an d random error term for observaton. ECO ote 4: Flename 452note4_sldes.doc Page of 27 pages

2 ECO OE 4: Probt and Logt Models he random error terms u are assumed to have zero condtonal means and constant condtonal varances for any set of regressor values E x : ( u x ) ( u x ) E( u x ) Var (2.) (2.2) In addton, the condtonal dstrbuton of the u s assumed to be symmetrc around ther zero condtonal mean. Symmetry around mean zero means that Pr( u a) Pr(u > a) Snce by defnton Pr( u > a) Pr(u a), symmetry means that Pr( u a) Pr(u a) or Pr( u a) Pr(u a). (2.3) ECO ote 4: Flename 452note4_sldes.doc Page 2 of 27 pages

3 ECO OE 4: Probt and Logt Models he observable outcomes of the bnary choce problem are represented by a bnary ndcator varable Y that s related to the unobserved dependent varable Y as follows: Y f Y > 0 (3.) Y 0 f Y 0 (3.2) he random ndcator varable Y represents the observed realzatons of a bnomal process wth the followng probabltes: > Pr( Y ) Pr(Y > 0) Pr(x + u 0) (5.) Pr( Y 0) Pr(Y 0) Pr(x + u 0) (5.2) What s requred to estmate the coeffcent vector are analytcal representatons of the bnomal probabltes (5.) and (5.2). ECO ote 4: Flename 452note4_sldes.doc Page 3 of 27 pages

4 ECO OE 4: Probt and Logt Models Interpretaton of the regresson functon Under the zero condtonal mean error assumpton (2.), equaton () mples that ( Y x ) E( x x ) + E( u x ) x E. (4) he regresson functon x s thus the condtonal mean value of the latent random varable Y for gven values of the regressors. he slope coeffcents j (j,, k) are the partal dervatves of the regresson functon (4) wth respect to the ndvdual regressors: E ( Y x ) X j x X j ( 0 + X + L+ jx X j j + L+ k X k ) j. ECO ote 4: Flename 452note4_sldes.doc Page 4 of 27 pages

5 ECO OE 4: Probt and Logt Models he bnomal probabltes 2. Analytcal Representaton of Bnomal Probabltes > Pr( Y ) Pr(Y > 0) Pr(x + u 0) (5.) Pr( Y 0) Pr(Y 0) Pr(x + u 0) (5.2) are represented analytcally n terms of the cumulatve dstrbuton functon, or c.d.f., for the random error term u n regresson equaton (): Y x + u () ECO ote 4: Flename 452note4_sldes.doc Page 5 of 27 pages

6 ECO OE 4: Probt and Logt Models he cumulatve dstrbuton functon (c.d.f.) for the random varable u s denoted n general by G(u) and s defned as G where ( a) Pr( u a) g( u) a du a g(u) du ( ) Pr( u ) g( u) du 0 G ( ) Pr( u ) g( u) du G ( a) G( b) G for a < b he probablty that Pr ( u a) Pr( u a) > s gven n terms of G(a) as Pr ( u > a) G( ) G( a) G( a) For a < b, the probablty Pr( a u b) Pr ( a u b) G( b) G( a). s gven as: ECO ote 4: Flename 452note4_sldes.doc Page 6 of 27 pages

7 ECO OE 4: Probt and Logt Models he frst dervatve of the c.d.f. equals the correspondng probablty densty functon, or p.d.f.: dg ( ) ( u) g u or g( a) du dg u du ( ) dg( a) u a da where g(a) s the value of d G(u) du evaluated at u a. he probablty densty functon (p.d.f.) for the random varable u s the functon g(u) defned over all real values of u such that:. g ( u) 0 2. g ( u) du 3. for any real values a and b where < a < b <, Pr b ( a u b) g( u) a du ECO ote 4: Flename 452note4_sldes.doc Page 7 of 27 pages

8 ECO OE 4: Probt and Logt Models Symmetry Property: In addton to the assumptons that the random varable u has zero mean and constant 2 (fnte) varance, t s assumed that the p.d.f. g(u) s symmetrc about ts zero mean. Symmetry of g(u) around mean zero means that ( a) g( a) and Pr ( u a) Pr( u > a) g Snce by defnton Pr. ( u a) G( a) and Pr( u a) Pr( u a) G( a) symmetry of g(u) mples that G >, ( a) G( a) or equvalently that G( a) G( a). Geometrcally, the symmetry property means that the lower tal area probablty that u a s equal to the upper tal area probablty that u > a. lower tal area Pr(u a) upper tal area Pr(u > a) ECO ote 4: Flename 452note4_sldes.doc Page 8 of 27 pages

9 ECO OE 4: Probt and Logt Models Representaton of the Bnomal Probabltes he bnomal probablty Pr( Y ) Pr( Y > 0) Pr( x + u 0) > can be represented n terms of the c.d.f. for the random varable u as follows: Pr( Y ) Pr( Y 0) > + u > Pr( x 0) Pr ( u > x Pr( u x G( x ( G by symmetry of ( u) x he bnomal probablty Pr( Y 0) Pr( Y 0) Pr( x + u 0) g (6.) can be represented n terms of the c.d.f. for the random varable u as follows: Pr ( Y 0) Pr( Y 0) Pr( x + u 0) Pr ( u x G ( x G( by symmetry of g ( u) x (6.2) he probablty densty functon, or p.d.f., for the bnary dependent varable Y can thus be wrtten as: g Y Y ( Y ) [ G( x ] [ G( x ] for Y 0,. (7) ECO ote 4: Flename 452note4_sldes.doc Page 9 of 27 pages

10 ECO OE 4: Probt and Logt Models 3. he Sample Lkelhood and Log-Lkelhood Functons he sample lkelhood functon for a sample of ndependent observatons {Y :,, } s: L( Y, Y, K, ) g ( ) 2 Y Y Y [ ( )] [ ( )] G x G x (8) Y G ( x ( G( x ) Y Y 0 he sample log-lkelhood functon for a sample of ndependent observatons {Y :,, } s: ln L( Y, Y, K, ) ln ( L) 2 Y ln g ( ) Y { Y ln G( x + ( Y )ln[ G( x ]} ( ) [ ( )] Y ln G x + ( Y )ln G x (9) ln G( x + ln[ G( x ] Y Y 0 ECO ote 4: Flename 452note4_sldes.doc Page 0 of 27 pages

11 ECO OE 4: Probt and Logt Models 4. Dstrbutonal Specfcatons of the Model o complete specfcaton of the model, a specfc probablty dstrbuton must be chosen for the random error terms u. he most commonly adopted dstrbutons n econometrc applcatons are the standard normal and the standard logstc.. he standard normal dstrbuton yelds the probt model. 2. he standard logstc dstrbuton yelds the logt model. ECO ote 4: Flename 452note4_sldes.doc Page of 27 pages

12 ECO OE 4: Probt and Logt Models Probt Model he standard normal dstrbuton has mean μ 0 and varance 2, and s symmetrc around ts zero mean. If the random varable x s normally dstrbuted wth mean μ and varance 2, then the standard normal varable z (x μ) s normally dstrbuted wth mean 0 and varance. hat s, 2 f x ~ ( μ, ), then ~ (0,) where z z (x μ). he standard normal p.d.f. s φ z 2. 2 ( ) ( π ) z 2 exp 2 he standard normal c.d.f. s Z Z 2 2 z ( ) ( ) ( ) ( ) Z Pr z Z φ z dz 2 π exp dz. 2 Choce of the standard normal for the dstrbuton of the random error terms u leads to the probt model. ECO ote 4: Flename 452note4_sldes.doc Page 2 of 27 pages

13 ECO OE 4: Probt and Logt Models Logt Model he standard logstc dstrbuton has mean μ 0 and varance π / 3, and s symmetrc around ts zero mean. 2 2 he standard logstc p.d.f. s f (x ) exp(x ) exp( x 2 ( + exp(x )) ( + exp( x )) 2 ). he standard logstc c.d.f. s F(X ) [ + exp( X ] ) ( + exp( X )) exp(x ) ( + exp(x )). Choce of the standard logstc for the dstrbuton of the random error terms u leads to the logt model. ECO ote 4: Flename 452note4_sldes.doc Page 3 of 27 pages

14 ECO OE 4: Probt and Logt Models 5. he Unvarate Probt Model Probt Representaton of the Bnomal Probabltes In the probt model, the bnomal probabltes Pr( Y ) and ( Y 0) terms of the standard normal c.d.f. ( Z ): Z Z 2 2 z ( Z ) Pr( z Z ) φ ( z) dz ( 2 π ) exp dz 2 Pr are represented analytcally n ECO ote 4: Flename 452note4_sldes.doc Page 4 of 27 pages

15 ECO OE 4: Probt and Logt Models he bnomal probablty ( Y ) follows: Pr( Y ) Pr( Y 0) > Pr( x + u > 0) Pr ( > u x Pr Pr( Y 0) Pr( x + u 0) > s represented n the probt model as > u x Pr > dvdng by > 0 u x Pr by defnton x u snce ~ (0,) x by symmetry of φ (z) (0) ECO ote 4: Flename 452note4_sldes.doc Page 5 of 27 pages

16 ECO OE 4: Probt and Logt Models he bnomal probablty ( Y 0) follows: Pr( Y 0) Pr( Y 0) ote that Pr( x + u 0) Pr ( u x Pr Pr( Y 0) Pr( x + u 0) u x Pr dvdng by > 0 x s represented n the probt model as u snce ~ (0,) x by symmetry of φ (z) () x Z 2 2 z Z exp dz where 2 ( ) ( 2 π ) x Z. ECO ote 4: Flename 452note4_sldes.doc Page 6 of 27 pages

17 ECO OE 4: Probt and Logt Models he contrbuton to the sample lkelhood functon of the -th sample observaton s: ( ) Y Y x x Y g Y 0, x for Y x for Y 0 ECO ote 4: Flename 452note4_sldes.doc Page 7 of 27 pages

18 ECO OE 4: Probt and Logt Models Probt Lkelhood Functon he probt lkelhood functon for a sample of ndependent observatons {Y :,, } s: ( ), L ( ) Y g Y Y x x (2) 0 Y Y x x ECO ote 4: Flename 452note4_sldes.doc Page 8 of 27 pages

19 ECO OE 4: Probt and Logt Models Probt Log-lkelhood Functon he probt log-lkelhood functon for a sample of ndependent observatons {Y :,, } s: ( ) L, ln ( ) [ ] L ln ( ) Y ln g + x Y )ln ( x ln Y + x Y )ln ( x ln Y (3) + 0 Y Y x ln x ln ECO ote 4: Flename 452note4_sldes.doc Page 9 of 27 pages

20 ECO OE 4: Probt and Logt Models A property of the probt log-lkelhood functon s that the coeffcent vector and the scalar parameter are not separately dentfable. Consequently, only the probt coeffcent vector can be estmated. However, t s conventonal to mpose the normalzaton, n whch case the probt coeffcent vector. ECO ote 4: Flename 452note4_sldes.doc Page 20 of 27 pages

21 ECO OE 4: Probt and Logt Models Computng Probt Coeffcent Estmates Maxmum lkelhood estmates of the probt coeffcent vector or are obtaned by maxmzng the probt log-lkelhood functon (3) wth respect to the K elements of or : Max{ } ln ln [ L( ) ] L( ) x x + Y ln ( Y )ln where ln ( x ) + ( Y )ln[ ( x )] Y (3.) or ln ( ) [ L(,)] Max{ } L, ln ln ( x + ( Y )ln[ ( x ] Y (3.2) Maxmzaton of the probt log-lkelhood functon (3.)/(3.2) wth respect to or requres the use of nonlnear optmzaton algorthms such as ewton's method. he result s an ML estmate ˆ ˆ of the probt coeffcent vector together wth an ML estmate of the covarance matrx for ˆ ˆ, Vˆ ( ˆ ) Vˆ (ˆ) Vˆ. ˆ ECO ote 4: Flename 452note4_sldes.doc Page 2 of 27 pages

22 ECO OE 4: Probt and Logt Models Logt Representaton of the Bnomal Probabltes 6. he Unvarate Logt Model In the logt model, the bnomal probabltes Pr( Y ) and ( Y 0) of the standard logstc c.d.f. F (Z ): ( ) F(Z ) Pr z Z exp(z) ( + exp(z ) he bnomal probablty ( Y ) follows: Pr( Y ) Pr( Y 0) > ). Pr Pr( Y 0) Pr( x + u 0) Pr( x + u > 0) Pr ( u > x Pr( u x F( x ( > by defnton snce x Pr are represented analytcally n terms > s represented n the logt model as u ~ f (z) F by symmetry of f(z) (4) ECO ote 4: Flename 452note4_sldes.doc Page 22 of 27 pages

23 ECO OE 4: Probt and Logt Models he bnomal probablty ( Y 0) follows: Pr( Y 0) Pr( Y 0) Pr( x + u 0) Pr ( u x ( x F( Pr Pr( Y 0) Pr( x + u 0) F by defnton of F(Z) x s represented n the logt model as by symmetry of f(z) (5) ECO ote 4: Flename 452note4_sldes.doc Page 23 of 27 pages

24 ECO OE 4: Probt and Logt Models he contrbuton to the sample lkelhood functon of the -th sample observaton s: g Y Y ( Y ) [ F( x ] [ F( x ] ( x F( Y 0, F for Y for Y 0 x Logt Lkelhood Functon he logt lkelhood functon for a sample of ndependent observatons {Y :,, } s: L ( g ( ) Y Y [ ( x ] [ F( x ] Y F (6) F ( x [ F( x ] Y Y 0 ECO ote 4: Flename 452note4_sldes.doc Page 24 of 27 pages

25 ECO OE 4: Probt and Logt Models Logt Log-lkelhood Functon he logt log-lkelhood functon for a sample of ndependent observatons {Y :,, } s: ln L( ln [ L( ] ln g ( ) Y Y ln [ F( x ] [ F( x ] Y { Y ln F( x + ( Y )ln[ F( x ]} ( ) [ ( )] Y ln F x + ( Y )ln F x (7) ln F( x + ln[ F( x ] Y Y 0 ECO ote 4: Flename 452note4_sldes.doc Page 25 of 27 pages

26 ECO OE 4: Probt and Logt Models Computng Logt Coeffcent Estmates by Maxmum Lkelhood Maxmum lkelhood estmates of the logt coeffcent vector are obtaned by maxmzng the logt loglkelhood functon (7) wth respect to the K elements of : ln ( ) ln[ L( ] Max{} L ( ) [ ( )] Y ln F x + ( Y )ln F x (7) ln F( x + ln[ F( x ] Y Y 0 ECO ote 4: Flename 452note4_sldes.doc Page 26 of 27 pages

27 ECO OE 4: Probt and Logt Models A convenent property of the logt log-lkelhood functon (7) s that t s globally concave wth respect to the coeffcent vector. L( ( ) [ ( )] ln Y ln F x + ( Y )ln F x (7) ln F( x + ln[ F( x ] Y Y 0 hs property makes nonlnear maxmzaton of the logt log-lkelhood functon (7) wth respect to farly straghtforward. he most commonly used nonlnear optmzaton algorthm for computng the ML estmates of the logt coeffcents s ewton's method, whch uses analytcal frst and second dervatves of ln L( wth respect to. he result s an ML estmate ˆ L of the logt coeffcent vector together wth an ML estmate of the covarance matrx for ˆ L, Vˆ (ˆ ) Vˆ ˆ. L L ECO ote 4: Flename 452note4_sldes.doc Page 27 of 27 pages

Maximum Likelihood Estimation of Binary Dependent Variables Models: Probit and Logit. 1. General Formulation of Binary Dependent Variables Models

Maximum Likelihood Estimation of Binary Dependent Variables Models: Probit and Logit. 1. General Formulation of Binary Dependent Variables Models ECO 452 -- OE 4: Probt and Logt Models ECO 452 -- OE 4 Mamum Lkelhood Estmaton of Bnary Dependent Varables Models: Probt and Logt hs note demonstrates how to formulate bnary dependent varables models for