Maximum Likelihood Estimation and Binary Dependent Variables

Transcription

1 MLE and Bnary Deendent Varables Maxmum Lkelhood Estmaton and Bnary Deendent Varables. Startng wth a Smle Examle: Bernoull Trals Lets start wth a smle examle: Teams A and B lay one another 0 tmes; A wns 4 of the games. The games are layed under smlar crcumstances, and the outcomes of each game are ndeendent. You are nterested n estmatng the (unknown) robablty,, that A wns a game. You don t know the value of, but you do know that whatever t s, t s n some sense consstent wth the outcomes you observe. So the challenge s to use the outcomes nformaton to come u wth your best estmate of the value of that s behnd all ths. Here s the data: Game Outcome for A Prob of that outcome: lkelhood lnlkelhood Wn ln() Loss - ln(-) 3 Loss - ln(-) 4 Loss - ln(-) 5 Wn ln() 6 Wn ln() 7 Loss - ln(-) 8 Loss - ln(-) 9 Wn ln() 0 Loss - ln(-) Prob Outcomes: Sum of lnlkelhoods: 4 6 ( ) 4ln( ) + 6ln( )

2 MLE and Bnary Deendent Varables Snce the outcomes of the games are ndeendent n the robablstc sense, the robablty of seeng ths artcular sequence of wns and losses for A s: 4 6 ( )( )( ) ( )( ) ( ) = ( ). Ths s the lkelhood of observng the sequence of wns and losses, condtonal on the unknown arameter. Wth Maxmum Lkelhood Estmaton (MLE) we estmate by answerng the queston: Whch arameter value generates the greatest lkelhood of the observed outcomes or more formally: max lkelhood( ) max ( ) 4 6 *. To solve for functon s strctly ncreasng f lnlkelhood. So we can restate the roblem: *, t s usually easer to frst take the ln(.) of the lkelhood. Snce the ln(.) * maxmzes the lkelhood, t wll also maxmze the 4 6 * [ lkelhood ] = + max ln ( ) max ln ( ) max 4 ln( ) 6 ln( ). We can use the Frst Order Condton (FOC) to solve for * : 4 6 * 4 FOC: = 0 = =.40 = % wns * * 0 The Second Order Condton s satsfed snce: d 4 6 SOC: ln [ lkelhood( ) ] = < 0 for > 0. d ( ) So we have: A s %wns s an MLE of the robablty that A wll beat B. %Wns as an MLE More generally: Suose that teams A and B lay one another n tmes; A wns k of the games. (Unlke the above, you don t know the sequence of wns and losses, you just know that there were k wns n n games.). As above, the games are layed under smlar crcumstances, and the outcomes of each game are ndeendent. And you want to estmate the robablty,, that A wns a game. Snce each game has two outcomes, A wns (wth robablty ) or B wns, wth robablty -, the robablty that A wns k out of n games s gven by the Bnomal dstrbuton: n k n k P( k wns n n games) = ( ) k Ths s also the lkelhood that A wns k out of n games, and so the lkelhood functon s: n k n k Lkn (, ) == ( ). k

3 MLE and Bnary Deendent Varables As dscussed above. wth Maxmum Lkelhood Estmaton (MLE), the goal s to fnd the value of the arameter value that makes most lkely what was observed, namely k wns * n n games. And so the MLE estmate of,, wll be defned by: max Lkn (, ) *. To solve ths otmzaton roblem, frst take ln(.) s as above and then use frst and * second order condtons (FOC and SOC) to fnd the that maxmzes the lnlkelhood: n max ln [ Lkn (, ) ] = max ln + kln( ) + ( n k) ln( ). k k n k * k FOC : = 0 = = % wns * * ( ) n k n k SOC : < 0 maxmum (snce SOC s everywhere < 0) * ( ) ( * ) We have the same result as above, that the wn ercent s an MLE estmator of, the robablty of a wn.. MLE More Generally Suose that Ly (, y,, y β, β,, β ) n k s the robablty (or lkelhood) of observng the outcomes { y } gven arameters { β j }. As the betas change, the lkelhood of gettng the outcomes that actually occurred wll change. The MLE estmates of those unknown arameters wll be the set of arameter values that would have wth the greatest robablty redcted the samle outcomes that were actually observed so the estmaton roblem s: max L { } { y } { } { * β j β β j } j. Tycally, t s convenent to take the ln of the lkelhood functon to solve the otmzaton roblem snce whatever maxmzes L() wll also maxmze lnl(): { L y } { * j j } { } { } { } j max ln β β β. In ths case, the maxmand s the called the lnlkelhood (or log-lkelhood) functon. Snce the overall lkelhood functon s often the roduct of a number of ndvdual lkelhood functons (as, say, n the case of ndeendent observatons), by takng ln s, we convert a messy multlcatve functon nto a much easer to evaluate addtve functon. And so standard ractce s n fact to take ln s and maxmze lnlkelhoods. 3

4 MLE and Bnary Deendent Varables 3. SLR Models and Maxmum Lkelhood Estmaton Consder the standard Smle Lnear Regresson (SLR) model: SLR: Y = β0 + βx + U Snce U N(0, σ ), condtonal on x, we have Y x N( β + β x, σ ) 0 The condtonal densty functon for Y s defned by: f( y x, β0, β) = ex ( y β 0 βx ) σ π σ And so the Lkelhood functon s: ({ } { } ) L y x, β0, β = f( y x, β0, β) = ex ( y β0 βx) σ π σ ln L(.) = nln ( y β0 βx) σ π σ and Log Lkelhood: ( ) Snce nln σ π s ndeendent of the arameters to be estmated, β0 and β, and snce s a negatve scalar, σ max β0, β nln ( y 0 x ) mn β0, β ( y 0 x ) σ π σ β β { β β } Note that to maxmze the lnlkelhood exresson s to mnmze SSR = ( y β0 βx). So the MLE estmates of β0 and β ˆ ˆ ( β0 and β ) wll also be the arameter values that mnmze SSRs. Or ut dfferently: For the standard SLR model, the OLS estmates and the MLE estmates of the ntercet and sloe are one and the same. 4

5 MLE and Bnary Deendent Varables 4. An Examle: Predctng NFL Game Wnners In ths examle, we ll buld some models that redct NFL game wnners as a functon of varous game stats, ncludng rushng yards, assng yards, and tme of ossesson. The data come from Warren Reole s webste, and cover every regular season NFL game from 00 through the 03 season. Here are the summary stats for the numerc varables n the dataset (the Home team s off and the Vstors are def ): Varable Obs Mean Std. Dev. Mn Max season scoreoff frstdownoff rushattoff rushydsoff assattoff asscomoff assydsoff assntoff fumblesoff sacknumoff sackydsoff enydsoff untavgoff scoredef frstdowndef rushattdef rushydsdef assattdef asscomdef assydsdef assntdef fumblesdef sacknumdef sackydsdef enydsdef lne totallne For each game, and for both teams, we have the followng data: game date, teams, fnal score, st down yardage, 3 rd down converson rates, rushng and assng attemts and yards, ntercetons, fumbles, sacks and sack yards, enalty yards, average unt yards, and tme of ossesson. In addton, we have venue nformaton and ont sreads and total (OU) lnes. Let s start by redctng the robablty of a home team vctory as a functon of the dfference n total rushng and assng yards: netyds = netrush + netass, where netrush = HomeRushYds VsRushYds = rushydsoff rushydsdef, and lkewse for netass. 5

6 MLE and Bnary Deendent Varables Take I: OLS - Lnear Probablty Models The frst estmated model s a standard Lnear Probablty Model named because the estmated robabltes are lnear n the exlanatory varables:. reg hwns netyds Source SS df MS Number of obs = F(, 3074) = Model Prob > F = Resdual R-squared = Adj R-squared = 0.49 Total Root MSE = hwns Coef. Std. Err. t P> t [95% Conf. Interval] netyds _cons netyds s hghly statstcally sgnfcant. The models estmates that on average, and controllng for nothng other than home feld advantage, an ncrease n net yards of 00 ncreases the chance of a vctory by 6% onts. An attractve feature of the LPM s that you can read the estmated effects rght off of the regresson outut. Here are the redcted values from the model: Ftted values netyds Rght away you can see a roblem. Some of the robabltes are below zero whle others are well above. That s a roblem. One way around ths roblem s to ntroduce second and thrd order terms: 6

7 MLE and Bnary Deendent Varables. reg hwns netyds* Source SS df MS Number of obs = F( 3, 307) = Model Prob > F = Resdual R-squared = Adj R-squared = 0.5 Total Root MSE =.4557 hwns Coef. Std. Err. t P> t [95% Conf. Interval] netyds netyds -4.44e e e e-07 netyds3-6.35e-09.9e e e-09 _cons Here are the redcted values from the two models: Ftted values netyds Ftted values Ftted values Whle the redcted robabltes are no longer outsde the [0,] nterval, we now have a new ssue: n the tals of the dstrbuton, the redcted robabltes move counter to what s exected (ncreases n net yardage redct decreases n wn robabltes). Notce, however, that for the overwhelmng majorty of the observatons, the redcted wn robabltes for the two models are vrtually the same. 7

8 MLE and Bnary Deendent Varables Take II: MLE - Logt The logt model s based on the logstc functon whch s defned by grahed below: f( x) = x e and It s a symmetrc functon ( f ( x) = f( x) ), bounded below by zero, ncreasng everywhere, and bounded above by one. In the logt secfcaton, P( Hwns netyds) = ( 0 netyds) e. + β + β ( β0+ βnetyds) e Snce P( Hloses netyds) = P( Hwns netyds) = =, + ( 0 netyds) e β + β ( β0+ βnetyds) + e the rato of the two robabltes s: OddsRato P( Hwns netyds) PHloses ( netyds) ( β0+ βnetyds) = =. e And so ln[ OddsRato] β0 β = + netyds lnear n the exlanatory varable! The logstc and logt functons are nverses of one another. The logt functon s defned by: logt( ) = ln( ). 8

9 MLE and Bnary Deendent Varables And so one way to thnk about the Logt Model s that t assumes that the lnoddsrato s lnear n the RHS varables and n that way, the model looks a bt lke the LPM. To llustrate how MLE/logt estmaton works, suose that the Home team wns the frst game, loses the second and wns the thrd, and that these results are ndeendent, so that the robablty of the three occurrng s just the roduct of the robabltes of each one haenng. Then the lkelhood of these three outcomes (condtonal on the dfferent netyds) s: P( Hwns netyds)* P( Hloses netyds)* P( Hwns3 netyds 3), or ( β0+ βnetyds) e 3 0 ( β0+ βnetyds) ( β0+ βnetyds) ( β0+ βnetyds3) (,, β, β ) L Hwns Hloses Hwns =. + e + e + e The MLE method wll estmate the coeffcent values by maxmzng ths exresson wth resect to β0 & β. If we take the ln( ) of ths exresson, and use the fact that ln()=0, we get the log-lkelhood functon ln( + e β + β ) ( β + β netyds ) ln( + e β + β ) ln( + e β + β ) ( 0 netyds ) ( 0 netyds ) ( 0 netyds3 ) 0 whch s much easer to evaluate. logt hwns netyds Iteraton 0: log lkelhood = Iteraton : log lkelhood = Iteraton : log lkelhood = Iteraton 3: log lkelhood = Iteraton 4: log lkelhood = Logstc regresson Number of obs = 3076 LR ch() = Prob > ch = Log lkelhood = Pseudo R = 0.98 hwns Coef. Std. Err. z P> z [95% Conf. Interval] netyds _cons Gven these arameter estmates, the redcted robablty that the home team wns s: 9

10 MLE and Bnary Deendent Varables P( Hwns netyds) = netyds) + e the redcted values are lotted below: In contrast to the LPM, t s dffcult to read margnal effects drectly from the estmated logt coeffcents. However, the sgns of the logt coeffcents are nterretable: f the netyds coeffcent s ostve (as t s above), then ncreases n netyds wll lead to ncreases n the redcted robablty of wns and had the coeffcent been negatve, then the drecton of the effect would have been negatve as well. More formally: Snce ˆ = ˆ ˆ ( + ex ( β + β x ) ), 0 ˆ ˆ ˆ ˆ = β ex ( ˆ β0 + βx) x, and snce ex ( ˆ β ˆ 0 βx) + > 0 and ˆ sgn = sgn x { ˆ β }. ˆ > 0, So the sgns of the logt coeffcents tell you drecton of effects, but coeffcents need to be rocessed before the magntudes of the effects can be determned netyds Ftted values Pr(hwns) Ftted values So all three models generate about the same robablty estmates for about -00 < netyds < 00, where most of the data resde and the Logt aroach avods the roblems assocated wth the two LPMs. 0

11 MLE and Bnary Deendent Varables We can use Stata s margns command to estmate the margnal mact of changes n netyds on the robablty that home wns, estmated at the means:. margns, dydx(_all) atmeans Condtonal margnal effects Number of obs = 3076 Model VCE : OIM Exresson : Pr(hwns), redct() dy/dx w.r.t. : netyds at : netyds =.8839 (mean) Delta-method dy/dx Std. Err. z P> z [95% Conf. Interval] netyds Note that the estmated mact coeffcent s 0.009, slghtly greater than the.006 estmated usng the LPM above. Just to confrm the lnlkelhood, we can use Solve n Excel to maxmze the lnlkelhood functon. Here are the results (I ve ncluded the LPM results as well): LPM Logt b b b b SSR sumlnlke (,849.75) hwns netyds rob resd robwn robloss lkelhood lnlke (0.79) (.36) (0.76) (.5) (0.60) (0.547) (0.88) (0.303) (0.47) (0.64) (0.7) (0.90) (0.84) (0.604) (0.977) Notce that n the Logt mlementaton, and consstent wth the MLE methodology, the lkelhood functon grabs robwn when the home team wns and robloss when they lose. Note the dydx n the syntax. Ths gves us an estmate of the margnal mact, as oosed to an elastcty, whch would requre eyex n the syntax.

12 MLE and Bnary Deendent Varables If you look closely, you ll see small dfferences between the estmated Logt coeffcents here and n the Stata outut above. You ll also see that the reorted lnlkelhood n the Stata outut s greater (less negatve) than what Excel s Solver routne roduced. Ths should not surrse as we have seen before that Solver s algorthm sometmes stos short of the mark. Take II: MLE - Probt In the robt secfcaton, [ β β ] [ β β ] P( Hwns netyds) =Φ + netyds = rob Z + netyds, 0 0 where Z s the standard Normal dstrbuton and Φ (.) s the Cumulatve Dstrbuton Functon (CDF) for Z. Here s what the robt functon looks lke: cdf z Let s see s usng the robt secfcaton of the lkelhood functon matters:

13 MLE and Bnary Deendent Varables. robt hwns netyds Iteraton 0: log lkelhood = Iteraton : log lkelhood = Iteraton : log lkelhood = Iteraton 3: log lkelhood = Probt regresson Number of obs = 3076 LR ch() = Prob > ch = Log lkelhood = Pseudo R = 0.96 hwns Coef. Std. Err. z P> z [95% Conf. Interval] netyds _cons As n the case of the logt functon, the sgns of the coeffcents n the robt tell you somethng drectonally, but to estmate macts, you have to do a bt of a calculaton. But we can look at redcted values for the two MLE aroaches: Pr(hwns) netyds Pr(hwns) Pr(hwns) Almost dentcal redcted values (robt v. logt)! And as before, we can use the margns command to estmate the margnal mact of changes n netyds on the robablty of a home team vctory: 3

14 MLE and Bnary Deendent Varables. margns, dydx(_all) atmeans Condtonal margnal effects Number of obs = 3076 Model VCE : OIM Exresson : Pr(hwns), redct() dy/dx w.r.t. : netyds at : netyds =.8839 (mean) Delta-method dy/dx Std. Err. z P> z [95% Conf. Interval] netyds As you can see, and consstent wth the fgure above, the estmated robt and logt margnal macts are vrtually dentcal. And for fun: And snce we have the data, here are the results from a few logt models of nterest (nettoss s net tme of ossesson). The neutral varable reflects the small number of games layed on neutral felds (not that those games have offcally desgnated home and away teams, deste that fact that nether team s at home):. esttab, comress () () (3) (4) (5) (6) hwns hwns hwns hwns hwns hwns neutral (-0.9) (0.04) (-0.48) (0.0) (-0.30) (0.00) netyds *** (9.87) netrush 0.054*** 0.066*** 0.035*** (3.) (3.90) (6.38) netass *** *** 0.008*** (4.05) (9.9) (4.35) nettoss 0.0*** *** (.00) (6.50) _cons 0.63*** 0.60*** 0.98*** 0.49*** 0.95*** 0.57*** (6.6) (6.) (8.) (5.88) (7.33) (6.00) N statstcs n arentheses * <0.05, ** <0.0, *** <0.00. corr hwns netyds netrush netass nettoss (obs=3076) hwns netyds netrush netass nettoss

15 MLE and Bnary Deendent Varables hwns.0000 netyds netrush netass nettoss esttab, comress For the fnal model we can use the margns command to calculate the elastctes at the mean to get some sense of economc sgnfcance:. margns, eyex(_all) atmeans Condtonal margnal effects Number of obs = 3076 Model VCE : OIM Exresson : Pr(hwns), redct() ey/ex w.r.t. : neutral netrush netass nettoss at : neutral = (mean) netrush = (mean) netass = (mean) nettoss = (mean) Delta-method ey/ex Std. Err. z P> z [95% Conf. Interval] neutral 4.8e netrush netass nettoss Readng Elastctes from Logt Models You may recall that for the OLS model wth Samle Regresson Functon ˆ ˆ x ˆ ˆ βx ˆ = β0 + βx, the elastcty of the redcted value wrt x s ε = =, and so ˆ x ˆ ε ˆ βx the rato of elastctes for two exlanatory varables s =. And so f these ε ˆ β x j j j elastctes are beng calculated at the means, the relatve magntudes wll be the roduct ε ˆ β x of the rato of the estmated coeffcents and the rato of the means: =. It turns ε ˆ β x out that ths s also the case for the logt model: For the logt model, the redcted robablty wll be defned by ( ˆ β ˆ ) 0 βx ˆ = + ex ( + ), and so j j j 5

16 MLE and Bnary Deendent Varables x ˆ x ε = = ˆ β ex ( ˆ β + ˆ β x ) ˆ = ˆ β x ex ( ˆ β + ˆ β x ) ˆ 0 0 ˆ x ˆ. But then ˆ ˆ ε ˆ β x ex ( β + β x ) ˆ ˆ β x = = ε ˆ β x ex ( ˆ β + ˆ β x ) ˆ ˆ β x 0 j j j 0 j j, whch s the same as above. So n both cases, f you know the means of the exlanatory varables, you can easly assess relatve elastctes by just lookng at the roduct of the estmated coeffcents and ther resectve means. Just to check to see f ths s correct, here are some calculatons of relatve elastctes n the LPM and logt models estmated above: coeffs β xbar elastctes means logt LPM logt LPM logt LPM home (0.00) (0.00) (0.00) 0.05 netrush netass nettoss Ratos wrt home home netrush (66.50).49 (66.53).49 netass (5.56) 0.88 (5.56) 0.88 nettoss (5.8) (5.8) Ratos wrt nettoss home (0.063).56 (0.063).56 netrush netass nettoss And the same s true for robt models: ε ˆ βx =. ε ˆ β x j j j 6

17 MLE and Bnary Deendent Varables Relatve Elastctes: Who Knew? More generally n fact, ths result really s qute general (who knew?): Suose that the Samle Regresson Functon s defned by: yˆ f ( ˆ β ˆ 0 βx) = +, for some functon f (.), whch s ncreasng n ts argument so that f (.) > 0. Note that OLS, logt and robt are all of ths tye. Then we have: and snce (.) 0 sgn yˆ = sgn ˆ β. So x x the sgn of the coeffcent tells you the drecton of the margnal effect: f sgn ˆ β > [ < ]0, then ncreases n x wll lead to ncreases [decreases] n ŷ. Snce yˆ = ˆ β ( ˆ ˆ f β0 + βx) j ( ) yˆ x = yˆ x f >, ( ) ˆ β. So the rato of the estmated coeffcents tells you the rato of the ˆ β j relatve margnal effects, the redcted macts of changes n the two RHS varables ε ˆ βx =. So the rato of the elastctes can be determned by the roduct of the ε ˆ j β j x j rato of the estmated coeffcents and the rato of the resectve RHS values at whch x ˆ yˆ βx ε ˆ βx those elastctes have been calculated. (Snce ε = = f (), =.) yˆ x yˆ ε ˆ β x j j j So whle we tycally need to evaluate thngs to determne estmated macts of changes n the RHS varables, we can often say a far amount about sgns of effects and relatve magntudes by just lookng at the estmated arameters. 7