Maximum Likelihood ML (Ch 13 Wooldridge)

Transcription

1 Maxmum Lkelhood ML (Ch 3 Wooldrdge) Motvaton: Dscusson last week focussed on dentfcaton, now we turn to estmaton, where we lke to use the most effcent estmator. Example: Y {,} ndcates whether an ndvdual bought a car W s an ndcator of marketng expenses (e.g. by regon) For dentfcaton we need to control for confoundng factors. For estmaton we want to use estmator that gves us the smallest confdence nterval (snce data collecton s costly)

2 Mcroeconometrcs Markus Frölch OLS and IV can be estmated wthout makng assumptons about the dstrbuton of y or the errors Only assumpton E( x u) = for OLS and Cov( z, u ) = for IV Why ML? onlnear models addtonal assumptons requred for dstrbuton of u x and y x ML s n general the most effcent estmator n the class of estmators that make assumptons on the dstrbuton of the endogenous varable gven the explanatory varables. Why not always ML? ML s nconsstent f dstrbutonal assumptons are volated!

3 Mcroeconometrcs Markus Frölch General framework for Condtonal MLE Let p o (y x) be the true condtonal densty of y gven x = x. K x χ ; y ϒ Result from probablty theory crucal for applyng analogy prncple Condtonal Kullback-Lebler nformaton nequalty (CKLI) For any non-negatve functon f ( x ) such that (3.8) f( y x) v( dy) =, x χ ϒ 3

4 Mcroeconometrcs Markus Frölch CKLI mples po ( y x) (3.9) K( f; x) log po ( ) v( d ), χ ϒ f ( ) y x y x y x K( f; x) s mnmsed at f = p o (3.9) can be appled to parametrc model for po ( x ): (3.) { P f ( x, θ), θ Θ, Θ } (3.) s a correctly specfed model of the condtonal densty p ( ) o, f for some θo Θ (3.) f( x, θ ) = p ( x), all x χ o o 4

5 Mcroeconometrcs Markus Frölch For all x χ K( f; x) can be wrtten as { [ o ] } [ ] { } E log p ( y x ) x = x E log f( y x ) x = x If the parametrc model s correctly specfed, then { [ θ ] } [ θ ] { } E log f( y x ; ) x E log f( y x, ) x or (3.) E[ l x ] E[ l x ] wth ( θ ) ( θ), θ Θ o (3.3) l ( θ ) l ( y,x, θ) log f ( y x; θ) 5

6 Mcroeconometrcs Markus Frölch l ( θ ) s the condtonal log lkelhood for observaton. By takng the expected value of (3.) wth respect to x und y, then we see that θ o solves (3.4) max E[ l ( θ )] θ Θ The sample analog of (3.4) s (3.5) max log f( y x ; θ ) θ Θ = The soluton to problem (3.5) s the condtonal maxmum lkelhood estmator of θ o denoted as ˆ θ 6

7 Mcroeconometrcs Markus Frölch General Framework for CMLE Kullback Lebler Informaton Crteron We want to learn somethng about the realty. We specfy thus a model. There are possbltes:. the model s an approxmaton of the realty, then P P wth P = f. x, θ = θ { ( ) } the true densty wth denstes assocated wth the model. { f., θ, θ } θ the true value of θ and ( x ) P = Θ the famly of. the model s not a good approxmaton. In ths case, P P We thus have to deal wth a specfcaton error. The KLIC s a useful measure to approxmate the dstance between the true densty P and the model P: f(y x) f(y x) KP / P = E log = log f(y ) dy f(y ) ϒ f(y ) x x x K ; K = f = f P/ P P / P MLE mnmzes KLIC! 7

8 Mcroeconometrcs Markus Frölch Tradtonal dervaton of ML Assumpton: random sample =,...,,..d treat x as constant n dstrbuton of y The probablty of observng the sample that s observed s = (, ) f y x θ and the sample log lkelhood st ( ) x log L( θ ) = l( θ) = l ( θ) = log f y, θ = = 8

9 Mcroeconometrcs Markus Frölch The ML estmator ˆ θ maxmses the log Lkelhood functon ˆ = max ( ) = max log, (3.6) θ l θ f ( y x θ) θ Θ θ Θ = The arguments explanng why solvng (3.6) should lead to a good estmator of θ o are heurstc. By contrast the analogy prncple leads drectly to problem (3.5) and we need not assume that the x are fxed. 9

10 Mcroeconometrcs Markus Frölch Examples Example : Lnear regresson model y = x β + u Assumpton: u ~ (, σ ); ths s the dstrbutonal assumpton θ = ( βσ, ) K+ x vector The condtonal densty s: f( y x ) = exp πσ ( y -xβ) σ

11 Mcroeconometrcs Markus Frölch Lkelhood Functon L = πσ ( βσ, ) exp = ( y x β ) σ = exp σ = ( ) πσ ( y x β) exp σ ( ) π σ ( y-xβ) ( y-xβ) = Log Lkelhood Functon l β σ π σ β β σ ( ), = ln ln ( y-x ) ( y-x )

12 Mcroeconometrcs Markus Frölch Example : Probt * Assume the latent varable y s defned as (3.) y * = xθ + e where e s ndependent of x and normally dstrbuted: e (,). We do not observe * y, but only an ndcator varable y, whch s ether equal to or to : (3.) y > = * wenn y * wenn y

13 Mcroeconometrcs Markus Frölch It may be helpful to wrte (3.) wth the ndcator functon: [ ] =, f the expresson n brackets s true, otherwse [ ] = * y = [ y > ]. Dstrbuton of y gven x : (3.4) * P( y = x) = P( y > x) = P( xθ + e > x ) = P( e > xθ x ) = Φ( x θ ) =Φ( x θ ) Φ(.) s the cumulatve dstrbuton functon (CDF) of the standard normal dstrbuton 3

14 Mcroeconometrcs Markus Frölch (3.5) P( y = x ) = Φ( x θ ) Condtonal densty of y gven x : (3.6) f y x θ xθ x θ y ( y) ( ; ) =Φ [ ( )] [ Φ ( )], =, y The log lkelhood functon for observaton s [ ] l( θ ) = ylog Φ ( xθ) + ( y)log Φ ( x θ) 4

15 Mcroeconometrcs Markus Frölch 3. Consstency It can be shown that the ML estmator ˆ θ s consstent under specfc assumptons (Wooldrdge, p. 39, Theorem 3.) plmθˆ = θ 5

16 Mcroeconometrcs Markus Frölch.4 Consstency of CMLE (Theorem 3. gves consstency proof) CMLE s an M-estmator. Use Theorem n Chapter for consstency proof {(, y ) : =,,..., } x random sample wth x K χ,y ϒ G parametrc model: ( ) (a) f (. ; θ ) P { f. x; θ, x χ, θ Θ, Θ } x s a densty for all x and θ (.e. ntegrates to one) θ Θ ( ) ( ) (b) for some f. x; θ = p. x, x χ and that θ max E l ( θ ) s the unque soluton of (c) Θ compact set θ Θ ( ) (d) for each θ Θ l(.,θ) s Borel measurable on ϒ xγ (e) for each (y,x) ϒ x γ l(y,x,.) s a contnuous functon on Θ 6

17 Mcroeconometrcs Markus Frölch Theorem (Consstency of CMLE) (contnued) w w for θ Θ and E b( w ) (f) l(, θ ) b( ) < under these condtons, there exsts a soluton to () whch s the CMLE ˆ θ and plm ˆ θ = θ 7

18 Mcroeconometrcs Markus Frölch 4. Asymptotc ormalty and Varance Estmaton Assume regularty condtons from Theorem.3. Specfcally, assume that θ s n the nteror of Θ, and l (θ) s twce dfferentable. The vector of frst dervatves s called score vector (or gradent) (3.7) s ( ) l l l l θ = θ ( θ)' = ( θ), ( θ),..., ( θ) θ θ θ, P where s ( θ ) s the P x score vector for observaton. 8

19 Mcroeconometrcs Markus Frölch l( θ ) ecessary condton for maxmum n the nteror: = θ Ths mples that = ( s ( ˆ θ ) x ) = [( ( ) )] E s θ x = (Ths s shown further below) 9

20 Mcroeconometrcs Markus Frölch Example : Lnear regresson model contnued l( θ ) = β σ X ( y-xβ ) l( θ) = + 4 σ σ σ ( y-xβ) ( y-xβ ) ecessary condton for maxmum: l( θ ) / θ = X ( y-xβ ) = σ + 4 ( y-xβ) ( y-xβ) = σ σ The soluton of the system s:

21 Mcroeconometrcs Markus Frölch ˆ β = ( XX) Xy ( ˆ) y-xβ ( y-xˆ β) uu ˆ σ = = For comparson the OLS estmator s: ˆ β = ( XX) Xy ( ˆ) y-xβ ( y-xˆ β) uu ˆ σ = = K K

22 Mcroeconometrcs Markus Frölch Example : Probt contnued l ( ) ( ) ( ) y φ xθ θ ( ) ( ) y φ θ x x x θ = Φ θ x [ Φ( xθ) ] (3.8) s ( θ ) [ y ] [ xθ ] = φ( xθ) x Φ( xθ) Φ ( xθ) Φ ( ) e e recall: φ() e = exp ; Φ () e = φ()d v v π there s no analytcal soluton for [( s ( ) x )] numercal optmsaton algorthm E θ =

23 Mcroeconometrcs Markus Frölch Mean zero property of the score ( θ ) E s x = θ Score of the log-lkelhood functon has a zero condtonal mean property: Proof: s( θ) x = s( y, x, θ) ( y x; θ) ϒ ( x θ) ( x θ) ( θ) ( θ) ϒ x ϒ x ϒ E f dy θ = θ log f y,, f y ; dy = θ f y,, dy = θ f y,, dy = θ= θ f ( y, x, θ) / f ( y x; θ) As a consequence: E ( θ ) ( θ ) s x s x = E = θ 3

24 Mcroeconometrcs Markus Frölch.5 Asymptotc ormalty and Asymptotc Varance Estmaton Hessan P x P matrx of the second dervatves of l (θ) (3.3) H s ( θ ) ( ) l θ θ = θ ( θ) Expectaton of Hessan: nformaton matrx (3.4) A = E[ H ( θ )] o o It can be shown that (3.7) A E[ H ( θ )] E[ s ( θ ) s ( θ ) ] = = o o o 4

25 Mcroeconometrcs Markus Frölch.5 Asymptotc ormalty and Asymptotc Varance Estmaton f l ( θ ) s twce contnuously dfferentable on Θ, the hessan for observaton s the PxP matrx: H θ s θ = θ ( ) ( ) ( ) θ θl Defnng A E H ( θ) under the standard regularty condtons, theorem 3 of chapter apples and ( ˆ ) (, θ θ ) such that ( ˆ ) ( θ θ, A ) = (, B ) A = B Var θ = E θ θ s s s A B A ( ) ( ) ( ) ' 5

26 Mcroeconometrcs Markus Frölch Proof of nformaton matrx equalty As derved above s( θ ) x E θ = ( ) Hence, θ s( θ) x = θ s( θ) ( y x ; θ) E f dy θ ( θ) f ( θ) dy ( θ) θ f ( θ) dy ϒ ( θ) ( y x; θ) s( θ) θ ln ( y x; θ) ( y x ; θ) ϒ = ( s ) y x; + s y x ; ϒ θ = l f dy + f f dy ϒ θ ϒ ow for θ ( ) ( x ) s( ) s( ) ( x ) ϒ = y ; + ' y ; ϒ θl θ f θ dy θ θ f θ dy ( θ ) E ( θ ) ( θ ) = E[ H x ] + [ s s '] 6

27 Mcroeconometrcs Markus Frölch Proof (contnued): ( ( θ) ) E ( θ) ( θ) ( ) E H x + s s ' x As a consequence, we obtan for E ( ) ( H θ x) E( s( θ) s( θ) ' ) = condtonal nformaton matrx equalty (CIME) by usng the law of terated expectatons, we obtan A = B equalty thought as the uncondtonal nformaton matrx equalty (UIME) x, 7

28 Mcroeconometrcs Markus Frölch Theorem (Asymptotc ormalty of CMLE): under condtons of Theorem + followng addtonal assumptons: (a) θ nt ( Θ) (b) for each ( y, ) ϒxχ l ( y,,.) (c) elements of θl ( y,, θ ) x x, twce contnuously dfferentable on Θ x bounded n absolute value by a functon b(y,x) wth fnte expectaton ( ) (d) A E H θ s postve defnte (e) the nterchanges of dervatve and ntegral s permtted for all θ nt ( Θ ) Then ( ˆ d ) (, θ θ A ) ( ˆ Avar θ ) A and / 8

29 Mcroeconometrcs Markus Frölch Under the assumptons n Wooldrdge (p. 395, Theorem 3.): (3.8) d ˆ ( θ θo) (, A o ) and therefore (3.9) ˆ θ = A Avar( ) o / In order to estmate the asymptotc varance Avar( ˆ θ ) we need an estmate of A o. 9

30 Mcroeconometrcs Markus Frölch Propertes of the ML estmator.the ML-estmator s consstent: plm ˆ θ ML = θ..the ML estmator s asymptotcally effcent and reaches the Cramér-Rao bound. Ths s gven by the Fsher nformaton matrx A o : o [ ( θ )] E[ ( θ ) ( θ ) ] A H s s = E = 3. The ML estmator s asymptotcally normally dstrbuted: d ˆ ( θ θo) (, o ) A. 3

31 Mcroeconometrcs Markus Frölch.5. Estmatng the Asymptotc Varance It can be shown that each of the three matrces (3.3) H s s A x = = = ( ˆ θ ); ( ˆ θ) ( ˆ θ) ; (, ˆ θ) converges to. o A (, θ) E ( y,, θ) A x H x x Therefore, Avar( ˆ θ ) can be estmated by ˆ ˆ ˆ () ; ()() ; ( ˆ H θ s θ s θ Ax θ) = = = 3

32 Mcroeconometrcs Markus Frölch Asymptotcally these three estmators are equvalent. Frst estmator s based on Hessan of log lkelhood functon and requres computaton of second dervatves. It s not generally postve semdefnte. Second estmator s based on the outer product of the gradents (opg). easest to compute but can be severely based n small samples. The thrd estmator s based on expectaton of Hessan, whch often cannot be determned analytcally. If there s an analytcal soluton ths estmator has best small sample propertes. 3

33 Mcroeconometrcs Markus Frölch Example : Lnear regresson model contnued The matrx of second dervatves of the lkelhood functon s: ln L ln L XX Xu β β β σ 4 σ σ = ln L ln L ux uu ( ) σ β σ σ σ σ If we take the expectaton of ths matrx the off-dagonal elements turn out to be zero, because E(u) = and cov( ux, ) =. 33

34 Mcroeconometrcs Markus Frölch The asymptotc varance of the ML-estmator s therefore: A ( βσ, ) ln L ln L β β β σ ( ) E σ XX = ln L ln L = 4 σ ( ) σ β σ The lnear regresson model s one of the cases where the expectaton of the Hessan has an analytcal soluton. 34

35 Mcroeconometrcs Markus Frölch Example : Probt contnued The Hessan for Probt s complated But the expected Hessan [ H ( ) ] [ θ ] E θo x can be expressed ncely: { φ( xθo) } x x xθ [ xθ ] E H( o) x = A( θo) = Φ ( ) Φ ( ) o o (3.33) Avar( ˆ ˆ θ ) { } φ ( x ˆ ) θ x x = ˆ ˆ = Φ( xθ) Φ( xθ) 35

36 Mcroeconometrcs Markus Frölch Hypothess Testng ull hypothess: H : c(θ) = q (ncludng nonlnear hypothess) where c(θ) s a functon of the parameters θ. ˆ θ s the unrestcted estmator, θ % s the restrcted estmator Lkelhood Rato (LR) Test (3.35) ( ˆ LR = l θ ) l( % θ) ~ χ = = Q wth Q nonredundant restrctons mposed nvarant to reparameterzatons of the model 36

37 Mcroeconometrcs Markus Frölch Wald Test (Unrestrcted model) ( ) ( ( ) ) ˆ W = c θ q V c ˆ θ q c( ˆ θ) q ~ χ ( ( ˆ) ) c( θ) ( ) ( ˆ) c θ V c θ q = V θ θˆ θˆ If c(θ) s a lnear functon Rθ: W = R q RV R R ( ˆ θ ) ( ˆ θ) ( ˆ θ q) where R s a J K Matrx. J requres the unrestrcted model to be estmated 37

38 Mcroeconometrcs Markus Frölch Lagrange Multpler (Score) Test Maxmse lkelhood functon under constrant c(θ) = q. The correspondng Lagrange functon s: [ ] l ( θ, λ) = l( θ) + λ c( θ) q, and the frst order condtons are: l ( θ, λ) l( θ) c( θ) = + λ ' θ θ θ l ( θ, λ) λ ( θ ) = c q= where λ s the correspondng vector of the Lagrange multplers. 38

39 Mcroeconometrcs Markus Frölch Under H : c(θ) = q. Formally ths mples that H l( θ ) : = θ % θ score functon at estmates of restrcted model Langrange Multpler (LM)Test l( θ) l( θ) LM = A( % θ ) ( θ) A( θ) ( θ ), θ θ θ = s % % s % % % θ = = 39

40 Mcroeconometrcs Markus Frölch For the LM tests, the 3 followng statstcs dstrbuted as χ Q under H o ' ( ) ( ) ( ) s % θ H % θ s % θ = = = ' ( ) ( ) ( ) s % θ A % θ s % θ = = = ' ' ( ) ( ) ( ) ( ) s % θ s % θ s % θ s % θ = = = 4

41 Mcroeconometrcs Markus Frölch dlnl( Θ)/ dθ lnl U Lkelhood rato lnl(θ) lnl r Restrkton c(θ) Lagrange multpler Wald Θ R Θ ML Θ 4

42 Mcroeconometrcs Markus Frölch.7 Specfcaton Testng (usng the full dstrbutonal assumpton) The dstrbutonal assumpton mples that the score s mean zero. There may be addtonal mplcatons that can be useful :, = wth g(w, θ) a Qx vector some mstakes n Wooldrdge regardng transpose Test: H E g( w θ) g ( w ) s unrestrcted but E ( θ ) H : E, θ a s x = s mantaned ths test s based on how far the sample average of ( ˆ, θ ) To derve the asymptotc dstrbuton: ( ˆ g θ) = g ( ˆ θ) s ( ˆ) Π snce s ( θ ) = = = / /. θ ' { θ θ } { E ( θ ) ( θ ) } ' wth E ( ) ( ) Π s s s g g w s from. ˆ = whch s PxQ matrx of populaton regresson from regressng g ( ) ' θ on ( ) ' θ s. 4

43 Mcroeconometrcs Markus Frölch. Usng a mean-value expanson about θ, we can wrte: ( ˆ θ) ( ˆ θ) ( θ ) ( θ ) / / g s Π g s Π = = = ( θ ) s ( θ ) Π ( ˆ θ θ ) () + E θg + o θ p when the densty s correctly specfed, the nd term on the rght-hand sde s. { } ' Ths latter result comes from the UIME mplyng that θs ( θ ) E s ( θ ) s ( θ ) = E θ θ E θ θ g x = g s x IP and from an extenson of the CIME mplyng that ( ) ( ) ( ) ' To see the latter, start from g( θ ) x and thus E ( θ ) = E θ θ θ g x and... As a consequence ( ˆ) ( ˆ θ θ) = ( θ) ( θ) + ( ) / g / s Π g s Π p = = o 43

44 Mcroeconometrcs Markus Frölch 3. Lettng ˆ ' ' ss ˆˆ ˆˆ sg = = Π under standard regularty condtons, plm Πˆ Π / Therefore, the asymptotc varance of ( ˆ θ ) = ( ˆ ˆ ˆ )( ˆ ˆ ˆ ) ' g s Π g s Π g s consstently estmated by = 4. ewey-tauchen-whte (TW) statstc ' ' ( ) ( )( ) g ˆ ˆ ˆ ˆ ˆ ˆ ˆ ( ˆ g sπ g sπ g ) Q = = = TW = θ θ χ note that ths statstc dentcal to SSR = R from the regresson on =,,..., sg ˆˆ ' 44

45 Mcroeconometrcs Markus Frölch To summarse: ML requres to get condtonal densty of y x rght (or alternatvely of u x) If ths s accomplshed estmaton s straghtforward - ether the log lkelhood functon s already mplemented n the software (e.g. Probt, Logt, Tobt, ) - of you have to specfy the log-lkelhood functon and feed t nto the ML algorthms provded by STATA (ml command) After estmaton you should test your specfcaton (f possble) 45

46 Mcroeconometrcs Markus Frölch.8 Partal Lkelhood Methods for Panel Data and Cluster Samples Relax the assumpton that full condtonal densty of y gven x s correctly specfed y s a T vector wth generc element yt y s a T random draw vector from the cross secton th y s the t element of t y ( ) D y x s the dstrbuton of and we assume to know the t t correctly specfed densty f( yt xt; θ ) wth x t a vector of observables y t 46

47 Mcroeconometrcs Markus Frölch Unlke ML, here we do not assume: Dy [ t xt ] to be a condtonal dstrbuton of y gven x T t= Ths means even though f( y x ; θ ) s correct densty for gven t t x for each t, ther product s not necessarly the densty of y. t y t Partal Lkelhood for each observaton: T l ( θ ) log f ( y x t; θ ) (3.44) t t t= the sum of log lkelhoods across t 47

48 Mcroeconometrcs Markus Frölch t By KLI θ maxmzes E[log ft( yt xt; θ )] over Θ for each so also maxmzes the sum of these over t. θ Identfcaton requres θ to be the unque maxmzer of the expected value of (3.44). The partal Maxmum lkelhood estmator (PMLE ) ˆ θ solves: max T log ft( yt x t; θ ) (3.45) θ Θ = t= and s generally consstent provded θ s dentfed and asymptotcal normal (Theorem.3) 48

49 Mcroeconometrcs Markus Frölch But the condtonal Informaton Matrx cannot be appled unless: T p ( y z) = f ( y x ; (3.46) o t= t t t (3.46) does not necessarly hold n panel settng: Suppose x t contans only contemporaneous condtonng varables z t Then: p ( y z... z ) = p ( y z) p ( y y, z)... p ( y y, y,..., y, z)... o o o o T t t t t o p T( yt yt, yt,... y, z... zt) for (3.46) to hold we should have strct exogenty n z t and absence o of dynamcs n p t : o pt ( yt yt, yt,..., y, z) = ft( yt zt; θ ), t =,..., T o 49

50 Mcroeconometrcs Markus Frölch (3.46) s more lkely to hold f x t contans lagged values of y t. If x t contans only lagged values of y t then: T po( y) = ft( yt x t; θo) t= o holds f ft( yt xt; θo) = pt ( yt yt, yt,..., y) for all t so that all dynamcs are captured by. f t If x t contans varables z t n addton to the lagged y t equaton (3.46) requres that all lags of yt and z t are accounted for n f( yt xt, θ o) AD strct exogenty of z t. 5