REVIEW OF MAXIMUM LIKELIHOOD ESTIMATION

Transcription

1 REVIEW OF MAXIMUM LIKELIHOOD ESIMAION [] Maximum Likelihood Esimaor () Cases in which θ (unknown parameer) is scalar Noaional Clarificaion: From now on, we denoe he rue alue of θ as θ o hen, iew θ as a ariable Definiion: (Likelihood funcion) Le {x,, x } be a sample from a populaion I does no hae o be a random sample x is a scalar Le f(x,x,, x,θ o ) be he join densiy funcion of x,, x he funcional form of f is known, bu no θ o hen, L (θ) f(x,, x, θ) is called likelihood funcion L (θ) is a funcion of θ gien x,, x he funcional form of f is known, bu no θ o Definiion: (log-likelihood funcion) l (θ) = ln[f(x,, x,θ)] MLE-

2 Example: {x,, x }: a random sample from a populaion disribued wih f(x,θ o ) f(x,, x, θ o ) = f (, θ ) = L (θ) = f(x,, x, θ) = = = l (θ) = ( f ( x, )) x o f ( x, θ ) ln θ = Σ ln f (, θ ) x Definiion: (Maximum Likelihood Esimaor (MLE)) MLE θˆ MLE maximizes l (θ) gien daa poins x,, x Example: {x,, x } is a random sample from a populaion following a Poisson disribuion [ie, f(x,θ) = e -θ θ x /x! (suppressing subscrip o from θ)] Noe ha E(x) = ar(x) = θ o for Poisson disribuion l (θ) = Σ ln[f(x,θ)] = -θ + (ln(θ))σ x - Σ x! FOC of max: / θ = + Σx = 0 θ Soling his, θˆ MLE = Σ x = x MLE-

3 () Exension o he Cases wih Muliple Parameers Definiion: θ = [θ,θ,, θ p ] L (θ) = f(x,, x,θ) = f(x,, x, θ,, θ p ) l (θ) = ln[f(x,, x,θ) = ln[f(x,, x, θ,, θ p )] x could be a ecor If {x,, x } is a random sample from a populaion wih f(x,θ o ), = l (θ) = ( f ( x, )) ln θ = Σ ln f (, θ ) x Definiion: (MLE) MLE θˆ MLE maximizes l (θ) gien daa (ecor) poins x,, x ha is, θˆ MLE soles ( θ ) θ = ( θ ) / θ ( θ ) / θ = : ( θ ) / θ p 0 0 : 0 p Example: Le {x,, x } be a random sample from N(μ,σ ) [suppressing subscrip o ] Since {x,, x } is a random sample, E(x ) = μ o and ar(x ) = σ o Le θ = (μ,), where = σ MLE-3

4 ( x μ) f( x, θ ) = exp π / / ( x μ) = ( π ) ( ) exp ( x μ) ln[ f( x, θ)] = ln( π) ln( ) Σ( x μ) ( θ) = ln( π) ln( ) MLE soles FOC: ( θ ) Σ( x μ) () = Σ ( x μ)( ) = = 0; μ () From (): ( θ) Σ( x μ) = + = 0 Σ x (3) Σ( x μ) = 0 Σ x - μ = 0 μˆ MLE = = x Subsiuing (3) in o (): (4) - + Σ (x -μˆ MLE ) = 0 hus, ˆ MLE = Σ ( x x) ˆ ˆ μ x MLE θ = = MLE ˆ Σ( x x) MLE MLE-4

5 (3) Exension o Condiional densiy Definiion: Condiional densiy of y : f( y θ, x ), θ = [θ,θ,, θ p ] L θ = f y ( ) =Π ( θ, x ) o l (θ) = L( θ) =Σ = ln( f( y θ, x )) Example: Assume ha ( y, x i ) iid and f( y x )~ N( x β, ) f(y x,β,) = i i o f( y,, x ) exp ( y x ) π l ( β, ) =Σ ln f( y β,, x ) i β = β i i = ln( π) ln Σ( y x β) = ln( π ) ln ( y Xβ) ( y Xβ) herefore, we hae he following likelihood funcion of y FOC: (i) l (β,)/ β = -(/)[-X y + X Xβ] = 0 k (ii) l (β,)/ = -(/) + (/ )(y-xβ) (y-xβ) = 0 From (i), X y - X Xβ = 0 k From (ii), ˆ MLE = SSE/ βˆ MLE = (X X) - X y = βˆ hus, we can conclude ha ˆ β and s = SSE/(-k) are asympoically efficien MLE-5

6 [] Large Sample Properies of he ML esimaor Definiion: ) Le g(θ) = g(θ,, θ p ) be a scalar funcion of θ Le g j = g/ θ j hen, g g g = θ : g p ) Le w(θ) =(w (θ),, w m (θ)) be a m ecor of funcions of θ Le w ij = w i (θ)/ θ j hen, w w w p w w w = θ : : : wm wm w mp w( θ ) p 3) Le g(θ) be a scalar funcion of θ where g ij = g(θ)/ θ i θ j hen, m p g g g p g g g = θθ : : : gp gp g pp g( θ ) p Called Hessian marix of g(θ) p p MLE-6

7 Example : Le g(θ) = θ + θ + θ θ Find g(θ)/ θ g( θ ) = θ θ + θ θ θ + Example : Le θ + θ w( θ ) = θ+ θ w( θ ) θ θ = θ Example 3: Le g(θ) = θ + θ + θ θ Find he Hessian marix of g(θ) g( θ ) = θθ Some useful resuls: ) c : p, θ: p (c θ is a scalar) (c θ)/ θ = c ; (c θ)/ θ = c ) R: m p, θ: p (Rθ is m ) (Rθ)/ θ = R 3) A: p p symmeric, θ: p (θ Aθ) (θ Aθ)/ θ = Aθ (θ'aθ)/ θ = θ'a (θ Aθ)/ θ θ = A MLE-7

8 Definiion: (Hessian marix of log-likelihood funcion) H l l ( θ ) = = θθ θi θ j p p heorem: Le ˆ θ be MLE hen, under suiable regulariy condiions, ˆ θ is consisen, and, ˆ ( θ θo) d N0 p, plim H( θo) Furher, ˆ θ is asympoically efficien Implicaion: ˆ θ N(θ o, [-H (θ o )] - ) ˆ θ N(θ o, [-H ( ˆ θ )] - ) heorem: Le s( θ ) = ln f( y θ, x )/ θ Define B =Σ s ( θ ) s ( θ ) hen, = plim B( θo) = plim H( θ o ) Implicaion: ˆ θ N(θ o, [-H ( ˆ θ )] - ) N(θ o, (B ( ˆ θ )) - ) MLE-8

9 Example: {x,, x } is a random sample from N(μ o,σ o ) Le θ = [μ,] and =σ l = Σ ln( π ) ln( ) ( x μ) he firs deriaies: l( θ ) Σ( x μ) l( θ) = ; = + Σ ( x μ μ ) he second deriaies: l ( θ ) = Σ( ) = μμ ; l( θ ) Σ( x μ) = ; μ l ( θ ) 0 4 ( ) = + Σ ( ) ( ) x μ = Σ 3 x μ herefore, MLE-9

10 3 ( ) ( ) ( ) ( ) x H x x μ ν θ μ μ ν ν Σ = Σ Σ + 0 ˆ ˆ ( ) 0 ˆ ML ML ML H θ = Hence, ˆ 0 ˆ ˆ, ˆ ˆ 0 ML o ML o ML ML N μ μ θ = MLE-0

11 [3] esing Hypoheses Based on MLE General form of hypoheses: Le w(θ) = [w (θ),w (θ),, w m (θ)], where w j (θ) = w j (θ, θ,, θ p ) = a funcion of θ,, θ p H o : he rue θ (θ o ) saisfies he m resrciions, w(θ) = 0 m (m p) Definiion: (Resriced MLE) Le θ be he resriced ML esimaor which maximizes l (θ) s w(θ) = 0 Wald es: W w ˆ W ˆ Co ˆ W ˆ w ˆ = ( θ )'[ ( θ) ( θ) ( θ)] ( θ ) If ˆ θ is a (unresriced) ML esimaor, W = w ˆ W ˆ H ˆ W ˆ w ˆ θ ( θ )[ ( θ){ ( θ)} ( θ)] ( ) Noe: Can be compued wih any consisen esimaor ˆ θ and Co( ˆ θ ) Likelihood Raio es: (LR) LR = [l ( ˆ θ ) - l (θ )] Lagrangean Muliplier (LM) es Define s l ( θ ) ( θ ) = hen, LM = s (θ ) [-H (θ )] - s (θ ) θ MLE-

12 heorem: Under H o : w(θ) = 0, W, LR, LM d χ (m) Implicaion: Gien significance leel (α), find a criical alue from χ able Usually, α = 005 or α = 00 If W > c, rejec H o Oherwise, do no rejec H o Commens: ) Wald needs only ˆ θ ; LR needs boh ˆ θ and θ ; and LM needs θ only ) In general, W LR LM 3) W is no inarian o how o wrie resricions ha is, W for H o : θ = θ may no be equal o W for H o : θ /θ = Example: () {x,, x }: RS from N(μ o, o ) wih o known So, θ = μ H o : μ = 0 w(μ) = μ l (μ) = -(/)ln(π) - (/)ln( o ) - {/( o )}Σ (x -μ) s (μ) = (/ o )Σ (x -μ) H ( μ) = o MLE-

13 [Wald es] Unresriced MLE: FOC: l (μ)/ μ = (/)Σ (x -μ) = 0 ˆ μ = x W(μ) = W( ˆμ ) = -H ( ˆμ ) = / o [LR es] Resriced MLE: μ = 0 l ( ˆμ ) = -(/)ln(π) - (/)ln( o ) - {/( o )}Σ (x - x ) l (μ ) = -(/)ln(π) - (/)ln( o )- {/( o )}Σ x [LM es] s (μ ) = (/ o )Σ x = (/ o ) x ; -H (μ ) = / o Wih his informaion, can show ha W = LR = LM = x o () Boh μ and unknown: θ = (μ,) H o : μ = 0 w(θ) = μ W(θ) = w(θ)/ θ = [ μ/ μ, μ/ ] = [, 0] l (θ) = -(/)ln(π) - (/)ln() - {/()}Σ (x -μ) MLE-3

14 Σ( x μ) s (θ) = + Σ ( ) x μ ; Σ( x μ) ν H ( θ ) = Σ( x μ) Σ( x μ) + 3 ν ν Unresriced MLE: ˆ μ = x and ˆ ( ) = Σ x x Resriced MLE: μ = 0, bu need o compue l ( μ,) = -(/)ln(π) - (/)ln() - {/()}Σ (x -μ ) l (0,) = -(/)ln(π) - (/)ln() - {/()}Σ x FOC: l (0,)/ = -/() + (/( ))/Σ x = 0 = (/)Σx [Wald es] w( ˆ θ ) = ˆμ = x ; W( ˆ θ ) = ( 0 ); -H ( ˆ θ ) = ˆ 0 0 ˆ W = w( ˆ θ ) [W( ˆ θ ){-H ( ˆ θ )} - W( ˆ θ ) ] - w( ˆ θ ) = x ˆ MLE-4

15 [LR es] l ( ˆ θ ) = -(/)ln(π) - (/)ln( ˆ ) - {/( ˆ )}Σ(x - x ) l (θ ) = -(/)ln(π) - (/)ln( ) - {/( )}Σx [LM es] s x Σx x ( θ ) = = = ; + Σ 0 x + Σx ν H( θml) = Σx ν ν LM = x s ( θ)[ H( θ)] s( θ) = x MLE-5