6. MAXIMUM LIKELIHOOD ESTIMATION

Transcription

1 6 MAXIMUM LIKELIHOOD ESIMAION [1] Maximum Likelihood Estimator (1) Cases in which θ (unknown parameter) is scalar Notational Clarification: From now on, we denote the true value of θ as θ o hen, view θ as a variable Definition: (Likelihood function) Let {x 1,, x } be a sample from a population It does not have to be a random sample x t is a scalar Let f(x 1,x,, x,θ o ) be the joint density function of x 1,, x he functional form of f is known, but not θ o hen, L (θ) f(x 1,, x, θ) is called likelihood function L (θ) is a function of θ given x 1,, x he functional form of f is known, but not θ o Definition: (log-likelihood function) l (θ) = ln[f(x 1,, x,θ)] MLE-1

2 Example: {x 1,, x }: a random sample from a population distributed with f(x,θ o ) f(x 1,, x, θ o ) = f (, θ ) t = 1 L (θ) = f(x 1,, x, θ) = t = t = 1 l (θ) = ( f ( xt, )) x t o 1 f ( x, θ ) ln θ = Σ ln f (, θ ) t t x t Definition: (Maximum Likelihood Estimator (MLE)) MLE θˆ MLE maximizes l (θ) given data points x 1,, x Example: {x 1,, x } is a random sample from a population following a Poisson distribution [ie, f(x,θ) = e -θ θ x /x! (suppressing subscript o from θ)] Note that E(x) = var(x) = θ o for Poisson distribution l (θ) = Σ t ln[f(x t,θ)] = -θ + (ln(θ))σ t x t - Σ t x t! 1 FOC of max: / θ = + Σtxt = 0 θ Solving this, θˆ MLE = Σ t x t = x MLE-

3 () Extension to the Cases with Multiple Parameters Definition: θ = [θ 1,θ,, θ p ] L (θ) = f(x 1,, x,θ) = f(x 1,, x, θ 1,, θ p ) l (θ) = ln[f(x 1,, x,θ) = ln[f(x 1,, x, θ 1,, θ p )] x t could be a vector If {x 1,, x } is a random sample from a population with f(x,θ o ), t = 1 l (θ) = ( f ( xt, )) ln θ = Σ ln f (, θ ) t x t Definition: (MLE) MLE θˆ MLE maximizes l (θ) given data (vector) points x 1,, x hat is, θˆ MLE solves ( θ ) θ = ( θ ) / θ1 ( θ ) / θ = : ( θ ) / θ p 0 0 : 0 p 1 Example: Let {x 1,, x } be a random sample from N(μ,σ ) [suppressing subscript o ] Since {x 1,, x } is a random sample, E(x t ) = μ o and var(x t ) = σ o Let θ = (μ,v), where v = σ MLE-3

4 1 ( xt μ) f( xt, θ ) = exp v π v 1/ 1/ ( xt μ) = ( π ) ( v) exp v 1 1 ( xt μ) ln[ f( xt, θ)] = ln( π) ln( v) v Σt( xt μ) ( θ) = ln( π) ln( v) v MLE solves FOC: ( θ ) 1 Σt( xt μ) (1) = Σt ( xt μ)( 1) = = 0; μ v v () From (1): ( θ) Σt( xt μ) = + = 0 v v v Σ x (3) Σt( x t μ) = 0 Σ t x t - μ = 0 μˆ t t MLE = = x Substituting (3) in to (): (4) -v + Σ t (x t -μˆ MLE ) = 0 hus, vˆ 1 MLE = Σt ( xt x) ˆ ˆ μ x MLE θ = = 1 MLE vˆ Σt( xt x) MLE MLE-4

5 [] Large Sample Properties of the ML estimator Definition: 1) Let g(θ) = g(θ 1,, θ p ) be a scalar function of θ Let g j = g/ θ j hen, g1 g g = θ : g p ) Let w(θ) =(w 1 (θ),, w m (θ)) be a m 1 vector of functions of θ Let w ij = w i (θ)/ θ j hen, w11 w1 w1 p w w w = θ : : : wm 1 wm w mp w( θ ) 1 p 3) Let g(θ) be a scalar function of θ where g ij = g(θ)/ θ i θ j hen, m p g11 g1 g1 p g g g = θθ : : : gp1 gp g pp g( θ ) 1 p Called Hessian matrix of g(θ) p p MLE-5

6 Example 1: Let g(θ) = θ 1 + θ + θ 1 θ Find g(θ)/ θ g( θ ) = θ θ + θ 1 θ θ + 1 Example : Let θ1 + θ w( θ ) = θ1+ θ w( θ ) θ 1 θ 1 = 1 θ Example 3: Let g(θ) = θ 1 + θ + θ 1 θ Find the Hessian matrix of g(θ) g( θ ) 1 = θθ 1 Some useful results: 1) c : 1 p, θ: p 1 (c θ is a scalar) (c θ)/ θ = c ; (c θ)/ θ = c ) R: m p, θ: p 1 (Rθ is m 1) (Rθ)/ θ = R 3) A: p p symmetric, θ: p 1 (θ Aθ) (θ Aθ)/ θ = Aθ (θ'aθ)/ θ = θ'a (θ Aθ)/ θ θ = A MLE-6

7 Definition: (Hessian matrix of log-likelihood function) H l l ( θ ) = = θθ θi θ j p p heorem: Let ˆ θ be MLE hen, under suitable regularity conditions, ˆ θ is consistent, and, ˆ 1 ( θ θo) d N0 p 1, plim H( θo) Further, ˆ θ is asymptotically efficient 1 Implication: ˆ θ N(θ o, [-H (θ o )] -1 ) ˆ θ N(θ o, [-H ( ˆ θ )] -1 ) Example: {x 1,, x } is a random sample from N(μ o,σ o ) Let θ = [μ,v] and v =σ l 1 1 = Σ v ln( π ) ln( v) t( x t μ) he first derivatives: l( θ ) Σt( xt μ) l( θ) 1 = ; = + Σ ( t x μ v v v v t μ ) MLE-7

8 he second derivatives: l ( θ ) 1 = Σt( 1) = μμ v v ; l( θ ) Σt( xt μ) = ; μ v v l ( θ ) 0 v 1 4v 1 vv v ( v) v v = + Σ ( ) ( ) t xt μ = Σ 3 t xt μ herefore, Hence, Σt( xt μ) v ν H ( θ ) = Σt( xt μ) Σt( xt μ) + 3 ν ν v 0 vˆ ( ˆ ML H θml) = 0 vˆ ML vˆ ML 0 ˆ ˆ μml μo θ = N, v ˆ ML v o vˆ ML 0 MLE-8

9 [3] esting Hypotheses Based on MLE General form of hypotheses: Let w(θ) = [w 1 (θ),w (θ),, w m (θ)], where w j (θ) = w j (θ 1, θ,, θ p ) = a function of θ 1,, θ p H o : he true θ (θ o ) satisfies the m restrcitions, w(θ) = 0 m 1 (m p) Definition: (Restricted MLE) Let θ be the restricted ML estimator which maximizes l (θ) st w(θ) = 0 Wald est: W w ˆ W ˆ Cov ˆ W ˆ w ˆ 1 = ( θ )'[ ( θ) ( θ) ( θ)] ( θ ) If ˆ θ is a (unrestricted) ML estimator, W = w ˆ W ˆ H ˆ W ˆ w ˆ θ 1 1 ( θ )[ ( θ){ ( θ)} ( θ)] ( ) Note: Can be computed with any consistent estimator ˆ θ and Cov( ˆ θ ) Likelihood Ratio est: (LR) LR = [l ( ˆ θ ) - l (θ )] Lagrangean Multiplier (LM) test Define s l ( θ ) ( θ ) = hen, LM = s (θ ) [-H (θ )] -1 s (θ ) θ MLE-9

10 heorem: Under H o : w(θ) = 0, W, LR, LM d χ (m) Implication: Given significance level (α), find a critical value from χ table Usually, α = 005 or α = 001 If W > c, reject H o Otherwise, do not reject H o Comments: 1) Wald needs only ˆ θ ; LR needs both ˆ θ and θ ; and LM needs θ only ) In general, W LR LM 3) W is not invariant to how to write restrictions hat is, W for H o : θ 1 = θ may not be equal to W for H o : θ 1 /θ = 1 Example: (1) {x 1,, x }: RS from N(μ o,v o ) with v o known So, θ = μ H o : μ = 0 w(μ) = μ l (μ) = -(/)ln(π) - (/)ln(v o ) - {1/(v o )}Σ t (x t -μ) s (μ) = (1/v o )Σ t (x t -μ) H ( μ) = v o MLE-10

11 [Wald est] Unrestricted MLE: FOC: l (μ)/ μ = (1/v)Σ t (x t -μ) = 0 ˆ μ = x W(μ) = 1 W( ˆμ ) = 1 -H ( ˆμ ) = /v o [LR est] Restricted MLE: μ = 0 l ( ˆμ ) = -(/)ln(π) - (/)ln(v o ) - {1/(v o )}Σ t (x t - x ) l (μ ) = -(/)ln(π) - (/)ln(v o )- {1/(v o )}Σ t x t [LM est] s (μ ) = (1/v o )Σ t x t = (/v o ) x ; -H (μ ) = /v o With this information, can show that W = LR = LM = x v o () Both μ and v unknown: θ = (μ,v) H o : μ = 0 w(θ) = μ W(θ) = w(θ)/ θ = [ μ/ μ, μ/ v] = [1, 0] l (θ) = -(/)ln(π) - (/)ln(v) - {1/(v)}Σ t (x t -μ) MLE-11

12 1 Σt( xt μ) v s (θ) = 1 + Σ ( ) t xt μ v v ; Σt( xt μ) v ν H ( θ ) = Σt( xt μ) Σt( xt μ) + 3 ν ν v Unrestricted MLE: ˆ μ = x and 1 vˆ ( ) = Σt xt x Restricted MLE: μ = 0, but need to compute v l ( μ,v) = -(/)ln(π) - (/)ln(v) - {1/(v)}Σ t (x t -μ ) l (0,v) = -(/)ln(π) - (/)ln(v) - {1/(v)}Σ t x t FOC: l (0,v)/ v = -/(v) + (1/(v ))/Σ t x t = 0 v = (1/)Σtx t [Wald est] w( ˆ θ ) = ˆμ = x ; W( ˆ θ ) = ( 1 0 ); -H ( ˆ θ ) = vˆ 0 0 vˆ W = w( ˆ θ ) [W( ˆ θ ){-H ( ˆ θ )} -1 W( ˆ θ ) ] -1 w( ˆ θ ) = x v ˆ MLE-1

13 [LR est] l ( ˆ θ ) = -(/)ln(π) - (/)ln( ˆv ) - {1/( ˆv )}Σt(x t - x ) l (θ ) = -(/)ln(π) - (/)ln(v ) - {1/( v )}Σtx t [LM est] s 1 x Σtx x t v v ( θ ) = = = v ; 1 + Σ 0 tx t + v v v v Σtxt v ν H( θml) = Σtxt ν ν LM = 1 x s ( θ)[ H( θ)] s( θ) = v x MLE-13

14 [4] Efficiency of OLS estimator under Ideal Conditions Assume that y t is iid N(x t β,v) conditional on x t f(y t x t,β,v) = 1 1 exp ( yt x ) π v v l ( β, v) =Σ ln f( y β, v, x ) t t ti ti β 1 = ln( π) ln v Σ( y x β) v 1 = ln( π ) ln v ( y Xβ) ( y Xβ) v t t ti herefore, we have the following likelihood function of y FOC: (i) l (β,v)/ β = -(1/v)[-X y + X Xβ] = 0 k 1 (ii) l (β,v)/ v = -(/v) + (1/v )(y-xβ) (y-xβ) = 0 From (i), X y - X Xβ = 0 k 1 From (ii), vˆ MLE = SSE/ βˆ MLE = (X X) -1 X y = βˆ hus, we can conclude that ˆ β and s = SSE/(-k) are asymptotically efficient MLE-14