6 More about likelihood

Transcription

1 6 More about lkelhood 61 Invarance property of mle s ( θ) Theorem If θ s an mle of θ and f g s a functon, then g s an mle of g(θ) Proof If g s one-to-one, then L(θ) =L ( g 1 (g(θ)) ) are both maxmsed by θ, so θ = g 1(ĝ(θ)) or g( θ) = ĝ(θ) If g s many-to-one, then θ whch maxmses L(θ) stll corresponds to g ( θ), so g( θ) stll corresponds to the maxmum of L(θ) Example Suppose X 1,X 2,,X n s a random sample from a Bernoull dstrbuton B(1,θ) Consder mle s of the mean, θ, and varance, θ(1 θ) Note, by the way, that θ(1 θ) s not a 1-1 functon of θ The log-lkelhood s and l(θ) = x log θ +(n x ) log(1 θ) dl(θ) dθ = x / θ (n x )/ (1 θ) so t s easly shown that the mle of θ s θ = X Puttng ν = θ(1 θ), dl(ν) dl (ν(θ)) = dθ dν dθ dν so t s easly seen that, snce dθ s not, n general, equal to zero, dν ν = ν( θ) = X ( 1 X ) 34

2 62 Relatve lkelhood If sup θ L(θ) <, therelatve lkelhood s RL(θ) = L(θ) sup L(θ) θ ; 0 RL(θ) 1 Relatve lkelhood s nvarant to known 1-1 transformatons of x, for f y s a 1-1 functon of x, f Y (y; θ) =f X (x(y); θ) dx dy dx s ndependent of θ, so RL X (θ) =RL Y (θ) dy 63 Lkelhood summares Realstc statstcal problems often have many parameters These cause problems because t can be hard to vsualse L(θ), and t becomes necessary to use summares Key dea In large samples, log-lkelhoods are often approxmately quadratc near the maxmum 35

3 Example Suppose X 1,X 2,,X n s a random sample from an exponental dstrbuton wth parameter λ e f X (x) =λe λx, x 0 Then l(λ) = n log λ x dl(λ) λ, = n/ λ dλ x, d 2 l(λ) dλ 2 = n/ λ 2, d 3 l(λ) dλ 3 = 2n/ λ 3 The log-lkelhood has a maxmum at λ = n / x,so (λ RL(λ) = λ)n e n λ x = (λ λ e1 λ/ λ)n, λ > 0 1 as λ λ Now, what happens as, for λ fxed, n? log RL(λ) = l(λ) l( λ) = l( λ) +l ( λ) (λ λ) l ( λ) (λ 1 λ)2 l( λ) λ λ usng Taylor seres, where λ 1 < λ Now l ( λ) =0and l ( λ) = n / λ2, so n(λ 1 λ)2 log RL(λ) = as n λ 2 unless λ = λ Thus, as n, RL(λ) { 1, λ = λ, 0, otherwse Concluson Lkelhood becomes more concentrated about the maxmum as n,and values far from the maxmum become less and less plausble 36

4 In general We call the value θ whch maxmses L(θ) or, equvalently, l(θ) = log L(θ) the maxmum lkelhood estmate, and J(θ) = 2 l(θ) θ 2 s called the observed nformaton Usually J(θ) > 0 and J( θ) measures the concentraton of l(θ) at θ Close to θ, we summarse 1 l(θ) l( θ) θ)2 J( θ) 2(θ 64 Informaton In a model wth log-lkelhood l(θ), the observed nformaton s J(θ) = 2 l(θ) θ 2 When observatons are ndependent, L(θ) s a product of denstes so l(θ) = log f (x ; θ) and J(θ) = 2 θ 2 log f (x ; θ) Snce 1 l(θ) l( θ) θ)2 J( θ), 2(θ for θ near to θ, we see that large J( θ) mples that l(θ) s more concentrated about θ Ths means that the data are less ambguous about possble values of θ, e we have more nformaton about θ 37

5 65 Expected nformaton 651 Unvarate dstrbutons Before an experment s conducted, we have no data so that we cannot evaluate J(θ) But we can fnd ts expected value ) I(θ) =E ( 2 l(θ) θ 2 Ths s called the expected nformaton or Fsher s nformaton If the observatons are a random sample, then the whole sample expected nformaton s I(θ) =n(θ) ) where (θ) =E ( 2 θ log f (X ; θ), 2 the sngle observaton Fsher nformaton Example Suppose X 1,X 2,,X n s a random sample n from a Posson dstrbuton wth parameter θ θ x e θ L(θ) =, x! = =1 gvng l(θ) = log L(θ) x log θ nθ log x! Thus J(θ) = 2 l(θ) = θ 2 To fnd I(θ), weneede (X )=θ and I(θ) = 1 θ 2 x / θ 2 E (X )= n θ 38

6 652 Multvarate dstrbutons If θ s a (p 1) vector of parameters, then I(θ) and J(θ) are (p ) p) matrces ( 2 l(θ) {J(θ)} rs = 2 l(θ) and {I(θ)} θ r θ rs = E s These matrces are obvously symmetrc We can also wrte the above as J(θ) = 2 l(θ) θ θ T and I(θ) =E ( 2 l(θ) θ θ T θ r θ s ) Example X 1,X 2,,X n s a random sample from a normal dstrbuton wth parameters µ and σ 2 We have already seen that so and ( 2) ( L µ, σ = 2πσ 2) n/2 exp [ 1 2σ 2 l ( µ, σ 2) = n 2 log 2π n 2 log σ2 1 2σ 2 l µ = 1 σ 2 (x µ), l σ 2 = n 2σ (x 2σ 4 µ) 2, 2 l µ 2 = n, σ 2 2 l µ σ 2 = 1 (x σ 4 µ) (x µ) 2], (x µ) 2 To fnd I(µ,σ 2 ),use 2 l (σ 2 ) 2 = n 1 (x 2σ 4 σ 6 µ) 2 J(µ, σ 2 )=( n σ σ 4 (x µ) (x σ 4 µ) 1 (x σ 6 µ) 2 n 2σ 4 ) so that E (X ) = µ, V (X ) = E [ (X µ) 2] = σ2, I(µ, σ 2 )=E ( J(µ, σ 2 ) ) =( n 39 σ 2 0 n 0 2σ 4 )

7 Example Censored exponental data Lfetmes of n components, safety devces, etc are observed for a tme c, when r have faled and (n r) are stll OK We have two knds of observaton: 1 Exact falure tmes x observed f x c, sothat 2 x unobserved f x >c, f(x; λ) =λe λx, x 0; P (X >c)=e λc }{{} Data are therefore x 1,,x r,c,,c n r tmes The (n r) components, safety devces, etc whch have not faled are sad to be censored The lkelhood s L(λ) = r =1 = λ r exp n [ e λc =r+1 λ( r λe λx =1 x +(n r)c )] l(λ) = r log λ λ ( r =1 x +(n r)c) l (λ) = r/ λ ( r =1 x +(n r)c) l (λ) = r / λ 2 / Thus J(λ) =r λ 2 > 0 f r>0 so we must observe at least one exact falure tme observed exactly) I(λ) =E ( r / λ 2) = 1 λ 2 E (#X Now P (X observed exactly) =P (X c) =1 e λc,so I c (λ) = n( 1 e λc) λ 2 40

8 No censorng f c, gvng I (λ) = n λ 2 >I c(λ) as one mght expect The asymptotc effcency when there s censorng at c relatve to no censorng s I c (λ)/i (λ) =1 e λc Example Events n a Posson process Events are observed for perod (0,T) n events occur at tmes 0 <t 1 <t 2 <<t n <T Two observers A and B A records exact tmes, B uses an automatc counter and goes to the pub (e B merely records how many events there are) A knows exact tmes, and tmes between events are ndependent and exponentally dstrbuted, so L A (λ) = λe λt 1 λe λ(t 2 t1) λe λ(t n tn 1) λe (λt t n) = λ n e λt B merely observes the event [N = n], where N Po(λT ), so Log-lkelhoods are L (λt e B (λ) = )n λt l A (λ) = n log λ λt, l B (λ) = n log λ + n log T λt log n! n! and J A (λ) =J B (λ) =n / λ 2 E(N) =λt,soi A (λ) =I B (λ) =T /λ, and both observers get the same nformaton As usual, the one who went to the pub dd the rght thng 41

9 66 Maxmum lkelhood estmates The maxmum lkelhood estmate θ of θ maxmses L(θ) and often (but not always) satsfes the lkelhood equaton wth for a maxmum θ( θ) l =0, J( θ) = 2 l θ 2( θ) > 0 In the vector case, θ solves smultaneously wth l θ r ( θ) =0, det J( θ) > 0 r =1,,p, (e J( θ) postve defnte) If the lkelhood equaton has many solutons, we fnd them all and check L(θ) for each Usually, the equaton has to be solved numercally One way s by Newton- Raphson Suppose we have a startng value θ 0 Then 0= l θ whch may be re-arranged to ( θ) ) l θ (θ 0)+ 2 l θ (θ 0)( θ θ0 2 θ = θ0 + U (θ 0) J(θ 0 ), where U(θ) = l θ J(θ) = 2 l θ 2 s the score functon, s the observed nformaton 42

10 Now we terate usng θ 0 as a startng value and θ n+1 = θ n + U(θ n) J(θ n ) Example Extreme value (Gumbel) dstrbuton Ths dstrbuton s used to model such thngs as annual maxmum temperature Data due to Blss on numbers of beetles klled by exposure to carbon dsulphde are ftted by ths model The cdf s and the densty s F (x) =exp ( e (x η)), x R, η R, f (x) =exp [ (x η) e (x η)], x R, η R The sample log-lkelhood s l(η) = (x η) e (x η), so that U (η) = n e (x η), J(η) = Startng at η 0 = x, terate usng e (x η) η n+1 = η n + n e (x η n ) e (x η n ) 661 Fsher scorng Ths smply nvolves replacng J(θ) wth I(θ) Example Extreme value dstrbuton We need = [ η)] I(η) = E [J(η)] E e (X = n [ (x η)] e (x η) exp (x η) e dx 43

11 Put u = e (x η) and the ntegral becomes I(η) =n so Fsher scorng gves the teraton η n+1 = η n +1 1 n 0 ue u du = n, e (x η n ) 44

12 67 Suffcent statstcs You have already seen a lkelhood whch cannot be summarsed by a quadratc Example f (x ; θ) =θ 1, 0 <x <θ, so L(θ) =θ n, 0 < max {x } <θ Clearly a quadratc approxmaton s useless here Suppose there exsts a statstc s(x) such that L(θ) only depends upon data x through s(x) Then s(x) s a suffcent statstc for θ and obvously always exsts The mportant queston s: Does s(x) reduce the dmensonalty of the problem? Defnton If S = s(x) s such that the condtonal densty f X S(x s; θ) s ndependent of θ, thens s a suffcent statstc 45

13 Example Suppose X 1,X 2 B(n,θ) and consder P (X 1 = x X 1 + X 2 = r) P (X 1 = x, X 1 + X 2 = r) = P (X 1 + X 2 = r) P (X 1 = x, X 2 = r x) = P (X 1 + X 2 = r) = = x) n x( r x) (n θ r) x n (1 θ) θ r x (1 θ) n r+x x)( r x) (2n θ r (1 θ) 2n r r) (n n (2n Ths does not contan θ, so that X 1 + X 2 s a suffcent statstc for θ 46

14 Example X 1,X 2,,X n U(0,θ), sothat L(θ) =θ n, 0 <x 1,,x n <θ Suppose we fnd the condtonal densty of X 1,X 2,,X n gven X (n) The jont densty of X (1),X (2),,X (n) s f (x (1),x (2),,x (n) )= n! θ, 0 <x (1),,x n (n) <θ and the densty of X (n) s nx n 1 /θ n so that the condtonal densty of X (1),,X (n 1) X(n) = y s n! θ n / nx n 1 θ n = Thus the densty of X 1,X 2,,X n X (n) s ( n ) f x 1,,x x(n) = y = 1 x n 1, (n 1)! x n 1, 0 <x (1),,x (n 1) <y 0 <x 1,,x n <y, whch s free of θ, sothatx (n) s a suffcent statstc for θ Factorzaton Theorem s(x) s a suffcent statstc for θ f and only f there exst functons g and h such that f (x; θ) =g (s(x); θ) h(x) for all x R n,θ Θ Proof for dscrete random varables () Let s(x) =a and suppose the factorzaton condton to be satsfed, so that f(x; θ) =g (s(x); θ) h(x) Then P (s(x) =a) = p(y) =g(a; θ) h(y) y s 1 (a) y s 1 (a) Hence P (X = x s(x) =a) = and ths does not depend upon θ 47 h(x) h(y) y s 1 (a)

15 () Let s(x) be a suffcent statstc for θ Then P (X = x) =P (X = x s(x) =a) P (s(x) =a) But suffcency P (X = x s(x) =a) does not depend upon θ so, wrtng P (s(x) =a) =g(a; θ) and P (X = x s(x) =a) =h(x) gves the result The proof n the contnuous case requres measure theory and s beyond the scope of ths course Example Suppose X 1,X 2,,X n s a random sample from a Bernoull dstrbuton Then p(x; θ) =θ x (1 θ) n x Trvally ths factorzes wth s(x) = x and h(x) =1 Example Suppose X 1,X 2,,X n s a random sample from a N(µ, σ 2 ) dstrbuton, where (µ, σ 2 ) T s a vector of unknown parameters Then ( f (x; µ, σ 2 ) = 2πσ 2) n/2 exp [ 1 2σ 2 ( = 2πσ 2) n/2 exp [ 1 2σ 2 (x µ) 2 ] (x x) 2 + n(x µ) 2 Agan ths factorzes where s(x) =(x, (x x) 2 ) T, a vector valued functon ] 48

16 68 The exponental famly The densty functon/probablty mass functon has the form f(x; ϕ) = exp [a(x)b(ϕ) c(ϕ) +d(x)], where x may be contnuous or dscrete and ϕ s n a sutable space (usually open reals) For a random sample X 1,X [b(ϕ) 2,,X n, we obtan + ] L(ϕ) =exp a(x ) nc(ϕ) d(x ), and, therefore, by the factorzaton theorem, a(x ) s suffcent for ϕ Example Let X B(n, θ) Then ( ) n p X (x) = θ x (1 θ) x n x [ = exp [ = exp x log θ +(n x) log(1 θ) + log( n x )] x log (θ/(1 θ)) + n log(1 θ) + log( n x Callng Y = a(x), θ = b(ϕ) the natural parametersaton, we can wrte the densty functon/probablty mass functon n the form f (y; θ) =exp[yθ k(θ)] m(y) Y Clearly s a suffcent statstc Note that, n the contnuous case, the moment generatng functon s E ( e ty) = e ty+θy k(θ) m(y)dy = e k(θ+t) k(θ) = e k(θ+t) k(θ) e ty+θy k(θ+t) m(y)dy The functon k(θ) s called the cumulant generator Letusseewhy 49 )]

17 The cumulant generatng functon If X s a random varable wth moment generatng functon M(t), then K(t) = log M(t) s sad to be the cumulant generatng functon Dfferentatng, K (t) = M (t) M (t), K (0) = M (0) M (0) = E(X), K (t) = M (t) M (t) M (t) 2 M(t) 2, K (0) = M (0) M (0) 2 M (0) M(0) 2 and so on The cumulants are generated drectly For the exponental famly, = V (X) K(t) =logm (t) =k(θ + t) k(θ) so that K (t) =k (θ + t), K (0) = k (θ), and so on The cumulants are generated by repeated dfferentaton of k(θ) Example Posson dstrbuton The pmf s so that p(x; µ) = µx e µ x!, x =0, 1, = exp [x log µ µ log x!] a(x) = x, b(µ) = log µ, c(µ) = µ, d(x) = log x! Under natural parametersaton, y = x, θ =logµ, k(θ) =e θ, m(y) = 1 y! Cumulants are gven by dervatves of k(θ), all of whch are e θ = µ Example Bnomal dstrbuton 50

18 The pmf s p(x; p) = ( ) n p x (1 p) n x, x [ = exp Natural parametersaton s For the cumulants, and so on x log( ) p 1 p y = x, θ =log( p 1 p k (θ) = ) x =0, 1,,n ne θ 1+e θ = np, k (θ) = ne θ + n log(1 p) + log( n x k(θ) =n log(1 + e θ ) )] = np(1 p), (1 + e θ 2 ) 69 Large sample dstrbuton of θ From the data summary pont of vew, the mle θ and J ( θ)have been thought of n terms of a partcular set of data We now wsh to thnk of θ n terms of repeated samplng (e as a random varable) Man results In many stuatons and subject to regularty condtons θ D N(θ, I(θ) 1 ), and an approxmate 95% confdence nterval for θ s gven by θ ± 196I( θ) 1/2 [or θ ± 196J( θ) 1/2, regarded by many as better, but not n the books] In the multvarate case, θ D N(θ, I(θ) 1 ) 51

19 Example Exponental dstrbuton For an exponental dstrbuton wth mean θ, L(θ) = θ n e x /θ, θ > 0, l(θ) = n log θ x /θ, so that U (θ) = n x θ +, J(θ) = n θ 2 θ + 2 x 2 θ 3 Thus n θ = x, J( θ) = x 2 and an approxmate 95% confdence nterval s x ± 196x / n Example Normal dstrbuton For a normal random sample, Therefore µ = x, σ 2 = n 1 (x x) 2, J ( µ, σ 2) = ( n /σ 2 σ 4 (x µ) σ 4 (x µ) σ 6 (x µ) 2 n /2σ 4 I ( µ, σ 2) = ( n / σ 2 / 0 n 2 σ 4 An approxmate 95% confdence nterval for µ s 0 ) ) and for σ 2 s x ± 196 σ / n, σ 2 ± 196 σ 2 2 n Note that the estmators µ and σ 2 are asymptotcally uncorrelated The exact nterval for µ s x ± S n t 0975 (n 1) 52

20 whch s not qute the same Proof of asymptotc normalty Suppose X 1,X 2,,X n s a random sample from a dstrbuton wth pdf f(x; θ) Then the log-lkelhood, score and observed nformaton are = l(θ) = U(θ) J(θ) = log f(x ; θ), θ log f(x ; θ), 2 θ 2 log f (x ; θ) Let U (θ) be the random varable U (θ) = θ log f (X ; θ), and, provded that condtons are such that ntegraton and dfferentaton are nterchangeable, and So 0 = θ = = E[ 2 E [U (θ)] = f(x; θ) θ = = θ f (x; θ) log f (x; θ)dx θ f(x; θ)dx θ log f(x; θ)dx f(x; θ)dx = θ 1=0 f(x; θ) 2 θ log f(x; ] θ)dx + θ)( 2 θ f (x; θ) log f (x; θ)dx θ )2 θ log f (X; θ) + f (x; 2 θ log f (x; θ) dx 0= (θ) +E [ U (θ) 2] and, therefore, V [U (θ)] = (θ) It follows that E [U(θ)]=0, V [U (θ)] = n(θ) =I(θ), and the CLT shows that U(θ) D N (0,I(θ)) Now the mle s a soluton of U( θ) =0, so that, Taylor expandng, U (θ) +U (θ)( θ θ) 0 53

21 or Re-arrangng, U (θ) J(θ)( θ θ) 0 I(θ)( θ θ) U (θ) I(θ) J(θ) / = U(θ) J(θ) I(θ) I(θ) From the CLT, and from WLLN U(θ) D N (0, 1) I(θ) J(θ) I(θ) P 1 Slutsky s Theorem therefore results n D I(θ)( θ θ) N(0, 1) or θ D N(θ, I(θ) 1 ) 54

22 Requrements of ths proof 1 The true value of θ s nteror to the partameter space 2 Dfferentaton under the ntegral s vald, so that E [U(θ)] = 0 and V [U(θ)] = n(θ) Ths allows a central lmt theorem to apply to U(θ) 3 Taylor expansons are vald for the dervatves of the log-lkelhood, so that hgher order terms may be neglected 4 A weak law of large numbers apples to J(θ) Example: Exponental famly In the natural parametersaton, the lkelhood has the form so that L(θ) =m(y)e θ y nk(θ) = U(θ) y nk (θ) J(θ) =nk (θ) θ solves U( θ) =0 k (θ) =y Expandng, so that Snce we have k (θ) +( θ θ)k (θ) y y k θ (θ) θ + k 1 (θ) E(Y )=n E(Y )=k (θ), E( θ) θ V ( θ) = 1 k (θ) 2 V (Y )=n 1 k (θ) k (θ) 2 = 1 nk (θ) whch, of course, we could have obtaned drectly from θ N (θ, I(θ) 1 ) Example: Exponental dstrbuton 55

23 so f(x; λ) =λe λx, x > 0, λ > 0, θ = λ, k(θ) = log λ = log( θ) so 1 θ = y, Thus, approxmately, gves a confdence nterval for θ, and gves a confdence nterval for λ k (θ) = 1 θ, k (θ) = 1 θ 2 I( θ) = n y 2 1 y ± z y α n 1 y ± z y α n 56