82 CHAPTER 4. MAXIMUM IKEIHOOD ESTIMATION Defiitio: et X be a radom sample with joit p.m/d.f. f X x θ. The geeralised likelihood ratio test g.l.r.t. of the NH : θ H 0 agaist the alterative AH : θ H 1, give the observed value x, rejects the NH at α -level of sigificace wheever θ Λ x sup θ H 0 f X x θ sup θ Θ f X x θ < c where c is such that Thus the g.l.r.t. has rejectio regio R sup Pr Λ X < c α θ H 0 x : Λ x sup θ H 0 f X x θ sup θ Θ f X x θ θ < c Example: Cosider the last example agai i which we wish to test whether productio i two plats is uiform i.e. test NH : θ 1 θ 2 agaist AH : θ 1 θ 2 where θ i is the parameter of the expoetial distributio of the compoet failure times produced i plat i, i 1, 2. Fid the g.l.r.t. procedure i this case. Solutio: We have see that the likelihood is θ f X x θ θ 1e θ 1T θ m 2 e θ 2S where T z i ad S m y i, ad the log-likelihood is l θ log θ 1 θ 1 T + log θ 2 θ 2 S Hece l θ θ 1 0 l θ θ2 0 θ 1 T 0 m θ 2 S 0 θ1 T θ 2 m S
4.4. RESTRICTED M..E AND THE G..R.T. 83 i.e. θ /T m/s ad e m m T T e m S S T S m m T S m e +m We have already see that whe the NH is true the restricted m.l.e. s are θ 1 θ 2 +m ad hece T+S m + m θ e +m T+S + m T e +m T+S S T + S T + S +m + m e +m T + S The g.l.r.t. statistic is therefore Λ x θ +m + m e +m T + S m m T S m e +m ad the rejectio regio of the g.l.r.t. is T S m costat T + S +m R {x :Λ x < c} { } T S m x :costat T + S +m < c { } T S m x : +m < c T + S { T/S } x : +m < c 1 + T/S { } U x : +m < c 1 + U where U T S z i m y i
84 CHAPTER 4. MAXIMUM IKEIHOOD ESTIMATION Plot of the statistic U /1+U +m U /1+U +m c 0 0 u 1 u 2 U Figure 4.8: The coditio U /1 + U +m < c is equivalet to U < u 1 ad U > u 2. From the figure above we see that the rejectio regio ca also be writte as R {x :U < u 1 or U > u 2 } { x :U m < v 1 or U m } > v 2 {x :V < v 1 or V > v 2 } where V U m T/ S/m z average of the z-values ȳ average of the y-values. To fid v 1 ad v 2 we must kow the distributio of V whe the NH is true. Sice the Z i s are expoetially distributed with parameter θ 1 the T Z i Gamma,θ 1 i.e. 2θ 1 T χ 2 2. Similarly 2θ 2 S χ 2 2m. Sice the Z i s ad Y i s are idepedet 2θ 1 T/ 2θ 2 S/m χ2 2/2 χ 2 2m/2m F 2,2m Whe the NH is true θ 1 θ 2 ad hece V T/ S/m 2θ 1T/ 2θ 2 S/m F 2,2m
4.4. RESTRICTED M..E AND THE G..R.T. 85 The values v 1 ad v 2 determiig the rejectio regio ca therefore be obtaied from the F 2,2m tables such that Pr v 1 F 2,2m v 2 1 α Example: et X be a radom sample of idepedet observatio from the Bi m,θ distributio. Fid the g.l.r.t. of NH : θ θ agaist the alterative AH : θ > θ for some fixed ad kow value θ. Solutio: The likelihood is θ;x where t x i. Cosequetly ad Thus sup θ;x θ Θ sup θ;x θ H 0 Λ x Λ t m x i θ x i 1 θ m x i θ t 1 θ m t sup θ;x θ H 0 sup θ 0,1 θ t 1 θ m t t t 1 t m t m m sup θ t 1 θ m t θ θ t t m 1 t m t t if m m θ θ t 1 θ m t if t m > θ θ;x t 1 if m θ θ t 1 θ m t sup t t θ Θ m 1 t m 1 if t mθ mθ t t m mθ m t m t if t > mθ m t if t m > θ Figure 4.9 gives the plot of Λ t from which it ca be see that it is a o-decreasig fuctio of t ad cosequetly the rejectio regio is give by R {x : Λ t < c α } {x : t > k α }
86 CHAPTER 4. MAXIMUM IKEIHOOD ESTIMATION ikelihood fuctio G.l.r.t. statistic 1 likelihood θ g.l.r.t. Λt c α 0 0 t/m 1 θ 0 0 mθ t k α m Figure 4.9: The likelihood of the biomial data ad the g.l.r.t statistic where k α is such that sup Pr X R sup Pr X i > k α α level of sigificace θ θ θ θ But X i Bi m,θ. Further, sice Pr Y > k θ is icreasig i θ whe Y Bi r, θ, it follows that ad k α satisfies sup Pr θ θ m jk a+1 X i > k α Pr m j θ j 1 θ m j α X i > k α θ θ Note that there may ot exist iteger k α which satisfies the above equatio. If it does ot, we the choose k α such that m jk a+1 m j θ j 1 θ m j < α ad m jk a m j θ j 1 θ m j > α
4.5. ASYMPTOTIC FORM OF THE G..R.T. 87 i.e. k α is the smallest iteger for which the is less tha α probability that a Bi m,θ radom variable will exceed it. If is large, we ca use the Normal approximatio to the Bi m,θ distributio ad hece obtai k α as the solutio to Pr N mθ,mθ 1 θ > k α α i.e. i.e. Pr N 0, 1 > k α mθ mθ 1 θ α k α mθ mθ 1 θ z α or k α mθ + z α mθ 1 θ where z α is such that Φ z α 1 α, Φ. beig the Stadard Normal distributio fuctio. Cosequetly the rejectio regio is { } R x : x i mθ mθ 1 θ > z α 4.5 Asymptotic form of the g.l.r.t. Result: I testig the NH : θ H 0 agaist the alterative AH ; θ Θ H 0 where H 0 {θ : h 1 θ 0,h 2 θ 0,...,h r θ 0}, provided the sample size o which the test is based is large, the uder mild regularity coditios 2 log Λ X 2 log θ 2[l θ l] is approximately chi-squared distributed with r degrees of freedom whe the ull hypothesis NH is true. The critical regio of the g.l.r.t ca therefore be take as R { } x : 2 log Λ x χ 2 r,α where χ 2 r,α is take from the chi-squared tables ad is such that if W χ 2 r the Pr W χ 2 r,α α ad α level of sigificace of test. The degrees
88 CHAPTER 4. MAXIMUM IKEIHOOD ESTIMATION of freedom are equal to the umber of idepedet side coditios used to specify the ull hypothesis. Note that 2 log Λ x χ 2 r idepedetly of θ as log as θ H 0. I the above Λ x is the g.l.r.t. statistic, θ is the likelihood fuctio, lθ is the lo-likelihood fuctio, θ the m.l.e. of θ ad θ the restricted m.l.e. i H 0. Sketch of proof: Sice θ Λ x sup θ θ H 0 sup θ θ Θ we have that { } 2 log Λ x 2 log log θ { } 2 l l θ where θ is the m.l.e of θ ad θ is the restricted m.l.e. However, we have see that whe bmθ ad θ emerge as turig poits of the likelihood θ ad hece of the log-likelihood l θ ad whe the ull hypothesis is true, θ ad θ are close to each other with high probability. Hece expadig l θ about θ we get to a secod order approximatio k 2 log Λ x 2 l l θi θ l i θ i 1 k k θi 2 θ i θj θ 2 l j θ j1 i θ j k k θi θ i θj θ 2 l j 4.19 θ i θ j j1 By argumets similar to those that produced 4.8 we have by the law of large umbers 2 l θ 2 l θ E I ij θ as θ i θ j θ i θ j
4.5. ASYMPTOTIC FORM OF THE G..R.T. 89 with I ij θ the i,jth elemet of the Fisher Iformatio matrix I θ. Hece for large 4.19 is further approximated by k k 2 log Λ x θi θ i θj θ j I ij j1 T θ θ I θ θ 4.20 Fially sice θ ad θ 0 are close to each other with high probability we ca itroduce the further approximatio T 2 log Λ x θ θ I θ0 θ θ But sice, as we have see, whe the ull hypothesis is true both θ ad θ are multivariate Normally distributed ad hece so is θ θ it follows that 2 log Λ x is approximately a quadratic i a Normally distributed radom vector of zero mea vector. It ca be show that such a quadratic is chisquared distributed. This is a extesio of the result that if W N 0,σ 2 the W 2 /σ 2 χ 2 1 Example et X ij, j 1, 2,..., i be the failure times of a radom sample of i electroic compoet produced selected from the productio lie of the ith maufacturer i 1, 2, 3 ad assume that they are idepedetly ad expoetially distributed with meas which may differ from maufacturer to maufacturer. Assumig that 1, 2 ad 3 are large, costruct a approximate test to test the ull hypothesis that the mea times to failure are the same for the three compaies. If 1 2 3 20 ad X 1j 3106, X 2j 5620 X 3j 3912 j j carry out the test ad report your coclusios. Solutio: {X ij } i j1 is a radom sample from the expoetial distributio with parameter θ i, mea 1/θ i i 1, 2, 3 ad we wish to test the hypothesis NH : θ 1 θ 2 θ 3 agaist AH : θ 1 θ 2 θ 3 Here θ θ 1,θ 2,θ 3 ad the likelihood of the observed values is θ 1 j1 2 3 θ 1 e θx 1j θ 2 e θx 2j θ 3 e θx 3j j1 j j1 θ 1 1 e θ 1t 1 θ 2 2 e θ 2t 2 θ 3 3 e θ 3t 3
90 CHAPTER 4. MAXIMUM IKEIHOOD ESTIMATION where t 1 1 j1 x 1j, t 2 2 j1 x 2j t 3 3 j1 x 3j. The log-likelihood is Hece l θ 1 log θ 1 + 2 log θ 2 + 3 log θ 3 θ 1 t 1 θ 2 t 2 θ 3 t 3 l θ θ i 0 i θ i t i θ i i t i i 1, 2, 3 Whe NH is true i.e. whe θ 1 θ 2 θ 3 θ say the likelihood fuctio reduces to θ θ 1+ 2 + 3 e θt 1+t 2 +t 3 ad Hece i.e. Cosequetly l θ log θ 1 + 2 + 3 log θ θ t 1 + t 2 + t 3 l θ θ θ 0 1 + 2 + 3 θ t 1 + t 2 + t 3 0 θ 1 + 2 + 3 t 1 + t 2 + t 3 θ 1 θ 2 θ 3 1 + 1 + 2 + 2 + 3 3 e 1+ 2 + 3 t 1 + t 2 + t 3 ad 1 t 1 1 e 1 2 t 2 2 e 2 1 1 2 2 3 3 e 1+ 2 + 3 t 1 1 t 2 2 t 3 3 3 t 3 3 e 3 Hece g.l.r.t. statistic is θ Λ x 1 + 2 + 3 1+ 2 + 3 t 1 1 t 2 2 t 3 3 1 1 2 2 3 3 t 1 + t 2 + t 3 1+ 2 + 3 1 2 3 t1 t2 t3 t 1 2 3
4.5. ASYMPTOTIC FORM OF THE G..R.T. 91 where 1 + 2 + 3 ad t t 1 + t 2 + t 3. The asymptotic form of the g.l.r.t. statistic is [ ] 2 log Λ x 2 l l θ [ ] t t1 t2 t3 2 log 1 log 2 log 3 log 1 2 3 χ 2 2 whe the ull hypothesis is true Note that NH is specified i terms of two coditios sice θ 1 θ 2 θ 3 { θ1 θ 2 0 θ 1 θ 3 0 distributio. For the data give { 3106 + 5620 + 3912 3106 2 log Λ 2 60 log 20 log 60 20 } 5620 3912 20 log 20 log 20 20 3.6209 Sice 3.6209 < χ 2 2;0.05 5.99 there is o evidece, at the 5% level, to reject the ull hypothesis. A importat example: Observatios fall idepedetly i oe of four categories C 1,C 2,c 3 ad C 4 with respective probabilities θ 1,θ 2,θ 3,θ 4 with θ 1 + θ 2 + θ 3 + θ 4 1. the followig hypothesis is put forward NH : θ 1 β 2, θ 2 1 2 β1 β, θ 3 3 2 β1 β, θ 4 1 β 2 with β 0.6 I a radom sample of 100 such observatios the umbers fallig i the four categories were x 1 10, x 2 13, x 3 37 ad x 4 40 Perform a test of approximate 5% level of sigificace to test this hypothesis agaist the alterative that at least oe of the equalities i NH does ot hold; show that there is evidece to reject NH.
92 CHAPTER 4. MAXIMUM IKEIHOOD ESTIMATION Solutio: Note that NH really says θ 1 0.36, θ 2 0.12, θ 3 0.36, θ 4 0.16 Note also that sice the θ i s add up to 1, the last oe is determied as soo as we kow the first three θ i s; cosequetly NH ivolves oly 3 idepedet equatios/coditios. The likelihood of θ θ 1,θ 2,θ 3,θ 4 T for the result x i observatios i the ith category i 1, 2, 3, 4 whe are sampled is give by the multiomial probability where x 1 + x 2 + x 3 + x 4 ad! θ x 1!x 2!x 3!x 4! θx 1 1 θ x 2 2 θ x 3 3 θ x 4 4 θ x 1 1 θ x 2 2 θ x 3 3 1 θ 1 θ 2 θ 3 x 4 lθ x 1 log θ 1 + x 2 log θ 2 + x 3 log θ 3 + x 4 log θ 1 θ 2 θ 3 + cost] Hece differetiatig w.r.t. θ i i 1, 2, 3 ad equatig to zero to obtai the m.l.e. we get i.e. lθ θ i 0 x i x 4 θ i 1 θ 1 θ 2 θ i 1, 2, 3 3 θ i x i x 4 θ 4 i 1, 2, 3, 4. 4.21 Sice 4 θ i 1 addig the four equatios above gives 1 x 1 + x 2 + x 3 + x 4 θ4 θ4 x 4 x θ 4 x 4 4 Replacig this i 4.21 we get the m.l.e. θ i x i i 1, 2, 3, 4 ad hece the maximised log-likelihood fuctio l x i log θ i + cost x i log xi + cost
4.5. ASYMPTOTIC FORM OF THE G..R.T. 93 Whe NH is true H 0 is a oe poit set, amely H 0 {θ 0.36, 0.12, 0.36, 0.16}. Hece θ 0.36, 0.12, 0.36, 0.16 π 1,π 2, π 3,π 5 ad the maximized log-likelihood uder the NH is l θ x i log π i + cost Therefore [ 2 log Λx 2[l l θ] 2 x i log Note that 2 x i log xi xi π i ] x i log π i χ 2 3 π i expected umber of observatios out of to fall i the ith category whe NH is true e i x i observed umber of observatios out of that fall i the ith category o i Thus the test statistic has the form 2 log Λx 2 o i log oi e i. whe NH is true For the give results 2 log Λx 2 10 log 10 13 37 40 + 13 log + 37 log + 40 log 25.90 36 12 36 16 > χ 2 3,0.05 7.815 There is, therefore, evidece at the 5% level of sigificace agaist the ull hypothesis which is therefore rejected.
94 CHAPTER 4. MAXIMUM IKEIHOOD ESTIMATION b We ow chage the ull hypothesis to NH : θ 1 β 2, θ 2 1 2 β1 β, θ 3 3 2 β1 β, θ 4 1 β 2 with β uspecified i 0,1. Notice that this ull hypothesis is specified i terms of oly two idepedet equatios/coditios. As before the equatio for theta 4 is redudat; further, sice β θ 1, the oly idepedet equatios are θ 2 1 2 θ1 1 θ 1, θ 3 3 2 θ1 1 θ 1 Thus whe the ull hypothesis NH is true the g.l.r.t. statistic 2 log Λx χ 2 2. Now as i part a whe NH is ot true l x i log xi + cost Whe NH is true the likelihood is θ β! x 1!x 2!x 3!x 4! β2x 1 1 β1 β 2 β 2x 1+x 2 +x 3 1 β x 2+x 3 +2x 4 β N 1 β M x2 3 x3 β1 β 1 β 2x 4 2 where N 2x 1 + x 2 + x 3 ad M x 2 + x 3 + 2x 4. Thus whe NH is true the log-likelihood is ad i.e. the m.l.e. β of β satisfies lβ N log β + M log1 β + costat lβ β 0 N β M 1 β 0 N1 β M β 0 β N N + M N 2 For the give data β N 2 70 200 0.35.
4.5. ASYMPTOTIC FORM OF THE G..R.T. 95 The restricted m.l.e. s of the θ i s whe NH is true are θ 1 β 2 π 1 β, θ2 1 2 β1 β π 2 β, θ 3 3 2 β1 β π 3 β, θ4 1 β 2 π 4 β Thus l θ x i log θ i + cost x i log π i β + cost so that 2 log Λx 2[l l θ] x i log xi xi x i log π i β x i log π β Note agai that π i β estimated expected umber fallig i the ith category e i Thus the g.l.r.t. statistic still has the form For the give data 2 log Λx 2 o i log oi e 1 100 0.35 2 12.25, e 2 100 1 0.35 0.65 11.375, 2 e i e 3 100 3 2 0.35 0.65 34.125, e 4 100 0.65 2 42.25 ad [ 2 log Λx 2 10 log 10 ] 12.25 + 13 log 13 11.375 + 37 log 37 40 + 40 log 34.125 42.25 1.02 < χ 2 2,0.05 5.991
96 CHAPTER 4. MAXIMUM IKEIHOOD ESTIMATION Thus there is o evidece at the 5% level of sigificace to reject the ull hypothesis. This example ca be geeralized. Result: The multiomial test Suppose each observatio ca fall i oe of k categories C 1,C 2,...,C k with PrC i θ i, i 1, 2,...,k, k θ i 1, ad that observatios are take idepedetly with x i fallig i C i, i 1, 2,...,k, k x i. Thus the sample joit p.m.f. is f X x θ! x 1!x 2!...x k! θx i 1 θ x2 θ x k k We formulate the ull hypothesis that states that the θ i s follow the model NH : θ i π i β i 1, 2,...,k with the π i s give fuctios ivolvig a ukow s-dimesioal parameter β if o β parameter is ivolved i the model the s 0 ad the π is i NH are give values. For large, the asymptotic form of the g.l.r.t. statistic is 2 log Λx k χ 2 k 1 s xi x i log π i β k o i log if s 0 the π i β π i, the give umerical value i NH where β is the m.l.e. of β uder the ull hypothesis i.e β maximises! Q k x i! k [π iβ] x i. oi e i