Computng MLE Bas Emprcally Kar Wa Lm Australan atonal Unversty January 3, 27 Abstract Ths note studes the bas arses from the MLE estmate of the rate parameter and the mean parameter of an exponental dstrbuton. Motvaton Although maxmum lkelhood estmaton (MLE) methods provde estmates that are useful, the estmates themselves are not guaranteed to be unbased. evertheless, MLE methods are stll hghly regarded n practce due to several of ther propertes, notably, the estmates are consstent and asymptotcally normal (Casella and Berger, 22; Panchenko, 26). The most popular example that llustrates the bas of the MLE methods s the MLE estmate of the varance parameter σ 2 of a normal dstrbuton (µ, σ 2 ), we refer the readers to Lang (22) for detals. Another example that s of nterest s that of an exponental dstrbuton. In ths case, the MLE estmate of the rate parameter λ of an exponental dstrbuton Exp(λ) s based, however, the MLE estmate for the mean parameter µ = /λ s unbased. Thus, the exponental dstrbuton makes a good case study for understandng the MLE bas. In ths note, we attempt to quantfy the bas of the MLE estmates emprcally through smulatons. For ths purpose, we wll use the exponental dstrbuton as example. 2 MLE for Exponental Dstrbuton In ths secton, we provde a bref dervaton of the MLE estmate of the rate parameter λ and the mean parameter µ of an exponental dstrbuton. We note that MLE estmates are values that maxmse the lkelhood (probablty densty functon) or loglkelhood of the observed data. Let {x } be..d. random varables that are exponentally dstrbuted, wrtten as The lkelhood functon assocated wth X = {x } can be wrtten as wth the followng log lkelhood: x Exp(λ). () p(x) = λ e λ x, (2) = L(λ, X) = log λ λ x. (3) =
Solvng for the MLE estmate ˆλ = arg mn λ L(λ, X) gves us ˆλ = (4) where s the mean of {x }: = x (5) = We note that by usng the nvarant property of the MLE, the MLE estmate ˆµ s smply 2. Bas of the MLE Estmates ˆµ =. (6) An estmate s unbased f the expectaton of the estmate equals to ts true value. We frst note that the MLE estmate ˆµ s unbased, evdenced by E[ˆµ] = E[] = For the MLE estmate ˆλ, ts expectaton s [ ] E[ˆλ] = E E[x ] = µ. (7) = E[] = λ. (8) The nequalty follows from Jensen s nequalty wth the convex functon f(x) = /x. To quantfy the bas of ˆλ, we can derve the bas B(ˆλ) drectly usng the propertes of gamma dstrbuton. Frst, we note that the exponental dstrbuton s a specal case of the gamma dstrbuton wth shape parameter, that s, x Gamma(, λ). (9) Snce the summaton of gamma random varables s also gamma dstrbuted (see Proposton 3., Taylor, 29), we have x Gamma(, λ). () = otce that the nverse of a gamma random varable s nverse-gamma dstrbuted (Wkpeda, 26): = x InvGamma(, λ), () wth expectaton [ ] E = x = λ, for >. (2) Wth ths establshed, we can derve the bas of the MLE estmate ˆλ as follows: [ ] [ ] B(ˆλ) = E λ = E = x λ = λ λ = λ. (3) We can see that the bas approaches zero as ncreases. 2
3 Emprcal Bas for Exponental Dstrbuton In ths secton, we perform experments to evaluate the bas of the MLE estmates emprcally through Monte Carlo method. The emprcally bas of an estmate ˆλ can be computed as ˆB(ˆλ) = M M ˆλ (j) λ, (4) j= where ˆλ (j) s the MLE estmate for λ n the j-th smulaton experment. To be precse, notng that x (j) ˆλ (j) = s the -th sample of the j-th smulaton experment., (5) = x(j) In the followng, we compute ˆB(ˆλ) by varyng from to and settng M =. For smplcty, we let λ = µ =. The experments are performed usng Matlab. Fgure below plots the mean of the MLE estmates ˆλ aganst the number of samples used n the experments. The mean ˆλ s computed as follows: ˆλ = M M ˆλ (j), (6) j= wth varyng. From the followng plot, we can see that the mean of the MLE estmates devate from ts true value. However, the devaton decreases as the number of samples ncreases..5.4 MLE Estmates of λ vs Ground Truth MLE estmate Ground truth.3 λ.2..99 2 3 4 5 6 7 8 9 ext, we plot the emprcal bas of the MLE estmator ˆλ (blue) and ts theoretcal counterpart (red) vs the number of samples n the fgure below. Here, we can see that the emprcal bas fluctuates around ts theoretcal values. Further, we also supermpose the emprcal bas of the MLE estmate ˆµ (green) for llustratve purpose. Snce the MLE estmator for µ s unbased, the emprcal bas fluctuates around (yellow). 3
-3 5 Emprcal Bas vs Theoretcal Bas of the MLE Estmator Bas of estmator µ Bas of estmator λ Theoretcal bas for µ Theoretcal bas for λ Bas 5-5 2 3 4 5 6 7 8 9 We note that the fluctuatons come from two ndependent sources. Frstly, the fluctuaton s due to varablty wthn the MLE estmates, that s, V[ˆλ]. The second source of varablty comes from the Monte Carlo smulatons, whch approaches zero only when M. To llustrate, n the fgure below, we compare the standard errors of ˆB(ˆλ) aganst ther theoretcal values, gven as [ ] [ V[ˆλ] = V = V = x ] = 2 λ 2 ( ) 2, for > 2. (7) ( 2) Ths corresponds to the frst source of varablty. The varablty from the Monte Carlo smulaton s n fact smaller, whch s gven by V[ ˆB(ˆλ)] = M V[ˆλ] = 2 λ 2 M( ) 2, for > 2. (8) ( 2) Emprcal Standard Error vs Theoretcal Standard Devaton of the MLE Estmator.7 Emprcal standard error Theoretcal standard devaton.6.5 Standard Error.4.3.2. 2 3 4 5 6 7 8 9 We note that the standard errors computed for the estmator ˆµ have smlar values as those of ˆλ (thus not plotted). Ths s because ther varances are very close to one another (due to λ = µ) for large : V[ˆµ] = V [ ] = V [ = x ] = λ 2 = µ2. (9) 4
3. Bas Correcton To recap, the expected value of the MLE estmator ˆλ s λ. We can smply multply a correcton factor to the MLE estmator to elmnate the bas, ths gves us an unbased estmator ˆλ = = = x. (2) In the followng graph, we compute the emprcal bas of the adjusted estmator. 5 x Emprcal Bas vs Theoretcal Bas of the Corrected Estmator 3 Bas of estmator λ * 4 Theoretcal bas for λ * 3 2 Bas 2 3 4 5 2 3 4 5 6 7 8 9 As expected, the MLE estmates centred around, verfyng that the estmator ˆλ s unbased. We note that the varance of the estmator s V[ˆλ ] = ( )2 λ 2 2 V[ˆλ] =, for > 2, (2) ( 2) whch s slghtly lower than that of the MLE estmator ˆλ, ths corrected estmator s thus better n terms of both bas and varablty. References Casella, G. and Berger, R. L. (22). Statstcal Inference, volume 2. Duxbury Pacfc Grove, CA. Lang, D. (22). Maxmum lkelhood estmator for varance s based: Proof. Retreved from http://dawenl.gthub.o/fles/mle_based.pdf. Panchenko, D. (26). Lecture 3 Propertes of MLE: consstency, asymptotc normalty. Fsher nformaton. Retreved from http://ocw.mt.edu/courses/mathematcs/ 8-443-statstcs-for-applcatons-fall-26/lecture-notes/ lecture3.pdf. Taylor, J. (29). Lecture 2: Sums of ndependent random varables. Retreved from https://math.la.asu.edu/ jtaylor/teachng/fall2/stp42/ lectures/lecture2.pdf. Wkpeda (26). Inverse-gamma dstrbuton. Retreved from https://en.wkpeda. org/wk/inverse-gamma_dstrbuton. 5