Bounds for the asymptotic normality of the maximum likelihood estimator using the Delta method

Transcription

1 ALEA, Lat. Am. J. Probab. Math. Stat. 14, Bous for the asymtotic normality of the maximum likelihoo estimator using the Delta metho Areas Anastasiou a Christohe Ley Loon School of Economics, Deartment of Statistics Columbia House, Houghton Street, WC2A 2AE Loon, Unite Kingom. aress: a.anastasiou@lse.ac.uk Ghent University, Deartment of Alie Mathematics, Comuter Science a Statistics Krijgslaan 281, S9, 9000 Gent, Belgium. aress: christohe.ley@ugent.be Abstract. The asymtotic normality of the Maximum Likelihoo Estimator MLE is a cornerstone of statistical theory. In the resent aer, we rovie shar exlicit uer bous on Zolotarev-tye istances between the exact, unknown istribution of the MLE a its limiting normal istribution. Our aroach to this fuamental issue is base on a sou combination of the Delta metho, Stein s metho, Taylor exansions a coitional exectations, for the classical situations where the MLE can be exresse as a function of a sum of ieeent a ientically istribute terms. This result is tailore for the broa class of one-arameter exonential family istributions. 1. Introuction The asymtotic normality of maximum likelihoo estimators MLEs is one of the best-known a most fuamental results in mathematical statistics. Uer certain regularity coitions given later in this section, we have the following classical theorem, first iscusse in Fisher Receive by the eitors Aril 4th, 2016; accete February 2, Mathematics Subject Classification. 62F12, 62E17. Key wors a hrases. Delta metho, Maximum likelihoo estimator, Normal aroximation, Stein s metho. A greatartof thiswork has been comlete whileareas Anastasiou wasworking at the University of Oxfor. Areas Anastasiou was suorte by a Teaching Assistantshi Bursary from the Deartment of Statistics, University of Oxfor, a EPSRC grant EP/K503113/1. Christohe Ley thanks the Fos National e la Recherche Scientifique, Communauté Française e Belgique, for financial suort via a Maat e Chargé e Recherche FNRS.. 153

2 154 Areas Anastasiou a Christohe Ley Theorem 1.1 Asymtotic Normality of the MLE. Let X 1,...,X n be i.i.. raom variables with robability ensity or mass function fx i θ, where θ is a scalar arameter. Its true value is enote by θ 0. Assume that the MLE exists a it is unique a coitions R1-R4, see below, are satisfie. Then niθ0 ˆθn X θ 0 N0,1, n where iθ 0 is the execte Fisher information quantity a means convergence in istribution. Theaimoftheresentaeristocomlementthisqualitativeresultwithaquantitative statement, in other wors, to fi the best ossible aroximation for the istance, at finite samle size n, between the istribution of niθ 0 ˆθn X θ 0 on the one ha a N0,1 on the other ha. In mathematical terms, for Z N0,1, we are intereste in the quantity H niθ0 ˆθn X θ 0,Z ] = su E h niθ0 ˆθn X θ 0 EhZ] 1.1 with h H H = {h : R R, absolutely continuous a boue}. 1.2 Distances of this tye are calle Zolotarev-tye istances. Our focus will lie on the classical situations where the MLE can be exresse as a function of a sum of ieeent a ientically istribute terms. Consier an i.i.. samle of observations X = X 1,...,X n. Writing ˆθ n X the MLE of the scalar arameter of interest θ Θ R, we are intereste in settings where there exists a one-to-one twice ifferentiable maing q : Θ R with q θ 0 θ Θ such that qˆθn X = 1 n n gx i 1.3 for some g : R R. Situations of this ki are all but rare; with fx θ the robability ensity or mass function, classical examles inclue thenormalistributionwithensityfx µ,σ 2 = 1 σ ex 1 2π 2σ x µ 2, 2 x R, for which µ R is our unknown arameter, whereas σ > 0 is consiere to be known. The MLE for θ = µ is ˆθ n X = 1 n X i ; n the normal istribution, where now the mean µ is known a θ = σ 2 reresents the unknown arameter, with ˆθ n X = 1 n X i µ 2 ; n the Weibull istribution with ensity fx α,σ = α x α 1ex σ σ x α σ, x 0, where σ is the unknown scale arameter a α > 0 is fixe. The

3 Delta metho for bous on MLE 155 MLE for θ = σ is efine through α 1 n ˆθn X = Xi α ; n the Lalace scale moel with ensity fx σ = 1 2σ ex x /σ,θ = σ > 0, over R, for which ˆθ n X = 1 n X i. n The broa one-arameter exonential families o satisfy coition 1.3; actually, it is ossible to show that MLEs of the form 1.3 are characteristic of these families, see Proosition 3.1 for etails. Hence, our results o aly to most of the wellknown istributions. We now resent in etail the notation use throughout the aer. We write E θ ] the exectation uer the secific value θ of the arameter. In line with the notation use above, the joint ensity or robability mass function of X is written fx θ. The true, unknown value of the arameter is θ 0 a Θ enotes the arameter sace. For X i = x i some observe values, the likelihoo function is enote by Lθ; x = fx θ a we enote its natural logarithm, calle the log-likelihoo function, by lθ; x. The erivatives of the log-likelihoo function with resect to θ are l θ;x,l θ;x,...,l j θ;x, for j any integer greater than 2, a iθ enotes the execte Fisher information number for one raom variable. Whenever the MLE exists a is also unique, we will write it as before uer the form ˆθ n X. For Θ being an oen interval, we use the results in Mäkeläinen et al to secure the existence a uniqueness of the MLE. Thus, it suffices to assume that: A1 The log-likelihoo function lθ;x is a twice continuously ifferentiable function with resect to θ a the arameter varies in an oen interval a,b, where a,b R {, } a a < b; A2 lim θ a,b lθ;x = ; A3 l θ;x < 0 at every oint θ a,b for which l θ;x = 0. Note that we tacitly assume those coitions in Theorem 1.1 when requiring existence a uniqueness of the MLE. Asymtotic normality further requires the following sufficient regularity coitions: R1 the arameteris ientifiable, which means that if θ θ, then x : fx θ fx θ ; R2 the ensity fx θ is three times ifferentiable with resect to θ, the thir erivative is continuous in θ a fx θx can be ifferentiate three times uer the integral sign; R3 for any θ 0 Θ a for X enoting the suort of fx θ, there exists a ositive number ǫ a a function Mx both of which may ee on θ 0 such that 3 θ 3 logfx θ Mx x X, θ 0 ǫ < θ < θ 0 +ǫ, with E θ0 MX] < ; R4 iθ > 0, θ Θ. These coitions, in articular R2, ensure that, rovie the resective exressions exist, E θ l θ;x] = 0 a Var θ l θ;x] = niθ. These coitions form the

4 156 Areas Anastasiou a Christohe Ley basis of Theorem 1.1 above; see age 472 of Casella a Berger 1990 for a basic sketch of the roof. The first aer to aress the roblem of assessing the accuracy of the normal aroximation for the MLE is Anastasiou a Reinert After eriving general bous on Zolotarev-tye istances, they use the boue Wasserstein istance bw, whichisalsoknownasfortet-mourieristancesee, e.g., NourinaPeccati, 2012 a is linke to the Kolmogorov istance H is the class of iicator functions of half-saces via K, 2 bw,. We state in Theorem 2.4 of Section 2 the bou obtaine in that aer. For the broa class of istributions satisfying 1.3, our bou is better than, or at least as goo as, the Anastasiou a Reinert 2017 bou hereafter referre to as AR-bou both in terms of sharness a simlicity. The tools we use to reach this result are the Delta metho, Stein s metho for normal aroximation, Taylor exansions a coitional exectations. The aer is organise as follows. Our new uer bou is escribe, rove a comare to the AR-bou in Section 2, a ractical statistical alications are exhibite. In Section 3 we then aly our results to the class of one-arameter exonential family istributions a treat some secific examles in etail. 2. New bous on the istance to the normal istribution for the MLE In orer to obtain bous on the aforementione istance, we artly emloy the following lemma. From now on, unless otherwise state, enotes the infinity norm. Also, for the sake of resentation we ro the subscrit θ 0 from the exectation a variance. Lemma 2.1 Reinert, Let Y 1,...,Y n be ieeent raom variables with EY i = 0, VarY i = σ 2 > 0 a E Y i 3] <. Let W = 1 n n Y i, with EW = 0, VarW = σ 2 a let K N0,σ 2. Then for any function h H, with H given in 1.2, one has EhW] EhK] h 2+ 1 Y n σ 3E 1 3]. As we shall see below, our strategy consists in benefiting from the secial form of qˆθ n X, which is a sum of raom variables a thus allows us to use the shar bou of this lemma. It is recisely at this oint that the Delta metho comes into lay: abusing notations a language, instea of comaring ˆθ n X to Z N0,1 we rather comare qˆθ n X to Z, a then bou the istance between ˆθ n X a qˆθ n X. The outcome of this aroach is the next theorem, the main result of the resent aer. Theorem 2.2. Let X 1,...,X n be i.i.. raom variables with robability ensity or mass function fx i θ a let Z N0,1. Assume that R1-R4 are satisfie. Furthermore, assume that the MLE exists a 1.3 is satisfie. In aition, the maing g : R R is such that E gx 1 qθ 0 3] < for θ 0 Θ the true value of the arameter. Then, for any h H as in 1.2,

5 Delta metho for bous on MLE if q is not uniformly boue but for any θ 0 Θ there exists 0 < ǫ = ǫθ 0 such that q θ <, we have E h niθ0 h 2+ iθ 0] 3 2 n su θ: θ θ 0 <ǫ ] ˆθn X θ 0 EhZ] q θ 0 3E gx 1 qθ 0 3] niθ0 ] 2 +E ˆθn X θ 0 2 h ǫ 2 ½{ θ Θ : qθ θ}+ h 2 q θ 0 su q θ ; θ: θ θ 0 ǫ if q is uniformly boue, with q θ B θ Θ for some B > 0, we have ] E h niθ0 ˆθn X θ 0 EhZ] h 2+ iθ 0] 3 2 n q θ 0 3E gx 1 qθ 0 3] + h B niθ0 2 q θ 0 E ˆθn X θ 0 2 ] ½{ θ Θ : qθ θ}. 2.2 Proof: The asymtotic normality of the MLE is exlicitly state in Theorem 1.1. Alying the wiely known Delta metho to this result in combination with the requirement q θ 0 0 yiels niθ0 qˆθn X qθ 0 q θ 0 with qˆθn X = 1 n n gx i. Using the triangle inequality, N0,1 2.3 n ] E h niθ0 ˆθn X θ 0 EhZ] E niθ0 h q qˆθn X qθ 0 ] EhZ] 2.4 θ 0 + E niθ0 h niθ0 ˆθn X θ 0 h q qˆθn X qθ 0 ]. θ We first obtain an uer bou for 2.4 using iirectly Stein s metho via Lemma 2.1. Some simle rewriting yiels niθ0 niθ0 1 n q qˆθn X qθ 0 = θ 0 q gx i qθ 0 θ 0 n { } = 1 n iθ0 n q θ 0 gx i qθ 0 = 1 n Y i, n

6 158 Areas Anastasiou a Christohe Ley iθ0 where Y i = q θ gx 0 i qθ 0,i = 1,2,...,n a, obviously, the Y i s are ieeent a ientically istribute raom variables. The Central Limit Theorem alie to 1 n n Y 1 i imlies that n n Y i EY 1 N0,VarY 1. n From 2.3 we know however that 1 n n Y i N0,1; comaring the two n asymtotic results reveals that, necessarilysince two normal istributions can only be equal if their exectations a variances are the same, we have EgX 1 ] = qθ 0 a VargX 1 ] = q θ 0 2, iθ 0 where coition R4 allows to ivie by iθ 0. Hence, as Lemma 2.1 requires, EY i ] = 0, VarY i ] = 1, a E Y i 3 ] < thanks to the aitional coition on g. Alying the result of the lemma we get E niθ0 h q qˆθn X qθ 0 ] EhZ] θ 0 h 2+ iθ 0] 3 2 n q θ 0 3E gx 1 qθ 0 3]. 2.6 Now we are searching for an uer bou on 2.5. Since the case qθ = θ θ Θ is obvious, we from here on assume that qθ θ. We enote A := Aq,θ 0,X niθ0 := h niθ0 ˆθn X θ 0 h q θ 0 qˆθn X qθ 0 a our scoe is to fi an uer bou for EA]. Case 1: θ Θ there exists 0 < ǫ = ǫθ 0 such that su q θ <. Then, θ: θ θ 0 <ǫ using the law of total exectation relate to coitioning on ˆθ n X θ 0 ǫ or ˆθ n X θ 0 < ǫ yiels EA] E A ] = E A ˆθ n X θ 0 ǫ ]P ˆθn X θ 0 ǫ +E A ˆθ n X θ 0 < ǫ ]P ˆθn X θ 0 < ǫ. Markov s inequality a the elementary results of P ˆθn X θ 0 < ǫ 1 a A 2 h further yiel E EA 2 h ˆθn X θ 0 2 ] ǫ 2 +E A ] ˆθ n X θ 0 < ǫ. 2.7 We now focus on the coitional exectation on the right-ha sie of 2.7. A seco-orer Taylor exansion of qˆθn x about θ 0 gives qˆθn x = qθ 0 + ˆθn x θ 0 q θ q ˆθn x θ 0 θ, 2.8 2

7 Delta metho for bous on MLE 159 for θ between ˆθ n x a θ 0. Since q θ 0 θ Θ is assume, we can multily niθ0 both sies in 2.8 with q θ 0. Rearranging the terms, niθ0 qˆθn x qθ 0 q θ 0 = niθ0 2. niθ 0 ˆθn x θ 0 + 2q θ 0 q θ ˆθn x θ 0 The above result along with another first-orer Taylor exansion recall that n ˆθn X θ 0 2 = op 1 as n gives niθ0 h niθ0 ˆθn x θ 0 h q θ 0 = qˆθn x qθ 0 niθ0 2h 2q θ 0 q θ ˆθn x θ 0 tx, 2.9 ˆθn x θ 0 where tx is between niθ 0 a niθ0 q θ 0 qˆθn x qθ 0. Equality 2.9 combine with Lemma 2.1 in Anastasiou a Reinert 2017 relate to coitional exectations yiel ] E A ˆθ n X θ 0 < ǫ ] = E niθ0 2h 2q θ 0 q θ ˆθn X θ 0 tx ˆθ n X θ 0 < ǫ h niθ 0 2 q E θ 0 h niθ 0 2 q θ 0 h niθ 0 2 q θ 0 2 ] q θ ˆθn X θ 0 ˆθn X θ 0 < ǫ 2 ] su q θ E ˆθn X θ 0 ˆθn X θ 0 < ǫ θ: θ θ 0 <ǫ su q θ E θ: θ θ 0 <ǫ ˆθn X θ 0 2 ] Combining the bous in 2.6, 2.7 a 2.10 we get the result in 2.1. Case 2: There exists B > 0 such that q is uniformly boue with q θ B θ Θ. In this case, we o not nee to take coitional exectations. The result in 2.9 gives E A ] h niθ 0 ] 2 E q θ ˆθn X θ 0 2 q θ 0 B h niθ 0 ] 2 2 q E ˆθn X θ θ 0 The above result a the bou in 2.6 give the bou in 2.2 for the case where q is uniformly boue in θ. Remark Theconvergenceofthe termsaartfromthe firstone isgoverne ] 2 by the asymtotic behaviour of the mean square error E ˆθn X θ 0, whose

8 160 Areas Anastasiou a Christohe Ley rate of convergence is O 1 n. This result is obtaine using the ecomosition E ˆθ n X θ 0 2] ] ] = Varˆθn X +bias 2ˆθn X Uer the staar asymtoticsfrom the regularity coitions the MLE is asymtotically efficient, meaning that ] nvarˆθn X iθ 0] 1, n a hence the variance of the MLE is of orer 1 n. In aition, from Theorem 1.1 the bias of the MLE is of orer 1 n ; see also Cox a Snell 1968, where no exlicit coitions are given. Combining these two results a using 2.12 shows that the mean square error of the MLE is of orer 1 n. 2 In the simlest ossible situation where ˆθ n X is alreay a sum of i.i.. terms, qx = x a hence our uer bou simlifies to ] E h niθ0 ˆθn X θ 0 EhZ] h 2+iθ 0 ] 3 2 E gx 1 θ 0 3], n which is equivalent to Lemma One may woer whether asking R1-R4 to hol is not a too strong assumtion given that we assume the MLE to be of the secial form 1.3. Iee, one can easily get from the Central Limit Theorem that n qˆθx EgX 1 ] n N0,VargX 1] without neeing the asymtotic normalityof ˆθX through R1-R4. However, without this asymtotic normality of ˆθX wecannot obtain 2.3, a without the latterexlicit asymtoticresult we cannotlink EgX 1 ] a VargX 1 ] to their corresoingq-basequantities. The sequel of the roof woul boil own. This is the erhas not so obvious reason why from the beginning of the roof we nee the regularity coitions R1-R4. In orer to areciate the sharness a simlicity of our new bous, we comare our result to the AR-bou. For ease of resentation the comarison is concerne with the case where q is not uniformly boue, but only boue in a neightbourhoo of θ 0. Therefore, we emloy the result in 2.1. Similar results hol when q is uniformly boue. We now state the main result of Anastasiou a Reinert Theorem 2.4 AnastasiouaReinert,2017. Let X 1,X 2,...,X n be i.i.. raom variables with ensity or frequency function fx i θ such that the regularity coitions R1-R4 are satisfie a that the MLE, ˆθ n X, exists a it is unique. Assume that E θ logfx 1 θ ] 3 ] 4 θ=θ 0 < a that E ˆθn X θ 0 <. Let 0 < ǫ = ǫθ 0 be such that θ 0 ǫ,θ 0 +ǫ Θ as in R3 a let Z N0,1.

9 Delta metho for bous on MLE 161 Then for any function h that is absolutely continuous a boue, ] E h niθ0 ˆθn X θ 0 EhZ] h 1 2+ E 3 n iθ 0 ] 3 2 θ logfx 1 θ θ=θ 0 ] 2 E ˆθn X θ 0 +2 h ǫ 2 + h ] niθ0 E R 2 θ 0 ;X ˆθ n X θ 0 ǫ ]]1 E su l 3 4 θ;x 2 θ: θ θ 0 ǫ ˆθ 2 n X θ 0 ǫ E ˆθn X θ 0, where R 2 θ 0 ;x = ˆθ n x θ 0 l θ 0 ;x+niθ Obvious observations are that the AR-bou requires finiteness of the fourth moment of ˆθ n X θ 0 a that this bou is more comlicate than ours. Let us now comment on the bous term by term. In the first term of the bous, the ifferent ositioning of the execte Fisher information number is exlaine by the fact that we aly Lemma 2.1 to the staariseversionof gx 1,gX 2,...,gX n, which havevariance q θ 0] 2 iθ 0, while Anastasiou a Reinert 2017 obtain the result by alying the lemma after staarising θ logfx 1 θ θ=θ0, θ logfx 2 θ θ=θ0,..., θ logfx n θ θ=θ0, which have variance equal to iθ 0. The seco a thir terms vanish in our bou when qθ = θ θ Θ, while the AR-bou oes not take this simlification into account. In aition, when qθ θ the seco term is the same in both bous, whereas the thir term in our bou reas ] 2 h E ˆθn niθ 0 X θ 0 2 q su q θ 2.14 θ 0 θ: θ θ 0 ǫ a is to be comare to E h 2 niθ 0 + h su θ: θ θ 0 ǫ l 3 θ;x 2 ˆθn X θ 0 ǫ] 1 2 ]]1 4 2 E ˆθn X θ 0 ] R E 2 θ 0 ;X ˆθ n X θ 0 ǫ n iθ where R 2 θ 0 ;x = ˆθ n x θ 0 l θ 0 ;x+niθ 0. The seco erivative, q θ, lays in our bou the role of l 3 θ;x, u to an imortant ifference: l 3 θ;x is a sum. Consequently, the first term in 2.15 has n in its numerator, exactly as in The istinct ositioning of the information quantity iθ 0 has the same reason as exlaine above. Besies the

10 162 Areas Anastasiou a Christohe Ley obvious aitional term in the AR bou the seco term in 2.15, our bou is also clearly sharer at the level of moments of ˆθ n X θ 0 since ] ]] E ˆθn X θ 0 E ˆθn X θ 0 by the Cauchy-Schwarz inequality. From this comarison one sees that our new bou is simler a, moreover, has one term less. This is articularly striking in the simlest ossible setting where ˆθ n X is a sum of i.i.. terms, where our bou clearly imroves on the AR-bou. An avantage of the AR-bou is its wier alicability as it works for all MLE settings, even when an analytic exression of the MLE is not known. We conclue this section by iicating ractical statistical alications of our bous in 2.1 a 2.2. This will be shown through their usefulness in the constructionofconfienceintervalsforθ 0. TheKolmogorovistancerelatesirectly to exact conservative confience intervals a the next roosition links our results to the Kolmogorov istance. Slightly ifferent results for conservative confience intervals were also given in Anastasiou a Reinert Proosition 2.5. Let W be any real-value raom variable a Z N0, 1. Assume that there exist δ 1 > 0 a δ 2 0 such that, for any function h with h a h boue, Then K W,Z 2 δ 1 +δ 2. EhW] EhZ] δ 1 h +δ 2 h Proof: The roof of this roosition follows the aroach exlaine in the roof of Theorem 3.3 on age 48 of Chen et al. 2011, where a similar result is shown that links the Wasserstein a the Kolmogorov istance. Let z R a for α = δ1 2π 1 4 let 1, if w z, h α w = 1+ z w α, if z < w z +α, 0, if w > z +α, so that h α is boue Lischitz with h α 1 a h α 1 α. The efinition of h α combine with 2.16 leas to P W z P Z z Eh α W] Eh α Z]+Eh α Z] P Z z Eh α W] Eh α Z] +Eh α Z] P Z z δ 1 h α +δ 2 h α +P z < Z z +α δ 1 α +δ 2 + α 2π Similarly, using now δ1 = 2 +δ 2π δ 1 +δ , if w z α, h α w = w z α α, if z α < w z, 1, if w > z,

11 Delta metho for bous on MLE 163 we can show that P W z P Z z 2 δ 1 +δ 2, which comletes the roof. Our bous are of the form 2.16, where in our case W = niθ 0 ˆθn X θ 0 1 a δ 1 a δ 2 are exlicitly given with δ 1 = O n a δ 2 = O 1 n in 2.1, while for 2.2 take δ 2 = 0. Using Proosition 2.5 we get K niθ0 ˆθ n X θ 0,Z 2 δ 1 +δ 2 B K, whereb K isasharbounoteeingonθ 0 ortheata. InviewofTheorem2.2, this bou can be obtaine by uer bouing 1/q θ 0 a su θ: θ θ0 ǫ q θ which is one on a case-by-case basis as well as the mean square error ] 2 E ˆθn X θ 0 e.g., through Theorem 5.1 in Anastasiou a Reinert, The resence of the Fisher information quantity is in many settings not a roblem, as it often oes not ee on θ 0 ; this is tyically the case for location moels. Therefore, for y R we have P niθ0 ˆθn X θ 0 y P Z y B K B K P niθ0 ˆθn X θ 0 y P Z y B K With Φ 1 the quantile function of the staar normal istribution, α 0,1] a α 2 B K > 0, alying 2.17 to y = Φ 1 α 2 B K a to y = Φ 1 1 α 2 +B K yiels P Φ 1 α 2 B K niθ 0 ˆθn X θ 0 Φ 1 1 α 2 +B K 1 α. Hence, if the execte Fisher information number for one raom variable, iθ 0, is known a oes not ee on θ 0, then ˆθ n X Φ 1 1 α 2 +B K, ˆθ n X Φ 1 α 2 B K niθ0 niθ0 is a conservative 1001 α% confience interval for θ Calculation of the bou in ifferent scenarios In this section we shall consier examles for which we exlicitly calculate our uer bous from Theorem 2.2 a comare them to the AR-bou. In articular, we shall show that our results are tailor-mae for one-arameter exonential families. To further assess their accuracy, we simulate ata from exonential istributions a comare our corresoing bou to the actual istance between the unknown exact law of the MLE a its asymtotic normal law, for istinct values of the samle size n Bous for one-arameter exonential families. The robability ensity or mass function for one-arameter exonential families is given by fx θ = ex{kθtx Aθ+Sx} ½ {x B}, 3.1 where the set B = {x : fx θ > 0} is the suort of the ensity a oes not ee on θ; kθ a Aθ are functions of the arameter; Tx a Sx are

12 164 Areas Anastasiou a Christohe Ley functions only of the ata. Whenever kθ = θ we have the so-calle canonical case, where θ a TX are calle the natural arameter a natural observation Casella a Berger, The ientifiability constraint in R1 entails that k θ 0 Geyer, 2013, an imortant etail for the following investigation. Proosition 3.1. Suose X 1,...,X n are i.i.. with robability ensity or mass function that can be exresse in the form of 3.1. Assume that the regularity coitions R1-R4 are satisfie a the MLE ˆθ n X exists a is unique. Let θ Dθ = A θ k θ be invertible. Then q = D, with q : Θ R as in Theorem 2.2. Moreover, uer the same regularity coitions, no istributions other than 3.1 amit an MLE of the form 1.3. Proof: Using 3.1, we have that a hence Lθ;x = { n n fx i θ = ex kθ Tx i naθ+ lθ;x = kθ n Tx i naθ+ l θ;x = k θ n Sx i, n Tx i na θ = 0 Dθ = 1 n } n Sx i, n Tx i, whichmeansthat ˆθ n X = D 1 1 n n TX i uerthe invertibilityassumtion for Dθ. It follows that q = D. For the seco statement, let us start from qˆθn X = 1 n n gx i. Such an MLE form necessarily comes from a log-likelihoo equation of the form qθ = 1 n n gx i n gx i nqθ = 0. Let us write the invertible function qθ as some ratio α θ/β θ for two ifferentiable functions α a β with β θ 0 θ Θ. Fromthisweeucethatlθ;x = βθ n gx i nαθ+fx for some function f of the observations. Since lθ;x is a log-likelihoo built from ieeent observations, the function f necessarily is a sum which we write n Sx iwhichcoulalsocontainconstants. Theclaimnowreailyfollows. This result hence shows that, as announce in the Introuction, the broa onearameter exonential families o satisfy 1.3 a there is no istribution outsie these families that can be of that form uer the given regularity coitions. This remarkable result reresents a new characterization of the exonential families through their MLE. With this in ha, Theorem 2.2 can be alie to 3.1, resulting in Corollary 3.2. Let X 1,...,X n be i.i.. raom variables with the robability ensity or mass function of a one-arameter exonential family. Assume that R1-R4 are satisfie a the MLE ˆθ n X exists a is unique. Then, with Z N0,1 a h H as efine in 1.2,

13 Delta metho for bous on MLE if D is not uniformly boue but for any θ 0 Θ there exists 0 < ǫ = ǫθ 0 such that D θ <, we have su θ: θ θ 0 <ǫ ] E h niθ0 ˆθn X θ 0 EhZ] h k θ 0 3 E TX 1 Dθ 0 3] 2+ n A θ 0 k θ 0 Dθ ] 2 +E ˆθn X θ 0 2 h ½{ θ Θ : Dθ θ} ǫ2 h n k θ A θ 0 k θ 0 Dθ 0 su θ: θ θ 0 ǫ D θ ; if D is uniformly boue, with D θ B θ Θ for some B > 0, we have ] E h niθ0 ˆθn X θ 0 EhZ] h k θ 0 3 E TX 1 Dθ 0 3] 2+ n A θ 0 k θ 0 Dθ h B n k θ 0 2 A θ 0 k θ 0 Dθ 0 E Proof: We reaily have ˆθn X θ 0 2 ] ½{ θ Θ : Dθ θ}. 3.3 iθ 0 = E l θ 0 ;X 1 ] = A θ 0 k θ 0 ETX 1 ] = A θ 0 k θ 0 k θ 0 A θ 0 k θ 0 a q θ 0 = A θ 0k θ 0 k θ 0A θ 0 k θ 0] 2. Combining these two results yiels iθ0 q θ 0 = k θ A θ 0 k θ 0 k θ 0 A θ 0 = k θ 0 A θ 0 k θ 0 Dθ 0. This result, along with the fact that gx = Tx a qθ = Dθ by Proosition 3.1, allows to euce the announce uer bou from Theorem 2.2. Remark 3.3. It is articularly interesting to sell out this bou in the canonical case kθ = θ. Since then k θ = 1, Dθ = A θ, we fi that if A is not uniformly boue but for any θ 0 Θ there exists 0 < ǫ = ǫθ 0 such that

14 166 Areas Anastasiou a Christohe Ley su A θ <, then θ: θ θ 0 <ǫ ] E h niθ0 ˆθn X θ 0 EhZ] h E TX 1 A θ 0 3] 2+ n A θ E ˆθn X θ 0 2 ] 2 h ǫ 2 ½{ θ Θ : A θ θ} + h n 2 su A θ A θ 0 θ: θ θ 0 ǫ Asimilar resultas the onein 3.3 hols when A is uniformly boue. Now, as iθ = A θ a l θ;x = na θ, R 2 θ;x = 0 a straightforwarmaniulations show that all terms in the AR-bou coincie with those in our bou 3.2, ]]1 4 2 excet for E ˆθn X θ 0, making the AR-bou less shar than ours. However, Anastasiou a Reinert 2017 have shown that, in the canonical exonential setting, their bou can actually have an E ] 2 ˆθn X θ 0 factor, imlying that both bous are exactly the same in the canonical case. In orer to get an iea of how our bou imroves on the AR-bou in non-canonical cases, we treat the exonential istribution uer a non-canonical arametrisation in subsection Bous for the Generalise Gamma istribution. Let us consier X 1,...,X n i.i.. raom variables from the Generalize Gamma GGθ,, istribution, where the shae arameters, > 0 are consiere to be known a the scale arameter θ is the unknown arameter of interest. The Generalise Gamma istribution inclues many other known istributions as secial cases: the Weibull for =, the Gamma when = 1, a the negative exonential when = = 1. Iee, with Γ enoting the Gamma function, the robability ensity function for x > 0 is { x } θ fx θ = x 1 θ ex Γ = ex { x +log logθ + 1logx log θ where, in the terminology of one-arameter exonential families, B = 0,, Θ = Γ. } 0,, Tx = x, kθ = 1 θ, Aθ = logθ a Sx = log + 1logx log Γ. Simle stes yiel lθ 0 ;x = 1 n θ x i +nlog logθ n 0 + 1log x i nlog Γ 0 1 l θ 0 ;x = n θ +1 x i n = 0 θ ˆθ n n x = x i. 0 0

15 Delta metho for bous on MLE 167 It is easy to verify that iee l ˆθ n x;x = n < 0, which shows that the ˆθ nx] 2 MLE exists a is unique. The regularity coitions R1-R4 are also satisfie a Dθ 0 = qθ 0 = θ 0 with its seco erivative not being uniformly boue but boue in a neighbourhoo of θ 0. Using therefore the first result of Corollary 3.2 for ǫ = θ0 2 we obtain ] E h niθ0 ˆθn X θ 0 Γ Γ + 8 h + h EhZ] h n Γ +2 ½{{ 1} { 1}} Γ 1 ½{ < 2} ½{ 2}] Let us briefly show how to obtain this bou. As alreay iicate, for the Generalise Gamma istribution, Dθ 0 = θ 0 TX Dθ a thus E 0 3] = X E 3]. This thir absolute moment is very comlicate to calculate. θ 0 Therefore, we use Höler s inequality a the fact that X GGθ 0,, a thus X Gamma, 1 to get θ 0 E X 3] θ 0 E X ]] θ 0 = = E X 4] θ 4 θ E X 2] θ 4 θ ]3 4 = θ 3 0 Simler calculations yiel ] 3 EX ] 4 θ 0 E X 3] k θ 0 A θ 0 k θ 0 Dθ 0 = θ 0 Using that X i Gamma, 1 n θ X i Gamma n 0, 1 θ 0 ] 2 E ˆθn X θ 0 = = θ n E 2 Γ Γ X i θ θ 2 0 2θ Γ. 3.6 n θ 0 E Γ +1, we get X i 1

16 168 Areas Anastasiou a Christohe Ley 1 = θ Γ Γ Γ Γ The seco erivative of the function Dθ is not uniformly boue a thus we use the result in 3.2. Regaring D θ, one has to be careful as the suremum ees on the value of : su θ: θ θ 0 ǫ su θ: θ θ 0 ǫ 1θ 2 = 1 su θ: θ θ 0 ǫ θ 2 θ 0 ǫ 2, if 0 < < 2 = θ 0 +ǫ 2, if 2. Thus, alying now the results of 3.5, 3.6, 3.7 a 3.8 on the general exression for the first uer bou in Corollary 3.2, we obtain the result in Remark The bou in 3.4 is O n. This is not obvious as we nee to comment on the orer of the term Γ 2 Γ+2. Using Γ + Γ the Taylor exansion for a ratio of Gamma functions see Tricomi a Erélyi, 1951 Γz +a Γz +b = za b 1+ a ba+b 1 +O z 2 2z for large z here, / a boue a a b, we can see that this term is of orer 1 n, leaing to the overall orer of n 1. 2 The iicator function in the last term of the bou in 3.4 comes from the fact that qθ = θ θ Θ, = Bous for the negative exonential istribution. In this subsection, we consier the most famous secial case of the Generalise Gamma istribution: the negative exonential istribution. First we will treat the canonical form of the istribution a then we will change the arametrisation to iscuss the more interesting non-canonical setting. In the canonical case D is not uniformly boue but it is boue in a neighbourhoo of θ 0 ; therefore, 3.2 of Corollary 3.2 is use. As for the non-canonical setting, we will show that Dθ = θ θ Θ which means that only the same first term of the bous in 3.2 a 3.3 survives The canonical case: Exθ. We start with X 1,...,X n exonentially istribute i.i.. raom variables with scale arameter θ > 0 a robability ensity function fx θ = θex{ θx} = ex{logθ θx} for x > 0, which we write Exθ. In terms of 3.1, this means B = 0,, Θ = 0,, Tx = x, kθ = θ, Aθ = logθ a Sx = 0. Further we have that l θ;x = n n θ x i, l θ;x = n θ 2, the unique MLE is given by ˆθ n X = 1/ X with X = 1 n n X i a R1-R4 are satisfie.

17 Delta metho for bous on MLE 169 With this in ha, we can easily see that Dθ 0 = qθ 0 = A θ 0 k θ 0 = 1 θ 0 a k θ 0 A θ 0 k θ 0 Dθ 0 = θ Simle calculations allow us here to byass the Höler inequality use for the Generalize Gamma case a to obtain E TX Dθ 0 3] 1 = E θ 0 X 3] Since X i Exθ i {1,2,...,n} then X Gamn,nθ, with Gamα,β being the Gamma istribution with shae arameter α a rate arameter β. Consequently, Eˆθ n X θ 0 2 ] = nθ 0 2 n 1n 2 2nθ2 0 n 1 +θ2 0 = n+2θ2 0 n 1n 2. Moreover, for ǫ > 0 such that 0 < ǫ < θ 0, we obtain Choosing ǫ = θ0 2, we get su θ: θ θ 0 ǫ Corollary 3.21 gives D θ = 16 θ 3 0 θ 3 0 su D 2 θ = θ 0 ǫ. 3 θ: θ θ 0 ǫ. Using this result a 3.9, ] E h niθ0 ˆθn X θ 0 EhZ] h n+2 +8 h n n 1n 2 nn+2 +8 h n 1n This bou is of orer O n a coincies, as iscusse in Remark 3.3, with the AR-bou The non-canonical case: Ex 1 θ. We now rocee to examine the more interesting case where X 1,...,X n are i.i.. raom variables from Ex 1 θ. The robability ensity function is fx θ = 1 { θ ex 1 } θ x = ex { logθ 1θ } x corresoing to B = 0,, Θ = 0,, Tx = x, kθ = 1 θ, Aθ = logθ a Sx = 0. As before, simle stes give that the MLE exists, it is unique a equal to ˆθ n X = X. The regularity coitions are satisfie a we obtain using will give the same result that E h niθ0 ] ˆθn X θ 0 EhZ] h n Iee, Dθ 0 = qθ 0 = θ 0, making the last two terms of both bous in Corollary 3.2 vanish. The result then follows from E TX Dθ 0 3] = E X θ 0 3] θ 3 0 a k θ 0 A θ 0 k θ 0 Dθ 0 = 1 θ 0. Remark The orer of the bou in terms of the samle size is, as execte, 1 n, corresoing to the orer obtaine for the Generalize Gamma istribution. The constant here is better than the one inherite from 3.4 for = = 1, thanks

18 170 Areas Anastasiou a Christohe Ley to a sharer bou for E TX Dθ 0 3]. 2 The AR-bou is given by h n +8 h n +2 h n +80 h n 6 n +3 1/2, showing that our new bou is an imrovement Emirical results. For a more comlete icture, we also assess the accuracy of our results using simulation-base ata. The rocess we follow is quite simle. We generate10000trials ofn = 10,100,1000,10000a raom i.i.. observationsfrom the exonential istribution Ex 1 2 non-canonicalcase. As function h we choose hx = 1 x 2 +2 with h H, h = 0.5 a h = With ẼhZ] being the aroximation of EhZ] u to three ecimal laces, simle calculations yiel ẼhZ] = Each trial gives an MLE ˆθ n X, hence we have em- ˆθn X θ 0 to comare to Taking the aver- ] ˆθn X θ 0 EhZ], niθ0 irical values of h age rovies a simulate estimation ofe h niθ0 which we comare to the uer bou given in Our bou rovies a very strong imrovement on the AR-bou see Table 3.1. Of course, this estimate istance is only a lower bou to the true istance, as we have chosen a articular function hinstea ofthe suremumoverallfunctions h H, but itscalculationstill rovies an iea of the accuracy of our bous. This closeness logically increases with the samle size a becomes quite shar for n 100. Table 3.1. Simulation results for the Ex 1 2 istribution treate as a non-canonical exonential family niθ0 ] n Ê h ˆθ n X θ 0 ẼhZ] New Bou AR-bou ACKNOWLEDGMENTS The authors woul like to thank the Eitor, an Associate Eitor a an anonymous referee for helful comments that le to an imrovement of the aer. They further thank Gesine Reinert for various insightful comments a suggestions. References A. Anastasiou a G. Reinert. Bous for the normal aroximation of the maximum likelihoo estimator. Bernoulli 23 1, MR G. Casella a R. L. Berger. Statistical inference. The Wasworth & Brooks/Cole Statistics/Probability Series. Wasworth& Brooks/Cole Avance Books& Software, Pacific Grove, CA ISBN MR

19 Delta metho for bous on MLE 171 L. H. Y. Chen, L. Golstein a Q.-M. Shao. Normal aroximation by Stein s metho. Probability a its Alications New York. Sringer, Heielberg ISBN MR D. R. Cox a E. J. Snell. A general efinition of resiuals. J. Roy. Statist. Soc. Ser. B 30, MR R. A. Fisher. Theory of Statistical Estimation. Mathematical Proceeings of the Cambrige Philosohical Society 22, DOI: /S C. J. Geyer. Asymtotics of maximum likelihoo without the LLN or CLT or samle size going to infinity. In Avances in Moern Statistical Theory a Alications: A Festschrift in honor of Morris L. Eaton, ages Es.: G. Jones a X. Shen, Beachwoo, OH: Institute of Mathematical Statistics DOI: /12-IMSCOLL1001. T. Mäkeläinen, K. Schmit a G. P. H. Styan. On the existence a uniqueness of the maximum likelihoo estimate of a vector-value arameter in fixe-size samles. Ann. Statist. 9 4, MR I. Nourin a G. Peccati. Normal aroximations with Malliavin calculus, volume 192 of Cambrige Tracts in Mathematics. Cambrige University Press, Cambrige ISBN MR G. Reinert. Coulings for normal aroximations with Stein s metho. In Microsurveys in iscrete robability Princeton, NJ, 1997, volume 41 of DIMACS Ser. Discrete Math. Theoret. Comut. Sci., ages Amer. Math. Soc., Provience, RI MR F. G. Tricomi a A. Erélyi. The asymtotic exansion of a ratio of gamma functions. Pacific J. Math. 1, MR