Closed-Form Estimators for the Gamma Distribution Derived from Likelihood Equations

Transcription

1 The American Statistician ISSN: (Print) (Online) Journal homepage: Closed-Form Estimators for the Gamma Distribution Derived from Likelihood Equations Zhi-Sheng Ye & Nan Chen To cite this article: Zhi-Sheng Ye & Nan Chen (216): Closed-Form Estimators for the Gamma Distribution Derived from Likelihood Equations, The American Statistician, DOI: 1.18/ To link to this article: Accepted author version posted online: 21 Jul 216. Submit your article to this journal Article views: 136 View related articles View Crossmark data Full Terms & Conditions of access and use can be found at Download by: [The University of Manchester Library] Date: 28 February 217, At: 1:36

2 Closed-Form Estimators for the Gamma Distribution Derived from Likelihood Equations Zhi-Sheng YE and Nan CHEN Department of Industrial & Systems Engineering National University of Singapore, Singapore, Abstract It is well-known that maximum likelihood (ML) estimators of the two parameters in a gamma distribution do not have closed forms. This poses difficulties in some applications such as real-time signal processing using low-grade processors. The gamma distribution is a special case of a generalized gamma distribution. Surprisingly, two out of the three likelihood equations of the generalized gamma distribution can be used as estimating equations for the gamma distribution, based on which simple closed-form estimators for the two gamma parameters are available. Intuitively, performance of the new estimators based on likelihood equations should be close to the ML estimators. The study consolidates this conjecture by establishing the asymptotic behaviours of the new estimators. In addition, the closed-forms enable biascorrections to these estimators. The bias-correction significantly improves the small-sample performance. Keywords: Estimating equations; bias-correction; generalized gamma distribution; asymptotic efficiency. 1

3 1 Introduction The gamma distribution, denoted as gam(α, β), is a two-parameter distribution with probability density function (PDF) f gam (x) = xα 1 exp ( x/β), x >, (1) β α Γ(α) where α > is the shape parameter, β > is the scale parameter and Γ( ) is the gamma function. Due to the moderate skewness, the gamma distribution is a useful model in many areas of statistics when the normal distribution is not appropriate. For example, it is often used to model frailty and random-effects. In queueing theory, the gamma distribution is often used as a distribution for waiting times and service times (Whitt 2). It is widely used in environmetrics such as environmental monitoring of rainfall sizes (Krishnamoorthy et al. 28). The gamma distribution is a useful model for lifetime (Chen and Ye 216; Meeker and Escobar 1998, Chapter 5.2). It is also used in signal processing (e.g. Vaseghi 28), and clinical trials (e.g., Wiens 1999). The most popular parameter estimation method is the maximum likelihood (ML) method. However, for the two-parameter gamma distribution, there are no closed-form expressions for the ML estimators. This poses difficulties in real-time data/signal processing using battery-constrained, memory and CPU deficient mobile hand-held devices (Song 28). On the other hand, when the computing power of the device is strong enough, the ML estimators may be obtained by numerically maximizing the gamma log-likelihood. However, our simulation experience suggests that the optimization algorithm may fail to converge when α is very small. Even when α is large enough to ensure convergence of the likelihood maximization, the maximization often takes a non-ignorable time. For example, to avoid very long computational time in detecting multiple changepoints in a gamma-distributed sample, previous changepoint detection algorithms usually assume a known 2

4 shape parameter α, e.g., Killick and Eckley (214). Although the moment estimators of the two gamma parameters have closed-forms, they are not efficient under either small samples or large samples, see Figures 1, 2, and 3 below. In order to obtain simple yet efficient estimators of the gamma parameters, we need to think outside the box of the two conventional inference methods. One way to accomplish this is through the use of the generalized gamma distribution, denoted as gg(α, β, γ), where γ > is a power parameter. It is a useful extension of the gamma distribution with PDF f gg (x) = γxαγ 1 β αγ Γ(α) exp [ (x/β) γ], x >. (2) The generalized gamma can be obtained by a power transformation of gamma: if X gam(α, β), then X γ gg(α, β, γ). This distribution, proposed by Stacy (1962), is a flexible model that contains the gamma, Weibull and lognormal distributions as special cases. Many studies have focused on parameter inference for the generalized gamma distribution. See Lawless (198) and Song (28), among others. Inference in this distribution is generally hard. Surprisingly, two estimating equations for the gamma distribution can be obtained by first treating the gamma-distributed data as if they are generalized gamma distributed and then obtaining the three likelihood equations based on the generalized gamma distribution. Estimators based on the two estimating equations have simple closed forms. We show that in terms of both small sample performance and asymptotic efficiency, the new estimators are comparable to the ML estimators. In addition, the closed-forms enable bias-correction to these estimators, which significantly improves the small-sample performance in terms of bias and mean squared errors (MSEs). The paper is organized as follows. Section 2 derives the new estimators for the gamma distribution by looking outside to the generalized gamma distribution. Large sample properties of the new estimators are investigated in Section 3. Section 4 studies bias-correction for the new estimators. 3

5 2 The New Estimators Let X gam(α, β) and X 1, X 2,, X n be n i.i.d. copies of X, where α and β are parameters of interest and need estimation. Obviously, X gg(α, β, γ) with γ = 1. For now, let us pretend that X follows the above generalized gamma distribution with unknown γ. Then the log-likelihood function based on the observed X 1, X 2,, X n is l gg (α, β, γ) = log γ αγ log β log Γ(α) + 1 n n [ (αγ 1) log Xi (X i /β) γ]. i=1 The likelihood equations are obtained by taking the partial derivatives of l gg with respect to α, β and γ, respectively: l gg (α, β, γ) α l gg (α, β, γ) β l gg (α, β, γ) γ = ψ(α) γ log β + γ n log X i, (3) n = α + 1 n = 1/γ + α n i=1 n (X i /β) γ, (4) i=1 n log(x i /β) 1 n i=1 n (X i /β) γ log(x i /β), (5) i=1 where ψ( ) = d log Γ(x)/dx is the digamma function. Setting these equal to zero and solving the resulting system of equations gives the ML estimators of (α, β, γ). In particular, by setting (4) equal to zero, we can express β as a function of α and γ: β(α, γ) = Substitute the above display into (5) to give ( X γ ) 1/γ i. nα α(γ) = n X γ i nγ X γ i log X i γ log X i X γ. i Now, return to the gamma distribution. We already know that γ = 1. Use this fact in the above 4

6 two displays to obtain the new estimators for α and β as ˆα = n X i n X i log X i log X i Xi, (6) and ˆβ = 1 n 2 ( n X i log X i log X i X i ). (7) From the viewpoint of estimating equations, ˆα and ˆβ are obtained based on the two estimating equations (4) and (5), which originate from the likelihood equations of the generalized gamma distribution. Another common parametrization of the gamma distribution is to replace β by a rate parameter λ = 1/β. Under this parametrization, we can go through the above procedure again to obtain an estimator for λ as ˆλ = n 2 n X i log X i log X i Xi, (8) which is simply the inverse of ˆβ. The estimator of α remains the same as (6) under the rate reparametrization. The above procedure can be easily extended to the estimation of stationary gamma processes (Ye et al. 214). Since the two estimating equations for the gamma parameters are essentially likelihood equations of the generalized gamma distribution, it is expected that the performance of the proposed estimators should be similar to the ML estimators. In the next section, we show that the asymptotic efficiency of the proposed estimators is almost the same as the ML estimator counterparts. 5

7 3 Large Sample Properties In this section, we first show that the new estimators are strongly consistent in Theorem 1. Then, the asymptotic normality is established and the asymptotic covariance matrix is derived in Theorem 2. Theorem 1 The estimators ˆα, ˆβ and ˆλ given in (6), (7) and (8) are strongly consistent estimators of α, β, and λ, respectively. Proof Given the n i.i.d. copies of X gam(α, β), let ˉX, Ȳ, ˉZ be the empirical means of X, log X, X log X, respectively. The mean of X is αβ. Based on the moment generating function of log X: M log X (z) = Γ(α + z) β z, (9) Γ(α) the mean of log X is ψ(α) + log β. To obtain E[X log X], note that E[X log X] = The above formula implies x α log x β α Γ(α) exp( x/β)dx = βγ(α + 1) Γ(α) x α log x β α+1 Γ(α + 1) exp( x/β)dx. E[X log X] = αβ[ψ(α + 1) + log β]. According to the strong law of large numbers, ( ˉX, Ȳ, ˉZ) a.s. (αβ, ψ(α) + log β, αβ[ψ(α + 1) + log β]) as n. Define two functions g 1 (x, y, z) = z xy, g 2 (x, y, z) = x/(z xy). Both g 1 and g 2 are continuous at (x, y, z) = (αβ, ψ(α) + log β, αβ[ψ(α + 1) + log β]). An application 6

8 of the continuous-mapping theorem yields that, when n, ˆβ = g 1 ( ˉX, Ȳ, ˉZ) a.s. αβ[ψ(α + 1) ψ(α)]. For the arguments on the right-hand side of the above display, ψ(α + 1) ψ(α) = d [ [ ] d log Γ(α + 1) log Γ(α) = log dt dt ] Γ(α + 1) = d [log α] = 1/α. Γ(α) dt we have ˆβ a.s. β. By the continuous-mapping theorem again, ˆα = g 2 ( ˉX, Ȳ, ˉZ) a.s. α. Since ˆλ = 1/ˆβ, its strong consistency is an immediate consequence of the continuous-mapping theorem. Theorem 2 When n, the two estimators ˆα and ˆβ in (6) and (7) are asymptotically normally distributed as n( ˆα α, ˆβ β) d N, α 2 [1 + αψ 1 (1 + α)] αβ[1 + αψ 1 (α + 1)] αβ[1 + αψ 1 (α + 1)] β 2 [1 + αψ 1 (α)]. (1) Proof Continue with the proof in Theorem 1 and let X gam(α, β). Then E[X] = αβ and E[X 2 ] = αβ 2 + α 2 β 2. Based on the moment generating function (9) of log X, define two quantities: v α E[log X] = ψ(α) + log β, u α E [ (log X) 2] = ψ 1 (α) + ψ 2 (α) + 2ψ(α) log β + log 2 β, where ψ 1 ( ) is the trigamma function equal to dψ(x)/dx. By making use of these two quantities, we have E[X log X] = αβv α+1, E[(X log X) 2 ] = αβ 2 (α + 1)u α+2, E[X log 2 X] = αβu α+1, and E[X 2 log X] = αβ 2 (α + 1)v α+2. Based on the above expectations, we can show after tedious calculations that n[( ˉX, Ȳ, ˉZ) (αβ, v α, αβv α+1 )] d N ( 3, Σ), 7

9 where 3 is a zero vector with 3 elements, and αβ 2 β αβ 2 (1 + v α+1 ) Σ = β ψ 1 (α) αβψ 1 (α + 1) + βv α+1. αβ 2 (1 + v α+1 ) αβψ 1 (α + 1) + βv α+1 αβ 2 [(α + 1)u α+2 αv 2 α+1 ] Because ˆα = g 2 ( ˉX, Ȳ, ˉZ) and ˆβ = g 1 ( ˉX, Ȳ, ˉZ), the partial derivatives of (g 1, g 2 ) with respect to the three arguments (x, y, z) and evaluated at (x, y, z) = (αβ, v α, αβv α ) are g 2 g 2 g 2 αv α+1 α 2 α x y z β β A g 1 x g 1 y g 1 z = v α αβ 1. An application of the delta method yields that n( ˆα α, ˆβ β) is normally distributed with mean 2 and variance matrix AΣA. After tedious simplifications, we can show that α 2 [1 + αψ 1 (1 + α)] αβ[1 + αψ 1 (α + 1)] AΣA = αβ[1 + αψ 1 (α + 1)] β 2 [1 + αψ 1 (α)]. Therefore, the theorem follows. Remark: From (6) and (7), it is interesting to observe that our estimators are mixed type logmoment estimators. The regular moment estimators use the first two moments E[X] = αβ and E[X 2 ] = αβ 2 +α 2 β 2. Our method replaces the second moment equation by E[X log X] E[X]E[log X] = β, after which our estimators are obtained by using sample moments of X, log X and X log X in place of the unknown population moments. Given that the means and variances of X, log X and X log X exist, the asymptotic results of the two estimators are not surprising. The generalized gamma device serves as a heuristic to obtain the two mixed-type log-moment estimating equations. Without the help of the generalized gamma, derivation of the two estimating equations would be difficult. 8

10 We compare the asymptotic efficiency of the new estimators, the ML estimators and the moment estimators. ML estimators of α and β have to be obtained by solving the likelihood equations numerically. The moment estimators of α and β are ˆα m = ( X i ) 2 n X 2 i ( X i ) 2, ˆβ m = n X 2 i ( X i ) 2 (. X i ) The asymptotic variance matrix, which is also the Cramer-Rao lower bound, for the ML estimators of (α, β) is obtained by first deriving the Fisher information matrix and then inverting it, which is given by 1 αψ 1 (α) 1 α β β β 2 ψ 1 (α). (11) The asymptotic variance matrix for the moment estimators can be obtained through the delta method. Figure 1 shows the asymptotic variances of the three different estimators for α and β. Because the variances of ˆα and ˆβ/β do not depend on β, we fix β = 1 and vary α over the interval [.1, 3], as shown in Figure 1. The asymptotic variances of the moment estimators are much higher than the others. In contrast, the variance curves of the proposed estimators and the ML estimators are almost the same. Simulation in the next section shows the same conclusion under small samples. Nevertheless, due to the simple closed forms, the proposed estimators can be calibrated to yield smaller biases under small samples, as shown in the next section. 4 Small Sample Properties In this section, an unbiased estimator for the scale parameter β is obtained by calibrating the new estimator ˆβ. Unbiased estimators for the rate and the shape parameters are not available. Never- 9

11 theless, we give a method to calibrate the corresponding new estimators by comparing the exact covariance and asymptotic covariance between the two estimators and ˆβ. A Monte Carlo simulation is used to show the good performance of the calibrated estimators in terms of bias and MSEs. 4.1 Bias correction Theorem 3 An unbiased estimator for the scale parameter β is β = n n 1 ˆβ = While an unbiased estimator for 1/α is 1 ( ) n X i log X i log X i X i. n(n 1) Proof First, express ˆβ as α 1 = n n 1 ˆα 1 = n Xi log X i log X i Xi (n 1). X i ˆβ = 1 n n (n 1) X 2 i log X i X i log X j. i=1 Note that X i are i.i.d. gam(α, β), and X i and log X j are independent when i j. According to the proof in Theorem 1, E[X log X] = αβ[ψ(α + 1) + log β], E[X] = αβ and E[log X] = ψ(α) + log β. Direct calculation yields i j E [ˆβ ] = 1 n 2 { (n 1)nαβ [ ψ(α + 1) + log β ] n(n 1)αβ [ ψ(α) + log β ]} = n 1 n β. Therefore, an unbiased estimator for β is β = nˆβ/(n 1). On the other hand, note that ˆα in (6) can be expressed as ˆα = n X i β n X i log X i log X i β β β Xi. β This expression suggests that ˆα is independent of the scale parameter β. Based on the results in 1

12 Pitman (1937, Section 6), ˆα is independent of i X i. Therefore, [ ] n Xi E ˆα = E [ ] n X i E[ ˆα 1 ] = n 2 αβe[ ˆα 1 ]. But based on (6), the above display is equal to E[n X i log X i log X i Xi ], which is equal to n(n 1)β. Therefore, E[ ˆα 1 ] = n 1 n α 1. An unbiased estimator for α 1 is then n n 1 ˆα 1. Next, we will show that the estimator ˆα can be calibrated to yield a smaller bias. First note that cov ( ˆα, ˆβ ) = E [ ˆαˆβ ] E [ ˆα ] E [ˆβ ] = αβ n 1 n βe[ ˆα ]. On the other hand, Theorem 2 suggests that the asymptotic covariance between ˆα and ˆβ is Acov ( ˆα, ˆβ ) = αβ [ 1 + αψ 1 (α + 1) ] /n. Equate the previous two displays to yield E [ ˆα ] = nα + α[ 1 + αψ 1 (α + 1) ]. n 1 If we expand ψ 1 ( ) as a Laurent series (Abramowitz and Stegun 1972, Eqn ) and keep the first term only, the right-hand side can be approximated by (n + 2)α/(n 1). Therefore, a biasedcorrected estimator for α can be Xi α = n 1 n + 2 ˆα = n(n 1) (n + 2) [ n X i log X i ]. log X i Xi Similarly, by looking into the covariance and asymptotic covariance between ˆβ and ˆλ, a biasedcorrected estimator for the rate parameter λ can be obtained as λ = n 1 n + 2 ˆλ = n 2 (n 1) (n + 2) [ n X i log X i log X i Xi ]. 11

13 4.2 Simulation A simulation is used to assess the performance of the proposed estimators and the effects of calibration. Because the variance of ˆα and the asymptotic variance of ˆβ/β are independent of β, we set β = 1 in the simulation and vary α from.2 to 5. We consider two sample sizes n = 2 and n = 5. The results under different sample sizes give the same conclusion. Under each sample size, the absolute biases and root MSEs (rmses) of different estimators of α, β and λ are obtained based on 1, simulation replications. The results are shown in Figures 2 and 3. According to the results, the performance of the proposed estimators ˆα and ˆβ, in terms of biases and rmses, is almost the same compared with the ML estimators. The bias calibration to ˆα, ˆβ and ˆλ significantly reduces their biases and improves the performance of these estimators. On the other hand, the moment estimators always have larger biases and rmses. It is interesting to observe that the unbiased estimator β has a larger rmse compared with ˆβ. This is because the weight n/(n 1) used in the calibration of ˆβ is larger than 1. The calibration decreases the bias but increases the variance. The increase in the variance overtakes the decrease in the bias, leading to an increase in the rmse. Acknowledgements We are grateful to the editor, an associate editor, and one referee for their helpful comments. Ye is supported by the National Research Foundation Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE). Chen is partially supported by Singapore AcRF Tier 1 grant R , and CREATE. 12

14 References Abramowitz, M. and Stegun, I. A. (1972), Handbook of Mathematical Functions: With Formulas, Graphs, and Mathematical Tables, no. 55, Courier Dover Publications: Washington, D.C. Chen, P. and Ye, Z.-S. (216), Estimation of field reliability based on aggregate lifetime data, Technometrics, to appear, DOI: 1.18/ Killick, R. and Eckley, I. (214), changepoint: An R package for changepoint analysis, Journal of Statistical Software, 58(3), Krishnamoorthy, K., Mathew, T., and Mukherjee, S. (28), Normal-based methods for a gamma distribution, Technometrics, 5(1), Lawless, J. F. (198), Inference in the generalized gamma and log gamma distributions, Technometrics, 22(3), Meeker, W. Q. and Escobar, L. A. (1998), Statistical Methods for Reliability Data, John Wiley & Sons: New York. Pitman, E. J. (1937), The closest estimates of statistical parameters, in Mathematical Proceedings of the Cambridge Philosophical Society, Cambridge Univ Press, vol. 33, pp Song, K.-S. (28), Globally convergent algorithms for estimating generalized gamma distributions in fast signal and image processing, IEEE Transactions on Image Processing, 17(8), Stacy, E. W. (1962), A generalization of the gamma distribution, The Annals of Mathematical Statistics,

15 Vaseghi, S. V. (28), Advanced Digital Signal Processing and Noise Reduction, John Wiley & Sons: New York. Whitt, W. (2), The impact of a heavy-tailed service-time distribution upon the M/GI/s waitingtime distribution, Queueing Systems, 36(1-3), Wiens, B. L. (1999), When log-normal and gamma models give different results: a case study, The American Statistician, 53(2), Ye, Z.-S., Xie, M., Tang, L.-C., and Chen, N. (214), Semiparametric estimation of gamma processes for deteriorating products, Technometrics, 56(4),

16 25 2 new esimator MLE moment 35 3 new esimator MLE moment 25 asymptotic var of α 15 1 asymptotic var of β α α (a) (b) Figure 1: Asymptotic variances of the new estimators, ML estimators and moment estimators under different values of α: The left panel is for α and the right panel is for β. 15

17 2.5 2 MLE new estimator calibrated moment.6.5 MLE new estimator calibrated moment 2 MLE new estimator calibrated moment bias/rmse of k bias/rmse of β.4.3 bias/rmse of λ k (a) α (b) k (c) Figure 2: Absolute values of the biases (thin lines) and the rmses (bold lines) of the new estimators, the calibrated estimators, the ML estimators and the moment estimators when the sample size is n = 2: The left panel is for α, the middle is for β and the right for λ. 16

18 1.2 1 MLE new estimator calibrated moment MLE new estimator calibrated moment MLE new estimator calibrated moment bias/rmse of α bias/rmse of β bias/rmse of λ α (a) α (b) α (c) Figure 3: Absolute values of the biases (thin lines) and the rmses (bold lines) of the new estimators, the calibrated estimators, the ML estimators and the moment estimators when the sample size is n = 5: The left panel is for α, the middle is for β and the right for λ. 17