A note on profile likelihood for exponential tilt mixture models

Transcription

1 Biometrika (2009), 96, 1,pp C 2009 Biometrika Trust Printed in Great Britain doi: /biomet/asn059 Advance Access publication 22 January 2009 A note on profile likelihood for exponential tilt mixture models BY Z. TAN Department of Statistics, Rutgers University, Piscataway, New Jersey 08854, U.S.A. ztan@stat.rutgers.edu SUMMARY Suppose that independent observations are drawn from multiple distributions, each of which is a mixture of two component distributions such that their log density ratio satisfies a linear model with a slope parameter and an intercept parameter. Inference for such models has been studied using empirical likelihood, and mixed results have been obtained. The profile empirical likelihood of the slope and intercept has an irregularity at the null hypothesis so that the two component distributions are equal. We derive a profile empirical likelihood and maximum likelihood estimator of the slope alone, and obtain the usual asymptotic properties for the estimator and the likelihood ratio statistic regardless of the null. Furthermore, we show the maximum likelihood estimator of the slope and intercept jointly is consistent and asymptotically normal regardless of the null. At the null, the joint maximum likelihood estimator falls along a straight line through the origin with perfect correlation asymptotically to the first order. Some key words: Empirical likelihood; Exponential tilt; Likelihood ratio statistic; Maximum likelihood; Mixture model; Profile likelihood. 1. INTRODUCTION Suppose that a sample of n j independent observations, {x j1,...,x jnj }, is drawn from each of m( 2) distributions P j ( j 1,...,m), and each P j is a mixture of two component distributions, G 0 and G 1, related in an exponential tilt form: dp j λ j dg 0 + (1 λ j )dg 1, where 0 λ j 1 are known mixture proportions, λ j λ k ( j k), and dg 1 exp(β 0 + β 1 x)dg 0, where G 0 is an unknown baseline distribution, β 1 is an unknown parameter, and β 0 log{ exp(β 1 x)dg 0 }. Note that β 1 0 implies that β 0 0 for any G 0.The jth sample, {x j1,...,x jnj }, is obtained from dp j {λ j + (1 λ j )exp(β 0 + β 1 x)} dg 0. (1) The special case of model (1) where m 2, λ 1 1, and λ 2 0 is known as the two-sample exponential tilt model (Qin, 1998), first studied by Anderson (1972) and Prentice & Pyke (1979) in the context of case-control studies. Inference for β (β 1,β 0 ) T and G 0 in model (1) has been studied using empirical likelihood (Owen, 2001), and mixed results have been obtained. Qin (1999) derived a profile empirical likelihood and the resulting maximum likelihood estimator of β in the case where m 3, λ 1 1, λ 2 0, and 0 <λ 3 < 1 may be unknown. However, Zou et al. (2002) pointed out that the profile empirical likelihood of β has an irregularity at the null hypothesis that G 1 G 0, i.e., β 0, and the asymptotic arguments in Qin (1999) for the maximum likelihood estimator of β are not applicable there. Zou & Fine (2002) claimed that

2 230 Z. TAN the maximum likelihood estimator of β is not consistent when β 0. To address this, Zou et al. (2002) proposed a partial profile empirical likelihood, and showed that the maximum partial likelihood estimator of β is consistent and asymptotically normal whether or not β 0, and the profile loglikelihood ratio for testing β 0 has a chi-squared distribution with one degree of freedom asymptotically. A limitation of this method is that, when β 0, the maximum partial likelihood estimator of β always has asymptotic variance no smaller than that of the maximum likelihood estimator in Qin (1999). See Zou & Fine (2002) for a further discussion on the construction of the maximum partial likelihood estimator and the efficiency comparisons of several related estimators. In this note we first derive a profile empirical likelihood and the resulting maximum likelihood estimator of β 1 alone, and show that the profile empirical likelihood of β 1 does not have an irregularity at β 1 0, in contrast to that of β. In fact, under mild regularity conditions, the profile empirical likelihood and the maximum likelihood estimator of β 1 behave like usual profile likelihoods and maximum likelihood estimators with the following properties, whether or not β 1 0: the maximum likelihood estimator is consistent and asymptotically normal; the inverse curvature of the profile loglikelihood at its maximum is a consistent estimator of the asymptotic variance of the maximum likelihood estimator; and the loglikelihood ratio statistic has a chi-squared distribution with one degree of freedom asymptotically. Second, we derive in our approach the maximum likelihood estimator of β, which agrees with that in Qin (1999), and show that it is consistent and asymptotically normal, and always has asymptotic variance no greater than that of the maximum partial likelihood estimator of β in Zou et al. (2002), whether or not β 0. A subtle point is that when β 0, the maximum likelihood estimator of β falls along a straight line through the origin with perfect correlation asymptotically to the first order. The proofs of the main results are provided in the Appendix. The loglikelihood of (β 1, G 0 )is 2. PROFILE LIKELIHOOD AND MAXIMUM LIKELIHOOD ESTIMATOR l(β 1, G 0 ) [log{λ j + (1 λ j )exp(β 0 + β 1 x ji )}+log G 0 ({x ji })], where β 0 log{ exp(β 1 x)dg 0 } is a function of (β 1, G 0 ). The profile loglikelihood of β 1 is defined for each fixed β 1 by maximizing l(β 1, G 0 ) over all possible probability distributions G 0, that is, pl(β 1 ) max G0 l(β 1, G 0 ). The maximum likelihood estimator of β 1 is defined by maximizing the profile loglikelihood, that is, ˆβ 1 arg max pl(β 1 ). We provide a formula for computing the profile loglikelihood pl(β 1 ) in Proposition 1 (i). The formula involves the function κ(β,α 0 ) log { } λ j + (1 λ j )exp(β 0 + β 1 x ji ) n log n, α 0 + (1 α 0 )exp(β 0 + β 1 x ji ) where β (β 1,β 0 ) T and 0 α 0 1 are free arguments. This function has a duality relationship with the loglikelihood l(β 1, G 0 ) as shown in Proposition 1 (ii). PROPOSITION 1. (i) For a fixed β 1, the loglikelihood l(β 1, G 0 ) achieves the maximum at Ĝ 0 Ĝ 0 (β 1 ) supported on {x ji : i 1,...,n j, j 1,...,m} with Ĝ({x ji }) n 1 ˆα 0 + (1 ˆα 0 )exp(ˆβ 0 + β 1 x ji ), where n m j1 n j, and ˆβ 0 ˆβ 0 (β 1 ) and ˆα 0 ˆα 0 (β 1 ) satisfy α 0 1 n λ j λ j + (1 λ j )exp(β 0 + β 1 x ji ), (2)

3 Miscellanea n 1 1 n 1 α 0 + (1 α 0 )exp(β 0 + β 1 x ji ), (3) exp(β 0 + β 1 x ji ) α 0 + (1 α 0 )exp(β 0 + β 1 x ji ). (4) The profile loglikelihood is pl(β 1 ) κ{β 1, ˆβ 0 (β 1 ), ˆα 0 (β 1 )}. (ii) Furthermore, for any G 0 and any (β 0,α 0 ) satisfying 0 α 0 1, max l(β 1, G 0 ) min max κ(β,α 0 ), G 0 α 0 β 0 where the equality holds when G 0 Ĝ 0 (β 1 ) and (β 0,α 0 ) {ˆβ 0 (β 1 ), ˆα 0 (β 1 )}. The profile loglikelihood pl(β 1 ) is a function of β 1 alone, with ˆβ 0 (β 1 ) and ˆα 0 (β 1 ) implicitly defined by equations (2) (4). Note that (3) and (4) are equivalent to each other, and both are equivalent to the equation 0 κ/ α 0, and that (2), (3) and (4) jointly are equivalent to 0 κ/ β 0 and 0 κ/ α 0 jointly. See the Appendix for the gradient and Hessian of κ(β,α 0 ). By implicit differentiation, the gradient and Hessian of pl(β 1 )are pl κ β 1 β, (5) 1 γ ˆγ (β1 ) { 2 pl 2 ( κ β1 2 β1 2 2 κ 2 ) 1 } κ 2 κ, (6) β 1 γ T γ γ T γ β 1 γ ˆγ (β1 ) where κ(β,α 0 ) is treated as κ(β 1,γ) with γ (β 0,α 0 ) T, and ˆγ (β 1 ) {ˆβ 0 (β 1 ), ˆα 0 (β 1 )} T. These formulas can be used in the Newton Raphson algorithm to numerically maximize pl(β 1 ) and find the maximum likelihood estimator ˆβ 1. In asymptotic considerations, assume that n j /n tends to a constant 0 <ρ j < 1asn ( j 1,...,m). Let β be the true value of β, α0 m j1 ρ jλ j, and γ (β0,α 0. The following lemma )T provides basic calculations used in Proposition 2, due to the law of large numbers and the central limit theorem. LEMMA 1. Suppose that β and α 0 are evaluated at the true values β and α 0. (i) As n, in probability, where S 11 m j1 S 12 S T 21 s 22 and z (x, 1) T. 1 n ( 2 κ β β T 2 κ β α 0 2 κ α 0 β T 2 κ α 2 0 ) ( U S11 S 12 S 21 s 22 λ j (1 λ j )exp(β T z)zz T α ρ j λ j + (1 λ j )exp(β T z) dg (1 α0 )exp(β T z)zz T α0 + (1 α 0 )exp(β T z) dg 0, exp(β T z)z α 0 + (1 α 0 )exp(β T z) dg 0, {1 exp(β T z)} 2 α 0 + (1 α 0 )exp(β T z) dg 0, ),

4 232 Z. TAN (ii) As n,n 1/2 ( κ/ β T, κ/ α 0 ) T converges in distribution to a multivariate normal variable with mean 0 and variance matrix ( ) V S11 δs 12 S 21 δs 12 s 22 δs 21 s 22 s 22 δs22 2, where δ m j1 ρ jλ 2 j α 2 0. We show in Proposition 2 (i) that (5) and (6) satisfy the second Bartlett identity: the variance of the gradient equals the expectation of the negative Hessian for a loglikelihood function, evaluated at the true value of the parameter. This leads directly to the negative Hessian representation for the asymptotic variance of the maximum likelihood estimator, and the chi-squared convergence of the loglikelihood ratio statistic in (ii) and (iii). PROPOSITION 2. Rewrite U and V in Lemma 1 by the partition (β 1,γ) instead of (β,α 0 ): ( ) ( ) ( ) U σ11 10, V U 0 S12 S12 δ (S S 21 2s 22 s 21, s 22 ). 22 Assume that U and 00 are nonsingular. (i) Equations (2) (4) for β 1 β1 admit a solution ˆγ (β 1 ) that converges to γ in probability, 1 pl n 1/2 β N (0, V ), 1 β1 β 1 in distribution, and 1 n 2 pl U, β1 β 1 β 2 1 in probability, where U V σ (ii) Assume that x 2 dg 0 < and x 2 dg 1 <. Then 0 pl/ β 1 admits a solution ˆβ 1 that converges in probability to β1, and n1/2 ( ˆβ 1 β1 ) converges in distribution to N (0, V ). (iii) Furthermore, 2{pl(β1 ) pl( ˆβ 1 )} converges in distribution to a chi-squared variable with one degree of freedom. COROLLARY 1. Under the same conditions in Proposition 2 (ii), n 1/2 { ˆβ 1 β1, ˆγ ( ˆβ 1 ) γ } converges in distribution to a multivariate normal variable with mean zero and variance matrix ( ) S s 22, (7) δ where s 22 and S 11 are defined by the partition of U 1 : ( U 1 S 11 S 12 S 21 s 22 Several interesting conclusions can be drawn. First, the maximum likelihood estimator of β is given by ˆβ {ˆβ 1, ˆβ 0 ( ˆβ 1 )} T, and it agrees with that derived in Qin (1999). In fact, { ˆβ 1, ˆγ ( ˆβ 1 )} satisfies 0 κ/ β 1 and 0 κ/ γ, which are equivalent to equations (2 2) and (2 5) in Qin (1999). By Corollary 1, n 1/2 ( ˆβ β ) converges in distribution to a multivariate normal variable with mean 0 and variance S 11, which equals S S 1 11 S 12s 22 S 21 S Second, ˆβ is asymptotically at least as efficient as ˆβ P, the maximum profile likelihood estimator of β in Zou et al. (2002), which is defined as a maximizer to κ(β, α 0 ) with α 0 n j1 (n j/n)λ j.asshown by Zou et al. (2002), n 1/2 ( ˆβ P β ) converges in distribution to a multivariate normal variable with mean zero and variance S 1 11 δs 1 11 S 12S 21 S 1 11, which is no smaller than S11 because δ s 22 by the positive semidefiniteness of (7) in Corollary 1. ).

5 Miscellanea 233 Third, when β 0, ˆβ falls along the line {β :(μ, 1)β 0} asymptotically to the order of n 1/2, where μ x dg 0.Letσ 2 x 2 dg 0 μ 2. The asymptotic variance (7) becomes 1/(δσ 2 ) μ/(δσ 2 ) 0 μ/(δσ 2 ) μ 2 /(δσ 2 ) 0. (8) As a result, ˆα 0 ( ˆβ 1 ) α0 + o p(n 1/2 ), and (μ, 1) ˆβ o p (n 1/2 ). Note that s 22 δ in this case. The asymptotic variance of n 1/2 ( ˆβ P β ) is identical to that of n 1/2 ( ˆβ β ), and therefore ˆβ P falls along the line {β :(μ, 1)β 0} in a similar manner as ˆβ. In the special case where m 2, λ 1 1, and λ 2 0, corresponding to the two-sample exponential tilt model (Qin, 1998), the assumption that U is nonsingular is violated for Proposition 2, because the 3 3 matrix δs xx δs x1 s x1 U δs 1x δs 11 s 11 s 1x s 11 δ 1 s 11 is singular, where s uv exp ( β0 + β 1 x) uv α0 + ( ) ( 1 α0 exp β 0 + β1 x) dg 0 for (u,v) (x, x), (x, 1), or (1,1). This reduction occurs because equation (2) implies that ˆα 0 (β 1 ) n 1 /n, independent of β 1. Nevertheless, similar conclusions to Proposition 2 and Corollary 1 can be obtained. The asymptotic variance of n 1/2 ( ˆβ 1 β1 )isu 1, where U δ(s xx sx1 2 /s 11). The asymptotic variance of n 1/2 ( ˆβ β )is { (sxx ) 1 ( ) } δ 1 s x1 0 (0, 1), s 1x s 11 1 which was previously obtained in Prentice & Pyke (1979) and Qin & Zhang (1997). When β 0, this matrix equals the first 2 2 principal submatrix of (8), and therefore ˆβ falls along the line {β :(μ, 1)β 0} asymptotically to the order of n 1/2. 3. DISCUSSION For model (1), the profile empirical likelihood of β 1 is well behaved with the usual properties of profile likelihoods whether or not β 0, in contrast to that of β, which has an irregularity when β 0. The reason for this difference is that β 0 is a function of (β 1, G 0 ) such that β 0 0 whenever β 1 0. The profile empirical likelihood of β is not defined on the line {(0,β 0 ) T : β 0 0}, which intersects each nonempty neighbourhood of β when β 0. Therefore, the profile empirical likelihood of β does not have a proper Hessian at β when β 0. The maximum likelihood estimator ˆβ is consistent and n 1/2 ( ˆβ β ) is asymptotically normal whether or not β 0. This estimator can be derived in two ways, either by combining ˆβ 1 and ˆβ 0 ( ˆβ 1 ) from the profile empirical likelihood of β 1 in Section 2 or by maximizing the profile empirical likelihood of β in Qin (1999). However, these two derivations are not equally useful for studying the asymptotic behavior of ˆβ. Qin (1999) aimed to show that the profile empirical likelihood of β is well-defined and has a maximizer in a small neighbourhood of β, but the first property fails when β 0. In contrast, such arguments can be properly applied to the profile empirical likelihood of β 1 to obtain Proposition 2 and Corollary 1. Alternatively in our proofs, we focus on the estimating equations and apply the theory of M-estimators. An interesting point is that when β 0, ˆβ falls along the line {β :(μ, 1)β 0} with perfect correlation asymptotically to the first order. As a result, ˆβ lies away from the line {(0,β 0 ) T : β 0 0}, on which the profile empirical likelihood of β is not defined. This separation allows ˆβ, obtained as a maximizer to the

6 234 Z. TAN profile empirical likelihood of β, to still behave asymptotically in the usual manner except with perfect correlation. Furthermore, the restriction of ˆβ to a straight line suggests that when β 0, the loglikelihood ratio statistic for β converges in distribution to a chi-squared variable with one degree of freedom. Formally, this result follows from Proposition 2 (iii), because the profile empirical likelihood of β at ˆβ, orβ 0, is identical to that of β 1 at ˆβ 1,orβ 1 0. ACKNOWLEDGEMENT The author thanks an associate editor and two referees for helpful comments. This research was supported by the U.S. National Science Foundation. APPENDIX Technical details Proof of Proposition 1. (i) If G 0 ({x}) > 0 for some point x outside the sample {x ji : i 1,...,n j, j 1,...,m}, then l(β 1, G 0 ) < l(β 1, G 0 ), where G is a probability distribution such that G 0 ({x}) 0 and G 0 ({x }) G 0 ({x })/{1 G 0 ({x})} for x x. Therefore, we restrict G 0 to probability distributions supported on the sample, and write w ji G 0 ({x ji }). For a fixed β 1, we maximize the loglikelihood log{λ j + (1 λ j )exp(β 0 + β 1 x ji )}+log w ji, over w ji (i 1,...,n j, j 1,...,m) and β 0 subject to the normalization conditions 1 w ji, 1 exp(β 0 + β 1 x ji )w ji. By introducing Lagrange multipliers nα 0 and nα 1 and setting the derivatives with respect to w ji and β 0 to 0, we obtain 0 1 w ji nα 0 nα 1 exp(β 0 + β 1 x ji ), α 1 1 n (1 λ j )exp(β 0 + β 1 x ji ) λ j + (1 λ j )exp(β 0 + β 1 x ji ). Multiplying the left-hand equation here by w ji and summing over the sample yields α 0 + α 1 1. Therefore, w ji s are of the claimed form, equation (2) results from the right-hand equation, and (3) and (4) correspond to the normalization conditions. (ii) The claimed inequality follows because for any distribution G 0 supported on the sample, any 0 α 0 1, and β 0 log{ exp(β 1 x)dg 0 },wehavel(β 1, G 0 ) κ(β,α 0 ). In fact, by Jensen s inequality, n n j l(β 1, G 0 ) κ(β,α 0 ) log G 0 ({x ji }) n{α 0 + (1 α 0 )exp(β 0 + β 1 x ji )} n n j n log G 0 ({x ji }){α 0 + (1 α 0 )exp(β 0 + β 1 x ji )} 0.

7 Miscellanea 235 Gradient and Hessian of κ(β,α 0 ) To find the Gradient and Hessian of κ(β,α 0 ), write z ji (x ji, 1) T (i 1,...,n j, j 1,...,m). By taking derivatives, we find 1 κ 1 n α 0 n 1 κ n β 1 n By taking second derivatives, we find 1 n 1 2 κ n α0 2 2 κ β β T 1 n 1 n 1 2 κ 1 n β α 0 n 1 + exp(β T z ji ) α 0 + (1 α 0 )exp(β T z ji ), { (1 λ j )exp(β T z ji )z ji λ j + (1 λ j )exp(β T z ji ) {(1 exp(β T z ji )} 2 {α 0 + (1 α 0 )exp(β T z ji )} 2, (1 α } 0)exp(β T z ji )z ji, α 0 + (1 α 0 )exp(β T z ji ) [ λ j (1 λ j )exp(β T z ji )z ji z T ji {λ j + (1 λ j )exp(β T z ji )} α ] 0(1 α 0 )exp(β T z ji )z ji z T ji, 2 {α 0 + (1 α 0 )exp(β T z ji )} 2 exp(β T z ji )z ji {α 0 + (1 α 0 )exp(β T z ji )} 2. Proof of Proposition 2. (i) Fix β 1 β1. Equations (2) (4) are equivalent to 0 κ/ γ β 1 β.the 1 individual terms in κ/ γ and 2 κ/( γ γ T ) are uniformly bounded by constants for γ in a neighbourhood of γ. By the asymptotic theory of M-estimators, the equation 0 κ/ γ β1 β admits a solution ˆγ (β 1 1 ) γ + O p (n 1/2 ). More specifically, ˆγ ( ( β1 ) γ 2 ) 1 κ κ γ γ T γ + o p (n 1/2 ). β1 β 1,γ γ By a Taylor expansion of ( pl/ β 1 )(β1 ), given by (5), with ˆγ (β 1 ) around γ, we find { 1 pl 1 ( κ 2 κ 2 ) 1 } κ κ + o n β 1 n β 1 β 1 γ T γ γ T γ p (n 1/2 ). β1 β 1,γ γ β1 β 1 By Lemma 1, as n, n 1/2 ( pl/ β 1 )(β1 ) converges in distribution to a multivariate normal variable with mean zero and variance matrix V ( 1, ( ) V ) σ , where the simplification is due to (1, )(S 12, s 22 ) T 0. Next, by Lemma 1 and (6), n 1 ( 2 pl/ β1 2)(β 1 ) converges in probability to U σ It follows immediately that U V. (ii) Note that ˆβ 1 satisfies 0 pl/ β 1 if and only if { ˆβ 1, ˆγ ( ˆβ 1 )} satisfies 0 κ/ β 1 and 0 κ/ γ. The individual terms in κ/ β 1 and κ/ γ and the second-order derivatives are uniformly bounded by quadratic functions of observations from G 0 and G 1 for (β 1,γ) in a neighbourhood of (β1,γ ). By the asymptotic theory of M-estimators, there exists a solution { ˆβ 1, ˆγ ( ˆβ 1 )}(β1,γ ) + O p (n 1/2 ). By a Taylor expansion of ( pl/ β 1 )(β1 ), we obtain ( ) 1 ˆβ 1 β1 2 pl pl β + o p (n 1/2 ), 1 β1 β 1 which together with (i) implies that n 1/2 ( ˆβ 1 β1 ) converges in distribution to N (0, V ). β 2 1

8 236 Z. TAN (iii) By a second-order Taylor expansion of pl(β1 ), we obtain pl( ˆβ 1 ) pl(β1 ) 1 2 pl ( 2 β1 2 ˆβ 1 β ) op (1), β1 β 1 which together with (i) implies that 2{pl(β1 ) pl( ˆβ 1 )} converges in distribution to a chi-squared distribution with one degree of freedom, as n. Proof of Corollary 1. The proof of Proposition 2 (ii) shows that n 1/2 { ˆβ 1 β1, ˆγ ( ˆβ 1 ) γ } converges in distribution to a multivariate normal variable with mean 0 and variance U 1 V U 1. The matrix U 1 V U 1 reduces to the claimed form, because U 1 (S 12, s 22 ) T (0, 0, 1) T and hence ( ) ( ) U 1 0 S12 U s 22 S 21 s 22. REFERENCES ANDERSON, J. A. (1972). Separate sample logistic discrimination. Biometrika 59, OWEN, A. (2001). Empirical Likelihood. New York: Chapman & Hall. PRENTICE, R. L.& PYKE, R. (1979). Logistic disease incidence models and case-control studies. Biometrika 66, QIN, J. (1998). Inferences for case-control and semiparametric two-sample density ratio models. Biometrika 85, QIN, J. (1999). Empirical likelihood ratio based confidence intervals for mixture proportions. Ann. Statist. 27, QIN, J.& ZHANG, B. (1997). A goodness-of-fit test for logistic regression models based on case-control data. Biometrika 84, ZOU, F.&FINE, J. P. (2002). A note on a partial empirical likelihood. Biometrika 89, ZOU, F., FINE, J. P.& YANDELL, B. S. (2002). On empirical likelihood for a semiparametric mixture model. Biometrika 89, [Received March Revised September 2008]