LIKELIHOOD RATIO INEQUALITIES WITH APPLICATIONS TO VARIOUS MIXTURES

Transcription

1 Ann. I. H. Poincaré PR 38, 6 00) Éditions scientiiques et médicales Elsevier SAS. All rights reserved S )05-/FLA LIKELIHOOD RATIO INEQUALITIES WITH APPLICATIONS TO VARIOUS MIXTURES Elisabeth GASSIAT Laboratoire de mathématiques, equipe de probabilités, statistique et modélisation, bâtiment 45, Université de Paris-Sud, 9405 Orsay cedex, France Received 0 March 00, revised 7 January 00 ABSTRACT. We give two simple inequalities on likelihood ratios. A irst application is the consistency o the maximum-penalized marginal-likelihood estimator o the number o populations in a mixture with Markov regime. The second application is the derivation o the asymptotic power o the likelihood ratio test under loss o identiiability or contiguous alternatives. Finally, we propose sel-normalized score tests that have exponentially decreasing level and asymptotic power. 00 Éditions scientiiques et médicales Elsevier SAS RÉSUMÉ. Nous donnons deux inégalités pour les rapports de vraisemblance. Une première application est la consistance de l estimateur de vraisemblance marginale pénalisée du nombre de composants dans un mélange de populations à régime markovien. Une deuxième application est le calcul de la puissance asymptotique du test de rapport de vraisemblance dans les cas de non identiiabilité des paramètres sous l hypothèse nulle, et pour des alternatives contigües. Enin, nous proposons un test du score auto-normalisé dont l erreur de première espèce décroit exponentiellement vite et la puissance asymptotique est égale à. 00 Éditions scientiiques et médicales Elsevier SAS. Introduction Let X i ) i N be a strictly stationary sequence o random variables with distribution P on a polish space X.LetG be a set o densities g with respect to some positive measure ν, such that the marginal density o all X i s is an element o G; will thus be ixed throughout the paper. For any g in G, set l n g) = log gx i ). ) When the variables X i ) are independently distributed, l n is the likelihood o the observations. Otherwise, call it marginal-likelihood. In this paper, we mainly investigate

2 898 E. GASSIAT / Ann. I. H. Poincaré PR 38 00) the use o l n as an inerence tool or the number o populations in various mixtures o populations. The testing problem o the number o populations in a mixture is a typical example o testing problems exhibiting a lack o identiiability o the alternative under the null hypothesis. This lack o identiiability leads to the degeneracy o the Fisher inormation o the model, so that the classical chi-square theory does not apply. In two recent papers, D. Dacunha-Castelle and E. Gassiat [,3] proposed a theory o reparametrization to solve such problems, that they called locally conic parametrization. Roughly speaking, the idea is to approach the null hypothesis using directional submodels in which the Fisher inormation is normalized to be uniormly equal to one. This theory provides a guideline to solve general testing problems in which non identiiability occurs, even using other statistics than likelihood ratios, or handling non identically distributed or dependent observations see [3] or an application to ARMA models). The theory was developed in [] and [3] to obtain the asymptotic level o the likelihood ratio test LRT) or the number o populations in a mixture with independent observations. Recently, Liu and Shao [7] proposed an alternative proo to derive this asymptotic distribution. The idea is to give an expansion o the likelihood ratio using a generalized dierentiable in quadratic mean condition, and to handle simultaneously Hellinger distances and Pearson distances. In this paper, we prove two general simple inequalities on likelihood ratios or marginal-likelihood ratios) that allow to derive, in a very simple way, upper bounds or the LRT statistic. We then apply it to several mixtures o populations. For any g G, deine g g s g = g = g, ) where is the norm in L dν). Let us now state the irst inequality. INEQUALITY.. : l n g) l n ) 0 g with s g ) x) = min{0,s g x)}. The second inequality is the ollowing. INEQUALITY.. s gx i ) s g) X i), ln g) l n ) ) s gx i )) s g) X i). Inequalities. and. are proved in Section 5. Inequality. allows to prove the tightness o the LRT or marginal LRT) statistic under simple assumptions. It is used in Section to prove the consistency o the estimator o the number o populations with Markov regime using penalized marginal-likelihood. This estimator has been used or instance by Leroux and Puterman [6], but was only known not to underestimate the number o populations with asymptotic probability, see [5,9]. Inequality. allows to

3 E. GASSIAT / Ann. I. H. Poincaré PR 38 00) prove that the Pearson distance o the maximum likelihood estimator o the marginal density to the true density is n-consistent under simple assumptions. We use it to derive the asymptotic expansion o the LRT statistic under contiguous alternatives or testing problems exhibiting a lack o identiiability o the alternative under the null hypothesis. The result applies in particular to the LRT or the number o populations with independent observations, so that it is possible to ind the asymptotic power o the test under contiguous alternatives. We also propose sel-normalized score tests which have asymptotic level 0 and power by a use o recent results on sel-normalized empirical processes obtained by Bercu, Gassiat and Rio [].. Maximum-penalized marginal-likelihood estimation o the number o populations in a mixture with Markov regime Let P ={p θ,θ } be a set o densities with respect to ν, being a compact inite dimensional set. Let also Z i ) i N be a sequence o stationary random variables taking values in a inite subset o such that, conditionally to Z i ), the variables X i are independent, with densities p Zi. Then, the marginal density o X is a inite mixture o populations, that is p = πi 0 p θi 0 where θ 0,...,θ0 p are the possible values or the Z i s, and π 0 i = PZ = θ 0 i ).IZ i) i N is a Markov chain, the sequence is said to be a inite mixture o populations with Markov regime, and the number o populations is the number o dierent states o the Markov chain having positive probability. Deine or any integer p { p G p = π i p θi : θ,...,θ p ) p,π,...,π p ) [0, ] p, } p π i =. In this section, the set G o possible densities or the marginal distribution o the X i s will be set to P G = G p = G P. p= For any g in G, deine the number o populations as pg) = min{p {,...,P}: g G p }, and let p 0 = p ).DeineS as the set o unctions s g see )) or g in G P. We now deine the maximum-penalized marginal-likelihood estimator o p 0,the number o populations, as a maximizer ˆp o T n p) = max{l n g): g G p } a n p) over {,...,P}. Let us introduce some assumptions. A) a n ) is increasing, a n p ) a n p ) tends to ininity as n tends to ininity or any p >p,anda n p)/n tends to 0 as n tends to ininity or any p.

4 900 E. GASSIAT / Ann. I. H. Poincaré PR 38 00) A) The parametrization θ p θ x) is continuous or all x, and one can ind a unction m in L dν) such that or any g G P, log g m. A3) Z i ) is an irreducible Markov chain such that, i β is its mixing rate unction, and H β u) is the entropy with bracketing o S with respect to the norm,β see [4] or deinitions), then 0 H β u) du < +. In [4] it is proved that,β, so that under A3), S admits an envelope unction F such that F dν <+. Assumption A3) also ensures the uniorm tightness o sx i))/ n) or s S. Indeed, it is proved by Doukhan, Massart and Rio [4] that under this assumption, sx i))/ n) s S satisies a uniorm central limit theorem. We now have: THEOREM.. Under A), A) and A3), ˆp converges in probability to the number o populations p 0. Proo. P ˆp>p 0 ) = P p=p 0 + P p=p 0 + P p=p 0 + P T n p) T n p 0 ) ) P p ln g) l n ) ) p0 ln g) l n ) ) a n p) a n p 0 ) ) P s S sx i)) ) s X i) a np) a n p 0 ) by applying Inequality. and since p0 l n g) l n )) 0. Now, using Theorem o [4], under A3) s S sx i )) = O P ), 3) n and since the usual norm in L dν) is upper bounded by,β, S is P-Glivenko Cantelli see [8]), so that in probability But under the assumptions, lim in n + s S n s X i) = in s S s. 4) in s S s > 0. 5) Indeed, i 5) does not hold, there exists a sequence s n o unctions in S such that s n ) converges to 0. This implies that s n ) converges to 0 in L dν) and ν-

5 E. GASSIAT / Ann. I. H. Poincaré PR 38 00) a.s. or a subsequence. But s n ) dν = s n ) + dν, with s + x) = max{0,sx)}, since all unctions s in S veriy s dν = 0. Thus s n ) + convergesalsoto0inl dν), and ν-a.s. or a subsequence. But under A3), S admits a square integrable envelope unction F, so that along a subsequence s n converges ν-a.s.to0,andinl dν). This contradicts the act that all unctions s in S veriy s dν=. Applying 3), 4), 5), s S sx i)) s X i) = O P) and P ˆp>p 0 ) tends to 0 as n tends to ininity using A). Also, P p ) 0 ˆp<p 0 p= l n g) l n ) P p n a ) np) a n p 0 ). n Under A), the set {logg/ ), g G p } is Glivenko Cantelli in ν-probability, so that p l n g) l n ))/n converges in probability to in log p g, which is negative using A) and the act that p<p 0. Finally, ˆp converges to p 0 in probability. 3. Asymptotic power o the LRT test under loss o identiiability In this section, we assume that the X i s are independent, and G is the set o possible densities o the X i s, not necessarily mixtures o populations. Deine the extended set o scores S as the set o unctions s g see )) or g in G. DeineH [], u) as the entropy with bracketing o S with respect to the norm in L dν). Let us introduce the assumption B): 0 H [], u) du < +. Under B), the set S is Donsker. Deine now the set o scores D as the set o unctions d in L dν) such that one can ind a sequence g n in G satisying g n 0and d s gn 0. With such a g n ), deine, or all t [0, ], g t = g n,wheren <n+. We thus have t that, or any d D, there exists a parametric path g t ) 0 t α such that or any t [0,α], g t G, t g t is continuous, tends to 0 as t tends to 0, and d s gt 0ast tends to 0. We irst prove that all densities in / n-neighborhoods o leading to a score d on a parametric path deine contiguous probabilities P n or the sequence X,...,X n.we reer the reader to [0] or the deinition o contiguity and general results on likelihood ratios or contiguous measures.

6 90 E. GASSIAT / Ann. I. H. Poincaré PR 38 00) Using the reparametrization g u = u, oranyd D, there exists a parametric path g u ) 0 u α such that gu ud) dν= o u ). Now, g u u ) g u gu d ud ) ) dν= dν ) gu ud dν+ ) g 4 u dν o u ) g u + u u ) gu / gu / + ) dν. Applying the dominated convergence theorem to the second term proves that the upper bound is ou ) so that the parametric path g u ) 0 u α is dierentiable in quadratic mean, with score unction d. As a consequence, any g c/ n along such a path deines a P n = g c/ n ν) n mutually contiguous to ν) n. Fix such a g c/ n,andletd 0 be the associated score. We now have the theorem: THEOREM 3.. Under B), ln g) l n ) ) = { max n dx i ); 0}) + o ). Pn d D Proo. Following the proo o Theorem., one can see that 3) and 4) hold again. Since G, one may apply inequality. to obtain : l n g) l n ) 0 g = O P n / ). 6) Taylor expansion gives that log + u) = u u + u Ru), with lim u 0 Ru) = 0. Thus or any g, l n g) l n ) = g s g X i ) + g g sg X i ) ) sg X i ) ) R g By B), S admits a square integrable envelope unction F, and max FX ) i) = o P n.,...,n ) s g X i ).

7 E. GASSIAT / Ann. I. H. Poincaré PR 38 00) Moreover, under B) n s gx i )) = O P ). These acts, together with 6) lead to so that : l n g) l n ) 0 ln g) l n ) ) = which implies that g { g sg X i ) ) R g ln g) l n ) ) g: l n g) l n ) 0 s g X i ) g ) s g X i ) = o P ), sg X i ) ) } + o P ), max { s gx i ) n ; 0 }) s gx i ) But under B), S is Glivenko Cantelli in probability so that s g X i) n = o P), n + o P ). and ln g) l n ) ) { max n s g X i ); 0}) + o P ). g: l n g) l n ) 0 Let G n ={g G: g n /4 }. Using 6), we obtain that ln g) l n ) ) { max n s g X i ); 0}) + o P ). n Now, n s g D tends to 0 as n tends to ininity so that or a sequence u n decreasing to 0, and with we obtain that ln g) l n ) ) n = { s g d: g G n,d D, s g d u n }, max { d D n dx i ) + δ n n δx i ); 0}) + o P ). But using B), the deinition o n and the maximal inequality p. 86 o [0], n δx i ) = o P ), δ n

8 904 E. GASSIAT / Ann. I. H. Poincaré PR 38 00) so that ln g) l n ) ) { max n dx i ); 0}) + o P ). 7) d D Moreover, using the dierentiability in quadratic mean along the parametric paths, one obtains that or a sequence o inite subsets D k increasing to D, one has or any k ln g) l n ) ) { max n dx i ); 0}) + o P ). d D k Thereore, equality holds in 7). Since ν) n and P n are mutually continguous, all o P ) are o Pn ). Deine Wd)) d D the centered gaussian process with covariance the scalar product in L dν). One has: COROLLARY 3.. Assume that B) holds. Under P n, l n g) l n )) converges in distribution to { max Wd)+ c dd 0 dν; 0}). d D The corollary ollows rom Theorem 3. by an application o Le Cam s third Lemma in metric spaces, see []. The result may be applied to mixtures o populations. 4. Sel-normalized score tests We still assume that the variables X i s are independent with common unknown density. Let us now consider the testing problem o H 0 : g = againsth : g, g G or some particular density. To investigate the asymptotic level and the asymptotic power o a test, one has to know about the large or moderate) deviations o the testing statistic. Let S be as in Section 3, vn) some sequence tending to +, and deine the test unction φ n by φ n = i and only i s S sx i)) vn), s X i ) and φ n = 0 otherwise. Let α n be the level o the test and β n g) its power unction. Introduce the assumption C): D is ν-donsker, and or any positive δ, there exists a inite covering B i ) i I o S such that, or any i I, one can ind unctions l i,u i ) satisying or any s B i, l i s u i, su i 0, sl i 0, and u i l i ) dν δ. One has:

9 E. GASSIAT / Ann. I. H. Poincaré PR 38 00) THEOREM 4.. Assume C).Letvn) + with vn) = on). Then or any g G such that g, one has lim n + vn) log α n = and lim n + β ng) =. Proo. Under C), the sel normalized ratio n s S i)/ sx n s X i ) obeys a moderate deviations principle by Theorem 3. o [], and the result on α n ollows. The result on β n g) is a consequence o C) and the act that vn) +. Remark. Assumption C) holds in particular when G is the set o mixtures o gaussian densities with dierent means and same variance. We have l n g) l n ) = 5. Proo o Inequalities. and. g log + g ) s g X i ) s g X i ) g s g ) X i), since or any real number u, log + u) u u. As soon as l ng) l n ) 0, g s g X i ) g s g ) X i) and Inequality. ollows. Now, or any g G, l n g) l n ) log + ps g X i ) ) 0 p g [ ] p s g X i ) p s g ) 0 p g i) [ ] and Inequality. ollows. p 0 p s g X i ) p s gx i )) n s g) X i) s g ) X i)

10 906 E. GASSIAT / Ann. I. H. Poincaré PR 38 00) REFERENCES [] B. Bercu, E. Gassiat, E. Rio, Concentration inequality, large and moderate deviations or sel-normalized empirical processes, Ann. Probab., to appear. [] D. Dacunha-Castelle, E. Gassiat, Testing in locally conic models, ESAIM Probab. Statist. 997). [3] D. Dacunha-Castelle, E. Gassiat, Testing the order o a model using locally conic parametrization: population mixtures and stationary ARMA processes, Ann. Statist. 7 4) 999) [4] P. Doukhan, P. Massart, E. Rio, Invariance principles or absolutely regular empirical processes, Ann. Inst. Henri Poincaré 3 ) 995) [5] B.G. Leroux, Consistent estimation o a mixing distribution, Ann. Statist. 0 99) [6] B.G. Leroux, M.L. Puterman, Maximum-penalized likelihood estimation or independent and Markov dependent mixture models, Biometrics 48 99) [7] X. Liu, Y. Shao, Asymptotics o the likelihood ratio under loss o identiiability with applications to inite mixture models, Preprint, 00. [8] G. Peskir, M. Weber, Necessary and suicient conditions or the uniorm law o large numbers in the stationary case, in: D. Budkovic, et al. Eds.), Functional Analysis IV, Proceedings o the Postgraduate School and Conerence Held at the Inter-university Center, Dubrovnic, Croatia, 993, Mat. Institut, in: Var. Publ. Ser., Aarhus Univ., Vol. 43, 994, pp [9] T. Ryden, Estimating the order o hidden Markov models, Statistics 6 995) [0] A. Van der Vaart, Asymptotic Statistics, Cambridge University Press, 998. [] A. Van der Vaart, J. Wellner, Weak Convergence and Empirical Processes, Springer, New York, 996.