PRESERVATION THEOREMS FOR GLIVENKO-CANTELLI AND UNIFORM GLIVENKO-CANTELLI CLASSES AAD VAN DER VAART AND JON A. WELLNER ABSTRACT We show that the P Gli

Transcription

1 PRESERVATION THEOREMS FOR GLIVENKO-CANTELLI AND UNIFORM GLIVENKO-CANTELLI CLASSES AAD VAN DER VAART AND JON A. WELLNER ABSTRACT We show that the P Glivenko property of classes of functions F ;::: ;F k is preserved by a continuous function ' from R k to R in the sense that the new class of functions x! '(f (x);::: ;fk(x)); fi 2F i; i=;::: ;k is again a Glivenko-Cantelli class of functions if it has an integrable envelope. We also prove an analogous result for preservation of the uniform Glivenko-Cantelli property. Corollaries of the main theorem include two preservation theorems of Dudley (998). We apply the main result to reprove a theorem of Schick and Yu (999) concerning consistency of the NPMLE in a model for mixed case" interval censoring. Finally a version of the consistency result of Schick and Yu (999) is established for a general model for mixed case interval censoring" in which a general sample space Y is partitioned into sets which are members of some VC-class C of subsets of Y.. Glivenko - Cantelli Theorems. Let (X ; A;P) be a probability space, and suppose that F ρ L (P ). For such a class of functions, let F 0;P ff Pf : f 2 Fg. We also let F F (x) sup f2f jf(x)j, the envelope function of F. Ifjfj»F for all f 2F with F measurable, then F is an envelope for F. Suppose that X ;X 2 ;::: are i.i.d. P. Glivenko-Cantelli theorems give conditions under which the empirical measure Pn converges uniformly to P over a class F, either in probability (in which case we say that F is a weak Glivenko-Cantelli class for P ) or almost surely: kpn P k F sup jpnf Pfj! a:s: Λ 0; f2f in this case wesay that wesay that F is a strong Glivenko-Cantelli class for P. Useful sufficient conditions for a class F to be a strong Glivenko-Cantelli class for P are that it has an integrable envelope and either log N [] (ffl; F;L (P)) < for all ffl>0

2 2 Glivenko-Cantelli preservation theorems or log N(ffl; F M ;L (Pn)) = o Λ P (n) for every ffl>0 and 0 <M<: Here F M = ff [F»M] : f 2Fg. In the second case some additional measurability conditions are necessary; see Van der Vaart and Wellner (996), Chapter 2.4, pages In particular, the condition in Theorem 2.4.3, page 23, is shown by Giné and inn (984) to be both necessary and sufficient, under measurability assumptions, for the class F to be a strong Glivenko-Cantelli class (see Talagrand (996) for another proof). Talagrand (987b) gives necessary and sufficient conditions for the Glivenko-Cantelli theorem without any measurability hypotheses. Theorem. (Giné and inn, 984). Suppose that F is L (P ) bounded and nearly linearly supremum measurable for P ; in particular this holds if F is image admissible Suslin. Then the following are equivalent: (a) F is a strong Glivenko-Cantelli class for P. (b) F has an envelope function F 2 L (P ) and the truncated classes F M ffff»mg: f2fgsatisfy n EΛ log N(ffl; F M ;L r (Pn))! 0 for all ffl>0; and for all M 2 (0; ) for some (all) r 2 (0; ] where kfk Lr(P ) kfk P;r fp(jfj r )g r ^. 2. Preservation of the Glivenko - Cantelli Property. Our goal in this section is to present several results concerning the stability of the Glivenko-Cantelli property of one or more classes of functions under composition with functions '. A theorem which motivated our interest is the following result of Dudley (998a). Theorem 2. (Dudley, 998a). Suppose that F is a strong Glivenko-Cantelli class for P with PF <, J is a possibly unbounded interval including the ranges of all f 2F,'is continuous and monotone on J, and for some finite constants c; d, j'(y)j»cjyj+dfor all y 2 J. Then '(F) is also a strong Glivenko-Cantelli class for P. Dudley (998a) proves this via the characterization of Glivenko-Cantelli classes due to Talagrand (987b). Dudley (998b) also uses Talagrand's characterization to prove the following interesting proposition. Proposition. (Dudley, 998b). Suppose that F is a strong Glivenko- Cantelli class for P with PF <, and g is a fixed bounded function (kgk < ). Then the class of functions g F fg f :f 2Fgis a strong Glivenko-Cantelli class for P. Yet another proposition in this same vein is:

3 van der Vaart and Wellner 3 Proposition 2. (Giné and inn, 984). Suppose that F is a uniformly bounded strong Glivenko-Cantelli class for P, and g 2 L (P ) is a fixed function. Then the class of functions g F fg f : f 2 Fg is a strong Glivenko-Cantelli class for P. Given classes F ;::: ;F k of functions f i : X! R and a function ' : R k! R, let '(F ;::: ;F k ) be the class of functions x! '(f (x);::: ;f k (x)) ; where f i 2F i,i=;::: ;k. Theorem 2 and Propositions and 2 are all corollaries of the following theorem. Theorem 3. Suppose that F ;::: ;F k are P Glivenko-Cantelli classes of functions, and that ' : R k! R is continuous. Then H '(F ;::: ;F k )is P Glivenko-Cantelli provided that it has an integrable envelope function. Proof. We first assume that the classes of functions F i are appropriately measurable. Let F ;::: ;F k and H be integrable envelopes for F ;::: ;F k and H respectively, and set F = F _ :::_F k.for M 2 (0; ), define Now H M f'(f) [F»M] : f =(f ;::: ;f k )2F F 2 F k Fg: k(pn P )'(f)k F» (Pn + P )H [F>M] +k(pn P)hk HM : The expectation of the first term on the right converges to 0 as M!. Hence it suffices to show that H M is P Glivenko-Cantelli for every fixed M. Let ffi = ffi(ffl) betheffiof Lemma 2 below for ' :[ M;M] k! R, ffl>0, and k kthe L -norm k k. Then for any (f j ;g j )2F j,j=;::: ;k, implies that It follows that Pnjf j g j j [Fj»M]» ffi ; j =;::: ;k k Pnj'(f ;::: ;f k ) '(g ;::: ;g k )j [F»M]»ffl: N(ffl; H M ;L (Pn))» ky j= N( ffi k ; F j [Fj»M];L (Pn)) : Thus E Λ log N(ffl; H M ;L (Pn)) = o(n) for every ffl>0, M<. This implies that E Λ log N(ffl; (H M ) N ;L (Pn)) = o(n) for (H M ) N the functions hfh» Ng for h 2 H M.Thus H M is strong Glivenko-Cantelli for P by Theorem. This concludes the proof that H = '(F) is weak Glivenko-Cantelli. Because it has an integrable envelope, it is strong Glivenko-Cantelli by, e.g., Lemma of Van der Vaart and Wellner (996). This concludes the proof for appropriately measurable classes F j, j =;::: ;k. We extend the theorem to general Glivenko-Cantelli classes using separable versions as in Talagrand (987a). (Also see van der Vaart and Wellner

4 4 Glivenko-Cantelli preservation theorems (996), pages 5-20 for a discussion.) As shown in the preceding argument, it is not a loss of generality to assume that the classes F i are uniformly bounded. Furthermore, it suffices to show that '(F ;::: ;F k )is weak Glivenko-Cantelli. We first need a lemma. Lemma. Any strong P -Glivenko Cantelli class F is totally bounded in L (P ) if and only if kp k F <. Furthermore for any r 2 (; ), if F has an envelope that is contained in L r (P ), then F is also totally bounded in L r (P ). Proof. A class that is totally bounded is also bounded. Thus for the first statement we only need to prove that a strong Glivenko-Cantelli class F with kp k F < is totally bounded in L (P ). It is well-known that such a class has an integrable envelope. E.g. see Giné and inn (983) or Problem 2.4. of van der Vaart and Wellner (996) to conclude first that P Λ kf Pfk F <. Next the claim follows from the triangle inequality kfk F»kf Pfk F +kpk F.Thus it is no loss of generality to assume that the class F possesses an envelope that is finite everywhere. If F is not totally bounded in L (P ), then there is " > 0 and G ρ F countable such that the packing number D("; G;L (P)) =. This is obvious, since if F is not totally bounded, then D("; F;L (P)) = for some ">0; then take a countable set of " separated points. So, now G has all the measurablity properties, is GC and L (P )-bounded, and moreover, D("; G;L )=. But this is a contradiction because, as shown in Giné and inn (984), first half of page 984, D("; G;L )must be finite for all ">0. This completes the proof of the first part of the lemma. If F has an envelope in L r (P ), then F is totally bounded in L r (P ) if the class F M of functions fff» Mg is totally bounded in L r (P ) for every fixed M. The class F M is P -GC by Theorem 3 and hence this class is totally bounded in L (P ). But then it is also totally bounded in L r (P ), because P jfj r» P jfjm r for any f that is bounded by M and we can construct the ffl-net over F M in L (P ) to consist of functions that are bounded by M. 2 Because a Glivenko-Cantelli class F with kp k F < is totally bounded in L (P )by Lemma, it is separable as a subset of L (P ). A minor generalization of Theorem in van der Vaart and Wellner (996) shows that there exists a bijection f $ ~ f of F onto a class ~ FρL (P) such that ffl f = ~ f P almost surely for every f 2F. ffl there exists a countable subset G ρ ~ F such that for every n there exists a measurable set N n ρ X n with P n (N n ) = 0 such that for all (x ;::: ;x n ) =2 N n and f 2 ~ F there exists fgm g ρ G such that P jg m ~ fj!0 and (g m (x );::: ;g m (x n ))! ( ~ f(x );::: ; ~ f(x n )).

5 van der Vaart and Wellner 5 By an adaptation of a theorem due to Talagrand (987a) (see Theorem in van der Vaart and Wellner (996)) a class F is weak Glivenko- Cantelli if and only if the class ~ F is weak Glivenko-Cantelli and sup Pnjf fj!0 ~ f2f in outer probability. Construct a pointwise separable version" ~ Fi for each of the classes F i. The classes ~ Fi possess enough measurability tomake the preceding argument work; in particular pointwise separable version" in the above sense is sufficient for the nearly linearly supremum measurable hypothesis of Giné and inn (984) for both F ;::: ;F k and '(F ;::: ;F k ). Thus the class '( ~ F ;::: ; ~ Fk ) is Glivenko-Cantelli for P. Now by Lemma 2 there exists for every ffl>0affi>0 such that Pnjf j ~ fj j < ffi k ; j =;::: ;k; implies Pnj'(f ;::: ;f k ) '( f ~ ;::: ; f ~ k )j<ffl: The theorem follows. 2 Lemma 2. Suppose that ' : K! R is continuous and K ρ R k is compact. Then for every ffl > 0 there exists ffi > 0 such that for all n and for all a ;::: ;a n ;b ;::: ;b n 2K ρr k implies n n nx i= nx i= ka i b i k <ffi j'(a i ) '(b i )j <ffl: Here k kcan be any norm on R k ; in particular it can be kxk r = ψ k X i= jx i j r! =r r 2 [; ) orkxk max»i»k jx i j for x =(x ;::: ;x k )2 R k. Proof. Let U n be uniform on f;::: ;ng, and set X n = a Un, Y n = b Un. Then we can write and n nx i= n nx i= ka i b i k = EkX n Y n k j'(a i ) '(b i )j = Ej'(X n ) '(Y n )j : ;

6 6 Glivenko-Cantelli preservation theorems Hence it suffices to show that for every ffl>0 there exists ffi>0 such that for all (X; Y ) random vectors in K ρ R k, EkX Y k <ffi implies Ej'(X) '(Y )j <ffl: Suppose not. Then for some ffl > 0 and for all m = ; 2;::: there exists (X m ;Y m ) such that EkX m Y m k < m ; Ej'(X m) '(Y m )j ffl: But since f(x m ;Y m )g is tight, there exists (X m 0;Y m 0)! d (X; Y ). Then it follows that EkX Y k = lim EkX m 0 m! 0 Y m0k =0 so that X = Y a.s., while on the other hand 0=Ej'(X) '(Y)j= lim Ej'(X m 0 m 0) '(Y m0)j ffl>0:! This contradiction means that the desired implication holds. 2 Another potentially useful preservation theorem is one based on building up Glivenko-Cantelli classes from the restrictions of a class of functions to elements of a partition of the sample space. The following theorem is related to the results of Van der Vaart (996) for Donsker classes. Theorem 4. Suppose that F is a class of functions on (X ; A;P), and fx i g is a partition of X : [ X i= i = X, X i X j = ; for i 6= j. Suppose that F j ff Xj : f 2 Fg is P Glivenko-Cantelli for each j, and F has an integrable envelope function F. Then F is itself P Glivenko-Cantelli. Proof. Since it follows that f = f X E Λ kpn P k F» Xj = X j= j= X j= f Xj ; E Λ kpn P k Fj! 0 by the dominated convergence theorem since each term in the sum converges to zero by the hypothesis that each F j is P Glivenko-Cantelli, and we have E Λ kpn Pk Fj»E Λ Pn(F Xj )+P(F Xj )»2P(F Xj ) where P j= P (F X j )=P(F)<. 2

7 van der Vaart and Wellner 7 3. Preservation of the Uniform Glivenko - Cantelli Property. We say that F is a strong uniform Glivenko-Cantelli class if for all ffl>0 sup PrP Λ sup kpm P k F >ffl!0 as n! P 2P(X ;A) m n where P(X ; A) is the set of all probability measures on (X ; A). For x = (x ;::: ;x n ) 2 X n, n = ;2;:::, and r 2 (0; ), we define on F the pseudo-distances ( ) X n r ^ e x;r (f;g) = n jf(x i ) g(x i )j r ; i= e x; (f;g) = max jf(x i) g(x i )j; f; g 2F:»i»n Let N(ffl; F;e x;r ) denote the ffl covering numberof(f;e x;r ), ffl>0. Then define, for n =;2;:::, ffl>0, and r 2 (0; ], the quantities N n;r (ffl; F) = sup x2x n N(ffl; F;e x;r ) : Theorem 5. (Dudley, Giné, and inn (99)). Suppose that F is a class of uniformly bounded functions such that F is image admissible Suslin. Then the following are equivalent: (a) F is a strong uniform Glivenko-Cantelli class. (b) log N n;r (ffl; F)! 0 for all ffl>0 n for some (all) r 2 (0; ]. For the definition of the image admissible Suslin property see Dudley (984), sections 0.3 and.. The following theorem gives natural sufficient conditions for preservation of the uniform Glivenko-Cantelli theorem. Theorem 6. Suppose that F ;::: ;F k are classes of uniformly bounded functions on (X ; A) such that F ;::: ;F k are image-admissible Suslin and strong uniform Glivenko-Cantelli classes. Suppose that ' : R k! R is continuous. Then H '(F ;::: ;F k ) is a strong uniform Glivenko-Cantelli class. Proof. It follows from Lemma 2 that for any ffl>0 there exists ffi>0 such that for any f j ;g j 2F j,j=;::: ;k, x2x n, e x; (f j ;g j )» ffi ; j =;::: ;k k implies that e x; ('(f ;::: ;f k );'(g ;::: ;g k ))» ffl:

8 8 Glivenko-Cantelli preservation theorems It follows that and hence that N n; (ffl; H)» n log N n;(ffl; H)» ky j= kx j= N n; ( ffi k ; F j) ; n log N n;( ffi k ; F j)! 0 where the convergence follows from part (b) of Theorem 5 with r =.If weshow that H = '(F) is image admissible Suslin, then the conclusion follows from (b) implies (a) in Theorem 5. We give a proof of a slightly stronger statement in the following lemma. 2 Lemma 3. If F ;::: ;F k are image admissible Suslin via (Y i ; S i ;T i ), i = ;::: ;k, and ffi : R k! R is measurable, then '(F ;::: ;F k ) is image admissible Suslin via ( Q k i= Y i; Q k i= S i;t) for T (y ;::: ;y k )='(T y ;::: ;T k y k ): Proof. There exist Polish spaces i and measurable maps g i : i! Y i which are onto. Then g : k! Y Y k defined by g(z ;::: ;z k )=(g (z );::: ;g k (z k )) is onto and measurable for S S k. So ( Q i Y i; Q i S i) is Suslin. It suffices to check that T is onto and ψ defined by the map ψ(x; y ;::: ;y k )='(T y (x);::: ;T k y k (x)) is measurable. Obviously T is onto, and ψ is measurable because each map (x; y i ) 7! T i y i (x) is measurable on X Y i and hence on X Y Y k, and hence (x; y ;::: ;y k )7! (T y (x);::: ;T k y k (x)) is measurable from X Yto R k Nonparametric Maximum Likelihood Estimation: a general result. Now we prove a general result for nonparametric maximum likelihood estimation in a class of densities. The main proposition in this section is related to results of Pfanzagl (988) and Van de Geer (993), (996). Suppose that P is a class of densities with respect to a fixed ff finite measure μ on a measurable space (X ; A). Suppose that X ;::: ;X n are i.i.d. P 0 with density p 0 2P. Let bp n argmax Pn log p:

9 van der Vaart and Wellner 9 For 0 <ff», let ' ff (t) =(t ff )=(t ff + ) for t 0, '(t) = for t<0. Then ' ff is bounded and continuous for each ff 2 (0; ]. For 0 < fi < define h 2 fi(p; q) p fi q fi dμ : Note that h =2 (p; q) h(p; q) is the Hellinger distance between p and q, and by Hölder's inequality, h fi (p; q) 0 with equality if and only if p = q a.e. μ. Proposition 3. Suppose that P is convex. Then h 2 (bp ff=2 n;p 0 )»(Pn P 0 ) ' ff ( bp n ) p 0 In particular, when ff =wehave, with ' ', h 2 (bp n ;p 0 ) h 2 =2 (bp n;p 0 )»(Pn P 0 ) : '( bp n ) : p 0 Corollary. Suppose that f'(p=p 0 ): p2pgis a P 0 Glivenko-Cantelli class. Then for each 0<ff», h ff=2 (bp n ;p 0 )! a:s: 0. Proof of Proposition 3. It follows from convexity of Pand convexity of t!' ff (=t) that () 0» Pn' ff (bp n =p 0 )=(Pn P 0 )' ff (bp n =p 0 )+P 0 ' ff (bp n =p 0 ); see van der Vaart and Wellner (996) page 330, and Pfanzagl (988), pages Now we show that P 0 ' ff (p=p 0 )= p ff p ff 0 (2) p ff +p ff dp 0» p fi 0 p fi dμ 0 for fi = ff=2. Note that this holds if and only if or But his holds if +2 p ff p ff 0 +pffp 0dμ» + p fi 0 p fi dμ 2 With fi = ff=2, this becomes p fi 0 p fi dμ ; p ff p ff 0 + pff p 0dμ : p fi 0 p fi 2 pff p 0 p ff 0 + pff : 2 (pff 0 + p ff ) p ff=2 0 p ff=2 = p p ff 0 pff ; and this holds by the arithmetic mean - geometric mean inequality. Thus (2) holds. Combining (2) with () yields the claim of the proposition. 2

10 0 Glivenko-Cantelli preservation theorems 5. Example: a result of Schick and Yu. Our goal in this section is to give another proof of the consistency result of Schick and Yu (999) for the Non-Parametric Maximum Likelihood Estimator (NPMLE) b Fn for mixed case" interval censored data. Our proof is based on the inequality of the preceding section, and is similar in spirit to results of Van de Geer (993), (996). Suppose that Y is a random variable taking values in R + = [0; ) with distribution function F 2 F = fall df's F on R + g. Unfortunately we are not able to observe Y itself. What we do observe is a vector of times T K =(T K; ;::: ;T K;K ) where K, the number of times, is itself random, and the interval (T K;j ;T K;j ] into which Y falls (with T K;0 0, T K;K+ ). More formally, we assume that K is an integer-valued random variable, and T = ft k;j ;j =;::: ;k;k =;2;:::g, is a triangular array of potential observation times", and that Y and (K; T) are independent. Let X = ( K;T K ;K), with a possible value x = (ffi k ;t k ;k), where k = ( k; ;::: ; k;k ) with k;j = (Tk;j ;T k;j ](Y ), j = ; 2;::: ;k +, and T k is the kth row of the triangular array T. Suppose we observe n i.i.d. copies of X; X ;X 2 ;::: ;X n, where X i = ( (i) ;T (i) ;K (i) ), i = K (i) K (i) ; 2;::: ;n. Here (Y (i) ;T (i) ;K (i) ), i = ; 2;::: are the underlying i.i.d. copies of (Y;T;K). We first note that conditionally on K and T K, the vector K has a multinomial distribution: where ( KjK; T K ) ο Multinomial K+ (; F K ) F K (F (T K; );F(T K;2 ) F (T K; );::: ; F(T K;K )) : Suppose for the moment that the distribution G k of (T K jk = k) has density g k and p k P (K = k). Then a density ofxis given by (3) p F (x) p F (ffi k ;t k ;k)= k+ Y j= where t k;0 0, t k;k+. In general, (4) p F (x) p F (ffi k ;t k ;k) = = (F (t k;j ) F (t k;j )) ffi k;j g k (t k )p k k+ Y j= k+ X j= (F (t k;j ) F (t k;j )) ffi k;j ffi k;j (F (t k;j ) F (t k;j )) is a density of X with respect to the dominating measure ν where ν is determined by the joint distribution of (K; T), and it is this version of the

11 van der Vaart and Wellner density of X with which we will work throughout the rest of the paper. Thus the log-likelihood function for F of X ;::: ;X n is given by where n l n(f jx) = n nx i= = Pnm F m F (X) = X K (i) + j= K+ X j= K+ X j= (i) K;j log F (T (i) (i) ) F (T K (i) ;j K ) (i) ;j K;j log (F (T K;j ) F (T K;j )) K;j log ( F K;j ) and where we have ignored the terms not involving F.We also note that, with P 0 P F0, 0 P 0 m F (X) =P K+ X F 0;K;j log ( F K;j ) A : j= The Nonparametric Maximum Likelihood Estimator (NPMLE) b Fn is the distribution function b Fn (t) which puts all its mass at the observed time points and maximizes the log-likelihood l n (F jx). It can be calculated via the iterative convex minorant algorithm proposed in Groeneboom and Wellner (992) for case 2 interval censored data. By Proposition 3 with ff = and ' ' as before, it follows that h 2 (pf bn ;p F0 )»(Pn P 0 ) '(pf bn =p F0 ) where ' is bounded and continuous from R to R. Now the collection of functions G fp F : F2Fg is easily seen to be a Glivenko-Cantelli class of functions: this can be seen by first applying Theorem 4 to the collections G k, k = ; 2;::: obtained from G by restricting to the sets K = k. Then for fixed k, the collections G k = fp F (ffi;t k ;k): F 2Fgare P 0 Glivenko-Cantelli classes since F is a uniform Glivenko-Cantelli class, and since the functions p F are continuous transformations of the classes of functions x! ffi k;j and x! F (t k;j ) for j =;::: ;k+, and hence G is P Glivenko-Cantelli by Theorem 3. Note that the single function p F0 is trivially P 0 Glivenko-Cantelli since it is uniformly bounded, and the single function (=p F0 ) is also P 0 GC since P 0 (=p F0 ) <. Thus by Proposition 2 with g = (=p F0 ) and F = G = fp F : F 2Fg,itfollows that G 0 fp F =p F0 : F 2Fgis P 0 Glivenko- Cantelli. Finally another application of Theorem 3 shows that the collection H fl (p F =p F0 ): F 2Fg=f'(p F =p F0 ): F 2Fg

12 2 Glivenko-Cantelli preservation theorems is also P 0 -Glivenko-Cantelli. When combined with Proposition 3, this yields the following theorem: Theorem 7. The NPMLE b Fn satisfies h(p b F n ;p F0 )! a:s: 0 : To relate this result to a recent theorem of Schick and Yu (999), it remains only to understand the relationship between their L (μ) and the Hellinger metric h between p F and p F0. Let B denote the collection of Borel sets in R. OnBwe define measures μ and eμ as follows: For B 2B, (5) and (6) μ(b)= eμ(b) = X k= X k= P (K = k) P (K = k) k kx j= kx j= P (T k;j 2 BjK = k) ; P (T k;j 2 BjK = k) : Let d be the L (μ) metric on the class F; thus for F ;F 2 2F, d(f ;F 2 )= jf (t) F 2 (t)jdμ(t) : The measure μ was introduced by Schick and Yu (999); note that μ is a finite measure if E(K) <. Note that d(f ;F 2 ) can also be written in terms of an expectation as: 2 X d(f ;F 2 )=E (K;T ) 4 K fi fi fi 3 (7) (F (T K;j ) F 2 (T K;j ) fi5 : j= As Schick and Yu (999) observed, consistency of the NPMLE b Fn in L (μ) holds under virtually no further hypotheses. Theorem 8. (Schick and Yu). Suppose that E(K) <. Then d( b Fn ;F 0 )! a:s: 0. Proof. We will show that Theorem 8 follows from Theorem 7 and the following Lemma. Lemma 4. ρ 2 Proof. We know that j b Fn F 0 jd~μff 2» h 2 (p b F n ;p F 0 ): h 2 (p b F n ;p F0 )»d TV (p b F n ;p F0 )» p 2h(p b F n ;p F0 )

13 where, with y k;0 =, y k;k+ =, while Note that h 2 (p b F n ;p F0 )= X k= d TV (p b F n ;p F0 )= j= P (K = k) X k= van der Vaart and Wellner 3 k+ X f[ Fn b (y k;j ) Fn b (y k;j )] =2 j= [F 0 (y k;j ) F 0 (y k;j )] =2 g 2 dg k (y) P (K = k) k+ X j[ Fn b (y k;j ) Fn b (y k;j )] j= [F 0 (y k;j ) F 0 (y k;j )]jdg k (y) : k+ X j[ Fn b (y k;j ) Fn b (y k;j )] [F 0 (y k;j ) F 0 (y k;j )]j = k+ X j= j( b Fn F 0 )(y k;j ;y k;j ]j max j Fn b (y k;j ) F 0 (y k;j )j ;»j»k+ so integrating across this inequality with respect to G k (y) yields k+ X j[ Fn b (y k;j ) Fn b (y k;j )] [F 0 (y k;j ) F 0 (y k;j )]j dg k (y) j= max»j»k k kx j= j b Fn (y k;j ) F 0 (y k;j )j dg k;j (y k;j ) j b Fn (y k;j ) F 0 (y k;j )j dg k;j (y k;j ) : By multiplying across by P (K = k) and summing over k, this yields and hence (8) d TV (p b F n ;p F0 ) h 2 (p b F n ;p F0 ) 2 ρ jb Fn F 0 jd~μ; jb Fn F 0 jd~μff 2 : The measure ~μ figuring in Lemma 4 is not the same as the measure μ of Schick and Yu (999) because of the factor =k. Note that this factor means that the measure ~μ is always a finite measure, even if E(K) =. It is clear that ~μ(b)» μ(b) 2

14 4 Glivenko-Cantelli preservation theorems for every Borel set B, and that μ ffiffi ~μ. The following lemma (Lemma 2.2 of Schick and Yu (999)) together with Lemma 4 shows that Theorem 7 implies the result of Schick and Yu once again: Lemma 5. Suppose that μ and ~μ are two finite measures, and that g; g, g 2 ;::: are measurable functions with range in [0; ]. Suppose that μ is absolutely continuous with respect to ~μ. Then R jg n gjd~μ! 0 implies that R jg n gjdμ! 0. Proof. Write jg n gjdμ = jg n gj dμ d~μ d~μ and use the dominated convergence theorem applied to a.e. convergent subsequences Example: generalizing the result of Schick and Yu. Our goal in this section is to give a generalization of the consistency result of Schick and Yu (999). Suppose that Y is a random variable taking values in Y. Suppose that Y has distribution Q on the measurable space (Y; B). Unfortunately we are not able to observe Y itself. What we do observe is a vector of random sets C K = (C K; ;::: ;C K;K ) where K, the number of sets is itself random, and the set fc K;j g K form a partition of Y: j= C K;j C K;j 0 = ; if j 6= j 0 and [ K j= C K;j = Y. More formally, we assume that K is an integer-valued random variable, and C = fc k;j ;j =;::: ;k;k =;2;:::g, is a triangular array of random sets", and that Y and (K; C) are independent. Let X =( K;C K ;K), with a possible value x =(ffi k ;c k ;k), where k =( k; ;::: ; k;k ) with k;j = Ck;j (Y ), j =;2;::: ;k, and C k is the k th row of the triangular array C. Suppose we observe n i.i.d. copies of X; X ;X 2 ;::: ;X n, where X =( (i) i ;C (i) ;K K (i) K (i) ), i =;2;::: ;n. Here (i) (Y (i) ;C (i) ;K (i) ), i =;2;::: are the underlying i.i.d. copies of (Y;C;K). We first note that conditionally on K and C K, the vector K has a multinomial distribution: ( KjK; C K ) ο Multinomial K (;Q K ) where Q K (Q(C K; );Q(C K;2 );::: ;Q(C K;K )) :

15 van der Vaart and Wellner 5 Suppose for the moment that the distribution G k of (C K jk = k) has density g k and p k P (K = k). Then a density ofxis given by (9) In general, (0) p Q (x) p Q (ffi k ;c k ;k)= p Q (x) p Q (ffi k ;c k ;k)= ky j= ky j= Q(c k;j ) ffi k;j g k (c k )p k : Q(c k;j ) ffi k;j = kx j= ffi k;j Q(c k;j ) is a density of X with respect to the dominating measure ν where ν is determined by the joint distribution of (K; C), and it is this version of the density of X with which we will work throughout the rest of the paper. Thus the log-likelihood function for Q of X ;::: ;X n is given by where n l n(qjx) = n m Q (X) = nx K X (i) i= j= KX j= (i) K;j log Q(C(i) K (i) ;j )=P nm Q K;j log Q(C K;j ) and where we have ignored the terms not involving Q. We also note that, with P 0 P Q0, 0 X P 0 m Q (X) =P K Q 0 (C K;j ) log Q(C K;j ) A : j= The Nonparametric Maximum Likelihood Estimator (NPMLE) b Qn is a probability measure b Qn which maximizes the log-likelihood l n (QjX). Here we will bypass the many interesting existence, characterization, and computational issues connected with the NPMLE b Qn, and focus instead on the issue of consistency once we have the NPMLE in hand. By Proposition 3 with ff = it follows that h 2 (p b Q n ;p Q0 )»(Pn P 0 ) '(p b Q n =p Q0 ) where ' is bounded and continuous. Now the collection of functions G fp Q : Q2Qg where Q is the collection of all probability distributions on (Y; B), is easily seen to be a Glivenko-Cantelli class of functions: this can be seen by first applying Theorem 4 to the collections G k, k =;2;::: obtained from G by restricting to the sets K = k. The next step is to show that the collections G k fp Q ( ; ;k): Q2Qg :

16 6 Glivenko-Cantelli preservation theorems are P 0 Glivenko-Cantelli for each k. To this end, suppose that (Y; B) bea measurable space and let C be a universal GC - class of sets in Y (i.e. Q GC for every probability measure Q on (Y; B)). Let (T;T ;G) be a probability space, and let t 7! C t be a map with values in C. Lemma 6. Suppose that the collection fft : y 2 C t g : y 2Ygis G GC. Then the collection of functions t 7! Q(C t ) where Q ranges over all probability measures Q on Y is G GC. Remark: By Assouad (983), Proposition 2.2, page 246, together with Corollary.0, page 24, it follows that fft 2 T : y 2 C t g : y 2Ygis a VC-class of subsets of T if and only if fc t : t 2 T g is a VC-class of subsets of Y.Thus if we assume that all the subsets C = C t arising in the partitions are elements of a VC-class C, then, subject to being suitably measurable, the hypothesis of Lemma 6 is satisfied (and the class in question is even universal Glivenko-Cantelli). Proof. For Q the Dirac measure at y (unit mass at y), the function t 7! Q(C t ) becomes t 7! Ct (y) : By assumption, the set of all such functions, with y ranging over Y, is G GC. Then so is the set of all functions t 7! m mx i= Ct (y i ); y ;::: ;y m 2Y;m2 N: Let Y ;::: ;Y m be an i.i.d. sample from a given Q. Because C is Q GC we have fi sup fi mx fi fy i 2 Cg P(C) fi!0 a.s. Λ : m C2C i= Then there exists a sequence y ;y 2 ;::: in Y such that and consequently sup fi m C2C fi fi supfi t2t m mx fi fy i 2 Cg P(C) fi!0; i= mx fi fy i 2 C t g P(C t ) fi!0: i= It follows that the maps t! Q(C t ) are the (uniform) limits of sequences of functions from a G GC class. Hence they are G GC. 2 Note on measurability: It is assumed implicitly that the sets ft : y 2 C t g are measurable in T for every y 2 Y. The proof shows that the maps t 7! P (C t ) are then also measurable.

17 van der Vaart and Wellner 7 It follows that for fixed k, the collections G k = fp Q (ffi;c k ;k) : Q 2 Qg are P 0 -Glivenko-Cantelli, and since the functions p Q are continuous transformations of the classes of functions x! ffi k;j and x! Q(c k;j ) for j = ;::: ;k, and hence G is P Glivenko-Cantelli by Theorem 3. Note that the single function p Q0 is trivially P 0 Glivenko-Cantelli since it is uniformly bounded. Now another application of Theorem 3 shows that the collection H fl (p Q =p Q0 ): Q2Qg=f'(p Q =p Q0 ): Q2Qg is also P 0 -Glivenko-Cantelli. When combined with Proposition 3, this yields the following theorem. Theorem 9. If all C K;j 2 C, a VC collection of subsets of X, then the NPMLE b Qn satisfies h(p b Q n ;p Q0 )! a:s: 0 : To obtain a statement analogous to the theorem of Schick and Yu (999), let ± denote a sigma algebra for the space C of subsets: we are assuming that P ( K j= [C K;j 2C]) =. Thus (C; ±) is a measurable space. On ± we define the measure μ as follows: () Note that (2) μ(b) = X k= d TV (p b Q n ;p Q0 ) = = P (K = k) X k= kx j= P (K = k) P (C k;j 2 BjK = k) ; B 2 ±: kx j= j b Qn (c k;j ) Q 0 (c k;j )jdg k (c) j b Qn (c) Q 0 (c)j dμ(c) d( b Qn ;Q 0 ): Here is a generalization of the theorem of Schick and Yu (999). Theorem 0. The NPMLE b Qn satisfies d( b Qn ;Q 0 )! a:s: 0. Proof. Theorem 0 follows from Theorem 9, (2) and the observation that we then have d( b Qn ;Q 0 )=d TV (p b Q n ;p Q0 )» p 2h(p b Q n ;p Q0 ): Example. (Mixed case interval censoring in R 2 ). Suppose that Y ο Q 0 takes values in R +2, and the partitions fc K;j g K j= are obtained as the natural rectangles formed from two increasing collections 0» S K;» :::» S K;K and 0» T K2;» :::» T K2;K on the two coordinate axes. Then K = (K + )(K 2 + ), and all the elements of the partitions are 2

18 8 Glivenko-Cantelli preservation theorems elements of the VC-class of rectangles in R 2. It follows that the NPMLE bq n of Q 0 satisfies s;t2r +2 j b Qn (s; t] Q 0 (s; t]jdμ(s; t)! 0 a.s. Λ : Acknowledgments: We owe thanks to Evarist Giné and Joel inn for suggesting the proof given for Lemma. References Assouad, P. (983). Densité et dimension. Annales de l'institut Fourier 33, Dudley, R. M. (984). A Course on Empirical Processes. Ecole d'été de probabilités de St.-Flour, 982. Lecture Notes in Mathematics 097, -42, Springer, Berlin. Dudley, R. M. (998a). Consistency of M-estimators and one-sided bracketing." In Proceedings of the First International Conference on High- Dimensional Probability, E. Eberlein, M. Hahn, and M. Talagrand, editors. Birkhäuser, Basel. Dudley, R. M. (998b). Personal communication. Dudley, R. M., Giné, E., and inn, J. (99). Uniform and universal Glivenko-Cantelli classes. J. Theoretical Probab. 4, Giné, E., and inn, J. (984). Some limit theorems for empirical processes. Ann. Probability 2, Groeneboom, P. and Wellner, J. A. (992). Information Bounds and Nonparametric Maximum Likelihood Estimation. DMV Seminar Band 9, Birkhäuser, Basel. Pfanzagl, J. (988). Consistency of maximum likelihood estimators for certain nonparametric families, in particular: mixtures." J. Statist. Planning and Inference 9, Schick, A. and Yu, Q. (999), Consistency of the GMLE with Mixed Case Interval-Censored Data," Scand. J. Statist., to appear. Talagrand, M. (987a). Measurability problems for empirical measures." Ann. Probability 5, Talagrand, M. (987b). The Glivenko-Cantelli problem." Ann. Probability 5, Talagrand, M. (996). The Glivenko-Cantelli problem, ten years later." J. Theor. Prob. 9,

19 van der Vaart and Wellner 9 Van de Geer, S. (993). Hellinger consistency of certain nonparametric maximum likelihood estimators." Ann. Statist. 2, Van de Geer, S. (996). Rates of convergence for the maximum likelihood estimator in mixture models." J. Nonparametric Statist. 6, Van der Vaart, A. W. (996). New Donsker classes. Ann. Probability 24, Van der Vaart, A. W., and Wellner, J. A. (996). Weak Convergence and Empirical Processes with Applications to Statistics, Springer-Verlag, New York. Department of Mathematics Free University De Boelelaan 08a 08 HV Amsterdam The Netherlands aad@cs.vu.nl Department of Statistics University of Washington P.O. Box Seattle, Washington U.S.A. jaw@stat.washington.edu