Statistics 581 Revision of Section 4.4: Consistency of Maximum Likelihood Estimates Wellner; 11/30/2001

Transcription

1 Statistics 581 Revision of Section 4.4: Consistency of Maximum Likelihood Estimates Wellner; 11/30/2001 Some Uniform Strong Laws of Large Numbers Suppose that: A. X, X 1,...,X n are i.i.d. P on the measurable space (X, A). B. For each θ Θ, f(x, θ) isameasurable, real-valued function of x, f(,θ) L 1 (P ). Let F = {f(,θ):θ Θ}. Since f(,θ) L 1 (P ) for each θ, g(θ) Ef(X, θ) = f(x, θ)dp (x) Pf(,θ) exists and is finite. Moreover, by the strong law of large numbers, P n f(,θ) f(x, θ)dp n (x) = 1 n f(x i,θ) n i=1 (0.1) Ef(X, θ) =Pf(,θ)=g(θ). a.s. It is often useful and important to strengthen (0.1) to hold uniformly in θ Θ: (0.2) sup P n f(,θ) Pf(,θ) a.s. 0. Note that the left side in (0.2) is equal to P n P F sup P n f Pf. f F Here is how (0.2) can be used: suppose that we have a sequence θ n of estimators, possibly dependent on X 1,...,X n, such that θ n a.s. θ 0. Suppose that g(θ) iscontinuous at θ 0.Wewould like to conclude that P n f(, θ n )= 1 n (0.3) f(x i, n θ n ) a.s. g(θ 0 ). i=1 The convergence (0.3) does not follow from (0.1); but (0.3) does follow from (0.2): P n f(, θ n ) g(θ 0 ) P n f(, θ n ) g( θ n ) + g( θ n ) g(θ 0 ) supp n f(,θ) g(θ) + g( θ n ) g(θ 0 ) = P n P F + g( θ n ) g(θ 0 ) a.s. 0+0=0. 1

2 The following theorems, due to Le Cam, give conditions on f and P under which (2) holds. The first theorem is a prototype for what are now known in empirical process theory as Glivenko-Cantelli theorems. Theorem 1. Suppose that: (a) Θ is compact. (b) f(x, ) iscontinuous in θ for all x. (c) There exists a function F (x) such that EF(X) < and f(x, θ) F (x) for all x X, θ Θ. Then (0.2) holds; i.e. sup P n f(,θ) Pf(,θ) a.s. 0. The second theorem is a one-sided version of theorem 1 which is useful for the theory of maximum likelihood estimation. Theorem 2. Suppose that: (a) Θ is compact. (b) f(x, ) isupper1inθ for all x. (c) There exists a function F (x) such that EF(X) < and f(x, θ) F (x) for all x X, θ Θ. (d) For all θ and all sufficiently small ρ>0 is measurable in x. Then limsup n sup sup f(x, θ ) P n f(,θ) a.s. sup Pf(,θ)=sup g(θ). We proceed by first proving Theorem 2. Then Theorem 1 will follow as a consequence of Theorem 2. Proof of Theorem 2. Let ψ(x, θ, ρ) sup f(x, θ ). Then ψ is measurable (for ρ sufficiently small), bounded by an integrable function F, and ψ(x, θ, ρ) f(x, θ) as ρ 0 by (b). Thus by the monotone convergence theorem ψ(x, θ, ρ)dp (x) f(x, θ)dp (x) =g(θ). 2

3 Let ɛ>0. For each θ, find ρ θ so that ψ(x, θ, ρ)dp (x) <g(θ)+ɛ. The spheres S(θ, ρ θ )={θ : θ θ <ρ θ } cover Θ, so by (a) there exists a finite sub cover: Θ m j=1s(θ j,ρ θj ). for each θ Θ there is some j, 1 j m, such that θ S(θ j,ρ θj ); hence from the definition of ψ it follows that f(x, θ) ψ(x, θ j,ρ θj ) for all x. Therefore and hence Hence P n f(,θ) P n ψ(,θ j,ρ θj ), sup P n f(,θ) sup P n ψ(,θ j,ρ θj ) 1 j m Pψ(,θ j,ρ θj ) limsup n sup Letting ɛ 0 completes the proof. a.s. sup 1 j m sup 1 j m sup g(θ j )+ɛ g(θ)+ɛ. P n f(,θ) a.s. sup g(θ)+ɛ. Proof of Theorem 1. Since f is continuous in θ, condition (d) of Theorem 2 is satisfied: for any countable set D dense in {θ : θ θ <ρ}, sup f(x, θ )=sup f(x, θ ) θ D where the right side is measurable since it is a countable supremum of measurable functions. Furthermore, g(θ) iscontinuous in θ: lim g(θ) = lim f(x, θ )dp (x) = f(x, θ)dp (x) θ θ θ θ 3

4 by the dominated convergence theorem. Now Theorem 1 follows from Theorem 2 applied to the functions h(x, θ) f(x, θ) g(θ) and h(x, θ): by Theorem 2 applied to {h(x, θ) :θ Θ}, limsup n sup(p n f(,θ) g(θ)) 0 a.s. By Theorem 2 applied to { h(x, θ) :θ Θ}, limsup n sup(g(θ)) P n f(,θ)) 0 a.s. The conclusion of Theorem 1 follows since 0 sup P n f(,θ) g(θ) = sup(p n f(,θ) g(θ)) sup(g(θ) P n f(,θ)). For our application of Theorem 2 to consistency of maximum likelihood, the following Lemma will be useful. Lemma 1. If the conditions of Theorem 2 hold, then g(θ) isupper-semicontinuous: i.e. limsup θ θg(θ ) g(θ). Proof. Since f(x, θ) is upper semicontinuous, i.e. limsup θ θf(x, θ ) f(x, θ) for all x ; liminf θ θ {f(x, θ) f(x, θ )} 0 for all x. Hence it follows by Fatou s lemma that 0 Eliminf θ θ {f(x, θ) f(x, θ )} liminf θ θe {f(x, θ) f(x, θ )} = Ef(X, θ) limsup θ θef(x, θ ); i.e. limsup θ θef(x, θ ) Ef(X, θ) =g(θ). 4

5 Now we are prepared to tackle consistency of maximum likelihood estimates. Theorem 3. (Wald, 1949). Suppose that X, X 1,...,X n are i.i.d. P θ0, θ 0 Θ with density p(x, θ 0 ) with respect to the dominating measure ν, and that: (a) Θ is compact. (b) p(x, ) isupper semi-continuous in θ for all x. (c) There exists a function F (x) such that EF(X) < and f(x, θ) log p(x, θ) log p(x, θ 0 ) F (x) for all x X, θ Θ. (d) For all θ and all sufficiently small ρ>0 sup p(x, θ ) is measurable in x. (e) p(x, θ) =p(x, θ 0 ) a.e. ν implies that θ = θ 0. Then for any sequence of maximum likelihood estimates θ n of θ 0, θ n a.s. θ 0. Proof. Let ρ > 0. The functions {f(x, θ) :θ Θ} satisfy the conditions of theorem 2. But we will apply Theorem 2 with Θ replaced by the subset Then S is compact, and by Theorem 2 S {θ : θ θ 0 ρ} Θ. P θ0 (limsup n sup ) P n f(,θ) sup g(θ) =1 where { } p(x, θ) g(θ) =E θ0 f(x, θ) = E θ0 log p(x, θ 0 ) = K(P θ0,p θ ) < 0 for θ S. Furthermore by the Lemma, g(θ) isupper semicontinuous and hence achieves its supremum onthe compact set S. Let δ = sup g(θ). Then by Lemma it follows that δ<0 and we have ) P θ0 (limsup n sup P n f(,θ) δ =1. 5

6 Thus with probability 1 there exists an N such that for all n>n But sup P n f(,θ) δ/2 < 0. P n f(, θ n ) = sup P n f(,θ) = sup 1 n {l n(θ) l n (θ 0 )} 0. Hence θ n / S for n>n; that is, θ n θ 0 <ρwith probability 1. Since ρ was arbitrary, θ n is a.s. consistent. Remark 3. Theorem 3 is due to Wald (1949). The present writeup is an adaptation of Chapters 16 and 17 of Ferguson (1996). For further Glivenko - Cantelli theorems, see chapter 2.4 of Van der Vaart and Wellner (1996). References: Le Cam, L. (1953). On some asymptotic properties of maximum likelihood estimates and related estimates. Univ. Calif. Publ. in Statist. 1, Ferguson, T. (1996). A Course in Large Sample Theory. Chapman and Hall, London. Van der Vaart, A. W. and Wellner, J. A. (1996). Weak Convergence and Empirical Processes. Springer-Verlag, New York. Wald, A. (1949). Note on the consistency of the maximum likelihood estimate. Ann. Math. Statist. 20,