February 26, 2017 COMPLETENESS AND THE LEHMANN-SCHEFFE THEOREM

February 26, 2017 COMPLETENESS AND THE LEHMANN-SCHEFFE THEOREM Abstract. The Rao-Blacwell theorem told us how to improve an estimator. We will discuss conditions on when the Rao-Blacwellization of an estimator is the best estimator. This is one of the major highlights of this course, and the classical theory of mathematical statistics. 1. Introduction Let F = (f θ ) θ Θ be a family of pdfs, as usual the family may be discrete or continuous. For each θ let Z = Z θ be a random variable with pdf f θ (z). The family F is complete if the condition: (1) E θ u(z) < and E θ u(z) = 0 for all θ Θ implies that for each θ Θ, we have u(z) = 0, P θ -almost surely; that is, (2) P θ (u(z) = 0) = 1 for all θ Θ. It is important to note that it in the definition, it is families which are complete, and the for all quantifier appears in the definition. Thus to chec if a family is complete, we consider any function u with Property (1), and using this property, we have to verify (2). If there is even one u for which (1) is satisfied, but (2) fails, then the family fails to be complete. From the statement of (2), we see that we will encounter some measure-theoretic difficulties. Let F = (f θ ) θ Θ be a family of pdfs. Let X = (X 1,..., X n ) be a random sample and T be a statistic. Here X, and T depend on θ. For each θ Θ, let h θ be the pdf for T. We say that T is complete statistic if the corresponding family H = (h θ ) θ Θ is complete. Let us remar that when considering the completeness of T, it is the family H that is important, not F. 1

2 COMPLETENESS AND THE LEHMANN-SCHEFFE THEOREM Theorem 1 (Lehmann-Schefee). Let X = (X 1,..., X n ) be a random sample from f θ, where θ Θ. Let T be a sufficient statistic for θ and suppose the corresponding family for T is complete. Let g : Θ R. If φ is a function that does not depend on θ and φ(t ) is an unbiased estimator of g(θ), the φ(t ) is the (almost surely) unique MVUE for g(θ). In particular, if Y is any unbiased estimator of g(θ), then E(Y T ) is the MVUE for g(θ). Proof of Theorem 1. First, we show the uniqueness using completeness. Let ψ(t ) be another unbiased estimator of g(θ). Then for all θ Θ, we have E θ φ(t ) = g(θ) = E θ ψ(t ), so that E θ (ψ(t ) φ(t )) = 0. Consider the function u(t) = ψ(t) φ(t). Since T is a complete family for every θ Θ, we have that u(t ) = 0, P θ -almost surely, from which we deduce that ψ(t ) = φ(t ) almost surely. Thus there is at most one unbiased estimator of g(θ) which is a function of T. Next, we show that it is a MVUE using the Rao-Blacwell theorem and the sufficiency of T. If Y is any unbiased estimator of g(θ), the Rao-Blacwell theorem gives that E(Y T ) is an unbiased estimator of g(θ) and Var θ (E(Y T )) Var θ (Y ) furthermore, since it is a conditional expectation, it is a function of T. Hence it must actually be φ(t ), and we have Var θ (φ(t )) = Var θ (E(Y T )) Var θ (Y ). Thus Theorem 1 gives us a powerful way of finding MVUE if we now that the corresponding family for a sufficient statistic is complete. It is not easy to show completeness. 2. Examples from the Binomial family Exercise 2. Fix n 1. Show that the Binomial family, given by f p () = p (1 p) n, where is an integer, with 0 n, and p (0, 1) is complete.

COMPLETENESS AND THE LEHMANN-SCHEFFE THEOREM 3 Solution. Let Z Bin(n, p). Assume that u : {0, 1,..., n} R is such that E p (u(z)) = 0 for all p (0, 1). We have that for all p (0, 1), 0 = E p (u(z)) = u() p (1 p) n =0 ( p ). = (1 p) n u() 1 p Set g(q) := =0 u() q. =0 We have that g is a polynomial in q of degree n, which means it must have at most n distinct roots, unless c := u() ( n ) = 0 for all. Since for all q (0, ), we have that g(q) = 0, we can conclude that c = 0, and infer that u() = 0, for all. Exercise 3. Let X = (X 1,..., X n ) be a random sample, where X 1 Bern(p), p (0, 1). Let X be the usual sample mean. Show that X is the MVUE for p. Solution. We already now that the sample sum T = X 1 + + X n is sufficient and T B(n, p). We now that X 1 is an unbiased estimator for p. By Theorem 1, we have that E(X 1 T ) = X is the MVUE. Exercise 4. Referring to Exercise 3, find the MVUE for p 2. Solution. Here we will try to guess. Observe that X 2 = T 2 /n 2 is a good place to start, since it is at least consistent, and X is the MVUE for p. Note that E X 2 = Var( X) + (E X) 2 = p(1 p)/n + p 2 = p/n + p 2 (1 1/n). So consider 1 φ(t) := n(n 1) (t2 t). Our calculations gives that Eφ(T ) = p 2, and by Exercise 3, T is a complete and sufficient statistic, thus φ(t ) is the MVUE. Exercise 5. Do Exercise 4, by considering Y = X 1 X 2 and E(Y T ). Exercise 6. Do Exercise 4, by solving Eφ(T ) = p 2, for φ by brute force. Solution. We want to solve for φ, using the equation E p (φ(t )) = φ() p (1 p) n = p 2. =0

4 COMPLETENESS AND THE LEHMANN-SCHEFFE THEOREM Again, with the change of variables q = p/(1 p) = 1/(1 p) 1 we obtain the relation that for all q (0, ), we have n 2 2 φ() q = q 2 (q + 1) n 2 = q j+2. j =0 Thus we equate coefficients, to obtain that φ(0) = φ(1) = 0 and for 2 n, we have φ() = from which we deduce that φ() = j=0 2, 2 ( 1) n(n 1). Note also that formula remains valid for = 0, 1. 3. Examples from the Poisson distribution Exercise 7. Show that the Poisson family, given by λ λn f λ (n) = e n!, where n is a nonnegative integer and θ (0, ), is complete. Solution. Let Z P oi(λ). Assume that u : {0, 1, 2...} R is such that E λ u(z) = 0 for all λ > 0. We need to show that u = 0. We have that for all λ > 0, E λ u(z) = e λ n=0 u(n) λ n = 0, n! where the sum is absolutely convergent for all λ > 0. Let a n = u(n)/n! and consider the power series g(x) = a n x n. n=0 Recall, the by the root test, power series have a radius of absolute convergence. Hence in the case of g, the radius is infinite, and furthermore, g(x) = 0 for all x > 0. From our nowledge of power series and analytic functions this is enough to conclude that g(x) = 0 for all x R, and that a n = 0 for all n 0. Thus u(n) = 0.

COMPLETENESS AND THE LEHMANN-SCHEFFE THEOREM 5 One way to see why g = 0 is to let c (0, ) and tae a Talyor series expansion of g at the point c, then g(x) = c n (x c) n, n=0 where c n = gn (c) n! = 0, since c > 0. Exercise 8. Let X = (X 1,..., X n ) be a random sample, where X 1 P oi(λ) Let X be the usual sample mean. Show that X is a MVUE for λ. Solution. We already now that the sample sum T = X 1 + + X n is sufficient. We now that T P oi(nλ), since the sum of independent Poisson random variables is again Poisson. Thus it follows from Exercise 7 that T is complete. We also now that the sample mean X = T/n is a unbiased estimator of λ, so by Theorem 1 it must be the MVUE. Exercise 9. Referring to Exercise 8, find the MVUE for e λ. Hint: you already did this without officially nowing. Just apply Theorem 1 to what you already calculated using the Rao-Blacwell theorem. Exercise 10. Referring to Exercise 8, find the MVUE for λ 2. Solution. We will just try to find a unbiased estimator of λ 2 by inspection. We now that X 2 is at least a consistent estimator of λ 2, so we start here. Let T = X 1 + + X n be the sample sum. Note that Var(T ) = n Var(X 1 ) = nλ. Thus E( X 2 ) = Var( X) + (E X) 2 = λ/n + λ 2. So consider Y := X 2 X/n. From our calculations we now that Y is an unbiased estimator for λ 2. Furthermore, since Y is a function of the sufficient statistic T, Y = (T/n) 2 T/n 2 T (T 1) =. n 2 we have by Theorem 1, it must be the MVUE. Exercise 11. Referring to Exercise 8, find E(X 1 X 2 T ), where T is the sample sum. Solution. We have that X 1 X 2 is an unbiased estimator for λ 2, thus by Theorem 1, and Exercise 10, we now that E(X 1 X 2 T ) = X 2 X/n.

6 COMPLETENESS AND THE LEHMANN-SCHEFFE THEOREM Exercise 12. Let Z P oi(λ) Let r 1, set g r (n) = n(n 1)(n 2) (n r + 1), where the product contains r terms. Show that Eg r (Z) = λ r. Exercise 13. Referring to Exercise 8, find the MVUE for λ r, where r is a positive integer. 4. Examples from the uniform family Exercise 14. Let X = (X 1,..., X n ) be a random sample, where X 1 U(0, θ), where θ > 0. Find the MVUE for θ. Solution. Consider M := max {X 1,..., X n }. First, we show that M is a sufficient statistic for θ. Let x (0, ) n. We now that L(x; θ) = 1 n 1[0 < x θ n i < θ]. i=1 Observe that if m = max {x 1,..., x n }, we have n 1[x i < θ] = 1[m < θ]. i=1 Hence L(x; θ) = g(m; θ) := 1 1[m < θ], θn and it follows from the Neyman factorization theorem that M is sufficient. Next, we show that M is complete. For this, we will need to now the distribution of M. An easy computation gives that M has pdf g(m) = nmn 1 1[0 < m < θ]. θ n We need to show that this family is complete. Suppose E θ u(m) = 0 for all θ, then θ u(m) nmn 1 dm = 0. 0 θ n This is equivalent to the statement that for all θ > 0, we have θ 0 u(m)m n 1 dm = 0. Tae a derivative on both sides with respect to θ, we have by the fundamental theorem of calculus tells us that if u is continuous, then u(θ)θ n 1 = 0,

COMPLETENESS AND THE LEHMANN-SCHEFFE THEOREM 7 from which we deduce that u = 0. If u is not continuous, we have to use a measure-theoretic version of the fundamental theorem of calculus, from which we can conclude that u = 0 Lebesgue-almost everywhere, from which we can deduce that u(z) = 0 P θ -almost surely, for every θ > 0. Finally, another easy calculation gives that E θ M = n θ. Hence, if n+1 we set Y = n+1 M, we have by Theorem 1 that M is the MVUE. n Exercise 15. Referring to Exercise 14, find the MVUE for θ 2. 5. Examples from the normal family Proposition 16. The normal family with nown variance σ 2, given by f(x; µ) = 1 σ (x µ) 2 2π e 2σ, where x R and µ R is complete. The proof of Proposition 16 requires nowledge of Laplace transforms, which is beyond the scope of this course. In general, showing that a family is normal may require advanced techniques from analysis. We will not be able to provide all the details, but we will be able to motivate why such statements are true. Setch Proof of Proposition 16. Let X N(µ, σ 2 ). Suppose that for all µ R, we have E µ u(x) = 1 u(x) σ (x µ) 2 2π e 2σ dx = 0. We have to show that u(x) = 0, P µ -almost surely. After some algebra, we find that the above is equivalent to the statement that for all µ R, we have u(x)e x2 /2σ e xµ/σ dx = 0. Setting g(x) := u(x)e x2 /2σ, for all µ R, we have ĝ(µ) := g(x)e xµ/σ dx = 0. The above expression may remind you of a Laplace transform. In the case that u is continuous, by appealing to theory of Laplace transforms, we can deduce that g = 0, from which we can deduce that u = 0. Exercise 17. Let X = (X 1,..., X n ) be a random sample, where X i N(µ, 1), where µ is unnown. Show that X is the MVUE for µ.

8 COMPLETENESS AND THE LEHMANN-SCHEFFE THEOREM Solution. We already now that X is a sufficient statistic for µ. We also now that X N(µ, 1/n) is a complete statistic by Proposition 16. Of course X is an unbiased estimator, so Theorem 1 tells us it is a MVUE. Exercise 18. Referring to Exercise 17. Find the MVUE for µ 2. Solution. As usual, we will try to modify X 2. We now that 1/n = Var( X) = E X 2 (E X) 2 = E X 2 µ 2. Thus set Y := X 2 1/n. Clearly, Y is an unbiased estimator for µ 2, and also it is a function of the sufficient statistic X, by Theorem 1, we are done.