Lecture 4: UMVUE and unbiased estimators of 0

Transcription

1 Lecture 4: UMVUE and unbiased estimators of Problems of approach 1. Approach 1 has the following two main shortcomings. Even if Theorem or its extension is applicable, there is no guarantee that the bound is sharp, i.e., there may be a UMVUE but it still cannot achieve the Cramér-Rao lower bound. The conditions for Theorem is somewhat strong. Example (a case where Theorem is not applicable) Let X 1,...,X n be iid with from uniform(,θ), where θ > is unknown. The pdf of X i is f θ (x i ) = θ 1 I( < x i < θ). Since P θ ( < X i < θ) = 1, we can focus on < x i < θ: Then logf θ (x i ) = logθ, θ logf θ (x i ) = 1 θ, [ ] 2 E θ θ logf θ (X i ) = 1 θ 2 < x i < θ UW-Madison (Statistics) Stat 61 Lecture / 16

2 Consider the estimation of g(θ) = θ, g (θ) = 1. According to Theorem (if it holds), for any unbiased estimator T (X) of θ, we should have Var θ (T ) [g (θ)] 2 [ ] 2 = θ 2 ne θ θ logf n θ (X 1 ) Let X (n) be the largest order statistic. From the result in Chapter 5, the pdf of X (n) is ny n 1 θ n, < y < θ Thus, θ ny n E θ (X (n) ) = θ n dy = n y n+1 θ θ n n + 1 = nθ n + 1 showing that n+1 n X (n) is an unbiased estimator of θ. E θ (X 2 (n) ) = θ ny n+1 θ n dy = n y n+2 θ n n + 2 θ = nθ 2 n + 2 UW-Madison (Statistics) Stat 61 Lecture / 16

3 Then ( ) n + 1 Var θ n X (n) = = = (n + 1)2 n 2 Var θ (X (n) ) (n + [ 1)2 n 2 E θ (X(n) 2 ) {E θ (X (n) )} 2] [ (n + 1)2 n 2 nθ 2 n + 2 ( ) ] nθ 2 n + 1 = θ 2 n(n + 2) < θ 2 n Hence, Theorem does not apply. What is wrong? Note that the key condition for Theorem is that θ E θ (T ) = T (x) f θ (x) θ dx θ Θ The right hand side is θ θ T (x) θ ( 1 θ n X ) dx 1 dx n = n θ n+1 θ θ T (x)dx 1 dx n UW-Madison (Statistics) Stat 61 Lecture / 16

4 The left hand side is [ 1 θ θ ] θ θ n T (x)dx 1 dx n = T (θ,...,θ) θ n n θ n+1 θ θ T (x)dx 1 dx n which is not the same as the right hand side unless T (θ,...,θ) = for all θ. The problem is the support of f θ (x), {x : f θ (x) > }, depends on θ. When this occurs, Theorem is typically not applicable. Approach 2: using unbiased estimators of To see when an unbiased estimator is best unbiased, we might ask how could we improve upon a given unbiased estimator? Suppose that T (X) is unbiased for g(θ) and U(X) is a statistic satisfying E θ (U) = for all θ, i.e., U is unbiased for. Then, for any constant a, is unbiased for g(θ). Can it be better than T (X)? T (X) + au(x) UW-Madison (Statistics) Stat 61 Lecture / 16

5 Var θ (T + au) = Var θ (T ) + 2aCov θ (T,U) + a 2 Var θ (U) If for some θ, Cov θ (T,U) <, then we can make 2aCov θ (T,U) + a 2 Var θ (U) < by choosing < a 2Cov θ (T,U)/Var θ (U). Hence, T (X) + au(x) is better than T (X) at least when θ = θ and T (X) cannot be UMVUE. Similarly, if Cov θ (T,U) > for some θ, then T (X) cannot be UMVUE either. Thus, Cov θ (T,U) = is necessary for T (X) to be a UMVUE, for all unbiased estimators of. It turns out that Cov θ (T,U) = for all U(X) unbiased for is also sufficient for T (X) being a UMVUE. Theorem An unbiased estimator T (X) of g(θ) is UMVUE iff T (X) is uncorrelated with all unbiased estimators of. UW-Madison (Statistics) Stat 61 Lecture / 16

6 Proof. We have shown the necessity (the only if part). We now show the sufficiency (the if part). Let W (X) be another unbiased estimator of g(θ). Then W (X) T (X) is unbiased for and Var θ (W ) = Var θ (T + (W T )) = Var θ (T ) + Var θ (W T ) + 2Cov θ (T,W T ) = Var θ (T ) + Var θ (W T ) Var θ (T ) The result follows since W is arbitrary. An unbiased estimator of is a first-order ancillary statistic and can be treated as a random noise. It makes sense that the most sensible way to estimate is with, not with a random noise. Theorem says that an estimator is correlated with a random noise, then it can be improved. UW-Madison (Statistics) Stat 61 Lecture / 16

7 The following two useful results are consequence of Theorem Corollary Let T j be a UMVUE of g j (θ), j = 1,...,m, where m is a fixed positive integer. Then T = m j=1 c jt j is a UMVUE of g(θ) = m j=1 c jg j (θ) for any constants c 1,...,c m. Proof. Let U(X) be any unbiased estimator of. First, the unbiasedness of T j s imply that T is unbiased for g(θ): E θ (T ) = E θ ( m j=1 c j T j ) = m j=1 E θ (T j ) = m j=1 c j g j (θ) = g(θ) By Theorem 7.3.2, Cov θ (T j,u) = for all j = 1,...,m, and θ Θ. ) Cov θ (T,U) = Cov θ ( m j=1 c j T j,u = m j=1 c j Cov θ (T j,u) = Again, by Theorem 7.3.2, this shows that T is a UMVUE of g(θ). UW-Madison (Statistics) Stat 61 Lecture / 16

8 Theorem If T (X) is a UMVUE of g(θ), then T (X) is unique in the sense that if T 1 (X) is another UMVUE of g(θ), then P θ (T (X) = T 1 (X)) = 1 for any θ Θ Proof. Let T (X) and T 1 (X) be both UMVUE of g(θ). Since both T (X) and T 1 (X) are unbiased, T (X) T 1 (X) is an unbiased estimator of. Since both T and T 1 are UMVUE, by Theorem 7.3.2, Then Cov θ (T,T T 1 ) =, Cov θ (T 1,T T 1 ) = θ Θ E θ (T T 1 ) 2 = E θ [(T T 1 )(T T 1 )] = E θ [T (T T 1 )] E θ [T 1 (T T 1 )] = Cov θ (T,T T 1 ) Cov θ (T 1,T T 1 ) = Hence, P θ (T (X) = T 1 (X)) = 1 for any θ Θ. UW-Madison (Statistics) Stat 61 Lecture / 16

9 Although Theorem provides an interesting characterization of best unbiased estimators and two useful results, its usefulness of checking whether a particular unbiased estimator T is a UMVUE is limited since it asks to check whether T is uncorrelated with all unbiased estimator of. However, Theorem is sometimes useful in determining that an estimator is not best unbiased or showing that no UMVUE exists. Example Let X be an observation from uniform(θ,θ + 1), θ R. We want to show that there is no UMVUE of g(θ) for any nonconstant differentiable function g. Note that an unbiased estimator U(X) of must satisfy θ+1 U(x)dx = for all θ R θ Differentiating both sizes of the previous equation, U(θ) = U(θ + 1), θ R, i.e., U(x) = U(x + 1), x R UW-Madison (Statistics) Stat 61 Lecture / 16

10 If T is a UMVUE of g(θ), then T (X)U(X) is unbiased for and, hence, T (x)u(x) = T (x + 1)U(x + 1), where U(X) is any unbiased estimator of. Since this is true for all U, Since T is unbiased for g(θ), g(θ) = T (x) = T (x + 1), θ+1 θ T (x)dx x R x R for all θ R Differentiating both sides of the previous equation we obtain that g (θ) = T (θ + 1) T (θ) =, Thus, g is a constant function. θ R We consider more for g(θ) = θ. Since E θ (X) = θ + 1 2, X is unbiased for θ, and its variance is 12. One unbiased estimator of is U = sin(2πx), i.e., E θ (U) = θ+1 sin(2πx)dx = θ R θ UW-Madison (Statistics) Stat 61 Lecture / 16

11 Cov θ (X 1 2,U) = Cov θ (X,sin(2πX)) = E θ [X sin(2πx)] = θ+1 θ x sin(2πx)dx θ+1 = x cos(2πx) 2π = cos(2πθ) 2π θ θ+1 + θ cos(2πx) dx 2π Hence X 1 2 is correlated with an unbiased estimator of, and cannot be a UMVUE of θ. In fact, X sin(2πx) 2π is unbiased for θ and has variance.71 < Sufficient statistics If there is a sufficient statistic T, then the Rao-Blackwell theorem says that E(W T ) is a better unbiased estimator than an unbiased estimator W. The following is a Rao-Blackwell theorem for unbiased estimators, a special case of what we have shown in Chapter 6. UW-Madison (Statistics) Stat 61 Lecture / 16

12 Theorem Let W be any unbiased estimator of g(θ) and T be a sufficient statistic for θ. Define ψ(t ) = E(W T ), which does not depend on θ since T is sufficient. Then ψ(t ) is an unbiased estimator of g(θ) and Var θ (T ) Var θ (W ). Thus, conditioning any unbiased estimator on a sufficient statistic will result in a uniform improvement. This and the next result indicate that we need consider only estimators that are functions of a sufficient statistic in our search for the UMVUE. An alternative to Theorem Suppose that T is a sufficient statistic for θ. An unbiased estimator ψ(t ) of g(θ) is a UMVUE iff Cov θ (ψ(t ),h(t )) = for any θ Θ and any h(t ) that is an unbiased estimator of. Proof. The only if" part follows from the only if" part of Theorem We now prove the if" part. UW-Madison (Statistics) Stat 61 Lecture / 16

13 Suppose that Cov θ (ψ(t ),h(t )) = for any θ Θ and any h(t ) that is an unbiased estimator of. For any unbiased estimator of U and any θ, Cov θ (ψ(t ),U) = E θ [ψ(t )U] = E θ {E[ψ(T )U T ]} = E θ {ψ(t )E(U T )}= since E(U T ) is a function of T and E θ [E(U T )] = E θ (U)=. If we have another sufficient statistic S, should we consider E[E(W T ) S]? If there is a function h such that S = h(t ), then by the properties of conditional expectation, E[E(W T ) S] = E(W S) = E[E(W S) T ] That is, we should always conditioning on a simpler sufficient statistic, such as a minimal sufficient statistic or a complete sufficient statistic. If we do have a sufficient and complete statistic T, then by the completeness, is essentially the only unbiased estimator of that is a function of T. UW-Madison (Statistics) Stat 61 Lecture / 16

14 Thus, by the alternative to Theorem 7.3.2, we have actually proved the following result useful for finding the UMVUE. Theorem (Lehmann-Scheffé Theorem) Let T be a complete sufficient statistic for θ. If ψ(t ) is an unbiased estimator of g(θ), then it is the unique UMVUE. By Example , Theorem does not hold if T is only minimal sufficient, since X is minimal sufficient for θ in Example This is because functions of minimal sufficient statistic can be first-order ancillary and can still be useful. Example Let X 1,...,X n be iid from uniform(,θ) with unknown θ >. Consider the estimation of θ. Previously we show that approach 1 is not applicable in this example. Since T = X (n) is the sufficient and complete statistic for θ and The UMVUE of θ is (1 + n 1 )X (n). E(X (n) ) = nθ/(n + 1) UW-Madison (Statistics) Stat 61 Lecture / 16

15 Suppose now that Θ = [1, ). Then X (n) is not complete, although it is still sufficient for θ. Thus, Theorem does not apply to X (n). We now illustrate how to use the alternative Theorem to find a UMVUE of θ. Let U(X (n) ) be an unbiased estimator of. Since X (n) has pdf nθ n x n 1 I (,θ) (x), = 1 U(x)x n 1 dx + θ This implies that U(x) = on [1, ) and 1 U(x)x n 1 dx =. 1 U(x)x n 1 dx for all θ 1. Consider T = h(x (n) ). To have E(TU) =, we must have 1 h(x)u(x)x n 1 dx =. Thus, we may consider the following function: UW-Madison (Statistics) Stat 61 Lecture / 16

16 { c x 1 h(x) = bx x > 1, where c and b are some constants. From the previous discussion, E[h(X (n) )U(X (n) )] =, θ 1. Since E[h(X (n) )] = θ, we obtain that θ = cp(x (n) 1) + be[x (n) I (1, ) (X (n) )] = cθ n + [bn/(n + 1)](θ θ n ). Thus, c = 1 and b = (n + 1)/n. The UMVUE of θ is then { 1 X(n) 1 h(x (n) ) = (1 + n 1 )X (n) X (n) > 1. This estimator is better than (1 + n 1 )X (n), which is the UMVUE when Θ = (, ) and does not make use of the information about θ 1. When Θ = (, ), this estimator is not unbiased. UW-Madison (Statistics) Stat 61 Lecture / 16