Proof In the CR proof. and

Transcription

1 Question Under what conditions will we be able to attain the Cramér-Rao bound and find a MVUE? Lecture 4 - Consequences of the Cramér-Rao Lower Bound. Searching for a MVUE. Rao-Blackwell Theorem, Lehmann-Scheffé Theorem. Corollary 1 There exists an unbiased estimator which attains the CR lower bound (under regularity conditions) if and only if Proof In the CR proof = I ( ) Cov[U, V ] 2 Var[U]Var[V ] and the lower bound is attained if and only equality is achieved. If U =, V =, the equality occurs when = c + d, where c, d are constants. E[V ] = 0 so c = d and = d( ). Jonathan Marchini (University of Oxford) BS2a MT / 20 Jonathan Marchini (University of Oxford) BS2a MT / 20 Multiply by / and take expectations. E [ ( ) ] 2 The LHS is I so we have d = I and [ ] [ ] = de de = d 1 0 = I ( ) Question What is the relationship between the CRLB and exponential families? Corollary 2 If there exists an unbiased estimator (X) which attains the CR lower bound (under regularity conditions) it follows that X must be in an exponential family Proof Taking n = 1 and log f (x; ) = = I ( ) log f (x; ) = A() + D() + C(x) which is an exponential family form. The constant of integration C(x) is a function of x. Jonathan Marchini (University of Oxford) BS2a MT / 20 Jonathan Marchini (University of Oxford) BS2a MT / 20

2 Question What is the relationship between the CRLB and MLEs? Corollary 3 Suppose (X) is an unbiased estimator that attains the CRLB, and so is a MVUE. Suppose that the MLE is a solution to / = 0 (so, not on boundary). Then =. i.e. if the CRLB is attained then it is generally the MLE that attains it. Proof must satisfy = I ( ). Setting = 0 and solving will give the MLE. Since I > 0 (in all but exceptional circumstances), this gives =. Question Do all MLEs attain the CRLB? No, because not all MLEs are unbiased. Example 11 Let X 1,..., X n be a random sample from N(µ, σ 2 ). Then we know the MLEs are µ = X, σ 2 = 1 n X 2 i X 2. Exercise µ is unbiased, but σ 2 is biased. CRLBs are 1/I µ = σ 2 and 1/I σ 2 = 2σ 4 /n. Var( µ) = σ 2 /n which equals the CRLB so is MVUE. Var( σ 2 ) = 2(n 1)σ 4 /n 2 is less than the CRLB. But σ 2 is biased. The sample variance S 2 = 1 n 1 (X i X) 2 is unbiased and has variance 2σ 4 /(n 1) which is larger than the CRLB. Question Is S 2 a MVUE? Jonathan Marchini (University of Oxford) BS2a MT / 20 Jonathan Marchini (University of Oxford) BS2a MT / 20 Efficiency Asymptotic normality of MLE Definition 13 The (Bahadur) efficiency of an estimator is defined as a comparison of the variance of with the CR bound I 1. That is Efficiency of = I 1 Var[ ] = 1 I Var[ ] The asymptotic efficiency is the limit as n. There are similar definitions for the relative efficiency of two estimators. Revision from Part A Statistics As the sample size n, the MLE N(, I 1 ). This is a powerful and general result. Assuming the usual regularity conditions hold then it tells us that the MLE has the following properties 1 it is asymptotically unbiased 2 it is asymptotically efficient i.e. it attains the CRLB asymptotically. 3 it has a normal distribution asymptotically. Jonathan Marchini (University of Oxford) BS2a MT / 20 Jonathan Marchini (University of Oxford) BS2a MT / 20

3 Extensions to the Cramér-Rao inequality 1. If is an estimator with bias b() = bias( ), then Var[ ] ( 1 + b ) 2 I 1 2. If ĝ(x) is an unbiased estimator for g(), then Var[ĝ(X)] ( ) g 2 I 1. Proof Begin with E ( (X)) = + b() (in 1.) and E (ĝ(x)) = g() (in 2.). Differentiate both sides and proceed as above to find Cov[U, V ] = (1 + b/) (in 1.) and Cov[U, V ] = g/ (in 2., with U = ĝ). The bound is against Cov[U, V ] 2 which leads to the results above. Fisher Information for a d-dimensional parameter Information matrix: [ ] [ 2 ] l I ij = E = E i j i j under regularity conditions. The CR inequality is Var( i ) [I 1 ] ii, i = 1,..., d. Exercise: verify that we have already proved Var( i ) [I ii ] 1. Note that [I 1 ] ii [I ii ] 1 (GJJ) so bound above is stronger. Exercise For an Exponential family in canonical form, 2 I ij = nd(φ). φ i φ j Jonathan Marchini (University of Oxford) BS2a MT / 20 Jonathan Marchini (University of Oxford) BS2a MT / 20 The CRLB may not be achievable but will still wish to search for an MVUE. Sufficiency plays an important role in the search for a MVUE. Theorem 3 : Rao-Blackwell Theorem (RBT) (GJJ 2.5.2) Let X 1,..., X n be a random sample of observations from f (x; ). Suppose that T is a sufficient statistic for and that is any unbiased estimator for. Define a new estimator T = E[ˆ T ]. Then 1. T is a function of T alone; 2. E[ T ] = ; 3. Var( T ) Var( ). Comment This says that estimators maybe be improved if we take advantage of sufficient statistics. Proof 1. T = E X [ˆ T = t] = = ˆ(x)f (x t, )dx χ χ ˆ(x)f (x t)dx 2. E[ T ] = E T [E[ˆ T ]] = E[ˆ] = (by law of total expectation) 3. Using the law of total variance Var( ) = Var(E[ˆ T ]) + E T [Var(ˆ T )] = Var( T ) + E T [Var(ˆ T )] Var( ) Var( T ) Jonathan Marchini (University of Oxford) BS2a MT / 20 Jonathan Marchini (University of Oxford) BS2a MT / 20

4 Example 12 Suppose X 1,..., X n be a random sample from Bernoulli(). It is easy to see that = X 1 is unbiased for. Also, we have seen before that T = N X i is sufficient for. We can use RBT to construct an estimator with smaller variance E[X 1 T = t] = P(X 1 = 1 T = t) = P(X 1 = 1, N X i = t) P( N X i = t) = P(X 1 = 1, N i=2 X i = t 1) N C t t (1 ) N t =.N 1 C t 1 t 1 (1 ) N t N C t t (1 ) n t Corollary 4 If an MVUE for exists, then there is a function T of the minimal sufficient statistic T for which is an MVUE. Proof If is a MVUE and T is minimal sufficient then by RBT we can construct T. Which implies T is a function of T alone, is unbiased and variance no larger than. Hence is also a MVUE. Comment This says that we can restrict our search for a MVUE to those based on minimal sufficient statistics. = t N Jonathan Marchini (University of Oxford) BS2a MT / 20 Jonathan Marchini (University of Oxford) BS2a MT / 20 Completeness Definition 14 : Complete Sufficient Statistics Let T (X 1,..., X n ) be a sufficient statistic for. The statistic T is complete if, whenever h(t ) is a function of T for which E[h(T )] = 0 for all, then h(t ) 0 almost everywhere. Lemma 4 Suppose T is a complete sufficient statistic for, and g(t ) unbiased for, so E[g(T )] =. Then g(t ) is the unique function of T which is an unbiased estimator of. Proof If there were two such unbiased estimators g 1 (T ), g 2 (T ), then E[g 1 (T ) g 2 (T )] = = 0 for all, so g 1 (T ) = g 2 (T ) almost everywhere. Question If we have an unbiased estimator what are the sufficient conditions for it to be MVUE? Lemma 5 If an MVUE for exists and T is a complete and minimal sufficient statistic for, and suppose h = h(t ) is unbiased for, then h(t ) is a MVUE. This Lemma combines the results of Corollary 4 and Lemma 4. Proof If an MVUE exists then there is a function of T which is an MVUE, by the RB Corollary 4. But h(t ) is the only function of T which is unbiased for (from Lemma 4). So h must be the function of T which an MVUE. Jonathan Marchini (University of Oxford) BS2a MT / 20 Jonathan Marchini (University of Oxford) BS2a MT / 20

5 Question Finally, how can we construct a MVUE? Theorem 4 : Lehmann-Scheffé theorem Let T be a complete sufficient statistic for, and let be an unbiased estimator for, then the unbiased estimator T = E[ˆ T ] has the smallest variance among all unbiased estimators of. That is, for all unbiased estimators. Var( T ) Var( ) Comment This theorem says that if we can find any unbiased estimator and a complete sufficient statistic T then we can construct a MVUE. Proof Suppose exists with Var( ) < Var( T ). Then by RBT we can construct T = E[ T ] such that Var( T ) Var( ) < Var( T ) But T and T are both unbiased and T is complete, so by Lemma 4 we have T = T and Var( T ) = Var( T ) which is a contradiction. Jonathan Marchini (University of Oxford) BS2a MT / 20 Jonathan Marchini (University of Oxford) BS2a MT / 20 Lemma 6 : Complete Sufficiency in EFs If the rv X has a distribution belonging to a k-parameter exponential family, then under the usual regularity conditions, the statistic ( n B 1 (X i ), n B 2 (X i ),..., n ) B k (X i ) which we already know is minimal sufficient, is complete. Comment We showed before that MLEs for exponential families were functions of this same statistic. Therefore, for a member of an exponential family if the MLE is unbiased (note : not all MLEs are unbiased), then by Lemma 5, the MLE will be MVUE. If there is an unbiased estimator that attains the CRLB then, by Corollary 3, the MLE will attain the CRLB. Example 13 Let X 1,..., X n be a random sample from N(µ, σ 2 ). Then we know the MLEs are µ = X, σ 2 = 1 n X 2 i X 2. µ is unbiased, but σ 2 is biased. Exercise ( The minimal sufficient complete statistic is n X i, ) n X i 2. So µ is MVUE and attains the CRLB with variance σ 2 /n. The sample variance S 2 = 1 n 1 (X i X) 2 is unbiased and is a function of the minimal sufficient complete statistic so is MVUE with variance 2σ 4 /(n 1) which is larger than the CRLB of 2σ 4 /n. Jonathan Marchini (University of Oxford) BS2a MT / 20 Jonathan Marchini (University of Oxford) BS2a MT / 20