Methods of evaluating estimators and best unbiased estimators Hamid R. Rabiee

Size: px

Start display at page:

Download "Methods of evaluating estimators and best unbiased estimators Hamid R. Rabiee"

Sara Warner
6 years ago
Views:

1 Stochastic Processes Methods of evaluating estimators and best unbiased estimators Hamid R. Rabiee 1

2 Outline Methods of Mean Squared Error Bias and Unbiasedness Best Unbiased Estimators CR-Bound for variance Sufficiency and UMVUE 2

3 Which estimator? Task of choosing between estimators! This chapter Introduces some basic criteria for evaluating estimators. 3

4 Mean Squared Error Mean squared error Of estimator W Of parameter θ E θ w θ 2 Mean squared error 4

5 Mean Squared Error Why MSE? Any increasing function on W θ could be good. But MSE: Analytically tractable E θ W θ 2 = Var θ W + (E θ W θ) 2 = Var θ W + (Bias θ W) 2 Two components of MSE variability of estimator (precision) and bias (accuracy). Bias: difference between the expected value of Wand θ. 5

6 MSE for an unbiased estimator Unbiased Estimator Bias 0 E θ W = θ, θ For an unbiased estimator: E θ W θ 2 = Var θ W 6

7 Normal distribution estimators Example: Let X 1, X 2,, X n be iid N μ, σ 2. X : unbiased estimator for μ S 2 = n i=1 n 1 X i X 2 and σ 2 = Are both estimator for σ 2 n i=1 X i X 2 σ 2 has lower MSE but S 2 is unbiased. n 7

8 Best Unbiased Estimator Comparison of estimators based on MSE is not practically possible, Since the class of all estimators is too large. To limit the class of estimators, we only consider unbiased estimators. We have to find an unbiased estimator with minimum MSE. It equivalent to find an unbiased estimator with minimum variance. Why? 8

9 Best Unbiased Estimator For any unbiased estimator the MSE is equal to the variance. Definition (UMVUE): An estimator W* is a best unbiased estimator of τ(θ) if it satisfies E θ W = τ(θ) for all θ and for any other estimator of W with E θ W = τ θ we have Var W Var W for all θ. It s also called MVUE 9

10 Best Unbiased Estimator Example: x 1, x 2,, x n are samples from iid poison(λ). Let X and S 2 be the sample mean and variance. We have to calculatevar X and Var(S 2 ). Is it sufficient?? 10

11 Best Unbiased Estimator Example: x 1, x 2,, x n are samples from iid poison(λ). Let X and S 2 be the sample mean and variance. We have to calculatevar X and Var(S 2 ). How can we sure that there are not better (with less variance) estimator?? We should find a lower band, B θ on the variance of any unbiased estimator. W is an UMVUE if Var θ W = B(θ) 11

12 Cramer-Rao Theorem Theorem (Cramer-Rao): Let X 1,, X n be a sample with pdf f(x θ) and let W(X) = W(X 1,, X n ) be any estimator where E θ (W(X)) is a differentiable function of θ. Suppose the joint pdf f x 1, x 2,, x n θ satisfies: d dθ h x f x θ dx 1 dx n = h x θ f x θ dx 1 dx n (1) for any function h(x) with E(h(X)) <. Then Var W X d dθ E θw X E θ (( θ log f(x θ))2 ) 2 12

13 Cramer-Rao bound for unbiased estimators If W is an unbiased estimator of θ then 1 Var W X E θ ( log f(x θ))2 θ Why? 13

14 Fisher information The Fisher information is a way of measuring the amount of information that an observable random variable X carries about an unknown parameter θ upon which the probability of X depends. The value I(θ) = E θ ( θ log f(x θ))2 is the fisher information and can also be derived by I(θ) = E θ 2 Note that: 0 I θ log f(x θ)) θ2 < 14

15 Fisher information and CR bound The variance of any unbiased estimator θ of θ is bounded by the inverse of Fisher information: Var θ 1 I(θ) 15

16 CR bound Example: Distribution N(μ, σ 2 ), μ is known, but σ 2 is unknown. Show T = n i=1 X i μ 2 What is variance of T? Find UMVUE for σ 2. n is unbiased for σ 2. 16

17 CR bound, iid case Let X 1,, X n be a sample with pdf f(x θ) and let W(X) = W(X 1,, X n ) be any estimator where E(W) is a differentiable function of θ. If the joint pdf satisfies the necessary condition for CR bound then: Why? Var W X d dθ E θw X ne θ (( θ log f(x θ))2 ) 2 17

18 CR bound, iid case Let X 1,, X n be a sample with pdf f(x θ) and let W(X) = W(X 1,, X n ) be any estimator where E(W) is a differentiable function of θ. If the joint pdf satisfies the necessary condition for CR bound then: E θ Var W X ne θ d dθ E θw X 2 ( log f(x θ))2 θ ( θ log f(x θ))2 = E θ ( θ log f(x i θ)) 2, independency E θ ( θ log f(x θ))2 = ne θ ( log f(x θ))2 θ 18

19 Best Unbiased Estimator (Cont.) Example: Let X 1,, X n be iid Poisson(λ). Prove that X is a UMVUE of λ. Solution: X is an unbiased estimator X has variance equal to the Cramer- Rao lower bound 19

20 When Cramer-Rao does not apply Example: Let X 1,, X n be iid with pdf f x θ = 1 θ The Cramer-Rao bound:, 0<x< θ. VarW θ2 n To find unbiased estimator there is a guess: Y = max X 1,, X n E θ Y = n n+1 θ n+1 n Y is unbiased. 20

21 When Cramer-Rao does not apply Var θ n+1 n Y = 1 n(n+2) θ2 This is uniformly smaller than θ2 n Cramer-Rao is not applicable θ 0 Because: d h x f x θ dx = d dθ dθ + θ h(x) h x = h(θ) θ 0 Unless h θ θ θ (1 θ )dx = 0 for all θ. θ 0 θ 0 θ h(x) 1 θ dx f x θ dx If the range of pdf, depends on the parameter, the theorem will not be applicable. 21

22 Sufficiency and Unbiasedness Theorem (Rao-Blackwell): Let W be any unbiased estimator of τ(θ) and let T be a sufficient statistic for all θ. Define ϕ T = E W T. Then E θ ϕ T = τ(θ) and Var(ϕ T ) Var(W) for all θ, that is ϕ T is a uniformly better unbiased estimator of τ θ. 22

23 Uniqeness of UMVUE Theorem: If W is the best unbiased estimator of τ(θ), then W is unique. Proof sketch: Suppose W is another UMVUE, and consider W = 1 2 W + W E W = τ θ Find the Variance of W. 23

24 How to find UMVUE Theorem: Let T be a complete sufficient statistic for a parameter θ, and let φ(t) be any estimator based on T. Then T is unique UMVUE of its expected value. What if there is no candidate estimator based on T? If T is a CSS for a parameter θ and h x 1, x 2,, x n is any unbiased estimator of τ(θ). Then φ T = E h x 1, x 2,, x n T is the UMVUE of τ(θ). 24

25 Binomial best unbiased estimation Example: Let X 1,, X n be iid binomial k, θ. Estimate the probability of exactly one success. τ θ = P θ X = 1 = kθ 1 θ k 1 Simple-minded estimator h X 1 = 1 if X 1 = 1 0 otherwise n φ X i i=1 n = E h X 1 X i i=1 = k k(n 1) X i 1 kn X i 25

26 Consistency Definition: A sequence of estimators W n = W n X 1,, X n is a consistent sequence of estimators of the parameter θ if for every ε > 0 and every θ Θ lim P θ( W n W < ε) = 1 n Theorem: If W n is a sequence of estimators of a parameter θ satisfying lim Var W n = 0 and lim Bias W n = 0. n n Then W n is a consistent sequence of estimators of θ. 26

Last Lecture. Biostatistics Statistical Inference Lecture 14 Obtaining Best Unbiased Estimator. Related Theorems. Rao-Blackwell Theorem

Last Lecture. Biostatistics Statistical Inference Lecture 14 Obtaining Best Unbiased Estimator. Related Theorems. Rao-Blackwell Theorem Last Lecture Biostatistics 62 - Statistical Inference Lecture 14 Obtaining Best Unbiased Estimator Hyun Min Kang February 28th, 213 For single-parameter exponential family, is Cramer-Rao bound always attainable?