Graduate Econometrics I: Unbiased Estimation

Transcription

1 Graduate Econometrics I: Unbiased Estimation Yves Dominicy Université libre de Bruxelles Solvay Brussels School of Economics and Management ECARES Yves Dominicy Graduate Econometrics I: Unbiased Estimation 1/26

2 Outline Elements of Decision Theory 1 Elements of Decision Theory 2 Yves Dominicy Graduate Econometrics I: Unbiased Estimation 2/26

3 Outline Elements of Decision Theory 1 Elements of Decision Theory 2 Yves Dominicy Graduate Econometrics I: Unbiased Estimation 3/26

4 Decision Theory Elements of Decision Theory An estimator of a function g(θ) of the parameters is a mapping δ(y ) from Y to g(θ) The comparison of different estimators is based on risk functions associated with loss functions When estimating a function of the parameter g(θ), one typically uses (but not exclusively) the quadratic loss function L(δ(Y ), θ) = (δ(y ) g(θ)) 2, θ R or L(δ(Y ), θ) = (δ(y ) g(θ))(δ(y ) g(θ)), θ R q Yves Dominicy Graduate Econometrics I: Unbiased Estimation 4/26

5 Decision Theory Elements of Decision Theory Definition An estimator δ (Y ) weakly dominates another estimator δ(y ) if and only if θ Θ : R(δ (Y ), θ) = E θ (δ (Y ) g(θ))(δ (Y ) g(θ)) R(δ(Y ), θ) = E θ (δ(y ) g(θ))(δ(y ) g(θ)) Yves Dominicy Graduate Econometrics I: Unbiased Estimation 5/26

6 Decision Theory Elements of Decision Theory The risk function may be decomposed into two parts : Property R(δ, θ) = V θ δ(y ) + (E θ δ(y ) g(θ))(e θ δ(y ) g(θ)) }{{}}{{} variance square bias The best estimator is the one with the smaller risk : No bias Minimum variance Yves Dominicy Graduate Econometrics I: Unbiased Estimation 6/26

7 Decision Theory Elements of Decision Theory Definition An estimator δ(y ) is an unbiased estimator of g(θ) if and only if : E θ δ(y ) = g(θ) θ Θ Thus, if δ(y ) is an unbiased estimator : R(δ, θ) = V θ δ(y ) From now on, we focus on unbiased estimators Yves Dominicy Graduate Econometrics I: Unbiased Estimation 7/26

8 Decision Theory Elements of Decision Theory Property If δ 1 (Y ) and δ 2 (Y ) are two unbiased estimators, then δ 1 (Y ) dominates δ 2 (Y ) if and only if : V θ δ 2 (Y ) V θ δ 1 (Y ) θ Θ, ie if and only if V θ δ 2 (Y ) V θ δ 1 (Y ) is a positive semi-definite matrix for every possible value of the parameter Yves Dominicy Graduate Econometrics I: Unbiased Estimation 8/26

9 Decision Theory Elements of Decision Theory The use of efficiency (ie variance) as a measure of the goodness of an estimator suffers from two drawbacks : It can only be used to compare two unbiased estimators since the variance of an estimator is the mean of the squared deviations from the expected value of the estimator This can only be regarded as a measure of the scatter of the estimator about the parameter, g(θ), which it estimates if E θ δ(y ) = g(θ) But there are biased estimators that may be acceptable if the sampling distribution is skewed The variance is not the only measure of the scatter of the distribution that could be used The mean absolute deviation of one estimator can be smaller that the corresponding for the another estimator ; and vice versa for the mean square deviation But in large samples almost all the estimators become normal distributed so that neither of these two objections are valid Yves Dominicy Graduate Econometrics I: Unbiased Estimation 9/26

10 Decision Theory Elements of Decision Theory Exercise : If a simple sample X 1, X 2,, X n iid has unknown finite variance σ 2, then we can consider the sample variance s 2 = 1 n n (X i X) 2 i=1 Suppose X is the sample mean and denote by µ the distribution mean Is this an unbiased estimator of σ 2? Yves Dominicy Graduate Econometrics I: Unbiased Estimation 10/26

11 Outline Elements of Decision Theory 1 Elements of Decision Theory 2 Yves Dominicy Graduate Econometrics I: Unbiased Estimation 11/26

12 The Cramer-Rao bound, which is based on the variance of the score function, known as the Fisher Information, gives a lower bound for the variance of an unbiased estimator Among unbiased estimators, one important goal is to find an estimator that has as small a variance as possible A more precise aim would be to find an unbiased estimator that has uniform minimum variance If the variance of ˆθ attains the minimum variance of the Cramer-Rao inequality we say that ˆθ is a minimum variance unbiased estimator of θ (MVUE) If ˆθ 1 and ˆθ 2 are both unbiased estimators of a parameter θ we say that ˆθ 1 is relatively more efficient if var(ˆθ 1 ) < var(ˆθ 2 ) We use the ratio var( ˆθ 1 ) var( ˆθ in order 2 ) to measure relative efficiency Yves Dominicy Graduate Econometrics I: Unbiased Estimation 12/26

13 It provides a lower bound to the variance covariance matrices of unbiased estimators for g(θ) This inequality holds under suitable regularity conditions : Definition A parametric model with likelihood l(y; θ), θ Θ, is said to be regular if i) Θ is an open subset of R p ii) l(y; θ) is differentiable with respect to θ iii) l(y; θ)dy as a function of θ is differentiable and Y l(y; θ)dy = l(y; θ)dy Y Y ( iv) I(θ) = E log l(y ;θ) log l(y ;θ) θ is the Fisher information θ matrix It exists and is nonsingular (ie positive definite) θ Θ θ ) Yves Dominicy Graduate Econometrics I: Unbiased Estimation 13/26

14 We also need some regularity conditions on unbiased estimators : Definition An unbiased estimator δ(y ) is said to be regular if i) It is square integrable : E θ δ(y ) 2 < + ii) δ(y)l(y; θ)dy is differentiable and Y δ(y)l(y; θ)dy = δ(y) l(y; θ)dy Y Y Yves Dominicy Graduate Econometrics I: Unbiased Estimation 14/26

15 The best possible estimator is one whose distribution is concentrated as closely as possible about the parameter it estimates In the limit it will be completely concentrated on the parameter (if it is unbiased) But we have seen that we can stabilize the distribution Then which is the smallest possible variance? Theorem Given a regular parametric model, every estimator δ(y ) that is regular and unbiased for g(θ) R q has a variance covariance matrix satisfying : V θ δ(y ) g(θ) I(θ) 1 g(θ) θ Θ In particular, if g(θ) = θ, then V θ δ(y ) I(θ) 1 This is the Cramer-Rao lower bound Yves Dominicy Graduate Econometrics I: Unbiased Estimation 15/26

16 PROOF Differentiating the unbiasedness condition, E θ δ(y ) = δ(y)l(y; θ)dy = g(θ), with respect to θ g(θ) = = Y Y Y δ(y) δ(y) = E θ ( δ(y ) = Cov θ ( δ(y ), l(y; θ) dy log l(y; θ) l(y; θ)dy ) log l(y; θ) log l(y; θ) ) The last equality follows because the score has zero mean Yves Dominicy Graduate Econometrics I: Unbiased Estimation 16/26

17 We need the Schwarz inequality : V (Y ) Cov(X, Y )V (X) 1 Cov(X, Y ) Using the Schwarz inequality : V θ δ(y ) Cov θ ( δ(y ), ) ( log l(y; θ) log l(y; θ) V θ ) ( log l(y; θ) Cov θ, δ(y ) it follows that this matrix is symmetric positive semidefinite ) 1 0, Yves Dominicy Graduate Econometrics I: Unbiased Estimation 17/26

18 Since : QED ( ) log l(y; θ) V θ = I(θ) V θ δ(y ) g(θ) I(θ) 1 g(θ) Yves Dominicy Graduate Econometrics I: Unbiased Estimation 18/26

19 Given a family of unbiased estimators, we may ask which is the best among all Definition Efficient estimator Given a regular parametric model, a regular unbiased estimator of g(θ) is efficient if its variance covariance matrix is equal to the Cramer-Rao lower bound, ie if V θ δ(y ) = g(θ) I(θ) 1 g(θ) θ Θ In particular, an efficient estimator of θ is an estimator of which the variance covariance matrix is equal to the inverse of the Fisher information matrix Yves Dominicy Graduate Econometrics I: Unbiased Estimation 19/26

20 Property If δ(y ) is a q 1 efficient estimator of its mean and if A and B are two constants matrices, the estimator AT (Y ) + B is also an efficient estimator (Ag(θ) + B) V θ (Aδ(Y ) + B) = I(θ) 1 (Ag(θ) + B) Yves Dominicy Graduate Econometrics I: Unbiased Estimation 20/26

21 Why the variance of an efficient estimator is the inverse of the Fisher information matrix? The calculations say so, but what does this mean? 1- We have seen that the Fisher information matrix is the variance-covariance of the score : ( ) ( ) log l(y; θ) log l(y; θ) log l(y; θ) V θ = E θ 2- In the score, θ (or g(θ)) is fixed 3- We evaluate the score in every observation Some will produce a positive score and others a negative score 4- Some will be close from the average while others will be far Yves Dominicy Graduate Econometrics I: Unbiased Estimation 21/26

22 4-The mean of all these fluctuations around the mean (ie the square root of the variance) gives us a measure of the information content on the observations 5- We have now an unbiased estimator δ(y ) of g(θ) (E θ (δ(y )) = g(θ)) that is also a function of the observation 6-There is therefore a relation between δ(y ) and the score : ( ) ( ) V log l(y;θ) θ = I(θ) E log l(y;θ) θ = 0 E θ (δ(y )) = g(θ) V θ (δ(y )) =? 7-Since the Fisher information matrix is a measure of the information content in the observations, δ(y ) will also exploit it Yves Dominicy Graduate Econometrics I: Unbiased Estimation 22/26

23 8-The better this information is exploited, the better δ(y ) is In other words, the better I(θ) is used, the smaller the variance of δ(y ) 9-In the best case, I(θ) is efficiently exploitedthere is a kind of perfect fit between δ(y ) and the score 10-It means that if we regress the estimator is log l(y; θ) δ(y i ) = β + ε i, ˆβ = Cov θ ( δ(y ), V θ ( log l(y;θ) log l(y;θ) ) ) Yves Dominicy Graduate Econometrics I: Unbiased Estimation 23/26

24 If there is a perfect fit ˆε i = 0 i = 1,, n δ ( y) β log l( y; θ ) Yves Dominicy Graduate Econometrics I: Unbiased Estimation 24/26

25 And perfect fit also means Var(ˆε) = 0 Therefore : ( ) V θ δ(y ) = ˆβ log l(y; θ) 2 V θ V θ δ(y ) = Or in matrix form : Cov θ ( δ(y ), V θ δ(y ) Cov θ ( δ(y ), V θ ( log l(y;θ) V θ ( log l(y; θ) ) 2 log l(y;θ) ) log l(y; θ) ) 2 V θ ( log l(y; θ) ) 1 Cov θ ( δ(y ), V θ δ(y ) g(θ) I(θ) 1 g(θ) = 0 ) ) log l(y; θ) = 0 Yves Dominicy Graduate Econometrics I: Unbiased Estimation 25/26

26 If δ(y ) does not use efficiently all the information, some residuals are not zero which means that V (ˆε) > 0 and ( ) log l(y; θ) V θ (δ(y )) = ˆβ 2 V θ + V (ˆε), which means that V θ (δ(y )) g(θ) I(θ) 1 g(θ) = V (ˆε) > 0, which means that δ(y ) is not efficient Yves Dominicy Graduate Econometrics I: Unbiased Estimation 26/26