Statistics and Econometrics I

Size: px

Start display at page:

Download "Statistics and Econometrics I"

Reynold Kelly
5 years ago
Views:

1 Statistics and Econometrics I Point Estimation Shiu-Sheng Chen Department of Economics National Taiwan University September 13, 2016 Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

2 Part I Estimation Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

3 Statistical Inference We now move our focus to statistical inference. Given a random sample with sample size n {X 1, X 2,..., X n } i.i.d. f(x; θ) where θ is an unknown population parameter. For instance, suppose we are interested in p = the (unknown) population proportion of NTU students, who have have a significant other. {X 1, X 2,..., X n } i.i.d. Bernoulli(p) (µ, σ 2 ) = the (unknown) population mean and variance of the S&P500 stock returns. {X 1, X 2,..., X n } i.i.d. N(µ, σ 2 ) Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

4 Statistical Inference We would like to find a good guess of the parameter θ: a point estimator. That is, we would like to come up with a random variable ˆθ = δ(x 1, X 2,..., X n ) that we expect to be close to θ. Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

5 Parameter Space Definition A parameter space Θ is the set of all possible values of parameters. It is denote by θ Θ. Examples: If µ denotes the mean grade point of all NTU students, then the parameter space (assuming a 100-point grading scale) is: Θ {µ : 0 µ 100} If p denotes the proportion of students who own a motorcycle, then the parameter space is: Θ {p : 0 p 1} Given X exp(β), then the parameter space is Θ {β : 0 < β < } Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

6 Estimator Definition Let {X 1, X 2,..., X n } be a random sample from the joint distribution indexed by a parameter θ Θ. A function ˆθ = δ(x 1, X 2,..., X n ) is called a point estimator of the parameter θ. When X 1 = x 1, X 2 = x 2,...,X n = x n are observed, then δ(x 1, x 2,... x n ) is called the point estimate of θ. Every estimator is also a statistic (by nature of being a function of a random sample). Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

7 How to Guess? Analogy Principle ( 類比原則 ) Method of Moments ( 動差法 ) Method of Maximum Likelihood ( 最大概似法 ) Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

8 Analogy Principle To estimate the population mean µ, use the sample mean n X n = 1 n i=1 X i To estimate the population variance σ 2, use the sample variance S 2 n = 1 n 1 n (X i X n ) 2 To estimate distribution function F X (x) = P (X x), use the empirical distribution function. i=1 ˆF n (x) = n i=1 I {X i x} n Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

9 Method of Moments The method of moments is simply an application of analogy principle. Suppose that {X i } n i=1 i.i.d. f(x, θ 1, θ 2,..., θ k ) j-th population moment, which is a function of unknown parameters E(X j ) = µ j (θ 1, θ 2,..., θ k ) j-th sample moment n M j = 1 n X j i i=1 By analogy principle, we use M j to estimate E(X j ). Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

10 Method of Moments The method of moments involves equating sample moments with population moments. We impose the condition so that the sample moment is equal to the population moment: the moment condition where θ = (θ 1, θ 2,..., θ k ), ˆθ = (ˆθ 1, ˆθ 2,..., ˆθ k ) Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

11 Method of Moments Given the following moment conditions: M j }{{} = µ j (ˆθ 1, ˆθ 2,..., ˆθ k ), j = 1, 2,..., k, }{{} sample moment population moment The solutions: (ˆθ 1, ˆθ 2,..., ˆθ k ) are the MM estimators of (θ 1, θ 2,..., θ k ) k unknown parameters with k moment conditions Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

12 Method of Moments The basic idea behind this form of the method is to: (1) Equate the first sample moment M 1 = 1 n n i=1 X i to the first theoretical moment E(X). (2) Equate the second sample moment M 2 = 1 n n i=1 X2 i to the second theoretical moment E(X 2 ). (3) Continue equating sample moments M j with the corresponding theoretical moments E(X j ), j = 3, 4,... until you have as many equations as you have parameters. (4) Solve for the parameters. Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

13 Method of Moments: Example Let {X i } n i=1 i.i.d. N(µ, σ 2 ), find the MMEs for µ and σ 2 The moment conditions are X = 1 n X i n i=1 }{{} M 1 1 n Xi 2 n i=1 }{{} M 2 We can solve for ˆµ and ˆσ 2 as = E(X) = µ }{{} µ 1 (µ,σ 2 ) = E(X 2 ) = µ 2 + σ 2 }{{} µ 2 (µ,σ 2 ) ˆµ = X ˆσ 2 = n i=1 X2 i n X 2 = n i=1 (X i X) 2 n Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

14 Method of Moments: Remarks Note that the Method of Moment Estimator is not unique. Different moment conditions may obtain different estimators. In general, we use the first few moments for simplicity. Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

15 Method of Maximum Likelihood Assume {X i } n i=1 i.i.d. f(x, θ) where f( ) is known but θ is an unknown parameter. Joint pmf/pdf (function of random sample) f(x 1,..., x n ; θ) = f(x 1 ; θ) f(x n ; θ) = i f(x i ; θ) We can also call it a likelihood function of θ: L(θ) = i f(x i ; θ) Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

16 Maximum Likelihood Estimator (MLE) The maximum likelihood estimator ˆθ ˆθ = arg max θ Θ L(θ) To find the value of θ such that the random sample {X 1 = x 1, X 2 = x 2,..., X n = x n } is most likely to be observed. Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

17 Example There are 5 balls in the urn. Let µ denote the portion of blue balls in the urn, and 1 µ be the portion of green balls in the urn. µ is the unknown parameter The random sample is {X 1, X 2,..., X 10 }, where { 1 If the ball is blue, X i = 0 If the ball is green. Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

18 Example It is clear that X i i.i.d. Bernoulli(µ) Let S S10 = X 1 + X X 10 = represents the number of blue ball, and i=1 X i S 10 Binomial(10, µ) Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

19 Likelihood Function Consider the following two possible samples Sample 1: S10 = 7 µ P (S10 = 7) = ( ) 10 7 µ 7 (1 µ) / / / / /5 0 Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

20 Likelihood Function Sample 2: S10 = 2 µ P (S10 = 2) = ( ) 10 2 µ 2 (1 µ) / / / / /5 0 Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

21 Likelihood Function Sample 1: S10 = 7 Sample 2: S10 = 2 ) µ 7 (1 µ) 3 P (Sn = 2) = ( ) 10 µ 2 (1 µ) 8 µ P (S n = 7) = ( / / / / / Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

22 Maximum Likelihood Estimator If L(θ) is differentiable, then the MLE is the solution to: Note that L(θ) θ = 0 ˆθ = arg max L(θ) = arg max log L(θ) So the MLE is also the solution to: log L(θ) θ = 0 where log L(θ) is called the log likelihood function Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

23 Examples Let {X i } n i=1 i.i.d. Bernoulli(µ) Then the likelihood function is n L(µ) = µ x i (1 µ) 1 x i = µ i x i (1 µ) n i x i i=1 The log likelihood function is ( ) ( log L(µ) = x i log µ + n i i x i ) log(1 µ) It can be shown that the estimate is ˆµ = 1 n i x i, and hence the estimator is ˆµ = 1 X i = n X Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40 i

24 Important Property of MLEs: Invariance Theorem If ˆθ is the MLE of θ, and let τ(θ) be a function of θ, then ˆτ = τ(ˆθ) is the MLE of τ(θ). Example: Given {X i } n i=1 i.i.d. Bernoulli(µ) The MLE of V ar(x1 ) = µ(1 µ) is V ar(x 1 ) = ˆµ(1 ˆµ) = X(1 X) Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

25 Part II Evaluating Estimators Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

26 Evaluating Estimators Criteria for Evaluating Estimators Unbiased Efficient Consistent Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

27 Evaluating Estimators Unbiasedness Definition (Unbiasedness) ˆθ is unbiased if E(ˆθ) = θ Hence, bias ca be defined as B(θ) = E(ˆθ) θ Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

28 Evaluating Estimators Unbiasedness Given {X i } n i=1 i.i.d. (µ, σ 2 ). By analogy principle, n X = i=1 Xi ˆσ 2 = n n i=1 (Xi X) 2 n It can be shown that E( X) = µ E(ˆσ 2 ) = n 1 n σ2 σ 2 Hence, to obtain unbiased estimator for σ 2, let ( ) n n S 2 = ˆσ 2 i=1 = (X i X) 2, n 1 n 1 so that E(S 2 ) = σ 2. Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

29 Evaluating Estimators Minimum Variance Unbiased Estimator (MVUE) Definition (MVUE) ˆθ is an MVUE of θ if and only if E(ˆθ) = θ V ar(ˆθ) V ar(ˆθ ) for all ˆθ such that E(ˆθ ) = θ Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

30 Evaluating Estimators Efficient Definition (Relatively Efficient Estimator) Given two unbiased estimators: ˆθ and θ. We say that ˆθ is more efficient than θ if V ar(ˆθ) V ar( θ) {X i } i.i.d. (µ, σ 2 ) Consider the following two estimators X = 1 n i X i X = X 1+X Both X and X are unbiased But V ar( X) = σ2 n, V ar( X) = σ2 2. That is, V ar( X) > V ar( X) when n > 2. X is more efficient. Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

31 Evaluating Estimators Unbiased vs. Efficient Given two unbiased estimators, the one with smaller variance is better. How to compare Unbiased estimator with higher variance Biased estimator with lower variance Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

32 Evaluating Estimators Mean Squared Error (MSE) Definition (Mean Squared Error) Mean Squared Error (MSE) is defined by MSE(ˆθ) E [(ˆθ θ) 2] Note that MSE(ˆθ n ) = V ar(ˆθ n ) + (B(θ)) 2 The estimator with smaller MSE is called an MSE efficient estimator. Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

33 Evaluating Estimators Example Given {X i } n i=1 i.i.d. N(µ, σ 2 ) Let n i=1 X = X n i, S 2 i=1 = (X i X) 2 n, ˆσ 2 i=1 = (X i X) 2 n n 1 n It can be shown that MSE(ˆσ 2 ) < MSE(S 2 ) That is, compared to S 2, ˆσ 2 is MSE efficient. Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

34 Evaluating Estimators Consitent Definition (Consistent Estimator) An estimator ˆθ is a consistent estimator of θ, if ˆθ n Example: {X i } n i=1 i.i.d. (µ, σ 2 ), By WLLN p θ X n p µ By WLLN and CMT S 2 n ˆσ 2 n p σ 2 p σ 2 Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

35 Part III Applications and Examples Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

36 Applications and Examples Example Consider the probability model for income (Y ) and education (X) E(Y X) = α + βx Suppose that we have random sample {Y i, X i } n i=1 i.i.d. f(x, y) with Y i X i = x (α + βx, σ 2 ) How to find the point estimator of β? Analogy Principle Method of Moments Maximum Likelihood Estimation Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

37 Applications and Examples Example Analogy Principle Recall that β = Cov(X, Y ) V ar(x) = E[(X E[X])(Y E[Y ])] E([(X E(X)] 2 ) Hence, the analog estimator is ˆβ = 1 n i (X i X)(Y i Ȳ ) i (X i X) 2 1 n Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

38 Applications and Examples Example Method of Moments Let ε = Y E(Y X) = Y α βx. Hence, E(ε) = 0, E(Xε) = 0 The moment conditions are 1 ε i = 1 (Y i α βx i ) n n i i }{{} sample moment 1 X i ε i = 1 X i (Y i α βx i ) n n i i }{{} sample moment We can obtain ˆβ = 1 n = E(ε) }{{} = 0 population moment i (X i X)(Y i Ȳ ) i (X i X) 2 1 n = E(Xε) }{{} = 0 population moment Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

39 Applications and Examples Example Maximum Likelihood Estimation If we further assume normality: Y i X i = x N(α + βx, σ 2 ) The conditional pdf is f Y X=x (y) = 1 e 2( 1 y α βx σ ) 2 2πσ The likelihood function is [ ] 1 n L(α, β, σ) = e 1 2 2πσ i ( ) yi α βx 2 i σ Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40

40 Applications and Examples Example Maximum Likelihood Estimation The log-likelihood function is log L(α, β, σ) = n log 2π n log σ 1 ( yi α βx i 2 σ The FOCs are log L α = i (y i α βx i ) = 0 log L = (y i α βx i )x i = 0 β i Again, we can obtain the estimator of β as 1 n i ˆβ = (X i X)(Y i Ȳ ) i (X i X) 2 1 n Shiu-Sheng Chen (NTU Econ) Statistics and Econometrics I September 13, / 40 i ) 2

STAT 512 sp 2018 Summary Sheet

STAT 512 sp 2018 Summary Sheet STAT 5 sp 08 Summary Sheet Karl B. Gregory Spring 08. Transformations of a random variable Let X be a rv with support X and let g be a function mapping X to Y with inverse mapping g (A = {x X : g(x A}