Introduction to Statistical Inference

Size: px

Start display at page:

Download "Introduction to Statistical Inference"

Job Gilbert
5 years ago
Views:

1 Introduction to Statistical Inference Ping Yu Department of Economics University of Hong Kong Ping Yu (HKU) Statistics 1 / 30

2 1 Point Estimation 2 Hypothesis Testing Ping Yu (HKU) Statistics 2 / 30

3 The Objective of Statistics The objective of statistics is to infer (characteristics of) the underlying probability law from observed data, and then use the obtained knowledge to explain what has happened (i.e., internal validity), and predict what will happen (i.e., external validity). The internal validity concerns three problems: - What is a plausible value for the parameter? (point estimation) - What are a plausible set of values for the parameter? (set/interval estimation) - Is some preconceived notion or economic theory on the parameter "consistent" with the data? (hypothesis testing). In other words, the objectives of statistics are estimation, inferences (including hypothesis testing and confidence interval (CI) construction) and prediction. Ping Yu (HKU) Statistics 2 / 30

4 Point Estimation Point Estimation Ping Yu (HKU) Statistics 3 / 30

5 Point Estimation There are two econometric traditions: the frequentist approach and the Bayesian approach. - the former treats the parameter as fixed (i.e., there is only one true value) and the samples as random. - the latter treats the parameter as random and the samples as fixed. This course will concentrate on the frequentist approach. Two main methods in the frequentist approach are the likelihood method and the method of moments (MoM). Ping Yu (HKU) Statistics 4 / 30

6 Point Estimation The Maximum Likelihood Estimator The MLE was popularized by R.A. Fisher ( ). The basic idea of the MLE is to guess the truth which could generate the phenomenon we observed most likely (practical examples here). Mathematically, R θ MLE = argmax E[ln(f (Xjθ))] = argmax f (x)lnf (xjθ)dx θ2θ R θ2θ (1) = argmax lnf (xjθ)df (x), θ2θ where X is a random vector, f (x) is the true pdf or the true pmf, f (xjθ) is the specified parameterized pdf or pmf, Θ is the parameter space, and F (x) is the true cdf. Ping Yu (HKU) Statistics 5 / 30

Point Estimation History of the MLE Ronald A. Fisher (1890-1962), UCL Ronald A. Fisher (1890-1962) is one iconic founder of modern statistical theory.

7 Point Estimation History of the MLE Ronald A. Fisher ( ), UCL Ronald A. Fisher ( ) is one iconic founder of modern statistical theory. The name of F -distribution was coined by G.W. Snedecor, in honor of R.A. Fisher. The p-value is also credited to him. Ping Yu (HKU) Statistics 6 / 30

8 Point Estimation The MoM Estimator The MoM estimator was introduced by Karl Pearson ( ). The original problem is to estimate k unknown parameters, say θ = (θ 1,,θ k ), in f (x). But we are not fully sure about the functional form of f (x). Nevertheless, we know the functional form of the moments of X 2 R as a function of θ: E[X ] = g 1 (θ), E[X 2 ] = g 2 (θ), (2). E[X k ] = g k (θ). There are k functions with k unknowns, so we can solve out θ uniquely in principle. Ping Yu (HKU) Statistics 7 / 30

Point Estimation History of the MoM Karl Pearson (1857-1936), UCL Karl Pearson (1857-1936) is also the inventor of the correlation

9 Point Estimation History of the MoM Karl Pearson ( ), UCL Karl Pearson ( ) is also the inventor of the correlation coefficient in Chapter 5, so the correlation coefficient is also called the Pearson correlation coefficient. Ping Yu (HKU) Statistics 8 / 30

10 Point Estimation Efficiency and Robustness The MoM estimator uses only the moment information in X, while the MLE uses "all" information in X, so the MLE is more efficient than the MoM estimator. However, the MoM estimator is more robust than the MLE since it does not rely on the correctness of the full distribution but relies only on the correctness of the moment functions. Efficiency and robustness are a common trade-off among econometric methods. Ping Yu (HKU) Statistics 9 / 30

11 Point Estimation A Microeconomic Example of the MoM Estimator Moment conditions often originate from the FOCs in an optimization problem. Suppose the firms are maximizing their profits conditional on the information in hand; then the problem for the firm i is π is the profit function, e.g., max d i E νjz [π(d i,z i,ν i ;θ)]. (3) π(d i,z i,ν i,θ) = p i f (L i,ν i ;θ) w i L i, where z i = (p i,w i ) is all information used in decision and can be observed by both the firm and the econometrician, p i is the output price and w i is the wage, ν i is the exogenous random error (e.g., weather, financial crisis, etc.) and cannot be observed or controlled by either the firm or the econometrician, and d i = L i is the decision of labor input. θ is the technology parameter, e.g., if f (L i,ν i ;θ) = L φ i exp(ν i ), then θ = φ, and is known to the firm but unknown to the econometrician. Our goal is to estimate θ, which is relevant to measure the causal effect - the effect of labor input on profit. Ping Yu (HKU) Statistics 10 / 30

12 Point Estimation continue... The FOCs of (3) are π(di,z E i,ν i,θ) νjz = m(d d i,z i jθ) = 0. i When there is randomness even in z i, 1 then the objective function changes to and the FOCs change to maxe [π(d i,z i,ν i ;θ)], d i which are a special set of moment conditions. E [m(d i,z i jθ)] = 0, (4) 1 The difference between z i and ν i is that z i can be observed ex post while ν i cannot. That z i is random means that the decision is made before z i is revealed, or the decision is made ex ante. Ping Yu (HKU) Statistics 11 / 30

13 Point Estimation A Macroeconomic Example of the MoM Estimator (*) max ρ t E 0 [u (c t )] fc t g t=1 t=1 s.t. c t+1 + k t+1 = k t R t+1, k 0 is known, ρ is the discount factor, E 0 [u()] is the conditional expected utility based on the information at t = 0, k t is the capital accumulation at time period t, c t is the consumption at t, and R t is the gross return rate at t. From dynamic programming, we have the Euler equation E 0 ρ u0 (c t+1 ) u 0 (c t ) R t+1 = 1. If u(c) = c1 α 1 1 α, α > 0, then we get α ct E 0 ρ R c t+1 = 1. (5) t+1 Suppose ρ is known while α is unknown; then (5) is a moment condition for α. Ping Yu (HKU) Statistics 12 / 30

14 Point Estimation Population Version vs Sample Version of Moment Conditions Equations (2), (4) and (5) are the population version of moment conditions. Although some econometricians treat "population" as a physical population (e.g., all individuals in the US census) in the real world, the term "population" is often treated abstractly, and is potentially infinitely large. Since the population distribution is unknown, we cannot solve the population moment conditions to estimate the parameters. In practice, we often have a set of finite data points from the population, so we can substitute the population distribution in the moment conditions by the empirical distribution of the data, which generates the sample version of the moment conditions. This is called the analog method. Ping Yu (HKU) Statistics 13 / 30

15 Point Estimation History of the Analog Method Charles F. Manski (1948-), Northwestern Ping Yu (HKU) Statistics 14 / 30

16 Point Estimation (The Sample Version of) the MoM Estimator Suppose the true distribution of X satisfies Z E [m(xjθ 0 )] = 0 or m(xjθ 0 )df (x) = 0, where m : Θ R k! R k, and F () is the true cdf of X. The essence of the MoM estimator is to substitute the true distribution F () by the empirical distribution F b n (x) = n 1 n 1(X i x): i=1 Z m(xjθ)df b n (x) = 0, which is equivalent to The MoM estimator b θ (X 1,,X n ) is the solution to (6). n 1 n m(x i jθ) = 0. (6) i=1 Ping Yu (HKU) Statistics 15 / 30

17 Point Estimation (The Sample Version of) the MLE Similarly, the MLE can be constructed as the maximizer of the average log-likelihood function `n(θ) = 1 n n lnf (X i jθ), i=1 which is equivalent to the maximizer of the log-likelihood function or the likelihood function L n (θ) = n lnf (X i jθ) i=1 n L n (θ) = expfl n (θ)g = f (X i jθ). i=1 If f (xjθ) is smooth in θ, the FOCs for the MLE are n 1 n s(x i jθ) = 0, i=1 where s(jθ) = lnf (jθ)/ θ is called the score function. 2 So the MLE is a special MoM estimator in this case. 2 More often, n i=1 s(x i jθ) is called the score function. Ping Yu (HKU) Statistics 16 / 30

18 Hypothesis Testing Hypothesis Testing Ping Yu (HKU) Statistics 17 / 30

19 Hypothesis Testing Hypotheses: The Null and Alternative Different from an estimation problem where nothing is known about the true parameter, in hypothesis testing, some restrictions about the true parameter are assessed. In other words, there is already a target to attack. Nevertheless, hypothesis testing and estimation are closely related since some test statistics are based on estimators. The null hypothesis, written as H 0, is often a point hypothesis θ = θ 0. The complement of the null hypothesis is called the alternative hypothesis. So the alternative hypothesis, written as H 1, is θ 6= θ 0. More generally, we express a null hypothesis as H 0 : θ 2 Θ 0 and the alternative hypothesis as H 1 : θ 2 Θ 1, where Θ 0 is a proper subset of Θ, Θ 0 \ Θ 1 = /0, and Θ 0 [ Θ 1 = Θ. For simplicity, we often refer to the hypotheses as "the null" and "the alternative". Ping Yu (HKU) Statistics 18 / 30

20 Hypothesis Testing Decisions A hypothesis test either accepts the null, or rejects the null in favor of the alternative. We can describe these two decisions as Accept H 0 and Reject H 0. Given the two possible states of the world (H 0 or H 1 ) and the two possible decisions (Accept H 0 or Reject H 0 ), there are four possible pairings of states and decisions: State of NaturenDecision Accept H 0 Reject H 0 H 0 is true Correct Decision Type I Error H 1 is true Type II Error Correct Decision Table: Hypothesis Testing Decisions Ping Yu (HKU) Statistics 19 / 30

21 Hypothesis Testing Acceptance Region and Rejection Region The decision is based on the data, and so is a mapping from the sample space to the decision set. [Check the MoM estimator] This splits the sample space into two regions A and R such that if the observed sample falls into A we accept H 0, while if the sample falls into R we reject H 0. The set A can be called the acceptance region and the set R the rejection or critical region. It is convenient to express this mapping as a real-valued function called a test statistic T n = T n (X 1,,X n ) relative to a critical value c. The hypothesis test then consists of the decision rule 1 Accept H 0 if T n c, 2 Reject H 0 if T n > c. A test statistic T n should be designed so that small values are likely when H 0 is true and large values are likely when H 1 is true. Ping Yu (HKU) Statistics 20 / 30

22 Hypothesis Testing Type I Error and Type II Error A false rejection of the null hypothesis H 0 (rejecting H 0 when H 0 is true) is called a Type I error. The probability of a Type I error is P (Reject H 0 jh 0 is true) = P (T n > cjh 0 is true). (7) The size of the test is defined as the supremum of (7) across all data distributions which satisfy H 0. (**)- For a set A R, the supremum or least upper bound of A, denoted as supa, is the smallest number y such that y x for every x 2 A, and the infimum or greatest lower bound of A, denoted as infa, is the largest number y such that x y for every x 2 A. - Although mina and maxa may not exist, infa and supa always exist. For example, minn 1n,n = 1,2, o does not exist, but infn 1n,n = 1,2, o = 0. Of course, if mina and maxa exist, then infa = mina and supa = maxa. (**) A primary goal of test construction is to limit the incidence of Type I error by bounding the size of the test. A false acceptance of the null hypothesis H 0 (accepting H 0 when H 1 is true) is called a Type II error. Ping Yu (HKU) Statistics 21 / 30

23 Hypothesis Testing Power The rejection probability under the alternative hypothesis is called the power of the test, and equals 1 minus the probability of a Type II error: π n (θ) = P (Reject H 0 jh 1 is true) = P (T n > cjh 1 is true). We call π n (θ) the power function and is written as a function of θ to indicate its dependence on the true value of the parameter θ under H 1. In the dominant approach to hypothesis testing, the goal of test construction is to have high power, subject to the constraint that the size of the test is lower than the pre-specified significance level. Generally, the power of a test depends on the true value of the parameter θ, and for a well behaved test the power is increasing both as θ moves away from the null hypothesis θ 0 and as the sample size n increases. Ping Yu (HKU) Statistics 22 / 30

24 Hypothesis Testing Trade-off Between Size and Power Given a test statistic T n, increasing the critical value c increases the acceptance region A while decreasing the rejection region R. This decreases the likelihood of a Type I error (decreases the size) but increases the likelihood of a Type II error (decreases the power). (why?) Thus the choice of c involves a trade-off between size and the power. This is why the significance level of the test cannot be set arbitrarily small. (Otherwise the test will not have meaningful power.) It is important to consider the power of a test when interpreting hypothesis tests, as an overly narrow focus on size can lead to poor decisions. For example, it is trivial to design a test which has perfect size yet has trivial power. Specifically, for any hypothesis we can use the following test: Generate a random variable U U[0,1] and reject H 0 if U < α. This test has exact size of α. Yet the test also has power precisely equal to α. When the power of a test equals the size, we say that the test has trivial power. Nothing is learned from such a test. Ping Yu (HKU) Statistics 23 / 30

25 Hypothesis Testing Scientific Reasoning of Hypothesis Testing To determine the critical value c, we need to pre-select a significance level α such that P (T n > cjh 0 is true) = α, yet there is no objective scientific basis for choice of α. Nevertheless, the common practice is to set α = 0.05 (5%). Alternative values are α = 0.10 (10%) and α = 0.01 (1%). These choices are somewhat the by-product of traditional tables of critical values and statistical software. The informal reasoning behind the choice of a 5% critical value is to ensure that Type I errors should be relatively unlikely - that the decision Reject H 0 has scientific strength - yet the test retains power against reasonable alternatives. The decision Reject H 0 means that the evidence is inconsistent with the null hypothesis, in the sense that it is relatively unlikely (1 in 20) that data generated by the null hypothesis would yield the observed test result. In contrast, the decision Accept H 0 is not a strong statement. It does not mean that the evidence supports H 0, only that there is insufficient evidence to reject H 0. So it is more accurate to use the label Do not Reject H 0 instead of Accept H 0. Ping Yu (HKU) Statistics 24 / 30

26 Hypothesis Testing Statistically Significant When a test rejects H 0 at the 5% significance level it is common to say that the statistic is statistically significant and if the test accepts H 0 it is common to say that the statistic is not statistically significant or that it is statistically insignificant. It is helpful to remember that this is simply a way of saying Using the statistic T n, the hypothesis H 0 can [cannot] be rejected at the 5% level. When the null hypothesis H 0 : θ = 0 is rejected it is common to say that the coefficient θ is statistically significant, because the test has rejected the hypothesis that the coefficient is equal to zero. Ping Yu (HKU) Statistics 25 / 30

27 Hypothesis Testing An Example Suppose we have only one data point z in hand and we know z N (µ,1). We want to test H 0 : µ = 0 against H 1 : µ > 0. A natural test is to reject H 0 if z is large. Rigorously, the test is 1(z > c), where 1() is the indicator function which equals 1 if the event in the parenthesis is true and 0 otherwise, 1 indicates rejection and 0 acceptance, the test statistic T n = z, and the critical value is c. Set the significance level α = 0.05; then c is chosen such that P(z > cjµ = 0) = E [1(z > c)jµ = 0] = 1 Φ(c) = 0.05, i.e., c = So if z > 1.645, we will reject H 0 ; otherwise, we cannot reject H 0. The power function π (µ) = P (z > cjµ) = P (z µ > c µ) = 1 Φ(c µ) which is an increasing function of µ and a decreasing function of c. [Figure here] It is understandable that π (µ) is increasing with µ since when µ is larger, it is easier to detect µ > 0. That π (µ) is decreasing in c indicates a trade-off between size and the power. Since the power equals 1 minus the probability of the Type II error, it is equivalent to study the trade-off between the probabilities of the Type I error and Type II error. Ping Yu (HKU) Statistics 26 / 30

28 Hypothesis Testing Figure: Trade-Off Between the Type I Error and Type II Error Ping Yu (HKU) Statistics 27 / 30

29 Hypothesis Testing continue... The acceptance region is z c and the critical region is z > c, which are quite trivial. To illustrate these regions in a more complicated example, suppose two data points y 1 and y 2 are observed and they follow N(µ,2). We want to test H 0 : µ = 0 against H 1 : µ 6= 0. A natural test is to reject H 0 if the absolute value of y = (y 1 + y 2 ) /2 is large. Given that y follows N(µ,1), the 5% critical value is 1.96 since The acceptance region is or [Figure here] P(jyj > 1.96jµ = 0) = f(y 1,y 2 )jjyj 1.96g f(y 1,y 2 )j 3.92 y 1 y y 1 g. Ping Yu (HKU) Statistics 28 / 30

30 Hypothesis Testing Figure: Acceptance Region Based on (y 1,y 2 ) and y Ping Yu (HKU) Statistics 29 / 30

31 Hypothesis Testing Summary One hypothesis testing includes the following steps. 1 specify the null and alternative. 2 construct the test statistic. 3 derive the distribution of the test statistic under the null. 4 determine the decision rule (acceptance and rejection regions) by specifying a level of significance. 5 study the power of the test. Step 2, 3 and 5 are key and step 1 and 4 are usually trivial. Of course, in some cases, how to specify the null and the alternative is also subtle, and in some cases, the critical value is not easy to determine if the asymptotic distribution is complicated. Ping Yu (HKU) Statistics 30 / 30

Introductory Econometrics. Review of statistics (Part II: Inference)

Introductory Econometrics. Review of statistics (Part II: Inference) Introductory Econometrics Review of statistics (Part II: Inference) Jun Ma School of Economics Renmin University of China October 1, 2018 1/16 Null and alternative hypotheses Usually, we have two competing