ECON 4160, Autumn term Lecture 1

Transcription

1 ECON 4160, Autumn term Lecture 1 a) Maximum Likelihood based inference. b) The bivariate normal model Ragnar Nymoen University of Oslo 24 August / 54

2 Principles of inference I Ordinary least squares (OLS) and the Method of Moments (MM) are classical principles of estimation and inference They lead to estimators, t-values and F-statistics that also are random variables. with known properties given the assumptions of the statistical model. In econometrics, the assumptions about the statistical model often appear in the disturbance term of a regression equation. In this Lecture we motivate another classic called Maximum Likelihood Estimation (MLE) And the Likelihood Ratio (LR) test principle. 2 / 54

3 Principles of inference II MLE and LR are introduced with the aid of models from basic statistics/economerics courses. The second main topic is the introduction of the bivariate normal model, that will be a main reference for the rest of the course. 3 / 54

4 References for Lecture 1 Background knowledge at the level of our Review of key concepts note. HN: Ch 1 and 2 (Bernoulli), Ch 3 (First regresssion model/location model), Ch 5 (2-variable regresion), Ch 4 (Logit model/logistic regression) Ch 10.2 and 10.2 (Bivariate normal model) BN(2011): Ch. 4.5,4.6, 4.8 and Ch 4.A; Ch , Ch 6.1, / 54

5 Ch 1 and 2 in HN Bernoulli Model I Assume that the purpose is to estimate the probability of girl or boy of newborn children. That population probability is the parameter of interest. To be of any relevance the statistical model must contain the parameter of interest (directly or as a derived parameter). The Bernoulli model meets that requirement. Let Y i denote the sex of child i and consider n random variables {Y 1,Y 2,...,Y n }. The model assumptions are: 1. Independence: Y 1,Y 2,...,Y n are mutually statistically independent; 2. Identical distribution: All children are drawn from the same distribution; 5 / 54

6 Ch 1 and 2 in HN Bernoulli Model II 3. Bernoulli distribution: Y i D = Bernoulli[θ]; θ = P(Yi = 1), where 1 indicates success (a girl). 4. Parameter space: 0 < θ < 1. The book also writes θ Θ = (0, 1). The parameter space symbol is Θ. The parameter of interest is θ. 6 / 54

7 Ch 1 and 2 in HN The likelihood function of the Bernoulli model I How probable are different outcomes y 1,y 2,...,y n of Y 1,Y 2,...,Y n for different values of θ? To answer this question we use the model specified by assumptions Write the probability density function (pdf) for the Bernoulli distribution as: hence: f θ (y i ) = θ y i (1 θ) (1 y i ), for y i = 1 or 0. P(Y i = 1) = f θ (1) = θ 1 (1 θ) 0 = θ P(Y i = 0) = f θ (0) = θ 0 (1 θ) 1 = 1 θ. 7 / 54

8 Ch 1 and 2 in HN The likelihood function of the Bernoulli model II With n = 2, the joint pdf becomes: f θ (y 1, y 2 ) = [θ ] [ ] y 1 (1 θ) (1 y 1) θ y 2 (1 θ) (1 y 2) = 2 i=1 θ y i (1 θ) (1 y i ) 8 / 54

9 Ch 1 and 2 in HN The likelihood function of the Bernoulli model III And in general, f θ (y 1,y 2,...,y n ): f θ (y 1, y 2,..., y n ) = = n f θ (y i ) = i=1 n θ y n i i=1 i=1 n i=1 θ y i (1 θ) (1 y i ) (1 θ) (1 y i ) = θ n i=1 y i (1 θ) n i=1 (1 y i ) = θ nȳ (1 θ) n(1 ȳ ) = {θȳ (1 θ) (1 ȳ )} n For known θ, we can calculate the joint density f θ (y 1, y 2,..., y n ) for any value of the average ȳ = n 1 n i=1y i. 9 / 54

10 Ch 1 and 2 in HN The likelihood function of the Bernoulli model IV When we use Maximum Likelihood Estimation (MLE), the premises are turned around: The aim is to find the most likely value of θ for the outcome of the random variables Y 1,Y 2,...,Y n that we can observe. Define the Likelihood-function: L Y1,Y 2,...,Y n (θ) = f θ (Y 1, Y 2,..., Y n ) = {θȳ (1 θ) (1 Ȳ )} n (1) Y i has replaced y i to indicate that the likelihood function is based on the random variables that represent the data In (1), θ is the argument of the function, n and Ȳ are the fixed parameters. 10 / 54

11 Ch 1 and 2 in HN The likelihood function of the Bernoulli model V Since the likelihood only depends on the random variables throughȳ, we say that Ȳ is a suffi cient statistic for θ. 11 / 54

12 Ch 1 and 2 in HN The log likelihood function of the Bernoulli model I The natural logarithm (1) is called the log-likelihood function l Y1,Y 2,...,Y n (θ) = ln [{θȳ (1 θ) (1 Ȳ )} n] (1 Ȳ = n ln {θȳ (1 θ) )} (2) = n [Ȳ ln(θ) + (1 Ȳ ) ln(1 θ)] (3) The MLE estimator ˆθ is given by the first order condition for a maximum of the log-likelihood function: (Ȳ n ˆθ 1 Ȳ ) = 0 (4) 1 ˆθ 12 / 54

13 Ch 1 and 2 in HN The log likelihood function of the Bernoulli model II We obtain the MLE of θ as: ˆθ = Ȳ. (5) DIY exercise 1.1: Why is it true that this ˆθ achieves maximum, and not minimum? 13 / 54

14 Ch 1 and 2 in HN Example: Statistics Norway, data for 2016: girl babies, and boys. Hence: ˆθ = = (6) very close to the estimate for UK in 2004 according to HN p. 10. The maximum value of the log-likelihood function: max l Y1,Y 2,...,Y n ( ˆθ) = θ Θ U = [ ln( ) + ( ) ln( )] = / 54

15 Ch 1 and 2 in HN ˆθ in (5) is the unrestricted MLE of θ, obtained from the unrestricted model, where θ is maximized over the whole parameter space that defines the model, we refer to it as Θ U = (0, 1). If the parameter space of θ is restricted we say that we analyse the restricted model. If the parameter space is restricted to a single point, e.g., Θ R = {0.5} the restricted log likelihood is: max l Y1,Y 2,...,Y n (0.5) = ln [{0.5Ȳ (1 0.5) (1 Ȳ )} n] θ Θ R (7) 15 / 54

16 Ch 1 and 2 in HN HN define the log-likelihood ratio test statistic: LR = 2 log(max θ Θ R l Y1,Y 2,...,Y n (θ) max θ ΘU l Y 1,Y 2,...,Y n (θ)) (8) = 2 log( max θ Θ U l Y1,Y 2,...,Y n (θ) max θ Θ R l Y1,Y 2,...,Y n (θ)) which takes non-negative values, LR 0. The closer LR is to zero, the more likely it is that the restricted model is acceptable, and that H 0 cannot be rejected. 16 / 54

17 Ch 1 and 2 in HN Inference in the Bernoulli model I Several of you will know the Bernoulli distribution as the binominal distribution. You will also remember that a natural test-statistic to use for the hypothesis test situation: H 0 : θ = θ 0 against H 0 : θ < θ 0 is Z = ˆθ θ 0 θ o (1 θ 0 ) n, 17 / 54

18 Ch 1 and 2 in HN Inference in the Bernoulli model II and that if nθ(1 θ) > 10 the distribution of Z is well approximated by the standard normal distribution N(0, 1), (de Moivres s theorem). Hence to test: Z D N(0, 1) H 0 : θ = 0.5 against θ < 0.5 with the Norwegian 2016-data we check how probable it is to observe = in a N(0, 1) distribution. We get: P(Z > 7.75) = 0.000, which is the p-value of the test. 18 / 54

19 Ch 1 and 2 in HN Inference in the Bernoulli model III Hence we can choose the significance level of the test (Type-I error probability) as low as we want, and still formally reject H 0. How do we relate this to the log-likelihood ratio test statistic.? A lengthy derivation in HN, that we do not need to follow in any detail, shows that : 2 ˆθ U θ R LR D θ R (1 θ R ) n which reads LR is asymptotically distributed as the square of the centred and standardized random variable ˆθ. But this is identical to Z above, since we have only added subscripts U and R for the Unrestricted and Restricted log-likelihoods. (9) 19 / 54

20 Ch 1 and 2 in HN Inference in the Bernoulli model IV Hence: ˆθ U θ R θ R (1 θ R ) n and using a well known result: LR D χ 2 (1), D N(0, 1) (10) the log-likelihood rate test statistic has an approximate distribution which is Chi-squared with one degree of freedom. 20 / 54

21 Ch 1 and 2 in HN Inference in the Bernoulli model V To perform the test on the Norwegian data, we use (8) with numbers: max l Y1,Y 2,...,Y n (θ) = θ Θ U max l Y1,Y 2,...,Y n (θ) = θ Θ R LR = 2 ( ( 40850)) = 122, 0 and the p-value is zero for all practical purposes, P(LR > 122) = / 54

22 Ch 1 and 2 in HN Summing up Maximum likelihood on the Bernoulli model (Chapter 1 and 2 in HN) I A statistical model is relevant if it contains a parameter of interest. The statistical model makes it possible for us to calculate the maximized value of the log-likelihood function. And the maximum likelihood estimator (MLE) of the parameter of interest. The restricted and unrestricted log-likelihood can be used to construct a Chi-squared distributed test statistic: The (log) Likelihood-ratio test. 22 / 54

23 Ch 1 and 2 in HN About the exercises to Ch 1 and 2 in HN I Ch 1: Since we use the B-model mainly to introduce MLE, maybe not dwell too much on the exercises here. But Exercise 1.2 might be fun to work with. Ch 2: Same remark, but Exercise 2.3 is useful! Many of the other theoretical exercises are either already covered by the warm-up question set, or seem to be too intricate (Exercise. 2.4, 2.9 and 2.11), unless you are particularly interested. 23 / 54

24 Purpose: Estimating the location parameter I Ch. 3 in HN ( A first regression model ) The statistical model is: 1. Independence: Y 1,Y 2,...,Y n are mutually statistically independent; 2. Identical distribution: Y 1,..., Y n have identical distributions; 3. Normal distribution: Y i D = N(β, σ 2 );. 4. Parameter space: β is a real number ( R) and σ 2 > 0; It is usual to call β a location parameter, and σ 2 a scale parameter, cf. the Figure 3.2 and the discussion on page 32 in HN. Unlike Ch. 1 and 2, we now have model equation, namely: Y i = β + ε i (11) 24 / 54

25 Purpose: Estimating the location parameter II DIY exercise 1.2: What are the properties of the random variable ε i in (11)? 25 / 54

26 The likelihood function and ML estimators I With exactly the same motivation as for the Bernoulli model, we want to write down the likelihood function of Y 1,...,Y n. But for a different statistical distribution! DIY exercise 1.3: Follow the steps on page 35 in HN to convince yourself that the log-likelihood function is l Y1,Y 2,...,Y n (β, σ 2 ) = n 2 log(2πσ2 ) 1 2σ 2 n i=1 (Y i β) 2, π 3.14 In this case, the likelihood can be maximized in two steps. First with respect to β 0 and second wrt to σ 2. (12) 26 / 54

27 The likelihood function and ML estimators II From the 1oc of (12) wrt to β, which is the same as the 1oc for the minimum of the sum of squared deviations we obtain the MLE of β ˆβ = Ȳ (13) which is identical to the OLS estimator (and Method of moments estimator.. The concentrated likelihood for σ 2 is l Y1,Y 2,...,Y n ( ˆβ, σ 2 ) = n 2 log(2πσ2 ) 1 n 2 2σ 2 ˆε i i=1 (14) where ˆε i = Y i ˆβ 27 / 54

28 The likelihood function and ML estimators III From the 1oc we get: ˆσ 2 = 1 n which is the MLE of the scale parameter σ 2. n ˆε 2 i = ˆε 2 (15) i=1 28 / 54

29 Inference about the location parameter I Read 3.4 in HN to review your knowledge about inference about β. Specifically: Approximate inference, using the standard normal distribution with referene to CLT. Exact inference: Keeping Y i D = N(β, σ 2 ): Inference by the t(n 1) distribution. Robustness ( 3.4.3). The reliability of the inference depends on how valid and relevant the assumptions of the model are for the data, in this case log transformed wage data for US individuals. 29 / 54

30 LR test for the location parameter I US wage data in HN Ch 3 as example. Denote the value of β that we have an hypothesis about: β R. And the MLE estimate ˆβ U The expression for the LR test of the hypothesis represented by β R : LR = 2 log( max θ Θ U l Y1,Y 2,...,Y n (β, σ 2 ) max θ Θ R l Y1,Y 2,...,Y n (β, σ 2 )) = 1 2σ 2 = 1 2σ 2 n i=1 ( n (Y i ˆβ U ) 2 ( 1 2σ 2 i=1 (Y i ˆβ U ) 2 n i=1 n i=1 (Y i β R ) 2 ) (Y i β R ) 2 ) 30 / 54

31 LR test for the location parameter II US wage data in HN Ch 3 as example. To be operational we need to decide which number to use for σ 2. Suggest to use the unrestricted estimate. From the data set, we obtain: n i=1 n i=1 (Y i 4) 2 = (Y i ˆβ U ) 2 = n i=1 σ 2 U = (0.7531) 2 (Y i 5.02) 2 = / 54

32 LR test for the location parameter III US wage data in HN Ch 3 as example. ( LR = 2 = (0.7531) with p-value = 0 in the χ 2 (1)-distribution. ) ( ) 2 32 / 54

33 About exercises to Ch 3 in HN I Unless you are very interested in the statistical theory, Exercise 3.4, 3.5 and 3.12 can be dropped Exercise 3.3: Note that this is exercise extends the model by one deterministic X-variable, so it is the model with deterministic regressor that many of you will have seen. Exercise 3.9: Take note about the point about weaker assumptions, even if you do not want to work with the proofs. 33 / 54

34 Partial modelling I Ch 5 in HN We can say that in the location model, Y is regressed on a constant. But the true nature of regression, as a partial model of the statistical system consisting of Y and X, of course requires that there is random variation also in X. When we specify a regression model, the purpose of the investigation is to estimate the parameters of the conditional density of one of the variables given the other. 34 / 54

35 Partial modelling II We say that our parameters of interest are in the conditional distribution of Y given X, even though the population distribution given by a joint probability density function f (x i, y i ): f (x i, y i ) }{{} = f (y i x i ) }{{} f (x i ) }{{} simultaneous pdf conditional pdf marginal pdf (16) Since f (y i x i ) is a perfectly valid statistical distribution, it is clearly allowed to focus only on f (y i x i ), and leaving f (x i ), the marginal pdf, unspecified, at least to begin with. Modelling f (y i x i ) is clearly a partial model compared to a full model of f (x i, y i ), which implies a multi-equation model. 35 / 54

36 Partial modelling III DIY exercise 1.4: Define independence between Y and X. What does independence imply for f (y i x i )? What about f (x i y i )? The expectation associated with the probability density distribution f (y i x i ) is the conditional expectation of Y given x. When the distribution of the system, i.e. f (x i, y i ), is bivariate normal, it is a fact that f (y i x i ) is a normal pdf and that the conditional expectation is a linear function of X. 36 / 54

37 Regression model specification I The statistical model is: 1. Independence: the pairs (Y 1,X 1 ), (Y 2, X 2 ),...,(Y n,x n ) are mutually independent;; 2. Conditional normality: Y i D = N(β1 + β 2 X i, σ 2 );. 3. Exogeneity: The conditioning variable X i is exogenous; 4. Parameter space: β 1, β 2 R 2 and σ 2 > 0; A central new concept here: Exogeneity: Heuristically: If we can estimate the parameters of interest without representing (or estimating) the marginal distribution, we say that the explanatory variable is exogenous. Later we will distinguish between weak and strong exogeneity, and we will also introduce super-exogeneity. 37 / 54

38 Regression model specification II But for the time being, we just concentrate on the intuitive idea that exogeneity represents the case where we do no throw away any useful information by only considering the conditional distribution. The model equation in for the regression model is where Y i = β 1 + β 2 X i + ε i (17) ε i def = Y i E (Y i X i ) (18) DIY exercise 1.5: What are the statistical properties of ε i? 38 / 54

39 MLE of the Gaussian regression model I The likelihood function is construction in the same manner as in the location model, because we condition on the X -values. The expression (12) is replaced by l Y1,Y 2,...,Y n (β 1, β 2, σ 2 ) = n 2 log(2πσ2 ) 1 2σ 2 n i=1 (Y i β 1 β 2 X i ) 2 (19) meaning that we find the MLEs for β 1 and β 2 by minimizing the second sum of squares. What does that imply for equivalence betwen ML and OLS estimators of β 1 and β 2? The MLE for σ 2 is ˆε 2 = 1/n n i=1 ˆε2 i, with ˆε i = Y i ˆβ 1 ˆβ 2 X i. 39 / 54

40 MLE of the Gaussian regression model II As you will be aware of, MLE for σ 2 is biased. The unbiased estimator replaces 1/n by 1/(n 2) 40 / 54

41 Reparameterization of models I In Chap 5.2 makes the point that you know (from elementary econometrics), that the OLS estimators using the original data requires the solution of two simultaneous equations in the two unknown ˆβ 1 and ˆβ 2. A well known trick is to add and subtract β 2 X on the right hand side of the model equation (17). This gives a re-parameterized model where the slope parameter is the same as in (17), but with a new intercept, denoted δ 1 in HN.The 1oc for ML/OLS for this model are two equations that can be solved separately for δˆ 1 and ˆβ 2, see Lecture note 1 for a more direct argument than the one in HN / 54

42 Reparameterization of models II The two estimators ˆδ and ˆ β 2 are uncorrelated ( orthogonalized ), while the estimators βˆ 1 and βˆ 2 are correlated. In the rest of the course we refer to a re-parameterization as an operation on the model equation that changes the parameters without changing the statistical properties of the disturbance. Orthogonalization of variables means to replace the original variables in a model by variable that are uncorrelated. Orthogonalization can sometimes be attained by reparameterization but, will often affect the properties of the disturbance, and therefore the statistical model. Care must be taken! 42 / 54

43 Inference in the simple regression model I All you know about estimation and testing holds as before! But we have a new interpretation of the OLS estimators ˆ β 1 and ˆβ 2 : They are ML-estimators under the assumption of the Gaussian regression model above. And in addition to t-value, we can use the LR-test statistic. 43 / 54

44 US wage data example I Let Y represent log-wage as above, and let X be the length of schooling (the variable educ in the data set on the HN-website). OLS estimation gives: ˆβ 1 = ˆβ 2 = ˆσ 2 = ( ) 2 with βˆ 1 and βˆ 2 as unrestricted ML estimates (and where we have used the unbiased estimate for σ 2. The LR test for the hypothesis that β 2 = β 2R = 0: ( LR = 2 = ( ) ) ( ) 2 44 / 54

45 US wage data example II Confirming what the t-value of the regression output would show: That educ is a significant regressor. 45 / 54

46 The logit model I Ch 4 in HN Not a very central model in our course. But included here since it illustrates that when: the parameter of interest is the probability of a dichotomous dependent variable: Y = 1(have job) Y = 0 (no job) And the success probability varies p varies with X, the log-likelihood function is well defined, and it is given in of HN, But unlike the other models in this lecture, the first order conditions have no analytical solutions. MLE is then obtained by numerical maximization, which is however easily done with good statitical software. 46 / 54

47 Variables and systems The inclusion of exogeneity of X i in the assumptions of the regression model is a sign that even when we are working with single-equation models, we need system thinking. We now introduce the binormal model as a framework for treating the system perspective in a statistical context. In HN, the bivariate normal model is in 10.2, which can be read while still bypassing the rest of Chapter / 54

48 The binormal distribution I We can indicate that the pair of random variables (Y i,x i ) has a bivariate normal distribution by writing: ( Yi X i ) N ( µy µ X } {{ } µ ) ( σ 2, Y σ XY σ XY σx 2 }{{} Σ ) where µ and is the vector with the expectations E (Y i ) = µ Y and E (X i ) = µ XY as elements. Σ is the covariance matrix, with the variances along the principal diagonal, and the covariance outside the diagonal. (20) 48 / 54

49 The binormal distribution II The joint pdf is found as equation (10.2.1) in HN (with slightly different symbols). The pdf is defined for non-negative variances, and for 1 < ρ XY < 1 (21) where ρ XY is the population correlation coeffi cient: ρ XY = σ XY σ 2 X σ2 Y = σ XY σ X σ Y where σ x and σ y are the two standard deviations. 49 / 54

50 The binormal distribution III A property of the binormal distribution is that the conditional distribution of Y i given X i is normal With parameters: (Y i X i ) D = N(µ Yi X i, σ 2 ) µ Y X = E (Y i X i ) = β 1 + β 2 X i (22) β 1 = µ Y β 2 µ X (23) β 2 = σ XY σx 2 (24) σ 2 = Var(Y i X i ) = σy 2 (1 ρ 2 XY ) (25) 50 / 54

51 The bivariate normal model I We can write the binormal distribution in terms of model equations by defining new variables ( εyi ε Xi ) def = ( Yi µ Y X i µ X ) ( ) ( ) 0 σ 2 N, Y σ XY 0 σ XY σ 2 X }{{}}{{} 0 Σ giving the model in terms of two marginal model equations: ( Yi = µ Y + ε Yi, ε Yi N(0, σ 2 Y ) and X i = µ X + ε Xi, ε Xi N(0, σ 2 X ) ) Cov(ε Yi, ε Xi ) = σ XY 51 / 54

52 The bivariate normal model II We can represent the same statistical system by first defining a new random variable ɛ i : ɛ i def = Y i µ Yi X i (26) and then formulating a 2-variable model in terms of a conditional equation and a marginal equation: Y i = β 1 + β 2 X i + ɛ i, ɛ i N(0, σ 2 ) (27) X i = µ X + ε Xi, ε Xi N(0, σ 2 X ) (28) and Cov(ɛ i, ε Xi ) = 0 (29) 52 / 54

53 The bivariate normal model III Intuitively (29) must be true, because ɛ i is what remains after we have extracted all correlation between ε Xi and Y i by conditioning on X i (and ε Xi is one-for-one with X i ). A formal argument: Note first that: E (ɛ i X i ) = E (Y i E (Y i X i ) X i ) = E (Y i X i ) E (Y i X i ) = 0 and that the marginal model for X i implies that E (ɛ i X i ) E (ɛ i ε Xi ). Finally, therefore: showing (29). E (ɛ i ε Xi ) = 0 Cov(ɛ i, ε Xi ) = 0 53 / 54

54 The bivariate normal model IV Extremely important take away: The error-term of the conditional model is uncorrelated with the disturbance of the marginal model and with the regressor. Note that in the formal argument, we have not used the properties of the binormal distribution. In fact, Cov(ɛ i, ε Xi ) = 0 is true for any conditional/marginal model, as long as the conditional expectation is correctly specified in the light of the system. What then about omitted variables bias, which we know is due to Cov(ɛ i, X i ) = 0. Where do those omitted variables come from? 54 / 54