Chapter 6 - Sections 5 and 6, Morris H. DeGroot and Mark J. Schervish, Probability and

Transcription

1 References Chapter 6 - Sections 5 and 6, Morris H. DeGroot and Mark J. Schervish, Probability and Statistics, 3 rd Edition, Addison-Wesley, Boston. Chapter 5 - Section 2, Bernard W. Lindgren, Statistical Theory, 3 rd Edition, MacMillan, New York. Method of Moments Estimation Assume that the joint density function f(x) depends on a K-vector of unknown parameters 2. Let m h = n -1 E n i=1x h i denote the sample moment of order h and : h = E(X h i) the corresponding population moment. The Method of Moments (MM) estimator is defined as follows: Step 1: Find the first K population moments of X. These moments will generally be functions of the unknown parameter vector 2. That is, : h = R h (2), for h=1 to K. (This is a theoretical exercise. The results of this step are specific to the form of the joint density.) Step 2: Use the first K sample moments to estimate the corresponding population moments, and solve for the MM estimates of 2. The MM estimates simultaneously solve: R h (2$) = m h for h=1 to K. The MM estimator is based on the intuitively pleasing idea of using sample moments to estimate population moments.

2 Application Assume that X~U(0,"). There is one unknown parameter. Step 1: Find the first population moment. For the uniform, this is E(X) = "/2. Step 2: Estimate the first population moment with the first sample moment and solve for the estimate of ". The MM estimator solves &X = "$/2, which gives "$ = 2&X. Maximum Likelihood Estimation The joint density function, f(x), defines a probability measure over the sample space of the random vector X. In parametric estimation problems, the joint density function is dependent on an unknown parameter vector, 2. For a given value of 2, the joint density function may be used to calculate the ex-ante probability of the observed sample vector X. When the joint density is considered as a function of 2 for a given value of X, it does not provide a probability measure over the parameter space of 2, and consequently, does not tell us the probability that a particular 2 has generated the sample vector X. Rather, it gives an intuitive measure of the degree to which the sample would have been "expected" for each value of 2. The Method of Maximum Likelihood is based on the simple idea that the best estimate of 2 is that which maximizes the ex-ante probability of the observed sample. We are implicitly assuming that the observed sample is not unusual. The Intuition for ML Estimation A coin is known to be either fair or two-headed. The coin is tossed 100 times, resulting in 100 heads. While we cannot rule out a fair coin generating 100 heads in 100 tosses, the exante probability of this event is extremely small relative to the ex-ante probability of a twoheaded coin giving 100 heads in 100 tosses. This event would be quite unusual for a fair coin

3 and the only possible outcome for a two-headed coin. According to the Maximum Likelihood Principle, the two-headed coin provides the "best" explanation of the observed sample. While this is an intuitively pleasing result, there is no right or wrong here. The observed sample could have been generated by either specification, and there is no way of determining the true specification based solely upon the sample information. Integral over the Parameter Space Consider X~U(0,"). Clearly, the uniform density must satisfy the second axiom of probability. That is, I " o " -1 dx = 1. Now consider variation in ". The parameter " can fall anywhere in R +. Note that I 4 o " -1 d" = ln(") 4 o = 4-(-4). When viewed as a function of ", the joint density does not satisfy the axioms of probability. The Likelihood Function The likelihood function is defined to be any function proportional to the joint density function of the observed data. That is, Clearly, L(2) = Kf(X) = K[ A i f(x i )] with SI observations. lnl(2) = lnk + lnf(x) = lnk + E i lnf(x i ) with SI observations. Since the log transformation is monotonic, the value of the vector 2 that maximizes lnl(2) also maximizes L(2). Furthermore, the maximizing value of 2 is unaffected by the value of K. Therefore, we may assume that K=1 without loss of generality.

4 The Score Equations The score equations are the vector of first partial derivatives of lnl(2) with respect to 2. That is, S(2) = MlnL(2)/M2. Sufficient Conditions for a Local MLE We will define a local maximum likelihood estimator as any root of S(2)=0 that corresponds to a local maximum of lnl(2). Sufficient conditions to establish that an estimator $2 is a local MLE are: 1. S($2) = 0 2. H($2) is negative definite where H(2) = M 2 lnl(2)/m2m2' denotes the Hessian matrix of lnl(2). Sufficient Conditions for a Global MLE We will define a global maximum likelihood estimator as any value of 2 that maximizes lnl(2) over the entire parameter space. Sufficient conditions to establish that an estimator $2 is a unique global MLE are: 1. S($2) = 0 2. H(2) is negative definite for all 2.

5 Application is Consider ML estimation of the parameter of an exponential. The log likelihood function The score equation is lnl(8) = n ln(8) - E n i=1 8X i The ML estimator solves MlnL(8)/M8 = n/8 - E n i=1 X i = n/8 - n&x S($8) = n/$8 - n&x = 0 Solving for $8 gives $8 = 1/&X. The ML estimator of 8 is the reciprocal of sample mean. The Hessian matrix, a scalar in this case, is H(8) = M 2 lnl(8)/m8 2 = - n/8 2 < 0 Since the Hessian is negative for all 8, the conditions for a unique global MLE are satisfied.