2.3 Methods of Estimation

Transcription

1 96 CHAPTER 2. ELEMENTS OF STATISTICAL INFERENCE 2.3 Methods of Estimation 2.3. Method of Moments The Method of Moments is a simple technique based on the idea that the sample moments are natural estimators of population moments. Thek-th population moment of a random variabley is µ k = E(Y k ), k =,2,... and thek-th sample moment of a sampley,...,y n is m k = Yi k n, k =,2,... If Y,...,Y n are assumed to be independent and identically distributed then the Method of Moments estimators of the distribution parameters ϑ,...,ϑ p are obtained by solving the set ofpequations: µ k = m k, k =,2,...,p. Under fairly general conditions, Method of Moments estimators are asymptotically normal and asymptotically unbiased. However, they are not, in general, efficient. Example 2.7. Let Y i iid N(µ,σ 2 ). We will find the Method of Moments estimators ofµandσ 2. We have µ = E(Y) = µ, µ 2 = E(Y 2 ) = σ 2 + µ 2, m = Y and m 2 = n Y i 2/n. So, the Method of Moments estimators ofµand σ2 satisfy the equations µ = Y Thus, we obtain µ = Y σ 2 = n σ 2 + µ 2 = n Y 2 i Y 2 = n Y 2 i. (Y i Y) 2.

2 2.3. METHODS OF ESTIMATION 97 Estimators obtained by the Method of Moments are not always unique. Example 2.8. Let Y i iid Poisson(λ). We will find the Method of Moments estimator of λ. We know that for this distribution E(Y i ) = var(y i ) = λ. Hence By comparing the first and second population and sample moments we get two different estimators of the same parameter, λ = Y λ 2 = n Y 2 i Y 2. Exercise 2.. Let Y = (Y,...,Y n ) T be a random sample from the distribution with the pdf given by { 2 (ϑ y), y [0,ϑ], f(y;ϑ) = ϑ 2 0, elsewhere. Find an estimator ofϑusing the Method of Moments Method of Maximum Likelihood This method was introduced by R.A.Fisher and it is the most common method of constructing estimators. We will illustrate the method by the following simple example. Example 2.9. Assume thaty i Bernoulli(p),i =,2,3,, with probability of iid success equal top, wherep Θ = {, 2, 3 } i.e.,pbelongs to the parameter space of only three elements. We want to estimate parameter p based on observations of the random sampley = (Y,Y 2,Y 3,Y ) T. The joint pmf is P(Y = y;p) = P(Y i = y i ;p) = p y i ( p) y i. The different values of the joint pmf for allp Θ are given in the table below

3 98 CHAPTER 2. ELEMENTS OF STATISTICAL INFERENCE p 8 27 P(Y = y;p) y i We see thatp( Y i = 0) is largest whenp =. It can be interpreted that when the observed value of the random sample is (0,0,0,0) T the most likely value of the parameter p is p =. Then, this value can be considered as an estimate of p. Similarly, we can conclude that when the observed value of the random sample is, for example,(0,,,0) T, then the most likely value of the parameter is p =. 2 Altogether, we have p = p = 2 p = 3 if we observe all failures or just one success; if we observe two failures and two successes; if we observe three successes and one failure or four successes. Note that, for each point (y,y 2,y 3,y ) T, the estimate p is the value of parameter p for which the joint mass function, treated as a function of p, attains maximum (or its largest value). Here, we treat the joint pmf as a function of parameter p for a given y. Such a function is called the likelihood function and it is denoted by L(p y). Now we introduce a formal definition of the Maximum Likelihood Estimator (MLE). Definition 2.. The MLE(ϑ) is the statistict(y) = ϑ whose value for a given y satisfies the condition L( ϑ y) = supl(ϑ y), ϑ Θ wherel(ϑ y) is the likelihood function forϑ. Properties of MLE The MLEs are invariant, that is MLE(g(ϑ)) = g(mle(ϑ)) = g( ϑ).

4 2.3. METHODS OF ESTIMATION 99 MLEs are asymptotically normal and asymptptically unbiased. Also, they are efficient, that is CRLB(g(ϑ)) eff(g( ϑ)) = lim =. n varg( ϑ) In this case, for largen,varg( ϑ) is approximately equal to the CRLB. Therefore, for largen, g( ϑ) N ( g(ϑ),crlb(g(ϑ)) ) approximately. This is called the asymptotic distribution ofg( ϑ). Example Suppose thaty,...,y n are independentpoisson(λ) random variables. Then the likelihood is n λ y i e λ L(λ y) = = λ n y i e nλ y i! n y. i! We need to find the value of λ which maximizes the likelihood. This value will also maximizel(λ y) = logl(λ y), which is easier to work with. Now, we have l(λ y) = y i logλ nλ log(y i!). The value of λ which maximizes l(λ y) is the solution of dl/dλ = 0. Thus, solving the equation dl n dλ = y i n = 0 λ yields the estimator ˆλ = T(Y) = n Y i/n = Y, which is the same as the Method of Moments estimator. The second derivative is negative for all λ hence, λ indeed maximizes the log-likelihood. Example 2.2. Suppose that Y,...,Y n are independent N(µ,σ 2 ) random variables. Then the likelihood is n { L(µ,σ 2 y) = exp (y } i µ) 2 2πσ 2 2σ 2 { } = (2πσ 2 ) n 2 exp (y 2σ 2 i µ) 2 and so the log-likelihood is l(µ,σ 2 y) = n 2 log(2πσ2 ) 2σ 2 (y i µ) 2.

5 00 CHAPTER 2. ELEMENTS OF STATISTICAL INFERENCE Thus, we have and l µ = 2σ 2 2(y i µ) = (y σ 2 i µ) l σ = n 2 2 2πσ 22π + 2σ Setting these equations to zero, we obtain (y i µ) 2 = n 2σ 2 + 2σ (y i µ) 2. ˆσ 2 (y i ˆµ) = 0 y i = nˆµ, so that ˆµ = Y is the maximum likelihood estimator ofµ, and n 2ˆσ 2 + 2ˆσ (y i y) 2 = 0 nˆσ 2 = (y i y) 2, so that ˆσ 2 = n (Y i Y) 2 /n is the maximum likelihood estimator ofσ 2, which are the same as the Method of Moments estimators. Exercise 2.2. LetY = (Y,...,Y n ) T be a random sample from a Gamma distribution,gamma(λ,α), with the following pdf Assume that the parameterαis known. f(y;λ,α) = λα Γ(α) yα e λy, for y > 0. (a) Identify the complete sufficient statistic forλ. (b) Find the MLE[g(λ)], where g(λ) =. Is the estimator a function of the λ complete sufficient statistic? (c) Knowing that E(Y i ) = α for all i =,...,n, check that the MLE[g(λ)] is λ an unbiased estimator ofg(λ). What can you conclude about the properties of the estimator?

6 2.3. METHODS OF ESTIMATION Method of Least Squares IfY,...,Y n are independent random variables, which have the same variance and higher-order moments, and, if each E(Y i ) is a linear function of ϑ,...,ϑ p, then the Least Squares estimates ofϑ,...,ϑ p are obtained by minimizing S(ϑ) = {Y i E(Y i )} 2. The Least Squares estimator of ϑ j has minimum variance amongst all linear unbiased estimators of ϑ j and is known as the best linear unbiased estimator (BLUE). If the Y i s have a normal distribution, then the Least Squares estimator of ϑ j is the Maximum Likelihood estimator, has a normal distribution and is the MVUE. Example Suppose thaty,...,y n are independentn(µ,σ 2 ) random variables and that Y n +,...,Y n are independent N(µ 2,σ 2 ) random variables. Find the least squares estimators ofµ and µ 2. Since E(Y i ) = { µ, i =,...,n, µ 2, i = n +,...,n, it is a linear function of µ and µ 2. The Least Squares estimators are obtained by minimizing S = n {Y i E(Y i )} 2 = (Y i µ ) 2 + (Y i µ 2 ) 2. i=n + Now, and S µ = 2 S µ 2 = 2 n i=n + (Y i µ ) = 0 µ = n Y i = Y n (Y i µ 2 ) = 0 µ 2 = n 2 i=n + Y i = Y 2, wheren 2 = n n. So, we estimate the mean of each group in the population by the mean of the corresponding sample.

7 02 CHAPTER 2. ELEMENTS OF STATISTICAL INFERENCE Example Suppose thaty i N(β 0 +β x i,σ 2 ) independently fori =,2,...,n, where x i is some explanatory variable. This is called the simple linear regression model. Find the least squares estimators ofβ 0 and β. Since E(Y i ) = β 0 + β x i, it is a linear function of β 0 and β. So we can obtain the least squares estimates by minimizing S = (Y i β 0 β x i ) 2. Now, S β 0 = 2 (Y i β 0 β x i ) = 0 Y i n β 0 β x i = 0 β 0 = y β x and S β = 2 x i (Y i β 0 β x i ) = 0 x i Y i β 0 x i β x 2 i = 0. Substituting the first equation into the second one, we have x i Y i (y β x) ( x i β x 2 i = 0 nx 2 ) β = nxy x i Y i. x 2 i Hence, we have the estimators β 0 = Y β x and β = n x iy i nxy n x2 i nx2. These are the Least Squares estimators of the regression coefficients β 0 and β. Exercise 2.3. Given data,(x,y ),...,(x n,y n ), assume thaty i N(β 0 +β x i,σ 2 ) independently fori =,2,...,n and that σ 2 is known. (a) Show that the Maximum Likelihood Estimators ofβ 0 andβ must be the same as the Least Squares Estimators of these parameters.

8 2.3. METHODS OF ESTIMATION 03 (b) The quench bath temperature in a heat treatment operation was thought to affect the Rockwell hardness of a certain coil spring. An experiment was run in which several springs were treated under four different temperatures. The table gives values of the set temperatures (x) and the observed hardness (y, coded). Assuming that hardness depends on temperature linearly and the variance of the r.vs is constant we may write the following model: E(Y i ) = β 0 +β x i, var(y i ) = σ 2. Calculate the LS estimates of β 0 and of β. What is the estimate of the expected hardness given the temperaturex = 0? Run x y In this section we were considering so called point estimators. We used various methods, such as the Method of Moments, Maximum Likelihood or Least Squares, to derive them. We may also construct the estimators using the Rao- Blackwell Theorem. The estimators are functions T(Y,...,Y n ) Θ, that is, their values belong to the parameter space. However, the values vary with the observed sample. If the estimator is MVUE we may expect that on average the calculated estimates are very close to the true parameter and also that the variability of the estimates is the smallest possible. Sometimes it is more appropriate to construct an interval which covers the unknown parameter with high probability and whose limits depend on the sample. We introduce such intervals in the next section. The point estimators are used in constructing the intervals.