Module 6: Methods of Point Estimation Statistics (OA3102)

Transcription

1 Module 6: Methods of Point Estimation Statistics (OA3102) Professor Ron Fricker Naval Postgraduate School Monterey, California Reading assignment: WM&S chapter Revision:

2 Goals for this Module Learn about two methods for finding good point estimators Methods of moments Maximum likelihood Revision:

3 Methods of Point Estimation Definition of unbiasedness does not help you find unbiased estimators Here we introduce two useful methods for finding estimators with good properties Method of moments Maximum likelihood Maximum likelihood estimators often more efficient than method of moments estimators, but cost is that they can be harder to calculate Revision:

4 Method of Moments Idea: For a theoretical distribution with k parameters, find parameter estimates so that first k population moments match the data s sample moments Population moments: E.g., the first population moment of Y is just the expected value of Y, E(Y) Sample moments: m k k EY ( ) E.g., the first sample moment of Y is just the sample mean, Revision: k 1 n k Yi n i 1 Y

5 Method of Moments Approach: Choose as estimates, those values of the parameters that are solutions to the equations m k k, for k=1,..,t, where t is the number of parameters I.e., find the values of the parameters so that n 1 E( Y) Y m 1 i 1 n i1 1 n E( Y ) Yi m 2 n i1 n t t ( ) Revision: 1-12 t E Y Yi m t 5 n i1 1

6 Again, What s a Population Moment? For a distribution with pdf f (y) The kth population moment is k k k E Y y f y dy ( ) So E(Y) is the first population moment, E(Y 2 ) is the second population moment, etc. Useful fact: We know Var(Y)=E(Y 2 )-[E(Y)] 2 so 2 Var( ) 2 E Y Y E Y For example, for Y with a normal distribution First population moment: EY Second population moment: E Y Var( Y) E Y Revision:

7 And, What s a Sample Moment? For Y 1, Y 2,, Y n, a random sample from the distribution, the kth sample moment is m k 1 n k Y n i 1 Thus, the first sample moment is 1 n i n i 1 m 1 Y Y The second sample moment is Etc. m 2 1 n 2 Yi n i 1 Revision:

8 Steps in Method of Moments 1. Determine how many parameters you need to estimate E.g., one for exponential (l), two for normal (, ), etc. 2. Write out as many equations equating the population moments to sample moments as there are parameters 3. Express the population moments in terms of the parameters and substitute into your equations 4. Solve the n equations for the n unknown population parameters Revision:

9 Method of Moments by the Numbers for the Normal Distribution 1. Normal has two parameters to estimate: and 2. So, we need two equations: m and 1 1 m 2 2 which are n E( Y) Y and E( Y ) Y n i 1 3. Two slides back we expressed population moments in terms of parameters, so substituting we have Y and 4. So, now we have to solve the two equations for the two unknown population parameters Revision: n 2 Yi n i1 i

10 Solving for Method of Moments Estimates for the Normal Distribution Revision:

11 Example #2 Let Y 1, Y 2,, Y n be a random sample of service times for n customers at a logistics facility Assume the underlying distribution is exponential with parameter l Since there is only one unknown parameter, we relate E( Y) Y and solve For an exponential distribution, we know So, we set 1/ l Y Solving, we get the estimator ˆ l 1 Y EY ( ) 1/ l Revision:

12 Example 9.11 Let Y 1, Y 2,, Y n be a random sample from a uniform distribution over the interval [0, q]. Use the method of moments to derive an estimator for q. Solution: Revision:

13 Example 9.12 (not req d to know) Show that ˆ q 2Y, from Example 9.12, is a consistent estimator for q. Solution: Revision:

14 Example 9.13 Let Y 1, Y 2,, Y n be a random sample from a gamma distribution with parameters a and b We know that E(Y)=ab and Var(Y)=ab 2 Find the moment estimators for a and b Revision:

15 Example 9.13 (cont d) Revision:

16 On Method of Moments Estimators Main benefit of Method of Moments (MoM) estimators is that they are often easy to derive Also, they re consistent estimators (a topic which we haven t really covered; just know it is another good characteristic for an estimator) However, MoM estimators are sometimes not very efficient estimators Another characteristic we haven t really covered It means they can have larger standard errors than other estimators Finally, in many cases, MoM estimators biased Revision:

17 Maximum Likelihood Idea: For a theoretical distribution, find the parameters of the distribution that make the observed data most likely In a picture: Which distribution is more likely to have generated the observed data? Revision:

18 Illustrating the Idea You ve got an urn with three balls in it Each ball is either red or white You pull out two and observe they re both red Let r be the total number of red balls Before we sampled the two balls, r 0,1, 2,3 After collecting our data, we know r 2,3 How to decide on a good estimate of the total number of red balls? Revision:

19 Illustrating the Idea (continued) The probability of observing y red balls out of n balls sampled from an urn with r red balls and N total balls follows a hypergeometric distribution (see chpt. 3.7) r N r N Pr( Y y) y n y n Here we have N=3 balls, we have observed y=n=2 red balls Revision:

20 Illustrating the Idea (continued) r 3 r 3 So, we know Pr( Y 2) and we ask: What value of r maximizes the probability of having observed two red balls? For r=2, !1! 3! 1 Pr( Y 2) !0!1!0! 2!1! 3 For r=3, Thus, we decide ! 3! Pr( Y 2) !1! 2!1! rˆ 3 because it is most likely Revision:

21 Maximum Likelihood (1) Let Y 1, Y 2,, Y n have a joint dist n f(y 1,,y n q 1,,q k ) with parameters q 1, q 2,, q k For observed sample values y 1, y 2,, y n, f(y 1,y 2,,y n ; q 1,q 2,,q k ) is called the likelihood function when treated as a function of q 1, q 2,, q k We write it as L(y 1,y 2,,y n ; q 1,q 2,,q k ) The maximum likelihood estimators (MLEs) ˆ q1, ˆ q2,, ˆ qk are those values that maximize the likelihood function, so that: L y, y,, y ; ˆ q, ˆ q,, ˆ q L y, y,, y ; q, q,, q 1 2 n 1 2 k 1 2 n 1 2 k for any values of q,, q Revision: k

22 Maximum Likelihood (2) Idea is to find parameter estimates that make the observed data most likely If Y1, Y2,..., Yn iid, the likelihood function is L( Y,..., Y ; q, q,, q ) f ( Y ; q, q,, q ) 1 n 1 2 k i 1 2 k i1 Usually easier to maximize the log-likelihood: l( Y,..., Y ; q, q,, q ) ln f ( Y ; q, q,, q ) 1 n 1 2 k i 1 2 k i1 To maximize, remember your calculus Take the (partial) derivative(s) of the log-likelihood function with respect to q Set derivative(s) equal to zero and solve for q(s) Revision: n n

23 Example: MLE for Exponential Given a random sample Y 1, Y 2,, Y n from an exponential distribution, find the MLE for l Revision:

24 Example 9.14 Let Y 1, Y 2,, Y n be iid according to a Bernoulli distribution with probability p. That is, p, y1 Pr( Y y) 1 p, y 0 Find the MLE for p. Revision:

25 Example 9.14 (continued) Revision:

26 Example 9.15 Given a random sample Y 1, Y 2,, Y n from a normal distribution, find the MLEs for and 2 Revision:

27 Example 9.15 (continued) Revision:

28 Example 9.16: Sometimes Calculus Does Not Work Back to the reaction time problem: X~U[0,q] and we want to estimate q with a random sample Y 1, Y 2,, Y n Since f(y;q) = 1/q, 0 < y < q (and 0 otherwise): 1, 0 y1 q,,0 y n n q L( y1,, yn; q ) q 0, otherwise But maxima occurs at a point of discontinuity, so calculus does not work Graphically, though, it s clear that ˆ q max( ) Y i Revision: * Figure from Probability and Statistics for Engineering and the Sciences, 7 th ed., Duxbury Press, 2008.

29 Large Sample Behavior of MLE Under very general conditions on the joint distribution, when n is large The MLE of any parameter q is approximately unbiased The MLE of q is nearly as small as can be achieved by any other estimator That is, the MLE of q is approximately the MVUE of q A good thing! Revision:

30 MLEs for Functions of Parameters The Invariance Property: Let ˆ q1, ˆ q2,, ˆ qk be the MLEs of the parameters q 1, q 2,, q k. Then the MLE of any one-to-one function h(q 1, q 2,, q k ) of these parameters is the function of the MLEs: h ˆ q ˆ q ˆ q Example: As we showed, in the normal case the MLEs for and 2 are So, if we want the MLE for, we have Thus: i1 Revision: ,,, k 1 2 n n ˆ Y Yi and ˆ Y 2 i Y n n ˆ 1 n n Y 2 i Y i1 i1 h, 2 2

31 What We Covered in this Module Learned about two methods for finding good point estimators Methods of moments Maximum likelihood Revision:

32 Homework WM&S chapter 9 Required exercises: 69, 71, 76, 80 part a, 81, 88 Also, do the following additional problem: Let Y 1, Y 2,, Y n be a random sample from a gamma distribution with parameters a and b. Find the Method of Moment estimators for a and b. Extra credit: 79, 92 Hints and instructions: Ex. 81: Ignore the hint in the book. Instead, use the Invariance Principle we discussed in class Revision:

33 Homework Hints and instructions continued: Ex. 88: To compare estimators, do so empirically with q =1 o To generate a random observation from the pdf f(y)=2y, 0<y<1, just take the square root of a random uniform o Then, first, empirically demonstrate that the estimators are unbiased by showing the sample mean of a large number of qˆ values is very close to 1 o Second, estimate the standard errors for both estimators via simulation and see if the standard error of one estimator is smaller than the standard error of the other Revision: