STAT 135 Lab 2 Confidence Intervals, MLE and the Delta Method

Transcription

1 STAT 135 Lab 2 Confidence Intervals, MLE and the Delta Method Rebecca Barter February 2, 2015

2 Confidence Intervals

3 Confidence intervals What is a confidence interval? A confidence interval is calculated in such a way that the interval contains the true value of θ with some specified probability (coverage probability). What kind of parameters can θ correspond to? θ = µ from N(µ, σ 2 ) θ = p from Binomial(n, p) θ typically corresponds to a parameter from a distribution, F, from which we are sampling X i F (θ)

4 Confidence intervals We usually write the coverage probability in the form of 1 α If the coverage probability is 95%, then α = Let q α be the number such that P(Z < q α ) = 1 α where Z N(0, 1) By symmetry of the normal distribution, we have also that q α = q (1 α)

5 Confidence intervals q α is the number such that P(Z < q α ) = 1 α where Z N(0, 1). Note also that by the symmetry of the normal distribution 1 α = P(Z < q α ) = P( q α/2 < Z < q α/2 ) For a 95% CI, we have: q 0.05/2 = 1.96 because P( 1.96 < Z < 1.96) = 0.95

6 Confidence intervals Suppose that our estimate, ˆθ n, of θ, asymptotically satisfies ˆθ n θ σˆθ n N(0, 1) So in all of the equations in the previous slides, we can replace Z with ˆθ n θ σ and rearrange so that θ is the subject. ˆθn

7 Confidence intervals Recall that 1 α = P( q α/2 < Z < q α/2 ) Given that ˆθ n θ σ N(0, 1), we have also the result that ˆθn ( 1 α = P q α/2 < ˆθ ) n θ < q α/2 σˆθ n rearranging to make θ the subject, we have 1 α = P (ˆθ n q α/2 σˆθn < θ < ˆθ n + q α/2 σˆθn)

8 Confidence intervals We have that 1 α = P (ˆθ n q α/2 σˆθn < θ < ˆθ n + q α/2 σˆθn) Recall that if we re looking for a 95% confidence interval (CI), then we are looking for an interval (a, b) such P(a < θ < b) = Thus, the 95% CI for θ can be found from 0.95 = P (ˆθ n q σˆθn < θ < ˆθ n + q σˆθn) For a general (1 α)% CI, the interval [ˆθ n q (1 α/2) σˆθn, ˆθ n + q (1 α/2) σˆθn ] contains θ with probability 1 α.

9 Exercise 1

10 Confidence intervals - exercise CI exercises: 1. In R, generate 1000 random samples, x 1, x 2,..., x 1000, from a (continuous) Uniform(5, 15) distribution 2. From the 1000 numbers you have just generated, draw 100 simple random samples (without replacement!), X 1,..., X 100. Repeat this 1000 times, so that we have 1000 samples of size For each sample of size 100, compute the sample mean, and produce a histogram (preferably using ggplot()) of the 1000 sample means calculated above. What distribution does the sample mean (approximately) follow, and why? 4. For each sample, calculate the 95% confidence interval for the population mean. 5. Of the 1000 confidence intervals, what proportion of them cover the true mean µ = = 10?

11 Maximum likelihood estimation

12 Maximum likelihood estimation Confidence interval for θ: calculate a range of values in which the true value of the parameter θ lies with some specified probability. Maximum likelihood estimator for θ: calculate a single value which estimates the true value of θ by maximizing the likelihood function with respect to θ i.e. find the value of θ that maximizes the likelihood of observing the data given.

13 Maximum likelihood estimation What is the likelihood function? The likelihood function, lik(θ), is a function of θ which corresponds to the probability of observing our sample for various value of θ. How to find the value of θ that maximizes the likelihood function?

14 Maximum likelihood estimation Assume that we have observed i.i.d. random variables X 1,..., X n and that their distribution has density/frequency function f θ. Suppose that the observed value of X i is x i for each i = 1, 2,..., n How do we write down the likelihood function? The (non-rigorous) idea: lik(θ) = P(X 1 = x 1,..., X n = x n ) = P(X 1 = x 1 )...P(X n = x n ) n = f θ (X i ) i=1 (Note that this proof is not rigorous for continuous variables since they take on specific values with probability 0)

15 Maximum likelihood estimation There are 4 main steps in calculating the MLE, ˆθ MLE, of θ. 1. Write down the likelihood function, lik(θ) = n i=1 f θ(x i ). 2. Calculate the log-likelihood function l(θ) = log(lik(θ)) (Note: this is because it is often much easier to find the maximum of the log-likelihood function than the likelihood function) 3. Differentiate the log-likelihood function with respect to θ. 4. Set the derivative to 0, and solve for θ.

16 Maximum likelihood estimation - example Example: Suppose X i Bernoulli(p). f p (x) = p x (1 p) 1 x Step 1: Write down the likelihood function: lik(p) = = n f p (X i ) i=1 n p X i (1 p) 1 X i i=1 = p n i=1 X i (1 p) n i=1 (1 X i )

17 Maximum likelihood estimation - example Example: Suppose X i Bernoulli(p). f p (x) = p x (1 p) 1 x Step 1: lik(p) = p n i=1 X i (1 p) n i=1 (1 X i ) Step 2: Calculate the log-likelihood function: l(p) = log(lik(p)) = n n X i log(p) + (1 X i ) log(1 p) i=1 i=1

18 Maximum likelihood estimation - example Example: Suppose X i Bernoulli(p). f p (x) = p x (1 p) 1 x Step 2: l(p) = n i=1 X i log(p) + n i=1 (1 X i) log(1 p) Step 3: Differentiate the log-likelihood function with respect to p: dl(p) dp = n i=1 X i p n i=1 (1 X i) 1 p

19 Maximum likelihood estimation - example Example: Suppose X i Bernoulli(p). Step 3: dl(p) n i=1 dp = X i p f p (x) = p x (1 p) 1 x n i=1 (1 X i ) 1 p Step 4: Set the derivative to 0, and solve for p: dl(p) n i=1 = 0 = ˆp MLE = X i = X dp n So the MLE for p where X i Bernoulli(p) is just equal to the sample mean.

20 Method of Moments (MOM)

21 Method of Moments Confidence interval for θ: calculate a range of values in which the true value of the parameter θ lies with some specified probability. Maximum likelihood estimator for θ: calculate a single value which estimates the true value of θ by maximizing the likelihood function with respect to θ. Method of moments estimator for θ: By equating the theoretical moments to the empirical (sample) moments, derive equations that relate the theoretical moments to θ. The equations are then solved for θ. Suppose X follows some distribution. The kth moment of the distribution is defined to be µ k = E[X k ] = g k (θ) which will be some function of θ.

22 Method of Moments MOM works by equating the theoretical moments (which will be a function of θ) to the empirical moments. Moment Theoretical Moment Empirical Moment first moment E[X ] second moment E[X 2 ] third moment E[X 3 ] n i=1 X i n n i=1 X i 2 n n i=1 X i 3 n

23 Method of Moments MOM is perhaps best described by example. Suppose that X Bernoulli(p). Then the first moment is given by E[X ] = 0 P(X = 0) + 1 P(X = 1) = p Moreover, we can estimate the E[X ] by taking a sample X 1,..., X n and calculating the sample mean : X = 1 n n i=1 X i We approximate the first theoretical moment, E[X ], by the first empirical moment, X, i.e. ˆp MOM = X which is the same as the MLE estimator! (note that this is not always the case...)

24 Exercise 2

25 Exercise Question 43, Chapter 8 (page 320) from John Rice The file gamma-arrivals contains a set of gamma-ray data consisting of the times between arrivals (interarrival times) of 3,935 photons (units are seconds) 1. Make a histogram of the interarrival times. Does it appear that a gamma distribution would be a plausible model? 2. Fit the parameters by the method of moments and by maximum likelihood. How do the estimates compare? 3. Plot the two fitted gamma densities on top of the histogram. Do the fits look reasonable? Hint 1: the gamma distribution can be written as f α,β (x) = βα Γ(α) x α 1 e βx Hint 2: the MLE for α has no closed-form solution - use: ˆα MLE = 1

26 The δ-method

27 The δ-method Recall that the CLT says n(x n µ) N(0, σ 2 ) What if we have some general function g( )? n(g(x n ) g(µ))?

28 The δ-method The δ-method tells us that n(g(x n ) g(µ)) N(0, σ 2 (g (µ)) 2 ) For a proof for the general case, see This method can be used to find the variance of a function of our random variables!