lim F n(x) = F(x) will not use either of these. In particular, I m keeping reserved for implies. ) Note:

Transcription

1 APPM/MATH 4/5520, Fall 2013 Notes 9: Convergence in Distribution and the Central Limit Theorem Definition: Let {X n } be a sequence of random variables with cdfs F n (x) = P(X n x). Let X be a random variable with cdf F(x). X n converges in distribution to X if for all x where F is continuous. We write X n D X. lim F n(x) = F(x) n (Other notations are X n L X as this is sometimes called convergence in law, and Xn X. I will not use either of these. In particular, I m keeping reserved for implies. ) Note: 1. Convergence in distribution is weaker in convergence in probability in the sense that X n P X X n D X but not necessarily the other way around. We will prove this. 2. This is the type of convergence used in the Central Limit Theorem! Examples: 1. Let X 1,X 2,...,X n iid Pareto(1). Consider Y n := nx (1) where X (1) = min(x 1,X 2,...,X n ). The Pareto(1) pdf is The cdf is f(x) = F(x) = γ (1+x) γ+1 I (0, )(x) γ=1 = x 0 (1+u) 2 du = 1 1+u We need to find the cdf of Y n and then let n... as n. 1 (1+x) 2 I (0, )(x). x 0 = x F Yn (y) = P(Y n y) = P(nX (1) y) = P(X (1) y/n) ) = 1 [P(X 1 > y/n)] n n = 1 ( 1 1+y/n = 1 (1+y/n) n 1 e y This is the cdf of the exponential distribution with rate 1. (If you don t recognize it, take the derivative to find the pdf.) So, Y n D Y exp(rate = 1).

2 2. Let X 1,X 2,...,X n iid unif(0,1). Consider Y n = X (n) = max(x 1,X 2,...,X n ). The cdf for the unif(0,1) distribution is F(x) = P(X x) = x 0 1du = x, for 0 x < 1. Note that the cdf is 0 for x < 0 and is 1 for x 1. The cdf of Y n is F Yn (y) = P(Y n y) = P(X (n) y) for 0 y < 1. The complete cdf is = [P(X 1 y)] n = y n 0, y < 0 F Yn (y) = y n, 0 y < 1 1, y 1 Letting n, we get the limiting cdf which looks like this F(y) = { 0, y < 1 1, y 1 This is the cdf of a random variable that equals 1 with probability 1. We say Y n D Y where Y = 1 w.p. 1 or simply Y n D 1. Theorem: X n P X Xn D X Proof: Let F n (x) = P(X n x). Let F(x) = P(X x). Let ε > 0. Let x be a point of continuity of F. Note that F n (x) = P(X n x) = P(X n x,x x+ε)+p(x n x,x > x+ε)

3 Also, note that the event that {X n x,x > x+ε} is contained in the event that { X n X > ε}. That is, if the first event is true then the second is too. (Not convinced? Trying sketching it out on a number line.) Thus, This is trivially since we have removed a restriction. So, F n (x) P(X n x,x x+ε)+p( X n X > ε). P(X x+ε)+p( X n X > ε) F n (x) = F(x+ε)+P( X n X > ε). On the other hand, F(x ε) = P(X x ε) = P(X x ε,x n x)+p(x x ε,x n > x) P(X n x)+p( X n X > ε) = F n (x)+p( X n X > ε). So, we have F(x ε) P( X n X > ε) F n (x)+f(x+ε)+p( X n X > ε). Letting n all the way across, and using the fact that X n P X, we get or F(x ε) 0 lim n F n(x) F(x+ε)+0 F(x ε) lim n F n(x) F(x+ε). (1) Notice that I did not write F(x) in the middle we are not yet sure that F n (x) has a limit! But... letting ε become arbitrarily small in (1) makes both the right and left-hand sides go to F(x), squeezing in of the limit in the center. So, lim F n(x) = F(x). n Therefore, X n D X, as desired! In general, X n D X Xn P X. For example, consider X1,X 2,... all iid with distribution P(X i = 1) = 1/2 P(X i = 1) = 1/2

4 Let X be a random variable with the exact same distribution. Clearly (since the cdfs are all the same and are not changing) X n D X. However, so X n P X. P( X n X 2) = 1/2 0 We have already proven that P D X n X implies Xn X and we have seen an example showing that D P X n X does not imply Xn X. However, if X is a constant, we do have the reverse implication! Theorem: Suppose that X D c where c is a constant. Then X n P c. Proof: Let ε > 0. P( X n c > ε) = P(X n > c+ε)+p(x n < c ε) P(X n > c+ε)+p(x n c ε) = 1 F Xn (c+ε)+f Xn (c ε) D Since X n c, limn F Xn (c+ε) = F(c+ε) and lim n F Xn (c ε) = F(c ε), where F(x) is the cdf of the constant random variable X = c. This cdf is { 0, if x < c F(x) = P(X x) = P(c x) = 1, if x c So, lim P( X n c > ε) lim [1 F X n n n (c+ε)+f Xn (c ε)] = = 0. Since a probability can t get less than 0, lim P( X n c > ε) 0 lim P( X n c > ε) = 0 n n so we have X n P c.

5 We have already claimed that moment generating functions uniquely determine a distribution. A distribution, for us, is defined by a pdf which we have then used to define a cdf. So, you might find the following theorem, which we will not prove, highly believable. Theorem: Let X 1,X 2,... be a sequence of random variables. Let F n (x) be the cdf of X n. Let M n (t) be the mgf of X n. Let X be a random variable with cdf F(x) and mgf M(t). Then, lim M n(t) = M(t) limf n (x) = F(x). n n Notes: 1. Just for the record, the first limit only needs to hold only for all t in an open interval containing zero. The second limit needs to hold at all points of continuity of F. 2. This result would go in both directions if moment generating functions always existed, but they don t. The mgf is defined as an expectation which is defined by an integral that sometimes doesn t diverges to! 3. This is how we prove the Central Limit Theorem! The Central Limit Theorem Let X 1,X 2,...,X n be a random sample from a distribution with mean µ and variance σ 2 <. Consider the sequence of random variables {Z n } where Then Z n D Z N(0,1) as n. Z n := X µ σ/ n. A proof of this can be found in the Extra Reading section of the course website. The proof uses moment generating functions and the previous theorem to show convergence in distribution. In the case that the mgfs do not exist, the CLT still hold and should be proven using characteristic functions. Characteristic function are complex-valued generalization of moment generating functions that always exist. However, they are outside the scope of our course this semester. So sad... Definition/Notation: We say that X n has an asymptotically normal distribution with mean µ n and variance σ 2 n if X n µ n σ n D N(0,1).

6 (Note that, even without the limit, that thing on the left side has mean 0 and variance 1.) If X n has an asymptotically normal distribution with mean µ n and variance σ 2 n, we will write X n asmyp N(µ n,σ 2 n ). Example 1: According to the Central Limit Theorem, if X 1,X 2,...,X n is a random sample from any distribution with mean µ and variance σ 2 <, X = X n asymp N(µ,σ 2 /n). Example 2: According to the Central Limit Theorem, if X 1,X 2,...,X n is a random sample from any distribution with mean µ and variance σ 2 <, we also have that n i=1 X i is asymptotically normal since and since Therefore, we see that X µ σ/ n d Z N(0,1) 1 X µ σ/ n = n Xi µ σ/ = n 1 n Xi µ n n σ/ Xi nµ = n nσ/ n i=1 X i asymp N(nµ,nσ 2 ). An Example of the Central Limit Theorem in Use: Let X denote the mean of a random sample of size 100 from the χ 2 (50) distribution. Give an approximate value for P(49 < <51). Answer: Just for the record, we can give an exact value since So, X 1,X 2,...,X 100 iid χ 2 (50) = Γ(25,1/2) P(49 < <51) = P(4900 < X i < 5100) 100 i=1 X i Γ(2500,1/2). = ( ) x Γ(2500) e 1 2 x dx (This computation requires 2, 500 iterations of integration by parts and was done by Mathematica!) With the Central Limit Theorem, we know that the distribution of X is approximately normal for large sample sizes. In statistics textbooks, a sample size of n 30 or sometimes n 40, is generally considered large. The mean for X, normal or not, is µ X = µ = 25/(1/2) = 50

7 and the variance is σ 2 X = σ2 n = 25/(1/4) 100 We can standardize X into something with mean 0 and variance 1, which, for large samples, will be approximately a standard normal randon variable. So, P(49 < X < 51) = P = P = 1. ( ) /10 < X µ σ/ n < /10 ( 1 < X µ σ/ n < 1 ) P( 1 < Z < 1) where Z N(0,1) is a standard normal random variable. Now, P( 1 < Z < 1) contin = P( 1 < Z 1) = P(Z 1) P(Z 1) z table = ( ) = Woo hoo! One final thing that I forgot to mention in class... Here is a Theorem that can be used for mixing convergence types. Slutsky s Theorem: P d Suppose that X n a and Yn Y. Then 1. X n +Y n d a+y 2. X n Y n d ay 3. Y n /X n d Y/a (if a 0) It should not be surprising that, in all cases we are only guaranteed the weaker type of convergence!