The n th moment M n of a RV X is defined to be E(X n ), M n = E(X n ). Andrew Dabrowski

Transcription

1 We ll get to the in a few slides, but first some preliminaries. The tool we need to prove the is the sequence of moments of a probability distribution.

2 We ll get to the in a few slides, but first some preliminaries. The tool we need to prove the is the sequence of moments of a probability distribution. Moments provide a way of describing distributions different from density functions or c.d.f.s, and better suited to the.

3 The n th moment M n of a RV X is defined to be E(X n ), M n = E(X n ).

4 The n th moment M n of a RV X is defined to be E(X n ), M n = E(X n ). This may not exist, especially when n is large.

5 The n th moment M n of a RV X is defined to be E(X n ), M n = E(X n ). This may not exist, especially when n is large. But if it does exit for all n, those numbers together tell you pretty much everything you need to know about X.

6 The n th moment M n of a RV X is defined to be E(X n ), M n = E(X n ). This may not exist, especially when n is large. But if it does exit for all n, those numbers together tell you pretty much everything you need to know about X. For example, µ X = M 1

7 The n th moment M n of a RV X is defined to be E(X n ), M n = E(X n ). This may not exist, especially when n is large. But if it does exit for all n, those numbers together tell you pretty much everything you need to know about X. For example, µ X = M 1 σ X = M 2 M 2 1.

8 The n th moment M n of a RV X is defined to be E(X n ), M n = E(X n ). This may not exist, especially when n is large. But if it does exit for all n, those numbers together tell you pretty much everything you need to know about X. For example, µ X = M 1 σ X = M 2 M 2 1. If g(x) = a 0 + a 1 x + a 2 x a k x k, then E(g(X )) = a 0 + a 1 M 1 + a 2 M a k M k.

9 The n th moment M n of a RV X is defined to be E(X n ), M n = E(X n ). This may not exist, especially when n is large. But if it does exit for all n, those numbers together tell you pretty much everything you need to know about X. For example, µ X = M 1 σ X = M 2 M 2 1. If g(x) = a 0 + a 1 x + a 2 x a k x k, then E(g(X )) = a 0 + a 1 M 1 + a 2 M a k M k. In other words we can compute the expected value of any polynomial function of X just using the M n s.

10 We can even compute things like How? P(a X a + dx).

11 We can even compute things like P(a X a + dx). How? Define b(x) = 1 2πɛ e 1 2 ( x a ɛ )2. This is just a very narrow bell curve, in which 99% of the bell is within 3ɛ of a.

12 We can even compute things like P(a X a + dx). How? Define b(x) = 1 2πɛ e 1 2 ( x a ɛ )2. This is just a very narrow bell curve, in which 99% of the bell is within 3ɛ of a. Note that b is equivalent to a normal p.d.f., but I m not using it here as a p.d.f., I m using it like the g on the previous slide: I m going to find E(b(X )).

13 We can even compute things like P(a X a + dx). How? Define b(x) = 1 2πɛ e 1 2 ( x a ɛ )2. This is just a very narrow bell curve, in which 99% of the bell is within 3ɛ of a. Note that b is equivalent to a normal p.d.f., but I m not using it here as a p.d.f., I m using it like the g on the previous slide: I m going to find E(b(X )). Since b is essentially a polynomial (an infinite one, but still...) it has the same property as g: that its expectation is determined by the M n s.

14 We can even compute things like P(a X a + dx). How? Define b(x) = 1 2πɛ e 1 2 ( x a ɛ )2. This is just a very narrow bell curve, in which 99% of the bell is within 3ɛ of a. Note that b is equivalent to a normal p.d.f., but I m not using it here as a p.d.f., I m using it like the g on the previous slide: I m going to find E(b(X )). Since b is essentially a polynomial (an infinite one, but still...) it has the same property as g: that its expectation is determined by the M n s. And the shape of b is useful because it focuses attention on a small range of x values.

15 Now suppose that X is a continuous RV with p.d.f. f. If ɛ is small enough, then we can assume that f is constant over almost the entire bell.

16 Now suppose that X is a continuous RV with p.d.f. f. If ɛ is small enough, then we can assume that f is constant over almost the entire bell. That means E(b(X )) = b(x)f (x) dx

17 Now suppose that X is a continuous RV with p.d.f. f. If ɛ is small enough, then we can assume that f is constant over almost the entire bell. That means = X a >3ɛ E(b(X )) = b(x)f (x) dx + b(x)f (x) dx X a 3ɛ b(x)f (x) dx

18 Now suppose that X is a continuous RV with p.d.f. f. If ɛ is small enough, then we can assume that f is constant over almost the entire bell. That means = X a >3ɛ E(b(X )) = b(x)f (x) dx f (a) X a 3ɛ b(x)f (x) dx X a 3ɛ b(x)f (x) dx b(x) dx f (a).

19 This means that the density function f of X is determined by the expected values of RV s like b(x ).

20 This means that the density function f of X is determined by the expected values of RV s like b(x ). And that in turn means that f is determined by the M n s, because although b(x) is not a polynomial, it is a limit of polynomials: b(x) = 1 2πɛ e 1 2 ( x a ɛ )2 = 1 2πɛ k=0 ( 1 2 ( x a ɛ ) 2 ) k k!

21 In a similar way it can be shown that those functions b also determine the probabilities in discrete distributions.

22 In a similar way it can be shown that those functions b also determine the probabilities in discrete distributions. I ll consider it established now that the M n s determine all the properties of a distribution. Hence if we can show that two distributions have finite and equal M n for all n we will know they are actually the same distribution. Now let s use the moments M n to prove the Central Limit Theorem.

23 Central Limit Theorem Theorem Let X i for i = 1, 2, 3,... be i.i.d. RVs with mean 0 and s.d. 1. Define A n = 1 n X i. n Then as n the distribution of A n approaches the standard normal. i=1

24 Central Limit Theorem Theorem Let X i for i = 1, 2, 3,... be i.i.d. RVs with mean 0 and s.d. 1. Define A n = 1 n X i. n Then as n the distribution of A n approaches the standard normal. Note that the particular distribution of the X i s does not matter the limiting distribution is the same no matter what you start out with. i=1

25 The basic idea is to show that the moments of the limiting distribution depend only on the first and second moments of the X i nothing else.

26 The basic idea is to show that the moments of the limiting distribution depend only on the first and second moments of the X i nothing else. Let s warm up by computing a few easy moments of A n. ( ) ( 1 n n ) E(A n ) = E X i = 1 E(X i ) = 0 n n i=1 i=1 because all the X i have mean 0.

27 Second moment: ( E(A 2 n) = E 1 n n i=1 ) 2 X i = 1 n n i=1 E(X 2 i ) + i j E(X i X j )

28 Second moment: ( E(A 2 n) = E 1 n n i=1 ) 2 X i = 1 n n i=1 E(X 2 i ) + i j E(X i X j ) = 1 [n + 0] = 1. n

29 Second moment: ( E(A 2 n) = E 1 n n i=1 ) 2 X i = 1 n n i=1 E(X 2 i ) + i j E(X i X j ) = 1 [n + 0] = 1. n So now we know that A n has mean 0 and s.d. 1 for all n.

30 Second moment: ( E(A 2 n) = E 1 n n i=1 ) 2 X i = 1 n n i=1 E(X 2 i ) + i j E(X i X j ) = 1 [n + 0] = 1. n So now we know that A n has mean 0 and s.d. 1 for all n. We conclude therefore the limiting distribution of the A n also has mean 0 and s.d. 1.

31 Third moment. ( E(A 3 n) = E 1 n n i=1 ) 3 X i

32 Third moment. = 1 n 3/2 n i=1 ( E(A 3 n) = E 1 n n i=1 E(Xi 3 ) + E(X i Xj 2 ) + i j ) 3 X i distinct i,j,k E(X i X j X k )

33 Third moment. = 1 n 3/2 = 1 n 3/2 n i=1 ( E(A 3 n) = E 1 n n i=1 E(Xi 3 ) + E(X i Xj 2 ) + i j ) 3 X i [ ne(x 3 1 ) + n(n 1)E(X 1 )E(X 2 2 ) + E(X i X j X k ) distinct i,j,k ( ] n )E(X 1 ) 3 3

34 Third moment. = 1 n 3/2 = 1 n 3/2 n i=1 ( E(A 3 n) = E 1 n n i=1 E(Xi 3 ) + E(X i Xj 2 ) + i j ) 3 X i [ ne(x 3 1 ) + n(n 1)E(X 1 )E(X 2 2 ) + = 1 n 3/2 (n M 3) where M 3 is the third moment of X 1. E(X i X j X k ) distinct i,j,k ( ] n )E(X 1 ) 3 3

35 Third moment. = 1 n 3/2 = 1 n 3/2 n i=1 ( E(A 3 n) = E 1 n n i=1 E(Xi 3 ) + E(X i Xj 2 ) + i j ) 3 X i [ ne(x 3 1 ) + n(n 1)E(X 1 )E(X 2 2 ) + = 1 n 3/2 (n M 3) where M 3 is the third moment of X 1. But this expression goes to 0 as n. E(X i X j X k ) distinct i,j,k ( ] n )E(X 1 ) 3 3

36 Third moment. = 1 n 3/2 = 1 n 3/2 n i=1 ( E(A 3 n) = E 1 n n i=1 E(Xi 3 ) + E(X i Xj 2 ) + i j ) 3 X i [ ne(x 3 1 ) + n(n 1)E(X 1 )E(X 2 2 ) + E(X i X j X k ) distinct i,j,k ( ] n )E(X 1 ) 3 3 = 1 n 3/2 (n M 3) where M 3 is the third moment of X 1. But this expression goes to 0 as n. So the third moment of the limiting distribution of A n is 0.

37 Note that so far the only higher moment of the X i that has appeared has been M 3 and that ended up cancelling as n.

38 Note that so far the only higher moment of the X i that has appeared has been M 3 and that ended up cancelling as n. One obvious pattern you ll have noticed is that any term involving a factor like E(X i ) drops out because that value is 0.

39 Note that so far the only higher moment of the X i that has appeared has been M 3 and that ended up cancelling as n. One obvious pattern you ll have noticed is that any term involving a factor like E(X i ) drops out because that value is 0. So the only terms to worry about are those in which each factor is a second or higher moment of X i.

40 Note that so far the only higher moment of the X i that has appeared has been M 3 and that ended up cancelling as n. One obvious pattern you ll have noticed is that any term involving a factor like E(X i ) drops out because that value is 0. So the only terms to worry about are those in which each factor is a second or higher moment of X i. There s another pattern lurking here though...

41 Consider an example: Suppose we re computing the fifth moment of A n. One expression we ll have to deal with is E(Xi 2 )E(Xj 3 ). i j

42 Consider an example: Suppose we re computing the fifth moment of A n. One expression we ll have to deal with is E(Xi 2 )E(Xj 3 ). i j Each summand is equal to M 2 M 3, and the number of summands is no more (actually fewer than) n 2. n 2 M 2 M 3

43 Consider an example: Suppose we re computing the fifth moment of A n. One expression we ll have to deal with is E(Xi 2 )E(Xj 3 ). i j Each summand is equal to M 2 M 3, and the number of summands is no more (actually fewer than) n 2. n 2 M 2 M 3 Now recall that A n includes a factor of 1 n, so that the fifth moment of A n in includes as factor of 1 n 5 = 1 n 5/2.

44 Consider an example: Suppose we re computing the fifth moment of A n. One expression we ll have to deal with is E(Xi 2 )E(Xj 3 ). i j Each summand is equal to M 2 M 3, and the number of summands is no more (actually fewer than) n 2. n 2 M 2 M 3 Now recall that A n includes a factor of 1 n, so that the fifth moment of A n in includes as factor of 1 n 5 = 1 n 5/2. Multiplying these two expressions together gives M n 2M 3 again approaches 0 as n. which once

45 Evidently very few terms survive the limit. In fact the only expressions that don t cancel in the limit are those involving only second moments of X i : 1 n k/2 E(Xi 2 1 )E(Xi 2 2 )... E(X 2 distinct i 1,i 2,...,i k/2 i k/2 )

46 Evidently very few terms survive the limit. In fact the only expressions that don t cancel in the limit are those involving only second moments of X i : 1 n k/2 E(Xi 2 1 )E(Xi 2 2 )... E(X 2 distinct i 1,i 2,...,i k/2 = 1 ( ) n n k/2 M k/2 2. k/2 i k/2 )

47 Evidently very few terms survive the limit. In fact the only expressions that don t cancel in the limit are those involving only second moments of X i : 1 n k/2 E(Xi 2 1 )E(Xi 2 2 )... E(X 2 distinct i 1,i 2,...,i k/2 = 1 ( ) n n k/2 M k/2 2. k/2 i k/2 ) Since ( n k/2) is on the order of n k/2, this term does not cancel as n.

48 Evidently very few terms survive the limit. In fact the only expressions that don t cancel in the limit are those involving only second moments of X i : 1 n k/2 E(Xi 2 1 )E(Xi 2 2 )... E(X 2 distinct i 1,i 2,...,i k/2 = 1 ( ) n n k/2 M k/2 2. k/2 i k/2 ) Since ( n k/2) is on the order of n k/2, this term does not cancel as n. Note that even this is only possible if k is even,

49 Evidently very few terms survive the limit. In fact the only expressions that don t cancel in the limit are those involving only second moments of X i : 1 n k/2 E(Xi 2 1 )E(Xi 2 2 )... E(X 2 distinct i 1,i 2,...,i k/2 = 1 ( ) n n k/2 M k/2 2. k/2 i k/2 ) Since ( n k/2) is on the order of n k/2, this term does not cancel as n. Note that even this is only possible if k is even, so we conclude that odd moments of A n approach 0 in the limit.

50 OK, that was the hard part, the rest is easy.

51 OK, that was the hard part, the rest is easy. Now we know that the limiting distribution of A n depends only on M 1 and M 2. That means that any X i s with the same mean (0) and the same s.d. (1) will have the same limiting distribution.

52 OK, that was the hard part, the rest is easy. Now we know that the limiting distribution of A n depends only on M 1 and M 2. That means that any X i s with the same mean (0) and the same s.d. (1) will have the same limiting distribution. Well it just so happens that we know a distribution with mean 0 and s.d. 1 which also just so happens to play very well with linear combinations of itself.

53 OK, that was the hard part, the rest is easy. Now we know that the limiting distribution of A n depends only on M 1 and M 2. That means that any X i s with the same mean (0) and the same s.d. (1) will have the same limiting distribution. Well it just so happens that we know a distribution with mean 0 and s.d. 1 which also just so happens to play very well with linear combinations of itself. I refer of course to the standard normal distribution. If the X i are standard normal, then A n, being a linear combination of independent standard normals is also normal.

54 OK, that was the hard part, the rest is easy. Now we know that the limiting distribution of A n depends only on M 1 and M 2. That means that any X i s with the same mean (0) and the same s.d. (1) will have the same limiting distribution. Well it just so happens that we know a distribution with mean 0 and s.d. 1 which also just so happens to play very well with linear combinations of itself. I refer of course to the standard normal distribution. If the X i are standard normal, then A n, being a linear combination of independent standard normals is also normal. Moreover we calculated the mean and s.d. of A n earlier and they turned out to be 0 and 1 respectively.

55 OK, that was the hard part, the rest is easy. Now we know that the limiting distribution of A n depends only on M 1 and M 2. That means that any X i s with the same mean (0) and the same s.d. (1) will have the same limiting distribution. Well it just so happens that we know a distribution with mean 0 and s.d. 1 which also just so happens to play very well with linear combinations of itself. I refer of course to the standard normal distribution. If the X i are standard normal, then A n, being a linear combination of independent standard normals is also normal. Moreover we calculated the mean and s.d. of A n earlier and they turned out to be 0 and 1 respectively. I.e., if the X i s are standard normal then so are the A n s.

56 OK, that was the hard part, the rest is easy. Now we know that the limiting distribution of A n depends only on M 1 and M 2. That means that any X i s with the same mean (0) and the same s.d. (1) will have the same limiting distribution. Well it just so happens that we know a distribution with mean 0 and s.d. 1 which also just so happens to play very well with linear combinations of itself. I refer of course to the standard normal distribution. If the X i are standard normal, then A n, being a linear combination of independent standard normals is also normal. Moreover we calculated the mean and s.d. of A n earlier and they turned out to be 0 and 1 respectively. I.e., if the X i s are standard normal then so are the A n s. And if every single A n is standard normal, then the limiting distribution is also.

57 To put this together: Since there is some distribution for the X i which produces the standard normal as the limit of A n, then in fact every distribution, once it s been standardized to have mean 0 and s.d. 1, must produce the standard normal in the limit also.

58 To put this together: Since there is some distribution for the X i which produces the standard normal as the limit of A n, then in fact every distribution, once it s been standardized to have mean 0 and s.d. 1, must produce the standard normal in the limit also. That s because we have shown that the limiting distribution depends only on the mean and s.d. of the original distribution.

59 To put this together: Since there is some distribution for the X i which produces the standard normal as the limit of A n, then in fact every distribution, once it s been standardized to have mean 0 and s.d. 1, must produce the standard normal in the limit also. That s because we have shown that the limiting distribution depends only on the mean and s.d. of the original distribution. Q.E.D.