Math 341: Probability Eighteenth Lecture (11/12/09)

Size: px

Start display at page:

Download "Math 341: Probability Eighteenth Lecture (11/12/09)"

Mercy May
6 years ago
Views:

1 Math 341: Probability Eighteenth Lecture (11/12/09) Steven J Miller Williams College Steven.J.Miller@williams.edu public html/341/ Bronfman Science Center Williams College, November 12, 2009

2 Summary for the Day

3 Summary for the day Complex Analysis: Review definitions / statements. Accumulation point theorems. Clicker question: Statement of the CLT. Clicker question on rate of convergence. Central Limit Theorem: Poisson example. Proof with MGFs. Proof with Fourier analysis.

4 Review

5 5 Summary for the Day Review Accumulation and Moments Clicker Questions CLT and MGF CLT and Fourier Analysis Accumulation points and functions Theorem Let f be an analytic function on an open set U, with infinitely many zeros z 1, z 2, z 3,... If lim n z n U, then f is identically zero on U. In other words, if a function is zero along a sequence in U whose accumulation point is also in U, then that function is identically zero in U.

6 Schwartz Space Schwartz space S(R): all infinitely differentiable functions f such that, for any non-negative integers m and n, (1+x 2 ) m d n f dx n <. sup x R Inversion Theorem for Fourier Transform: Let f S(R). Then f(x) = ˆf(y)e 2πixy dy. f, g S(R) withˆf = ĝ then f(x) = g(x).

7 7 Summary for the Day Review Accumulation and Moments Clicker Questions CLT and MGF CLT and Fourier Analysis Accumulation and Moments

8 Accumulation Points and Moments Theorem Assume the MGFs M X (t) and M Y (t) exist in a neighborhood of zero (i.e., there is some δ such that both functions exist for t < δ). If M X (t) = M Y (t) in this neighborhood, then F X (u) = F Y (u) for all u. As the densities are the derivatives of the cumulative distribution functions, we have f = g.

9 Accumulation Points and Moments Theorem Let {X i } i I be a sequence of random variables with MGFs M Xi (t). Assume there is a δ > 0 such that when t < δ we have lim i M Xi (t) = M X (t) for some MGF M X (t), and all MGFs converge for t < δ. Then there exists a unique cumulative distribution function F whose moments are determined from M X (t) and for all x where F X (x) is continuous, lim n F Xi (x) = F X (x).

10 Accumulation Points and Moments Theorem: X and Y continuous random variables on [0, ) with continuous densities f and g, all of whose moments are finite and agree, and 1 C > 0 st c C, e (c+1)t f(e t ) and e (c+1)t g(e t ) are Schwartz functions. 2 The (not necessarily integral) moments μ r n (f) = 0 x rn f(x)dx and μ r n (g) = 0 x rn g(x)dx agree for some sequence of non-negative real numbers {r n } n=0 which has a finite accumulation point (i.e., lim n r n = r < ). Then f = g (in other words, knowing all these moments uniquely determines the probability density).

11 Application to equal integral moments Return to the two densities causing trouble: f 1 (x) = 1 2πx 2 e (log2 x)/2 f 2 (x) = f 1 (x)[1+sin(2π log x)].

12 Application to equal integral moments Return to the two densities causing trouble: f 1 (x) = 1 2πx 2 e (log2 x)/2 f 2 (x) = f 1 (x)[1+sin(2π log x)]. Same integral moments: e k 2 /2. Have the correct decay. Using complex analysis (specifically, contour integration), we can calculate the (a+ib) th moments: For f 2 : e (a+ib)2 /2 + i 2 For f 1 : e (a+ib)2 /2 ( e (a+i(b 2π))2 /2 e (a+i(b+2π))2 /2 ).

13 Application to equal integral moments Return to the two densities causing trouble: f 1 (x) = 1 2πx 2 e (log2 x)/2 f 2 (x) = f 1 (x)[1+sin(2π log x)]. No sequence of real moments having an accumulation point where they agree. a th moment of f 2 is e a2 /2 + e (a 2iπ)2 /2 ( 1 e 4iaπ), and this is never zero unless a is a half-integer. Only way this can vanish is if 1 = e 4iaπ.

14 Clicker Questions

15 Normalization of a random variable Normalization (standardization) of a random variable Let X be a random variable with mean μ and standard deviation σ, both of which are finite. The normalization, Y, is defined by Note that Y := X E[X] StDev(X) = X μ. σ E[Y] = 0 and StDev(Y) = 1.

16 Statement of the Central Limit Theorem Normal distribution A random variable X is normally distributed (or has the normal distribution, or is a Gaussian random variable) with mean μ and variance σ 2 if the density of X is f(x) = ( 1 exp 2πσ 2 (x μ)2 2σ 2 ). We often write X N(μ,σ 2 ) to denote this. If μ = 0 and σ 2 = 1, we say X has the standard normal distribution.

17 Statement of the Central Limit Theorem Central Limit Theorem Let X 1,...,X N be independent, identically distributed random variables whose moment generating functions converge for t < δ for some δ > 0 (this implies all the moments exist and are finite). Denote the mean by μ and the variance by σ 2, let and set X N Z N = X 1 + +X N N = X N μ σ/ N. Then as N, the distribution of Z N converges to the standard normal.

18 Statement of the Central Limit Theorem Why are there only tables of values of standard normal?

19 Statement of the Central Limit Theorem Why are there only tables of values of standard normal? Answer: normalization. Similar to log tables (only need one from change of base formula).

20 Alternative Statement of the Central Limit Theorem Central Limit Theorem Let X 1,...,X N be independent, identically distributed random variables whose moment generating functions converge for t < δ for some δ > 0 (this implies all the moments exist and are finite). Denote the mean by μ and the variance by σ 2, let S N = X 1 + +X N and set Z N = S N Nμ. Nσ 2 Then as N, the distribution of Z N converges to the standard normal.

21 Key Probabilities: Key probabilities for Z N(0, 1) (i.e., Z has the standard normal distribution). Prob( Z 1) 68.2%. Prob( Z 1.96) 95%. Prob( Z 2.575) 99%.

22 Convergence to the standard normal Question: Let X 1, X 2,... be iidrv with mean 0 and variance 1, and let Z N = X N /(1/ N). By the CLT Z N N(0, 1); which choice converges fastest? Slowest? 1 Uniform: X Unif( 3, 3). 2 Laplace: f X (x) = e 2 x / 2. 3 Normal: X N(0, 1). 4 Millered Cauchy: f X (x) = a = a sin(π/8) π 1, 1+(ax) 8

23 Convergence to the standard normal Question: Let X 1, X 2,... be iidrv with mean 0 and variance 1, and let Z N = X N /(1/ N). By the CLT Z N N(0, 1); which choice converges fastest? Slowest? 1 Uniform: X Unif( 3, 3). Kurtosis: Laplace: f X (x) = e 2 x / 2. Kurtosis: 6. 3 Normal: X N(0, 1). Kurtosis: 3. 4 Millered Cauchy: f X (x) = 4a sin(π/8) π 1, 1+(ax) 8 a = 2 1. Kurtosis:

24 Convergence to the standard normal Question: Let X 1, X 2,... be iidrv with mean 0 and variance 1, and let Z N = X N /(1/ N). By the CLT Z N N(0, 1); which choice converges fastest? Slowest? 1 Uniform: X Unif( 3, 3). Kurtosis: Laplace: f X (x) = e 2 x / 2. Kurtosis: 6. 3 Normal: X N(0, 1). Kurtosis: 3. 4 Millered Cauchy: f X (x) = 4a sin(π/8) π 1, 1+(ax) 8 a = 2 1. Kurtosis: log M X (t) = t2 2 + (μ 4 3)t 4 4! + O(t 6 ).

25 Convergence to the standard normal Question: Let X 1, X 2,... be iidrv with mean 0 and variance 1, and let Z N = X N /(1/ N). By the CLT Z N N(0, 1); which choice converges fastest? Slowest? 1 Uniform: X Unif( 3, 3). Excess Kurtosis: Laplace: f X (x) = e 2 x / 2. Excess Kurtosis: 3. 3 Normal: X N(0, 1). Excess Kurtosis: 0. 4 Millered Cauchy: f X (x) = 4a sin(π/8) π 1, 1+(ax) 8 a = 2 1. Excess Kurtosis: log M X (t) = t2 2 + (μ 4 3)t 4 4! + O(t 6 ).

26 Convergence to the standard normal Figure: Convolutions of 5 Uniforms.

27 Convergence to the standard normal Figure: Convolutions of 5 Laplaces.

28 Convergence to the standard normal Figure: Convolutions of 5 Normals.

29 Convergence to the standard normal Figure: Convolutions of 1 Millered Cauchy.

30 Convergence to the standard normal Figure: Convolutions of 2 Millered Cauchy.

31 Central Limit Theorem

32 MGF and the CLT Moment generating function of normal distributions Let X be a normal random variable with mean μ and variance σ 2. Its moment generating function satisfies M X (t) = e μt+σ2 t 2 2. In particular, if Z has the standard normal distribution, its moment generating function is M Z (t) = e t2 /2.

33 MGF and the CLT Moment generating function of normal distributions Let X be a normal random variable with mean μ and variance σ 2. Its moment generating function satisfies M X (t) = e μt+σ2 t 2 2. In particular, if Z has the standard normal distribution, its moment generating function is M Z (t) = e t2 /2. Proof: Complete the square.

34 Poisson Example of the CLT Example Let X, X 1,...,X N be Poisson random variables with parameter λ. Let X N = X 1 + +X N, Y = X E[X] N StDev(X). Then as N, Y converges to having the standard normal distribution.

35 Poisson Example of the CLT Example Let X, X 1,...,X N be Poisson random variables with parameter λ. Let X N = X 1 + +X N, Y = X E[X] N StDev(X). Then as N, Y converges to having the standard normal distribution. Moment generating function: M X (t) = exp(λ(e t 1)). Independent formula: M X1 +X 2 (t) = M X1 (t)m X2 (t). Shift formula: M ax+b (t) = e bt M X (at).

36 General proof via Moment Generating Functions X i s iidrv, Z N = X μ σ/ N = N n=1 X i μ σ N.

37 General proof via Moment Generating Functions X i s iidrv, Z N = X μ σ/ N = Moment Generating Function is: M ZN (t) = N n=1 e μt σ N ( ) t MX σ N N n=1 X i μ σ N. ( ) = e μt N N t σ M X σ N

38 General proof via Moment Generating Functions X i s iidrv, Z N = X μ σ/ N = Moment Generating Function is: M ZN (t) = N n=1 e μt σ N ( ) t MX σ N N n=1 X i μ σ N. ( ) = e μt N N t σ M X σ N Taking logarithms: log M ZN (t) = μt N σ ( ) t + N log M X σ. N

39 General proof via Moment Generating Functions (cont) Expansion of MGF: M X (t) = 1+μt + μ 2 t 2 2! + = 1+t ( μ+ μ 2 t ) 2 +.

40 General proof via Moment Generating Functions (cont) Expansion of MGF: M X (t) = 1+μt + μ 2 t 2 + = 1+t 2! Expansion for log(1+u) is log(1+u) = u u2 2 + u3 3!. ( μ+ μ 2 t ) 2 +.

41 General proof via Moment Generating Functions (cont) Expansion of MGF: M X (t) = 1+μt + μ 2 t 2 + = 1+t 2! Expansion for log(1+u) is ( μ+ μ 2 t ) 2 +. Combining gives log M X (t) = t log(1+u) = u u2 2 + u3 3!. ( ( μ+ μ 2 t ) t μ+ μ 2 t 2 + ) 2 2 = μt + μ 2 μ2 t 2 + terms in t 3 or higher. 2 +

42 General proof via Moment Generating Functions (cont) = ( ) t log M X σ N μt σ N + σ2 t 2 2 σ 2 N + terms in t 3 /N 3/2 or lower in N.

43 General proof via Moment Generating Functions (cont) = ( ) t log M X σ N μt σ N + σ2 t 2 2 σ 2 N + terms in t 3 /N 3/2 or lower in N. Denote lower order terms by O(N 3/2 ). Collecting gives log M ZN (t) = μt ( N μt + N σ σ N + t ) 2 2N + O(N 3/2 ) = μt N σ + μt N σ = t O(N 1/2 ). + t O(N 1/2 )

44 Central Limit Theorem and Fourier Analysis

45 Convolutions Convolution of f and g: h(y) = (f g)(y) = R f(x)g(y x)dx = R f(x y)g(x)dx.

46 Convolutions Convolution of f and g: h(y) = (f g)(y) = R f(x)g(y x)dx = R f(x y)g(x)dx. X 1 and X 2 independent random variables with probability density p. Prob(X i [x, x +Δx]) = Prob(X 1 + X 2 ) [x, x +Δx] = x+δx x x+δx x1 x 1 = p(t)dt p(x)δx. x 2 =x x 1 p(x 1 )p(x 2 )dx 2 dx 1.

47 Convolutions Convolution of f and g: h(y) = (f g)(y) = R f(x)g(y x)dx = R f(x y)g(x)dx. X 1 and X 2 independent random variables with probability density p. Prob(X i [x, x +Δx]) = Prob(X 1 + X 2 ) [x, x +Δx] = x+δx x x+δx x1 x 1 = p(t)dt p(x)δx. As Δx 0 we obtain the convolution of p with itself: Prob(X 1 + X 2 [a, b]) = b a x 2 =x x 1 p(x 1 )p(x 2 )dx 2 dx 1. (p p)(z)dz. Exercise to show non-negative and integrates to 1.

48 Statement of Central Limit Theorem WLOG p has mean zero, variance one, finite third moment and decays rapidly so all convolution integrals converge: p infinitely differentiable function satisfying xp(x)dx = 0, x 2 p(x)dx = 1, X 1, X 2,... are iidrv with density p. Define S N = N i=1 X i. x 3 p(x)dx <. Standard Gaussian (mean zero, variance one) is 1 2π e x2 /2.

49 Statement of Central Limit Theorem WLOG p has mean zero, variance one, finite third moment and decays rapidly so all convolution integrals converge: p infinitely differentiable function satisfying xp(x)dx = 0, x 2 p(x)dx = 1, X 1, X 2,... are iidrv with density p. Define S N = N i=1 X i. x 3 p(x)dx <. Standard Gaussian (mean zero, variance one) is 1 2π e x2 /2. Central Limit Theorem Let X i, S N be as above and assume the third moment of each X i is finite. Then S N / N converges in probability to the standard Gaussian: lim Prob N ( SN N [a, b] ) = 1 b e x2 /2 dx. 2π a

50 Proof of the Central Limit Theorem The Fourier transform: ˆp(y) = p(x)e 2πixy dx.

51 Proof of the Central Limit Theorem The Fourier transform: ˆp(y) = p(x)e 2πixy dx. Derivative of ĝ is the Fourier transform of 2πixg(x); differentiation (hard) is converted to multiplication (easy). ĝ (y) = 2πix g(x)e 2πixy dx; g prob. density, ĝ (0) = 2πiE[x], ĝ (0) = 4π 2 E[x 2 ].

52 Proof of the Central Limit Theorem The Fourier transform: ˆp(y) = p(x)e 2πixy dx. Derivative of ĝ is the Fourier transform of 2πixg(x); differentiation (hard) is converted to multiplication (easy). ĝ (y) = 2πix g(x)e 2πixy dx; g prob. density, ĝ (0) = 2πiE[x], ĝ (0) = 4π 2 E[x 2 ]. Natural: mean and variance simple multiples of derivatives of ˆp at zero: ˆp (0) = 0, ˆp (0) = 4π 2.

53 Proof of the Central Limit Theorem The Fourier transform: ˆp(y) = p(x)e 2πixy dx. Derivative of ĝ is the Fourier transform of 2πixg(x); differentiation (hard) is converted to multiplication (easy). ĝ (y) = 2πix g(x)e 2πixy dx; g prob. density, ĝ (0) = 2πiE[x], ĝ (0) = 4π 2 E[x 2 ]. Natural: mean and variance simple multiples of derivatives of ˆp at zero: ˆp (0) = 0, ˆp (0) = 4π 2. We Taylor expand ˆp (need technical conditions on p): ˆp(y) = 1+ p (0) 2 y 2 + = 1 2π 2 y 2 + O(y 3 ). Near origin, ˆp a concave down parabola.

54 Proof of the Central Limit Theorem (cont) Prob(X 1 + +X N [a, b]) = b a (p p)(z)dz.

55 55 Summary for the Day Review Accumulation and Moments Clicker Questions CLT and MGF CLT and Fourier Analysis Proof of the Central Limit Theorem (cont) Prob(X 1 + +X N [a, b]) = b a (p p)(z)dz. The Fourier transform converts convolution to multiplication. If FT[f](y) denotes the Fourier transform of f evaluated at y: FT[p p](y) = ˆp(y) ˆp(y).

56 Proof of the Central Limit Theorem (cont) Prob(X 1 + +X N [a, b]) = b a (p p)(z)dz. The Fourier transform converts convolution to multiplication. If FT[f](y) denotes the Fourier transform of f evaluated at y: FT[p p](y) = ˆp(y) ˆp(y). Do not want the distribution of X 1 + +X N = x, but rather S N = X 1+ +X N N = x.

57 57 Summary for the Day Review Accumulation and Moments Clicker Questions CLT and MGF CLT and Fourier Analysis Proof of the Central Limit Theorem (cont) Prob(X 1 + +X N [a, b]) = b a (p p)(z)dz. The Fourier transform converts convolution to multiplication. If FT[f](y) denotes the Fourier transform of f evaluated at y: FT[p p](y) = ˆp(y) ˆp(y). Do not want the distribution of X 1 + +X N = x, but rather S N = X 1+ +X N N = x. If B(x) = A(cx) for some fixed c = 0, then ˆB(y) = 1 c Â( y c).

58 Proof of the Central Limit Theorem (cont) Prob(X 1 + +X N [a, b]) = b a (p p)(z)dz. The Fourier transform converts convolution to multiplication. If FT[f](y) denotes the Fourier transform of f evaluated at y: FT[p p](y) = ˆp(y) ˆp(y). Do not want the distribution of X 1 + +X N = x, but rather S N = X 1+ +X N N = x. If B(x) = A(cx) for some fixed c = 0, then ˆB(y) = 1 c Â( y c). ) Prob( X1 + +X N N = x = ( Np Np)(x N).

59 Proof of the Central Limit Theorem (cont) Prob(X 1 + +X N [a, b]) = b a (p p)(z)dz. The Fourier transform converts convolution to multiplication. If FT[f](y) denotes the Fourier transform of f evaluated at y: FT[p p](y) = ˆp(y) ˆp(y). Do not want the distribution of X 1 + +X N = x, but rather S N = X 1+ +X N N = x. If B(x) = A(cx) for some fixed c = 0, then ˆB(y) = 1 c Â( y c). ) Prob( X1 + +X N N = x = ( Np Np)(x N). [ FT ( Np Np)(x ] N) (y) = ( )] [ˆp y N. N

60 Proof of the Central Limit Theorem (cont) Can find the Fourier transform of the distribution of S N : [ ( )] N y ˆp. N

61 Proof of the Central Limit Theorem (cont) Can find the Fourier transform of the distribution of S N : [ ( )] N y ˆp. N Take the limit as N for fixed y.

62 Proof of the Central Limit Theorem (cont) Can find the Fourier transform of the distribution of S N : [ ( )] N y ˆp. N Take the limit as N for fixed y. Know ˆp(y) = 1 2π 2 y 2 + O(y 3 ). Thus study [1 2π2 y 2 ( )] y 3 N N + O. N 3/2

63 Proof of the Central Limit Theorem (cont) Can find the Fourier transform of the distribution of S N : [ ( )] N y ˆp. N Take the limit as N for fixed y. Know ˆp(y) = 1 2π 2 y 2 + O(y 3 ). Thus study [1 2π2 y 2 ( )] y 3 N N + O. For any fixed y, [1 2π2 y 2 lim N N N 3/2 ( )] y 3 N + O = e 2π2 y 2. N 3/2

64 Proof of the Central Limit Theorem (cont) Can find the Fourier transform of the distribution of S N : [ ( )] N y ˆp. N Take the limit as N for fixed y. Know ˆp(y) = 1 2π 2 y 2 + O(y 3 ). Thus study [1 2π2 y 2 ( )] y 3 N N + O. For any fixed y, [1 2π2 y 2 lim N N N 3/2 ( )] y 3 N + O = e 2π2 y 2. N 3/2 Fourier transform of 1 2π e x2 /2 at y is e 2π2 y 2.

65 Proof of the Central Limit Theorem (cont) We have shown: the Fourier transform of the distribution of S N converges to e 2π2 y 2 ; the Fourier transform of 1 2π e x2 /2 at y is e 2π2 y 2. Therefore the distribution of S N equalling x converges to 1 2π e x2 /2.

66 Proof of the Central Limit Theorem (cont) We have shown: the Fourier transform of the distribution of S N converges to e 2π2 y 2 ; the Fourier transform of 1 2π e x2 /2 at y is e 2π2 y 2. Therefore the distribution of S N equalling x converges to 1 2π e x2 /2. We need complex analysis to justify this inversion. Must be careful: Consider { e 1/x2 if x = 0 g(x) = 0 if x = 0. All the Taylor coefficients about x = 0 are zero, but the function is not identically zero in a neighborhood of x = 0.

Math 341: Probability Seventeenth Lecture (11/10/09)

Math 341: Probability Seventeenth Lecture (11/10/09) Steven J Miller Williams College Steven.J.Miller@williams.edu http://www.williams.edu/go/math/sjmiller/ public html/341/ Bronfman Science Center Williams