Lecture 2: CENTRAL LIMIT THEOREM

Transcription

1 A Theorist s Toolkit (CMU 8-859T, Fall 3) Lectre : CENTRAL LIMIT THEOREM September th, 3 Lectrer: Ryan O Donnell Scribe: Anonymos SUM OF RANDOM VARIABLES Let X, X, X 3,... be i.i.d. random variables (Here i.i.d. means independent and identically distribted ), s.t. Pr[X i = ] = p, Pr[X i = ] = p. X i is also called Bernolli random variable. Let S n = X + + X n. We will be interested in the random variable S n which is called Binomial random variable (S n B(n, p)). If yo toss a coin for n times, and X i = represents the event that the reslt is head in the ith trn, then S n is jst the total nmber of appearance of head in n times. Recall some basic facts on expectation and variance, where Y, Y, Y are random variables. E[Y + Y ] = E[Y ] + E[Y ] E[Y Y ] = E[Y ] E[Y ] if random variables Y and Y are independent (Y Y ) E[cY ] = ce[y ], E[c + Y ] = c + E[Y ], where c is a constant If we denote µ = E[Y ], the variance of Y Var[Y ] = E[(Y µ) ] = E[Y µy + µ ] = E[Y ] µ E[Y ] + µ = E[Y ] E[Y ] If Y Y Var[Y + Y ] = E[(Y + Y ) ] E[(Y + Y )] = (E[Y ] + E[Y Y ] + E[Y ]) (E[Y ] + E[Y ] E[Y ] + E[Y ] ) = (E[Y ] E[Y ] ) + (E[Y ] E[Y ] ) = Var[Y ] + Var[Y ] Var[cY ] = c Var[Y ], Var[c + Y ] = Var[Y ] The standard derivation σ = stddev[y ] = Var[Y ]

2 For any X i we have the expectation E[X i ] = Pr[X i = ] + Pr[X i = ] = p, therefore [ n ] E[S n ] = E X i = i= n E[X i ] = np i= Since the variance of X i Var[X i ] = E[Xi ] E[X i ] = p p = p( p) and X i s are independent, the variance of S n is [ n ] n Var[S n ] = Var X i = Var[X i ] = np( p) i= We wold like to somehow normalize the random variable S n s.t. its mean is and its variance is. Let Z n := S n µ σ where µ and σ is the mean and standard derivation of S n. It is easy to see that and Since S n = σz n + pn, we have i= E[Z n ] = E[(S n µ)/σ] = (E[S n ] µ)/σ = Var[Z n ] = Var[(S n µ)/σ] = Var[S n ]/σ = Pr[S n ] = Pr[σZ n + pn ] = Pr [ Z n pn ] σ So if we know the probability distribtion of Z n, we may also know the probability distribtion of S n, vice versa. Example.. Sppose p =, E[S n] = n, Var[S n] = n, Z n = X + + X n n n = (X ) + + (X n ) n It can be seen as X i = { + w.p. w.p. Recall from Lectre, the probability that Z n is is Pr[Z n = ] = Θ( n ), when n is even, as in Figre.

3 . Θ ( / n)..8 density z n Figre : Histogram of Z n for n = 6 and p = / GAUSSIAN DISTRIBUTION Definition. (Gassian Distribtion). A random variable Z is Gassian distribted with parameters µ and σ, (abbreviated N(µ, σ )), if it is continos with p.d.f. (probability density fnction) φ(z) = π e z where µ refers to the mean and σ refers to the variance of Gassian. Particlarly we call Z N(, ) a standard Gassian. Fact.. Let Z,..., Z d be i.i.d. standard Gassians, Z = (Z,..., Z d ). Then Z s distribtion is rotationally symmetric, which means for all Z = r, the probability density of Z is the same. Figre shows the probability density of -dimension standard Gassian. Proof. p.d.f. of Z at (Z,..., Z d ) with Z = r is φ( ( ) d d Z ) = φ(z )φ(z )... φ(z d ) = π i= = e Z i ( ) d ( ) d ( ) d e (Z + +Z d ) = e (Z,...,Z d ) = e r / π π π Therefore the probability density of Z only depends on r, which means Z s distribtion is rotationally symmetric. Corollary.3. φ(x)dx = 3

4 y r x Figre : Distribtion of D-standard Gassian is rotationally symmetric Proof. Consider abot the -dimension standard Gassian Z = (Z, Z ), which has p.d.f. π e (Z +Z ). We can first integrate the p.d.f. in a natral way. The integral becomes π e ( (z +z ) dz dz = Therefore, in order to prove π e z dz) = φ(x)dx =, it sffices to prove e (z +z ) dz dz = π ( φ(z)dz since φ(x)dx >. On the otherhand, as in Figre 3, we can intergrate the fnction e (z +z ) by height. The height of each cylinder is dh, while the radim of it is r which satisfies e r / = h. Therefore we have r = ln h Since < h = e r /, we have e (z +z ) dz dz = πr dh = ) π ln h dh = π ( h ln h + h ) = π Corollary.4 (Sm of Independent Gassian). Sopose X N(µ, σ ), Y N(µ, σ ) are independent. We have ax + by (aµ + bµ, a σ + b σ ) 4

5 .5 density..5 h = e r / dh h r y x Figre 3: Integration of D Gassian by height Proof. If µ = µ =, σ = σ =, then X, Y N(, ). We want to prove ax + by N(, a + b ), and we have Z = ax + by = (a, b) (X, Y ) Becase D standard Gassian (X, Y ) is rotationally symmetric, we can rotate (a, b) to ( a + b, ) as in Figre 4. Now Z = ( a + b, ) (X, Y ) = a + b X = N(, a + b ). Sppose X N(µ, σ ), Y N(µ, σ ). Since (X µ )/σ, (Y µ )/σ N(, ), we have Z aµ bµ = a(x µ ) + b(y µ ) = aσ X µ σ + bσ Y µ σ N(, a σ + b σ ) Therefore Z (aµ + bµ, a σ + b σ ). 3 CENTRAL LIMIT THEOREM Theorem 3. (Central Limit Theorem). For any i.i.d. X,..., X n, Z n n is a standard Gassian N(, ), i.e. R, lim n Pr[Z n ] = Pr[Z ] Z, where Z Remember the Central Limit Theorem is kind of seless since we don t know how qickly Z n will converge to a standard Gassian. 5

6 .5 y.5 (a, b) ( (a +b ),).5 (X, Y) x Figre 4: Rotating (a, b) to ( a + b, ) since (X, Y ) is rotationally symmetric The name of the sefl version is Berry-Esseen Theorem. Theorem 3. (Berry-Esseen Theorem). Let X,..., X n be independent r.v.s, Assme (W.O.L.O.G.) E[X i ] =. we write σi = E[Xi ] = Var[X i ] and assme n i= σ i =. Let S = X +... X n so E[S] =, Var[S] =. Then R, where Z N(, ), β = n i= E[ X i 3 ] Pr[S ] Pr[Z ] O() β Remember β is not always small. Here is an example. { + w.p. Example 3.3. Let X = and X w.p.,..., X n. In this scenario, S jst has the same distribtion as X. Then β = is a big nmber. On the other hand, this theorem does work in some cases. { + n w.p. Example 3.4. X i =, n w.p. E[X i ] = n, E[ X i 3 ] = Therefore β = / n. In this case β is small, and n 3 Pr[S ] Pr[Z ] = O, i ( n ) 6

7 The most recent pper bond of O() is O().554 [She3]. In this scenario, we have Pr[S ] Pr[Z ].56 n We can find that Pr[Z ]. when = 3 by compter or in standard normal table. Sppose H is the nmber of appearance of head when tossing a coin for n times. As in Example., we have H n = S n which means H n + n Assign = 3, we have Pr[H n/.5 n]., which is qite small. Sometimes β might be extremely large. Example 3.5. Let +n w.p. n X = n w.p. n otherwise and X,..., X n. n i= E[X i ] = E[X] =. In this scenario, Pr[S = ], which means S does not converge to a normal distribtion. In this case β +. From this example, we can see that the constraint on β = n i= E[ X i 3 ] captre two things: The r.v.s will not become extremely hge with small probability; The sm does not only depend on finite nmber of random variables. Notice that Berry-Esseen Theorem is good becase it does not care abot the vale of. We have [ Pr H n n + even thogh =. n which is spertiny. ] O( n ) φ(z)dz 4 CUMULATIVE DISTRIBUTION Definition 4. (Cmlative Distribtion Fnction of Standard Gassian). We denote Φ() as the c.d.f. (cmlative distribtion fnction) of standard Gassian N(, ) and define Φ() = Pr[Z ] = Φ() = + φ(z)dz = Pr[Z ] = Φ( ) 7

8 It is trivial that Φ() = Φ() =. Fact 4.. when >. ( ) φ() Φ() = O.8 density.6.4 e / / (π). 3 +δ 3 x Figre 5: Find a δ s.t. φ( + δ) c φ(), where c is a constant and less than Proof. We want to find a δ s.t. φ( + δ) c φ(), where c is a constant and less than (Figre 5). δ = / satisfies or conditions. φ( + ) = π e (+ ) / = e φ() e e φ() In general, we have φ(+ k ) e k φ() where > and k N. Since φ() is descreasing when >, sing the method in Lectre, we have Φ() = + φ()d k= φ( + k ) φ() e k = k= e e φ() ( ) φ() = O Proposition 4.3. when +. In fact, Φ() φ() ( 3 )φ() Φ() φ() when +. 8

9 References [She3] I. G. Shevtsova. On the absolte constants in the berry esseen ineqality and its strctral and nonniform improvements. Inform. Primen., 7():4 5, 3. 9