2.1. Convergence in distribution and characteristic functions.

Transcription

1 3 Chapter 2. Cetral Limit Theorem. Cetral limit theorem, or DeMoivre-Laplace Theorem, which also implies the wea law of large umbers, is the most importat theorem i probability theory ad statistics. For idepedet radom variables, Lideberg-Feller cetral limit theorem provides the best results. Throughout this chapter, radom variables shall ot tae values i or with positive chace Covergece i distributio ad characteristic fuctios. Covergece i distributio, which ca be geeralized slightly to wea covergece of measures, has bee itroduced i Sectio 1.2. This sectio provides a more detailed descriptio. (i. Defiitio, basic properties ad examples. Recall that i Sectio 1.3, we have already defied covergece i distributio for a sequece of radom variables. Here we preset the same defiitio i terms of wea covergece of their distributios. We first ote that a fuctio F is a cdf if ad oly if it is right cotiuous, odecreasig with F (t 1 ad whe t ad, respectively. Defiitio. A sequece of distributio fuctio F is called covergig to aother distributio fuctio F wealy, if (1 F (t F (t for every cotiuity poits of F ; or (2, lim if F (B F (B for every ope set B i (, ; or (3 lim sup F (C F (C for every closed set C i (, ; or (4 g(xdf (x g(xdf (x for every cotiuous fuctio g. Here F (A is defied as A df (x = 1 x A df (x for ay Borel set A. The above four claims are equivalet to each other, as proved i Sectio 1.3. Remar. If F is cotiuous, the iequalities i (2 ad (3 are actually equalities. O the other had, if X all taes iteger values, the X X i distributio is equivalet to P (X = P (X = for all iteger values. Remar. (Sheffe s Theorem Suppose X has desity fuctio f ( ad f (t f(t for every fiite t ad f is a desity fuctio. The, X X i distributio, where X has desity f. This ca be show quite straightforwardly as follows: 2 = lim if(f + f f (x f(x dx lim if (f (x + f(x f (x f(x dx ( = lim if 2 f (x f(x dx Certaily, for ay Borel set B, P (X B P (X B = B = 2 lim sup f (x f(x dx. (f (x f(xdx f (x f(x dx. I the above proof, we have used Fatou lemma with Lebesgue measure. I fact, the mootoe covergece theorem, Fatou lemma ad domiated covergece theorem that we have established with probability measure all hold with σ-fiite measures, icludig Lebesgue measure. Remar. (Slutsy s Theorem Suppose X X i distributio ad Y c i probability. The, X Y cx i distributio ad X + Y X c i distributio. We leave the proof as a exercise. I the followig, we provide some classical examples about covergece i distributio, oly to show that there are a variety of importat limitig distributios besides the ormal distributio as the limitig distributio i CLT.

2 31 Example 2.1. (Covergece of maxima ad extreme value distributios Let M = max 1 i X i where X i are iid r.v.s with c.d.f. F (. The, P (M t = P (X 1 t = F (t. As, the limitig distributio of properly scaled M, should it coverge, should oly be related with the right tail of the distributio of F (, i.e., the F (x whe x is large. The followig are some examples. (a. F (x = 1 x α for some α > ad all large x. The, for ay t >, P (M / 1/α < t = (1 1 t α e t α (b. F (x = 1 x β for x [ 1, ] ad some β >. The, for ay t <, P ( 1/β M t = (1 1 t β e t β (c. F (x = 1 e x for x >, i.e., X i follows expoetial distributio. The for all t, P (M log t e e t These limitig distributios are called extreme value distributios. Example 2.2. (Birthday problem Suppose X 1, X 2,... are iid with uiform distributio o the itegers {1, 2,..., N} with < N ad, Let The, for N, T N = mi{ : there exists a j < such that {X j = X } }. P (T N > = P ( X 1,..., X all tae differet values ( = 1 P ( X j taes oe of the values of X 1,.., X j 1 = j=2 (1 j 1 1 N = exp{ log(1 j/n} j=2 The, for ay fixed x >, as N, j=1 P (T N /N 1/2 > x = P (T N > N 1/2 x exp{ exp{ 1 j<n 1/2 x 1 j<n 1/2 x log(1 j/n} j/n} exp{ (1/NN 1/2 x(n 1/2 x + 1/2} exp{ x 2 /2} I other words, T N /N 1/2 coverges i distributio to a distributio F (t = 1 exp( t 2 /2 for t. Suppose ow N = 365. By this approximatio, we have P (T 365 > ad P (T 365 > 5.326, meaig that, with 22 (5 people there is about half (3% probability that all of them have differet birthday. Example 2.3. (Law of rare evets Suppose there are totally flights worldwide each year, ad each flight has chace p to have a accidet, idepedet of rest flights. There is o average λ accidets a year worldwide. The distributio of the umber of accidets is B(, p with p close to λ. The this distributio approximates Poisso distributio with mea λ, amely, Bi(, p P(λ if ad p λ >.

3 32 Proof. For ay fixed, ad ( P (Bi(, p = = = 1! p (1 p =! (p (1 p!(! (1 p ( 1 ( + 1 (p e log(1 p (1 p λ e λ!, as. Example 2.4. (The secretary/marriage problem Suppose there are secretary to be iterviewed oe by oe ad, right after each iterview, you must mae immediate decisio of hire or fire the iterviewee. You observe oly the relative ras of the iterviewed cadidates. What is the optimal strategy is maximize the chace of hirig the best of the cadidates? (Assume o ties of performace. Oe type of strategy is to give up the first m cadidates, whatever their performace i the iterview. Afterwards, the oe that outperforms all previous cadidates is hired. I other words, startig from m + 1-th iterview, the first cadidate that outperforms the first m cadidates is hired. Or else you settle with the last cadidate. The chace that the -th best amog all cadidates is hired is = P = j=m+1 j=m+1 P ( the -th best is the j-th iterviewee ad is hired 1 P (the best amog first j 1 appears i the first m, the j-th cadidate is the -th best, ad the 1 best all appear after the j-th cadidate. m j 1 1 ( j 1 j=m+1 Let, ad m c where c is the percetage of the iterviews to be give up. The the probability of hirig the -th best P c j=m 1 j (1 j/ 1 c 1 c (1 x 1 dx = ca, x Sice A +1 = A (1 c /, for 1, ad A 1 = log c, it follows that ( P c 1 log c j=1 (1 c j j, as. I particular, P 1 c log c. The fuctio c log c is maximized at c = 1/e =.368. The best strategy is to give up the first 36.8% of the iterviews ad the hire the best to date. The chace of hirig the best overall is also 36.8%. The chace of hirig the last perso is also c. This pheomeo is also called 1/e law. You might please formulate this problem i terms of a sequece of radom variables. (ii. Some theoretical results about covergece i distributio. (a. Fatou Lemma Suppose X ad X X i distributio. The E(X lim if E(X. Proof. E(X = Write P (X tdt lim if P (X tdt = lim if say. P (X tdt lim if E(X.

4 33 The domiated covergece theorem also holds with covergece i distributio, which is left as a exercise. (b. Cotiuous mappig theorem: X X i distributio ad g( is a cotiuous fuctio. The, g(x g(x i distributio. Proof. For ay bouded cotiuous fuctio f, f(g( is still bouded cotiuous fuctio. Hece E(f(g(X E(f(g(X, provig that g(x g(x i distributio. (c. Tightess ad coverget subsequeces. I studyig the covergece of a sequece of umbers, it is very useful that boudedess of the sequece, guaratees a coverget subsequece. The same is true for uiformly bouded mootoe fuctios, such as, for example, distributio fuctios. This is the followig Helly s Selectio theorem, which is useful i studyig wea covergece of distributios. Helly s Selectio Theorem. A sequece of cumulative distributio fuctios F always cotais a subsequece, say F, that coverges to a fuctio, say F, which is odecreasig ad right cotiuous, at every cotiuity poit of F. If F ( = ad F ( = 1. The, F is a distributio fuctio ad F coverges to F wealy. Proof Let t 1, t 2,... be all ratioal umbers. I the sequece F (t 1, 1, there is always a coverget subsequece. Deote oe of them as, say (1, = 1, 2,... Amog this subsequece there is agai a further subsequece, deoted as (2, = 1, 2,..., with (2 1 > (1 1, such that F (t (2 2 is coverget. Repeat this process of selectio ifiitely. Let = ( 1 be the first elemet of the -th sub-sub-sequece. The, for ay fixed m, { : m} is always a subsequece of { (l : 1} for all l m. Hece F is coverget o every ratioal umber. Deote the limit as F (t l o every ratioal t l. Mootoicity of F implies the mootoicity of F o ratioal umbers. Defie, for all t, F (t = if{f (t l : t l > t, t l are ratioal}. Tha, F is right cotiuous ad o-decreasig. The right cotiuity of F esures that, if s is a cotiuity poit of F, F (s F (s. Not all sequece of distributios F would coverge wealy to a distributio fuctio. The easiest example is F ({} = F ( F ( = 1, i.e., P (X = = 1. The, F (t for all t (,. If F all have little probability mass ear or, the the covergece to a fuctio which is ot a distributio fuctio ca be avoided. A sequece of distributio fuctios F is called tight if, for ay ɛ >, there exists a M > such that lim sup (1 F (M+F (M < ɛ; Or, i other words, sup(1 F (x + F ( x as x. Propositio. Every tight sequece of distributio fuctios cotais a a subsequece that wealy coverges to a distributio fuctio. Proof Repeat the proof Helly s Selectio Theorem. The tightess esures the limit is a distributio fuctio. (iii. Characteristic fuctios. Characteristic fuctio is oe of the most useful tools i developig theory about covergece i distributio. The techical details of characteristic fuctios ivolve some owledge of complex aalysis. We shall view them as oly a tool ad try ot to elaborate the techicalities. 1. Defiitio ad examples. For a r.v. X with distributio F, its characteristic fuctio is ψ(t = E(e itx = E(cos(tX + isi(tx = e itx df (x, t (, where i = 1.

5 34 Some basic properties are: ψ( = 1; ψ( 1; ψ( is cotiuous o (, If ψ is characteristic fuctio of X, the e itb ψ(at is characteristic fuctio of ax + b. Product of characteristic fuctios is still a characteristic fuctio. Ad the characteristic fuctio of X X is the product of those of X 1,..., X. The followig table lists some characteristic fuctios for some commoly used distributios: Distributio Desity/Probability fuctio characteristic fuctio (of t Degeerate P (X = a = 1 e iat Biomial Bi(, p P (X = = ( p (1 p, =, 1,..., (pe it + 1 p Poisso P(λ: P (X = = λ e λ /!, =, 1,... exp(λ(e it 1 Normal N(µ, σ 2 : f(x = e (x µ2 /(2σ 2 / 2πσ 2, x (, e iµt σ2 t 2 /2 Uiform Uif[, 1]: f(x = 1, x [, 1] (e it 1/(it Gamma : f(x = λ α x α 1 e λx /Γ(α, x > (1 it/λ α Cauchy: f(x = 1/[π(1 + x 2 ], x (, e t 2. Levy s iversio formula. Propositio Suppose X is r.v. with characteristic fuctio ψ(. The, for all a < b, lim 1 T 2π T e ita e itb it ψ(tdt = P (a < X < b + 1 (P (X = a + P (X = b. 2 Proof. The proof uses Fubii s theorem to iterchage the the expectatio with the itegratio ad the fact that si(x/xdx = π/2. We omit the proof. The above theorem clearly implies that two differet distributio caot have same characteristic fuctio, as formally preseted i the followig corollary. Corollary. fuctios. There is oe-to-oe correspodece betwee distributio fuctios ad characteristic 3. Levy s cotiuity theorem. Theorem 2.1 Levy s cotiuity theorem. Let F, F be cdf with characteristic fuctio ψ, ψ. The, (a. If F F wealy, the ψ (t ψ(t for every t. (b. If ψ (t ψ(t for every t, ad ψ( is cotiuous at, the F F wealy, where F is a cdf with characteristic fuctio ψ. Proof. Part (a directly follows from the defiitio of covergece i distributio sice e itx is a cotiuous fuctio of x for every t. Proof of part (b uses the Levy iversio formula. We omit the details. Remar. Levy s cotiuity theorem eables us to show covergece of distributio through poitwise covergece of characteristic fuctios. This shall be our approach to establish the cetral limit theorem. DIY Exercises: Exercise 2.1. Prove Slutsy s Theorem. Exercise 2.2. (Domiated covergece theorem Suppose X X i distributio ad X Y with E(Y <. Show that E(X E(X.

6 35 Exercise 2.3. Suppose X is idepedet of Y, ad X is idepedet of Y. Use characteristic fuctios to show that, if X coverges to X i distributio ad Y coverges to Y i distributio ad, the X + Y coverges i distributio to X + Y. Exercise 2.4. Suppose a r.v. X has characteristic fuctio ψ. Show that C P (X = x = lim e itx ψ(tdt. C C