Lecture 28: Convergence of Random Variables and Related Theorems

Transcription

1 EE50: Pobability Foundations fo Electical Enginees July-Novembe 205 Lectue 28: Convegence of Random Vaiables and Related Theoems Lectue:. Kishna Jagannathan Scibe: Gopal, Sudhasan, Ajay, Swamy, Kolla An impotant concept in Pobability Theoy is that of convegence of andom vaiables. Since the impotant esults in Pobability Theoy ae the limit theoems that concen themselves with the asymptotic behaviou of andom pocesses, studying the convegence of andom vaiables becomes necessay. We begin by ecalling some definitions petaining to convegence of a sequence of eal numbes. efinition 28. Let {x n, n } be a eal-valued sequence, i.e., a map fom N to R. sequence {x n } conveges to some x R if thee exists an n 0 N such that fo all ɛ > 0, We say that the x n x < ɛ, n n 0. We say that the sequence {x n } conveges to + if fo any M > 0, thee exists an n 0 N such that fo all n n 0, x n > M. We say that the sequence {x n } conveges to if fo any M > 0, thee exists an n 0 N such that fo all n n 0, x n < M. We now define vaious notions of convegence fo a sequence of andom vaiables. It would be helpful to ecall that andom vaiables ae afte all deteministic functions satisfying the measuability popety. Hence, the simplest notion of convegence of a sequence of andom vaiables is defined in a fashion simila to that fo egula functions. Let (Ω, F, P) be a pobability space and let {X n } n N be a sequence of eal-valued andom vaiables defined on this pobability space. efinition 28.2 [efinition 0 (Point-wise convegence o sue convegence)] A sequence of andom vaiables {X n } n N is said to convege point-wise o suely to X if X n (ω) X(ω), ω Ω. Note that fo a fixed ω, {X n (ω)} n N is a sequence of eal numbes. Hence, the convegence fo this sequence is same as the one in definition (28.). Also, since X is the point-wise limit of andom vaiables, it can be poved that X is a andom vaiable, i.e., it is an F-measuable function. This notion of convegence is exactly analogous to that defined fo egula functions. Since this notion is too stict fo most pactical puposes, and neithe does it conside the measuability of the andom vaiables no the pobability measue P( ), we define othe notions incopoating the said chaacteistics. efinition 28.3 [efinition (Almost sue convegence o convegence with pobability )] A sequence of andom vaiables {X n } n N is said to convege almost suely o with pobability (denoted by o w.p. ) to X if P ({ω X n (ω) X(ω)}) =. 28-

2 28-2 Lectue 28: Convegence of Random Vaiables and Related Theoems Almost sue convegence demands that the set of ω s whee the andom vaiables convege have a pobability one. In othe wods, this definition gives the andom vaiables feedom not to convege on a set of zeo measue! Hence, this is a weakened notion as compaed to that of sue convegence, but a moe useful one. In seveal situations, the notion of almost sue convegence tuns out to be athe stict as well. So seveal othe notions of convegence ae defined. efinition 28.4 [efinition 2 (convegence in pobability)] A sequence of andom vaiables {X n } n N is said to convege in pobability (denoted by ) to X if lim P ( X n X > ɛ) = 0, ɛ > 0. As seen fom the above definition, this notion concens itself with the convegence of a sequence of pobabilities! At the fist glance, it may seem that the notions of almost sue convegence and convegence in pobability ae the same. But the two definitions actually tell vey diffeent stoies! Fo almost sue convegence, we collect all the ω s wheein the convegence happens, and demand that the measue of this set of ω s be. But, in the case of convegence in pobability, thee is no diect notion of ω since we ae looking at a sequence of pobabilities conveging. To claify this, we do away with the shot-hand fo pobabilities (fo the moment) and obtain the following expession fo the definition of convegence in pobability: lim P ({ω X n(ω) X(ω) > ɛ}) = 0, ɛ > 0. Since the notion of convegence of andom vaiables is a vey inticate one, it is woth spending some time pondeing the same. efinition 28.5 [efinition 3 (convegence in th mean)] A sequence of andom vaiables {X n } n N is said to convege in th mean to X if lim E [ X n X ] = 0. In paticula, when = 2, the convegence is a widely used one. It goes by the special name of convegence in the mean-squaed sense. The last notion of convegence, known as convegence in distibution, is the weakest notion of convegence. In essence, we look at the distibutions (of andom vaiables in the sequence in consideation) conveging to some distibution (when the limit exists). This notion is extemely impotant in ode to undestand the Cental Limit Theoem (to be studied in a late lectue). efinition 28.6 [efinition 4 (convegence in distibution o weak convegence)] A sequence of andom vaiables {X n } n N is said to convege in distibution to X if lim F X n (x) = F X (x), x R whee F X ( ) is continuous.

3 Lectue 28: Convegence of Random Vaiables and Related Theoems 28-3 That is, the sequence of distibutions must convege at all points of continuity of F X ( ). Unlike the pevious fou notions discussed above, fo the case of convegence in distibution, the andom vaiables need not be defined on a single pobability space! Befoe we look at an example that seves to claify the above definitions, we summaize the notations fo the above notions. () Point-wise Convegence: X n p.w.. (2) Almost sue Convegence: X n o X n w.p.. (3) Convegence in pobability: X n. (4) Convegence in th mean: X n. When = 2, X n m.s.. (5) Convegence in istibution: X n. Example: Conside the pobability space (Ω, F, P) = ([0, ], B([0, ]), λ) and a sequence of andom vaiables {X n, n } defined by X n (ω) = { n, if ω [ 0, n], 0, othewise. Since the pobability measue specified is the Lebesgue measue, the andom vaiable can be e-witten as { n, with pobability n, 0, with pobability n. Clealy, when ω 0, lim X n(ω) = 0 but it diveges fo ω = 0. This suggests that the limiting andom vaiable must be the constant andom vaiable 0. Hence, except at ω = 0, the sequence of andom vaiables conveges to the constant andom vaiable 0. Theefoe, this sequence does not convege suely, but conveges almost suely. Fo some ɛ > 0, conside lim P ( X n > ɛ) = lim P ( n), ( ), n = lim = 0. Hence, the sequence conveges in pobability. Conside the following two expessions: lim X n 2] = lim (n 2 n ) + 0, =.

4 28-4 Lectue 28: Convegence of Random Vaiables and Related Theoems lim E [ X n ] = lim (n n ) + 0, =. Since the above limits do not equal 0, the sequence conveges neithe in the mean-squaed sense, no in the sense of fist mean. Consideing the distibution of X n s, it is clea (though visualization) that they convege to the following distibution: F X (x) = { 0, if x < 0,, othewise. Also, this happens at each x 0 i.e. at all points of continuity of F X ( ). Hence, the sequence of andom vaiables convege in distibution. So fa, we have mentioned that cetain notions ae weake than cetain othes. Let us now fomalize the elations that exist among vaious notions of convegence. It is immediately clea fom the definitions that point-wise convegence implies almost sue convegence. Figue (28.) is a summay of the implications that hold fo any sequence fo andom vaiables. No othe implications hold in geneal. We pove these, in a seies of theoems, as below. p.w. th mean ( ) s th mean ( > s ) Figue 28.: Implication iagam Theoem 28.7 X n = X n,. Poof: Conside the quantity lim P ( X n X > ɛ). Applying Makov s inequaliy, we get whee (a) follows since X n Theoem 28.8 X n Poof: Fix an ɛ > 0. lim P ( X E [ X n X ] n X > ɛ) lim ɛ, ɛ > 0,. Hence poved. = X n. F Xn (x) = P(X n x), (a) = 0, = P(X n x, X x + ɛ) + P(X n x, X > x + ɛ), F X (x + ɛ) + P( X n X > ɛ).

5 Lectue 28: Convegence of Random Vaiables and Related Theoems 28-5 Similaly, Thus, F X (x ɛ) = P(X x ɛ), = P(X x ɛ, X n x) + P(X x ɛ, X n > x), F Xn (x) + P( X n X > ɛ). F X (x ɛ) P( X n X > ɛ) F Xn (x) F X (x + ɛ) + P( X n X > ɛ). As n, since X n, P( X n X > ɛ) 0. Theefoe, F X (x ɛ) lim inf F X n (x) lim sup F Xn (x) F X (x + ɛ), ɛ > 0. If F is continuous at x, then F X (x ɛ) F X (x) and F X (x + ɛ) F X (x) as ɛ 0. Hence poved. Theoem 28.9 X n s = X n, if > s. Poof: Fom Lyapunov s Inequality [, Chapte 4], we see that (E[ X n X s ]) /s (E[ X n X ]) /, > s. Hence, the esult follows. Theoem 28.0 X n = X n in geneal. Let X n be an independent sequence of andom vaiables defined as { n 3, w.p. n, 2 0, w.p. - n. 2 Then, P( X n > ɛ) = n 2 fo lage enough n, and hence X n 0. On the othe hand, E[ X n ] = n, which diveges to infinity as n gows unbounded. Theoem 28. X n = Xn in geneal. Let X be a Benoulli andom vaiable with paamete 0.5, and define a sequence such that X i =X i. Let Y = X. Clealy, X i Y. But, Xi Y =, i. Hence, X i does not convege to Y in pobability. Theoem 28.2 X n = X n in geneal. Let {X n } be a sequence of independent andom vaiables defined as {, w.p. n, 0, w.p. - n. lim P( X n > ɛ) = lim P(X n = ) = lim n = 0. So, X n 0. Let A n be the event that { }. Then, A n s ae independent and n= P(A n ) =. By Boel-Cantelli Lemma 2, w.p. infinitely many A n s will occu, i.e., { } i.o.. So, X n does not convege to 0 almost suely.

6 28-6 Lectue 28: Convegence of Random Vaiables and Related Theoems Theoem 28.3 X n s = X n if > s in geneal. Let {X n } be a sequence of independent andom vaiables defined as { n, w.p. Hence, E[ X s n ] = n s 2 0. But, E[ X n ] = n s 2. Theoem 28.4 X n m.s. = X n in geneal. n +s 2 0, w.p. - n +s 2 Let {X n } be a sequence of independent andom vaiables defined as {, w.p. n, 0, w.p. - n. E[Xn]= 2 n. So, X m.s. n 0. As seen peviously (duing the poof of Theoem (28.2)), X n does not convege to 0 almost suely. Theoem 28.5 X n m.s. = X n in geneal. Let {X n } be a sequence of independent of andom vaiables defined as { n, ω (0, X n (ω) = n ), 0, othewise. We know that X n conveges to 0 almost suely. E[X 2 n]=n. So, X n does not convege to 0 in the mean-squaed sense. Befoe poving the implication X n = X n, we deive a sufficient condition followed by a necessay and sufficient condition fo almost sue convegence.,. Theoem 28.6 If ɛ > 0, n P(A n (ɛ)) <, then X n, whee A n (ɛ) = {ω : X n (ω) X(ω) > ɛ}. Poof: By Boel-Cantelli Lemma, A n (ɛ) occus finitely often, fo any ɛ > 0 w.p.. Let B m (ɛ) = Theefoe, P(B m (ɛ)) P(A n (ɛ)). n=m n m A n (ɛ). So, P(B m (ɛ)) 0 as m, wheneve P(A n (ɛ)) <. An equivalent way of poving almost sue ( n ) convegence is to fist conside lim P {ω : X n (ω) X(ω) > ɛ} = 0, ɛ > 0. Hence, P(B m (ɛ)) 0 m n m as m. This implies almost sue convegence.

7 Lectue 28: Convegence of Random Vaiables and Related Theoems 28-7 Theoem 28.7 X n iff P(B m (ɛ)) 0 as m, ɛ > 0. Poof: Let A(ɛ) = { ω Ω : ω A n (ɛ) fo infinitely many values of n } = A n (ɛ). m n=m If X n ( X, then it is easy to see that P(A(ɛ))=0, ɛ > 0. ) Then, P B m (ɛ) = 0. m= Since {B m (ɛ)} is a nested, deceasing sequence, it follows fom the continuity of pobability measues that lim P(B m(ɛ)) = 0. m Convesely, let C={ ω Ω : X n (ω) X(ω) as n }. Then, ( ) P(C c ) = P A(ɛ) = P ɛ>0 ( m= P m= ( A m ( A ( m ) ) as A(ɛ) A(ɛ ) if ɛ ɛ )). Also, P(A(ɛ)) = lim m P(B m(ɛ))=0. Conside ( ( )) ( P A = P k So, P(C c ) = 0. Hence, P(C) =. m= = lim m P ) ) ( B m k ( ( B m k )) = 0. k, by assumption Coollay 28.8 X n = X n. Poof: X n = lim P(B m(ɛ))=0. m As A m (ɛ) B m (ɛ), it is implied that lim P(A m(ɛ))=0. m Hence, X n. Theoem 28.9 If X n, then thee exists a deteministic, inceasing subsequence n, n 2, n 3,... such that X a.s ni. X as i. Poof: The eade is efeed to Theoem 3 in Chapte 7 of [] fo a poof. Example: Let {X n } be a sequence of independent andom vaiables defined as {, w.p. n, 0, w.p. - n.

8 28-8 Lectue 28: Convegence of Random Vaiables and Related Theoems It is easy to veify that, X n 0, but X n X. Howeve, if we conside the subsequence {X, X 4, X 9,... }, this (sub)sequence of andom vaiables conveges almost suely to 0. This can be veified as follows. Let n i = i 2, Y i = X ni = X i 2. Thus, P(Y i = ) = P(X i 2 = ) = i. 2 P(Y i ) = i <. Hence, by BCL-, X 2 2 i 0. i N i N Although this is not a poof fo the above theoem, it seves to veify the statement via a concete example. Theoem [Skookhod s Repesentation Theoem] Let {X n, n } and X be andom vaiables on (Ω, F, P) such that X n conveges to X in distibution. Then, thee exists a pobability space (Ω, F, P ), and andom vaiables {Y n, n } and Y on (Ω, F, P ) such that, a) {Y n, n } and Y have the same distibutions as {X n, n } and X espectively. b) Y n Y as n. Theoem 28.2 [Continuous Mapping Theoem] If X n X, and g : R R is continuous, then g(xn ) g(x). Poof: By Skookhod s Repesentation Theoem, thee exists a pobability space (Ω, F, P ), and {Y n, n }, Y on (Ω, F, P ) such that, Y n Y. Futhe, fom continuity of g, {ω Ω g(y n (ω)) g(y (ω))} {ω Ω Y n (ω) Y (ω)}, P({ω Ω g(y n (ω)) g(y (ω))}) P({ω Ω Y n (ω) Y (ω)}), P({ω Ω g(y n (ω)) g(y (ω))}), g(y n ) g(y ), g(y n ) g(y ). This completes the poof since, g(y n ) has the same distibution as g(x n ), and g(y ) has the same distibution as g(x). Theoem X n E[g(X)]. X iff fo evey bounded continuous function g : R R, we have E[g(Xn )] Poof: Hee, we pesent only a patial poof. Fo a full teatment, the eade is efeed to Theoem 9 in chapte 7 of []. Assume X n X. Fom Skookhod s Repesentation Theoem, we know that thee exist andom vaiables {Y n, n } and Y, such that Y n Y. Fom Continuous Mapping Theoem, it follows that g(y n ) g(y ), since g is given to be continuous. Now, since g is bounded, by CT, we have E[g(Y n )] E[g(Y )]. Since, g(y n ) has the same distibution as g(x n ), and g(y ) has the same distibution as g(x), we have E[g(X n )] E[g(X)]. Theoem If X n, then CXn (t) C X (t), t. Poof: If X n, fom Skookhod s Repesentation Theoem, thee exist andom vaiables {Yn } and Y such that Y n Y. So, cos(y n t) cos(y t), cos(x n t) cos(xt), t.

9 Lectue 28: Convegence of Random Vaiables and Related Theoems 28-9 As cos( ) and sin( ) ae bounded functions, We get, E[cos(Y n t)] + ie[sin(y n t)] E[cos(Y t)] + ie[sin(y t)], t. C Yn (t) C Y (t), t. C Xn (t) C X (t), t, since distibutions of {X n } and X ae same as those of {Y n } and Y espectively, fom Skookhod s Repesentation Theoem. Theoem Let {X n } be a sequence of RVs with chaacteistic functions, C Xn (t) fo each n, and let X be a RV with chaacteistic function C X (t). If C Xn (t) C X (t), then X n. Theoem Let {X n } be a sequence of RVs with chaacteistic functions C Xn (t) fo each n, and suppose lim C X n (t) exists t, and is denoted by φ (t). Then, one of the following statements is tue: (a) φ ( ) is discontinuous at t = 0, and in this case, X n does not convege in distibution. (b) φ ( ) is continuous at t = 0, and in this case, φ is a valid chaacteistic function of some RV X. Then. X n Remak In ode to pove that the φ(t) above is indeed a valid chaacteistic function, we need to veify the thee defining popeties of chaacteistic functions. Howeve, in the light of Theoem (28.25), it is sufficient to veify the continuity of φ(t) at t = 0. Afte all φ(t) is not an abitay function; it is the limit of the chaacteistic functions of X n s, and theefoe inheits some nice popeties. ue to these inheited popeties, it tuns out it is enough to veify continuity at t = 0, instead of veifying all the conditions of Bochne s theoem! Note: Theoems (28.24) and (28.25) togethe ae known as Continuity Theoem. Fo poof, efe to []. 28. Execises. (a) Pove that convegence in pobability implies convegence in distibution, and give a counteexample to show that the convese need not hold. (b) Show that convegence in distibution to a constant andom vaiable implies convegence in pobability to that constant. 2. Conside the sequence of andom vaiables with densities f Xn (x) = - cos(2πnx), x (0, ). o X n s convege in distibution? oes the sequence of densities convege? 3. [Gimmett] A sequence {X n, n } of andom vaiables is said to be completely convegent to X if n P( X n X > ɛ) <, ɛ > 0

10 28-0 Lectue 28: Convegence of Random Vaiables and Related Theoems Show that, fo sequences of independent andom vaiables, complete convegence is equivalent to almost sue convegence. Find a sequence of (dependent) andom vaiables that convege almost suely but not completely. 4. Constuct an example of a sequence of chaacteistic functions φ n (t) such that the limit φ(t) = lim φ n (t) exists fo all t, but φ(t) is not a valid chaacteistic function. Refeences [] G. G.. Stizake and. Gimmett. Pobability and andom pocesses. Oxfod Science Publications, ISBN 0, 9(853665):8, 200.