ECE5: Aalysis of Radom Sigals Fall 6 Lecture Chapter 6: Covergece of Radom Sequeces Dr Salim El Rouayheb Scribe: Abhay Ashutosh Doel, Qibo Zhag, Peiwe Tia, Pegzhe Wag, Lu Liu Radom sequece Defiitio A ifiite sequece X, =,,, of radom variables is called a radom sequece Covergece of a radom sequece Example Cosider the sequece of real umbers X =, =,,, + This sequece coverges to the limit l = We write lim X = l = This meas that i ay eighbourhood aroud we ca trap the sequece, ie, ɛ >, (ɛ) st for (ɛ) X l ɛ We ca pick ɛ to be very small ad make sure that the sequece will be trapped after reachig (ɛ) Therefore as ɛ decreases (ɛ) will icrease For example, i the cosidered sequece: Almost sure covergece ɛ =, (ɛ) =, ɛ =, (ɛ) = Defiitio A radom sequece X, =,,, 3,, coverges almost surely, or with probability oe, to the radom variable X iff P ( lim X = X) = We write X X
Example Let be a radom variable that is uiformly distributed o, ] Defie the radom sequece X as X = So X =, X =, X =, X 3 = 3, Let us take specific values of For istace, if = X =, X =, X = 4, X 3 = 8, We ca thik of it as a ur cotaiig sequeces, ad at each time we draw a value of, we get a sequece of fixed umbers I the example of tossig a coi, the output will be either heads or tails Whereas, i this case the output of the experimet is a radom sequece, ie, each outcome is a sequece of ifiite umbers Questio: Does this sequece of radom variables coverge? Aswer: This sequece coverges to if with probability = P ( ) X = if = with probability = P ( = ) Sice the pdf is cotiuous, the probability P ( = a) = for ay costat a Notice that the covergece of the sequece to is possible but happes with probability Therefore, we say that X coverges almost surely to, ie, X Covergece i probability Defiitio 3 A radom sequece X coverges to the radom variable X i probability if ɛ > lim P r X X ɛ} = We write : X p X Example 3 Cosider a radom variable uiformly distributed o, ] ad the sequece X defied by: with probability X = with probability Distributed as show i Figure Notice that oly X or X 3 ca be equal to for the same value of Similarly, oly oe of X 4, X 5, X 6 ad X 7 ca be equal to for the same value of ad so o ad so forth Questio: Does this sequece coverge?
X7() X6() X5() X4() X3() X() X() 5 3 4 5 6 7 8 9 5 3 4 5 6 7 8 9 5 3 4 5 6 7 8 9 5 3 4 5 6 7 8 9 5 3 4 5 6 7 8 9 5 3 4 5 6 7 8 9 5 3 4 5 6 7 8 9 Figure : Plot of the distributio of X () Aswer: Ituitively, the sequece will coverge to Let us take some examples to see how the sequece behave for = : for = 3 : = = = = =3 =3 =4 =4 From a calculus poit of view, these sequeces ever coverge to zero because there is always a jump showig up o matter how may zeros are precedig (Fig ); for ay : X () does ot coverge i the calculus sese Which meas also that X does ot coverge to zero almost surely () 3
5 X 5 3 4 5 6 Figure : Plot of the sequece for = This sequece coverges i probability sice lim P ( X ) = ɛ > Remark The observed sequece may ot coverge i calculus sese because of the itermittet jumps ; however the frequecy of those jumps goes to zero whe goes to ifiity 3 Covergece i mea square Defiitio 4 A radom sequece X coverges to a radom variable X i mea square sese if lim E X X ] = We write: X ms X Remark I mea square covergece, ot oly the frequecy of the jumps goes to zero whe goes to ifiity; but also the eergy i the jump should go to zero Example 4 Cosider a radom variable uiformly distributed over, ], ad the sequece X () defied as: a for X () = otherwise Note that P (X = a ) = ad P (X = ) = Questio: Does this sequece coverge? 4
Figure 3: Plot of the sequece X () Aswer: Let us check the differet covergece criteria we have see so far Almost sure covergece: X because Covergece i probability: X because lim P (X = ) = lim P X ɛ} = (Flash Forward: almost sure covergece covergece i probability) X X X X 3 Mea Square Covergece: E X ] = a (P (X = a ) + P (X = )), = a If a = lim E X ] ms = X, If a = lim E X ] = X does ot coverge i ms to I this example, the covergece of X i the mea square sese depeds o the value of a 4 Covergece i distributio Defiitio 5 (First attempt) A radom sequece X coverges to X i distributio if whe goes to ifiity, the values of the sequece are distributed accordig to a kow distributio We say d X X Example 5 Cosider the sequece X defied as: X i B( X = ) for i = (X i + ) mod = X for i > 5
Questio: I which sese, if ay, does this sequece coverge? Aswer: This sequece has two outcomes depedig o the value of X : X =, X : X =, X : Almost sure covergece: X does ot coverge almost surely because the probability of every jump is always equal to Covergece i probability: X does ot coverge i probability because the frequecy of the jumps is costat equal to 3 Covergece i mea square: X does ot coverge to i mea square sese because lim X E ] = E X X + ], 4 = EX ] EX ] + 4, = + + 4, 4 Covergece i distributio: At ifiity, sice we do ot kow the value of X, each value of X ca be either or with probability Hece, ay umber X is a radom variable B( ) We say, X coverges i distributio to Beroulli( ) ad we deote it by: = X d Ber( ) Example 6 (Cetral Limit Theorem)Cosider the zero-mea, uit-variace, idepedet radom variables X, X,, X ad defie the sequece S as follows: S = X + X + + X The CLT states that S coverges i distributio to N(, ), ie, Theorem Note: Almost sure covergece Covergece i mea square } S d N(, ) Covergece i probability covergece i distributio There is o relatio betwee Almost Sure ad Mea Square Covergece The relatio is uidirectioal, ie, covergece i distributio does ot imply covergece i probability either almost sure covergece or mea square covergece 6
3 Covergece of a radom sequece Example : defied as: Questio: Let the radom variable U be uiformly distributed o, ] Cosider the sequece X() = ( ) U Does this sequece coverge? if yes, i what sese(s)? Aswer: Almost sure covergece: Suppose The sequece becomes U = a I fact, for ay a, ] therefore, X X = a, X = a, X 3 = a 3, X 4 = a 4, lim X =, Remark 3 X because, by defiitio, a radom sequece coverges almost surely to the radom variable X if the sequece of fuctios X coverges for all values of U except for a set of values that has a probability zero Covergece i probability: Does X? Recall from theorem 3 of lecture 7: } d ms which meas that by provig almost-sure covergece, we get directly the covergece i probability ad i distributio However, for completeess we will formally prove that X coverges to i probability To do so, we have to prove that lim P ( X ɛ) = ɛ >, lim ɛ) = ɛ > 7
By defiitio, Thus, X = U ( ) ( ) U lim P X ɛ = lim P ɛ, () = lim P (U ɛ), () = (3) Where equatio 3 follows from the fact that fidig U, ] 3 Covergece i mea square sese: Does X coverge to i the mea square sese? I order to aswer this questio, we eed to prove that We kow that, Hece, X ms lim E X ] = lim E X ] = lim E X = lim E = lim = lim = lim U ], ], E U ], u du, ] u 3 3 = lim 3, = 4 Covergece i distributio: Does X coverge to i distributio? The formal defiitio of covergece i distributio is the followig: Hereafter, we wat to prove that X d, X d X lim F X (x) = F X (x) Recall that the limit rv X is the costat ad therefore has the followig CDF : Sice X = ( ) U, the distributio of the X i ca be derived as followig: 8
Figure 4: Plot of the CDF of Remark 4 At the CDF of X will be flip-floppig betwee (if is eve) ad (if is odd) (cf figure 5) which implies that there is a discotiuity at that poit Therefore, we say that X coverges i distributio to a CDF F X (x) except at poits where F X (x) is ot cotiuous Defiitio 6 X coverges to X i distributio, ie, X] d X iff lim F X (x) = F X (x) Remark 5 It is clear here that except at poits where F X (x) is ot cotiuous lim F X (x) = F x (x) except for x = Therefore, X coverges to X i distributio We could have deduced this directly from covergece i mea square sese or almost sure covergece Theorem a) If X X X X b) If X ms X X X c) If X X X d X d) If P X Y } = for all for a radom variable Y with E Y ] <, the ms X X X X Proof The proof is omitted Remark 6 Covergece i probability allows the sequece, at, to deviate from the mea for ay value with a small probability; whereas, covergece i mea square limits the amplitude of this deviatio whe (We ca thik of it as eergy we ca ot allow a big deviatio from the mea) 9
CDF of U CDF of X CDF of X CDF of X 3 4 Back to real aalysis Figure 5: Plot of the CDF of U, X, X ad X 3 Defiitio 7 A sequece (x ) is Cauchy if for every ɛ, there exists a large umber N st m, > N, x m x < ɛ lim,m x m x = Claim Every Cauchy sequece is coverget Couter example Cosider the sequece X Q defied as x =, x + = x+ x The limit of this sequece is give by: l = l + l, l = l +, l = ± / Q This implies that the sequece does ot coverge i Q
Couter example Cosider the sequece x = / i (, ) Obviously it does ot coverge i (, ) sice the limit l = / (, ) Defiitio 8 A space where every sequece coverges is called a complete space Theorem 3 R is a complete space Proof The proof is omitted Theorem 4 Cauchy criteria for covergece of a radom sequece ] a) X X P lim x m x = m, b) X ms X c) X X lim E x m x ] = m, lim P x m x ε] = m, = ɛ Proof The proofs are omitted Example 7 Cosider the sequece of example from last lecture, X i B( X = ) for i = (X i + ) mod = X for i > Goal: Our goal is to prove that this sequece does ot coverge i mea square usig Cauchy criteria This sequece has two outcomes depedig o the value of X : X =, X : X =, X : Therefore, E X X m ] = E X] ] + E X m E Xm X ], = + E X mx ] Cosider, without loss of geerality, that m > E X X m ] = if m is odd, E X X m ] = E X] = if m is eve Hece, lim E X X m ] if m is odd, =,m if m is eve, which implies that X does ot coverge i mea square by theorem 4-b)
Lemma Let X be a radom sequece with E X ] < X Theorem 5 Weak law of large umbers ms X iff lim m, E X mx ] exists ad is fiite Let X, X, X 3,, X i be iid radom variables E X i ] = µ, i Let The Usig the laguage of this chapter: S = X + X + + X P S µ ɛ] S µ Theorem 6 Strog law of large umbers Let X, X, X 3,, X i be iid radom variables E X i ] = µ, i Let The Usig the laguage of this chapter: S = X + X + + X ] P lim S µ ɛ = S µ Theorem 7 Cetral limit theorem Let X, X, X 3,, X i be iid radom variables E X i ] =, i Let The Usig the laguage of this chapter: Z = X + X + + X P Z z] = z Z d N(, ) π e z dz