ECE 330:541, Stochastic Signals and Systems Lecture Notes on Limit Theorems from Probability Fall 2002

Transcription

1 ECE 330:541, Stochastic Sigals ad Systems Lecture Notes o Limit Theorems from robability Fall 00 I practice, there are two ways we ca costruct a ew sequece of radom variables from a old sequece of radom variables. Suppose that we have a sequece {X } of radom variables, the we may defie the sum ad the average by: 1. S = X 1 + X + + X. M = 1 S These sequeces play a domiat role i the limit theory of probability ad lead to two classes of theorems: the cetral limit theorems, ad the laws of large umbers. We start with the cetral limit theorems. 1 Cetral Limit Theorems Suppose that we have a sequece of radom variables {X } that are idepedet ad idetically distributed iid, with mea µ ad variace σ. We have show that the pdf of S = X 1 + X + + X is simply the covolutio of the pdfs of X 1, X,, X. If oe plays aroud ad covolves a fuctio, such as χ 0,1] the uit box fuctio, with itself may times oe will see that the shape gets closer ad closer to beig bell shaped. This pheomeo is what the cetral limit theorems are all about. Specifically, the cetral limit theorems state that as becomes large, the cdf of a ormalized versio of S approaches the cdf of a Gaussia radom variable that is, we have covergece i distributio. I order to show the cetral limit theorems, we eed some tools about covergece i distributio. We start by itroducig a ew tool for studyig discrete radom variables, called the probability geeratig fuctio. The probability geeratig fuctio G X z, for a o-egative iteger valued radom variable X with pmf p x k = p k is give by G X z = p k z k where z is the complex variable with z 1. Observe that for z = e iω we have G X z = φ x ω, the characteristic fuctio. G X z is called the probability geeratig fuctio sice: k=0 p k = 1 d k G0 k! dz k. A further ote: Sice k p k = 1, we have that G X z coverges absolutely for z < 1. For iteger-valued radom variables, covergece i distributio is equivalet to covergece of the pmfs. Here is how to see this. Let F X k be the cdf of X, ad such that F X k F X k for some radom variable X. This implies F X k F X k 0 for all k as. We wat to show p X k p X k 0. Let us expad each of the pmfs as: p X k p X k = F X k F X k 1 F X k F X k 1 F X k F X k + F X k 1 F X k 1 0. The reverse implicatio, that p X k p X k implies F X k F X k ca be similarly show. We ow prove a key tool for showig covergece i distributio of o-egative, iteger-valued radom variables. 1

2 Theorem 1. The sequeces of pmfs p X k coverges to the limitig pmf p X k if ad oly if where G X z is the geeratig fuctio for p X k. lim G X z = G x z for z < 1, roof. = : Suppose p X k p X k the we must show G X z G x z. G X z G x z = p X k p X k z k k=0 K p X k p X k + k=0 k=k+1 z k. Let us take z r < 1, the for ay ɛ > 0 we may choose K large eough so that k=k+1 z k rk+1 1 r < ɛ/. For this choice of ɛ, sice p X k p X k, we may fid a N such that for all N p X k p X k < for k = 0, 1,, K. Therefore, substitutig we get ɛ K + 1, ɛ G X z G x z < K + 1 K ɛ = ɛ for N, ad hece we have show G X z G X z. =: Now suppose that we have G X z G x z for z < 1, we must show p X k p X k. To hadle this directio, we apply a famous theorem from complex variables due to Weierstrass see below for the statemet of this theorem about coverget sequeces of aalytic fuctios. But ad similarly for p X k. Thus, we have d t G X z lim dz t = dt G X z dz t for z < 1 ad for all k. p X k = 1 d k G X 0 k! dz k lim p X k = p X k. For referece, we state Weierstrass s Theorem o coverget sequeces of aalytic fuctios: Theorem Weierstrass. If {f } is a sequece of aalytic fuctios over a subset G of the complex plai, ad if f f the f is aalytic ad for every iteger k 1. d k dz k f dk dz k f

3 I order to show the cetral limit theorems, we eed a more geeral versio of the previous result, kow as the Levy-Cramer Theorem. We oly state the theorem as the proof requires mathematical machiery sigificatly more complicated tha is eeded for this class. Theorem 3 Levy-Cramer Theorem. Let {X } be a sequece of radom variables ad let φ X ω ad F X ω be the characteristic fuctio ad cumulative distributio fuctio of X. The F X x F x x if ad oly if, for every ω, φ X ω φ X ω, where φ X ω is the characteristic fuctio of the limit distributio F X x. There are several cetral limit theorems. The first oe, the De Moivre-Laplace Theorem is a special case of the Lideberg-Levy Cetral Limit Theorem. Theorem 4 De Moivre-Laplace Theorem. Let {X } be a sequece of iid Beroulli radom variables with X = 1 = p ad q = 1 p. Let The, for every x Z = S p pq = S ES V ars. lim {Z x} = 1 x e z / dz. π Rather tha prove this theorem, we shall observe that it is a cosequece of the followig theorem, which we shall prove. Theorem 5 Lideberg-Levy Theorem or The Cetral Limit Theorem. Let {X } be a sequece of iid radom variables such that EX = µ ad V arx = σ. The, for every x lim F Z x = 1 x e z / dz. π where Z = S ES V ars. roof. Let us deote by X the radom variable that the X are iid to for simplicity i otatio. Observe that we may rewrite Z as Z = 1 σ X j µ. j=1 Usig the followig properties of the characteristic fuctio: ] 1. φ X1 +X + +X ω = E e iωx 1+X + +X = φ X ω]. φ ax ω = φ X aω, we get ] ω φ Z ω = φ X µ σ. Now let us use the series expasio for Ee iωx ], amely Ee iωx ] = E 1 + iωx + iωx! + = 1 + iωex] + i ω EX ] +! 3 ]

4 ad we thus get ω φ X µ σ Hece = 1 + iω σ EX µ + i ω = 1 ω + O 1 3/ φ Z ω = 1 σ EX µ + O 3/ error terms. 1 ω + error terms ]. error terms We may discared the error terms sice they are domiated by the ω / term. Takig the limit, we get Recall that ad thus lim φ Z ω = lim 1 + ω lim 1 + x = e x lim φ Z ω = e ω /. ] 1. Fially, observe that this is the characteristic fuctio of a N0, 1 radom variable, ad ow by the Levy-Cramer Theorem we have the desired result. If the sequece {X } are ot iid, but merely idepedet, the the Lideberg-Levy theorem does ot hold, eve if all the radom variables have fiite variace. There are, however, some cases where it does. We state o of them for referece: Theorem 6 Lyapoov s Cetral Limit Theorem. Let {X } be a sequece of idepedet radom variables with µ = EX < ad σ = V arx <, ad β = E X µ 3 <. Defie 1/3 B = β i If lim B /C = 0, the for every x C = Z = σi 1/ X i µ i C. lim F Z x = 1 x e z / dz. π To illustrate the behavior ad priciples of the cetral limit theorem, we refer to the pictures o pg 50 ad 51 of the Yates ad Goodma textbook. 4

5 Laws of Large Numbers Havig discussed the cetral limit theorems, we ow tur our attetio to the other class of limit theorems, the laws of large umbers. The laws of large umbers are cocered with the sample mea sequece M. Theorem 7 Beroulli s Law of Large Numbers. Let {X } be a i.i.d. sequece of Beroulli radom variables with X i = 1 = p ad q = 1 p. For every ɛ > 0, lim {ω S : M ω p < ɛ} = 1. roof. Observe that EM = p ad V arm = 1 V ars = pq. Now apply Chebyshev s iequality to get: { M p ɛ} pq ɛ. Therefore, lim { M p ɛ} = 0. I provig Beroulli s Law of Large Numbers, there was othig special about the fact that the radom variables were i.i.d. or Beroulli. If we relax the requiremets ad let {X } be a sequece of idepedet radom variables, we get a more geeral form of the Law of Large Numbers. Theorem 8 Chebyshev s Law of Large Numbers. Let {X } be a sequece of idepedet r.v.s. with EX i = µ i ad V arx i = σi c <. The for ɛ > 0, { M 1 } µ i < ɛ = 1. roof. As i the proof of Beroulli s Law of Large Numbers, we shall make use of Chebyshev s Iequality. Observe that EM = 1 µ i ad By Chebyshev s Iequality, we have V arm = 1 σ i c. which yields Takig the limit, we have ad the desired result follows. M EM ɛ V arm ɛ, M 1 µ i ɛ lim c ɛ. M 1 µ i ɛ = 0, I Chebyshev s Law of Large Numbers, we assumed V arx i c for every i. I may cases, however, we caot uiformly boud V arx i. However, there is still hope provided that 1 V ar X i 0. 5

6 Theorem 9 Markov s Law of Large Numbers. Let {X } be a sequece of idepedet r.v.s. with EX i = µ i, V arx i = σi 1 ad such that V ar X i 0. The { M 1 } µ i < ɛ = 1. roof. As before, apply Chebyshev s Iequality: which yields Hece M EM ɛ V arm ɛ, M 1 µ i ɛ lim σ i ɛ 0. M 1 µ i < ɛ = 1. The two laws of large umbers that we have stated so far are examples of weak laws of large umbers sice they ivolve weak stochastic covergece, or covergece i probability. We ow move oto discussig the strog laws of large umbers which ivolve almost sure covergece. Due to the amout of machiery eeded to prove these theorems, we merely state them. Theorem 10 Borel s Strog Law of Large Numbers. Let {X } be a sequece of i.i.d. Beroulli r.v.s. with X i = 1 = p, the lim M = p = 1. Theorem 11 Kolmogorov s Law of Large Numbers. Let {X } be a sequece of idepedet r.v.s. with V arx < the =1 lim M 1 = lim EX i = 1. A special cosequece of Kolmogorov s Law of Large Numbers is what is commoly referred to as the Strog Law of Large Numbers. Theorem 1 Covetioal Strog Law of Large Numbers. Let {X } be a sequece of i.i.d. r.v.s. with V arx <, the lim M 1 = lim EX i = 1. 6