MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 21 11/27/2013

MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 21 11/27/2013 Fuctioal Law of Large Numbers. Costructio of the Wieer Measure Cotet. 1. Additioal techical results o weak covergece 2. Fuctioal Strog Law of Large Numbers 3. Existece of Wieer measure (Browia motio) 1 Additioal techical results o weak covergece Give two metric spaces S 1, S 2 ad a measurable fuctio f : S 1 S 2, suppose S 1 is equipped with some probability measure P. This iduces a probability measure o S 2 which is deoted by Pf 1 ad is defied by Pf 1 (A) = P(f 1 (A)) for every measurable set A S 2. The for ay radom variable X : S 2 R, its expectatio E Pf 1 [X] is equal to E P [X(f)]. (Covice yourself that this is the case by lookig at the special case whe f is a simple fuctio). Theorem 1 (Mappig Theorem). Suppose P P for a sequece of probability measures P, P o S 1 ad suppose f : S 1 S 2 is cotiuous. The P f 1 Pf 1 o S 2. Proof. We use Portmetau theorem, i particular weak covergece characterizatio usig bouded cotiuous fuctios. Thus let g : S 2 R be ay bouded cotiuous fuctio. Sice it is cotiuous, it is also measurable, thus it is also a radom variable defied o (S 2, B 2 ), where B 2 is the Borel σ-field o S 2. We have, E Pf 1 [g] = E P [g(f)]. Sice g is a bouded cotiuous, the the compositio is also bouded cotiuous. Therefore, by Portmateau theorem E P [g(f)] E P [g(f)] = E Pf 1 [g] 1

Defiitio 1. A sequece of probability measures P o metric space S is defied to be tight if for every E > 0 there exists 0 ad a compact set K S, such that P (K) > 1 E for all > 0. Theorem 2 (Prohorov s Theorem). Suppose sequece P is tight. The it cotais a weakly coverget subsequece P (k) P. The coverse of this theorem is also true, but we will ot eed this. We do ot prove Prohorov s Theorem. The proof ca be foud i [1]. Recall that Arzela-Ascoli Theorem provides a characterizatio of compact sets i C[0, T ]. We ca use it ow for characterizatio of tightess. Propositio 1. Suppose a sequece of measures P o C[0, T ] satisfies the followig coditios: (i) There exists a 0 such that lim P ( x(0) a) = 0. (ii) For each E > 0, lim δ 0 lim sup P ({x : w x (δ) > E}) = 0. The the sequece P is tight. Proof. Fix E > 0. From (i) we ca fid â ad 0 large eough so that P ( x(0) > â) < E for all > 0. For every m 0 we ca also fid a m large eough so that P m ( x(0) > a m ) < E. Take a = max(â, a m ). The P ( x(0) > a) < E for all. Let B = {x : x(0) a}. We just showed P (B) 1 E. Similarly, for every k > 0 we ca fid δˆk ad k such that P (w x (δˆk) > E/2 k ) < E/2 k for all > k. For every fixed k we ca fid a small eough δ > 0 such that P (w x (δ ) > E/2 k ) < E/2 k sice by uiform cotiuity of x we have δ>0 {w x (δ) > E/2 k } = Ø a.s. Let δ k = mi(δˆk, mi k δ ). Let B k = {x : w x (δ k ) E}. Sice P (B c ) < E/2 k the P ( k Bc k k ) < E ad P ( k B k ) 1 E, for all. Therefore P (B k B k ) 1 2E for all. The set K = B k B k is closed (check) ad satisfies the coditios of Arzela-Ascoli Theorem. Therefore it is compact. 2 Fuctioal Strog Law of Large Numbers (FSLLN) We are about to establish two very importat limit results i the theory of stochastic processes. I probability theory two corerstoe theorems are (Weak or Strog) Law of Large Numbers ad Cetral Limit Theorem. These theorems have direct aalogue i the theory of stochastic processes as Fuctioal Strog Law of Large Numbers (FSLLN) ad Fuctioal Cetral Limit Theorem (FCLT) also kow as Dosker Theorem. The secod theorem cotais i it the fact that Wieer Measure exists. 2

We first describe the setup. Cosider a sequece of i.i.d. radom variables X 1, X 2,..., X,.... We assume that E[X 1 ] = 0, E[X 2 1 ] = σ 2. We ca view each realizatio of a ifiite sequece (X (ω)) as a sample i a product space R equipped with product type σ-field F ad probability measure P i.i.d., iduced by the probability distributio of X 1. Defie S = 1 X k. Fix a iterval [0, T ] ad for each 1 ad t [0, T ] cosider the followig fuctio S t (ω) X t +1(ω) N (t) = + (t t ). (1) This is a piece-wise liear cotiuous fuctio i C[0, T ]. Theorem 3 (Fuctioal Strog Law of Large Numbers (FSLLN)). Give a i.i.d. sequece (X ), 1 with E[X 1 ] = 0, E[ X 1 ] <, for every T > 0, the sequece of fuctios N : [0, T ] R coverges to zero almost surely. Namely P(IN (ω)i T 0) = P( sup N (t, ω) 0) = 1 0 t T As we see, just as SLLN, the FSLLN holds without ay assumptios o the variace of X 1, that is eve if σ =. Here is aother way to state FSLLN. We may cosider fuctios N defied o etire [0, ) usig the same defiig idetity (1). Recall that sets [0, T ] are compact i R. A equivalet way of sayig FSLLN is N coverges to zero almost surely uiformly o compact sets. Proof. Fix E > 0 ad T > 0. By SLLN we have that for almost all realizatios ω of a sequece X 1 (ω), X 2 (ω),..., there exists 0 (ω) such that for all > 0 (ω), S (ω) E < T We let M(ω) = max 1 m 0 (ω) S m (ω). We claim that for > M(ω)/E, there holds sup N (t) < E. 0 t T We cosider two cases. Suppose t [0, T ] is such that t > 0 (ω). The ( S t (ω) S t +1(ω) ) N (t) max,. 3

We have S t (ω) S t (ω) t ɛ = t T t ɛ. Usig a similar boud o S t +1 (ω), we obtai N (t) ɛ. Suppose ow t is such that t 0 (ω). The M(ω) N (t) < ɛ, sice, by our choice > M(ω)/ɛ. We coclude sup 0 t T N (t) < ɛ for all > M(ω)/ɛ. This cocludes the proof. 3 Weier measure FSSLN was a simpler fuctioal limit theorem. Here we cosider istead a Gaussia scalig of a radom walk S ad establish existece of the Weier measure (Browia motio) as well as FCLT. Thus suppose we have a sequece of i.i.d. radom variables X 1,..., X with mea zero but fiite variace σ 2 <. Istead of fuctio (1) cosider the followig fuctio S t (ω) X t +1 (ω) N (t, ω) = + (t t ) σ σ, 1, t [0, T ]. (2) This is agai a piece-wise liear cotiuous fuctio. The for each we obtai a mappig ψ : R C[0, T ]. Of course, for each, the mappig ψ depeds oly o the first T + 1 coordiates of samples i R. Lemma 1. Each mappig ψ is measurable. Proof. Here is where it helps to kow that Kolmogorov field is idetical to Borel field o C[0, T ], that is Theorem 1.4 from the previous lecture. Ideed, ow it suffices to show that that ψ 1 (A) is measurable for each set A of the form 4 4

A = π 1 t ψ 1 (π 1 t (, y], as these sets geerate Kolmogorov/Borel σ-field. Each set (, y]) is the set of all realizatios N (ω) such that 1 k N m X k(ω) (t, ω) = + (t m)xm+1 (ω) y. σ where m = t. This defies a measurable subset of R m+1 R. Oe way to see this is to observe that the fuctio f : R m+1 R defied by x f(x 1,..., x m ) = 1 k m k σ + (t m)x m+1 is cotiuous ad therefore is measurable. We coclude that ψ is measurable for each. Thus each ψ iduces a probability measure o C[0, T ], which we deote by P. This probability measure is defied by P (A) = P i.i.d. (ψ 1 (A)) = P i.i.d. (N (ω) A). We ow establish the pricipal result of this lecture existece of Weier measure, amely, the existece of a Browia motio. Theorem 4 (Existece of Wieer measure). A sequece of measures P has a weak limit P which satisfies the property of Wieer measure o C[0, T ]. The proof of this fact is quite ivolved ad we give oly its scheme, skippig some techical results. First let us outlie the mai steps i the proof. I the previous lecture we cosidered projectio mappigs P t : C[0, T ] R. Similarly, for ay collectio 0 t 1 < < t k we ca cosider π t1,...,t k (x) = (x(t 1 ),..., x(t k )) R k. 1. We first show that the sequece of measures P o C[0, T ] is tight. We use this to argue that there exists a subsequece P (k) which coverges to some measure π. 2. We show that π satisfies the properties of Wieer measures. For this purposes we look at the projected measures π t1,...,t k (π ) o R k ad show that these give a joit Gaussia distributio, the kid arisig i a Browia motio (that is the joit distributio of (B(t 1 ), B(t 2 ),..., B(t k ))). At this poit the existece of Wieer measure is established. 3. We the show that i fact the weak covergece π π holds. Proof sketch. We begi with the followig techical ad quite delicate result about radom walks. 5

Lemma 2. The followig idetity holds for radom walks S = X 1 + +X : lim lim sup λ 2 P(max S k λσ ) = 0. (3) λ Note, that this is ideed a very subtle result. We could try to use submartigale iequality, sice S k is sub-martigale. It will give E[S 2 ] 1 P(max S k λσ ) =. λ 2 σ 2 λ 2 So by takig a product with λ 2 we do ot obtai covergece to zero. O the other had ote that if the radom variables have a fiite fourth momet E[X 4 ] <, the the result follows from the sub-martigale iequality by co siderig S 4 i place of S 2 (exercise). The proof of this lemma is based o the followig fact: Propositio 2 (Etemadi s Iequality,[1]). For every α > 0 P(max S k 3α) 3 max P( S k α) Proof. Let B k be the evet S k 3α, S j < 3α, j < k. The P(max S k 3α) P( S α) + P(B k S < α) P( S α) + P(B k S S k > 2α) = P( S α) + P(B k )P( S S k > 2α) P( S α) + max P( S S k 2α) P( S α) + max (P( S α) + P( S k α)) 3 max P( S k α). Now we ca prove Lemma 2. Proof of Lemma 2. Applyig Etemadi s Iequality P(max S k λσ ) 3 max P( S k (1/3)λσ ) 6

Fix E > 0. Let Φ deote the cumulative stadard ormal distributio. Fid λ 0 large eough so that 2λ 2 (1 Φ(λ/3)) < E/3 for all λ λ 0. Fix ay such λ. By the CLT we ca fid 0 = 0 (λ) large eough so that P( S (1/3)λσ ) 2(1 Φ(λ/3)) + E/(3λ 2 ) for all 0, implyig λ 2 P( S (1/3)λσ ) 2E/3. Now fix ay 27 0 /E ad ay k. If k 0, the from the derived boud we have λ 2 P( S k (1/3)λσ ) λ 2 P( S k (1/3)λσ k) 2E/3. O the other had, if k 0 the λ 2 E[S 2 ] σ 2 k k λ 2 P( S k (1/3)λσ ) = E/3. (λ 2 /9)σ 2 (1/9)σ 2 We coclude that for all 27 0 /E, λ 2 max P( S k (1/3)λσ ) E/3, 1 from which we obtai λ 2 lim sup P(max S k λσ ) E Sice E > 0 was arbitrary, we obtai the result. The ext result which we also do ot prove says that the property (3) implies tightess of the sequece of measures P o C[0, T ]. Lemma 3. The followig covergece holds for every E > 0. lim lim sup P(w N (δ) E) = 0. (4) δ 0 As a result the sequece of measures P is tight. Proof. Observe that X k i k j w N (δ) max. i j T :j i δ σ Exercise 1. Use this to fiish the proof of the lemma. Hit: partitio iterval [0, T ] ito legth δ itervals ad use Lemma 2. 7

Let us ow see how Lemma 2 implies tightess. We use characterizatio give by Propositio 1. First for ay positive a, P ( x(0) a) = P( N (0) > 0) = 0 sice N (0) = 0. Now P (w x (δ) > ɛ) = P(w N (δ) > ɛ). From the first part of the lemma we kow that the double covergece (4) holds. This meas that coditio (ii) of Propositio 1 holds as well. We ow retur to the costructio of Wieer measure. Lemma 3 implies that the sequece of probability measures P o C[0, T ] is tight. Therefore, by Prohorov s Theorem, it cotais a weakly coverget subsequece P (k) P. Propositio 3. P satisfies the property of the Wieer measure. Proof. Sice P is defied o the space of cotiuous fuctios, the cotiuity of every sample is immediate. We eed to establish idepedece of icremets ad the fact that icremets are statioary Gaussia. Thus we fix 0 t 1 < < t k ad y 1,..., y k R k. To preserve the cotiuity, we still deote elemets of C[0, T ] by x, x(t) or x(ω, t) whereas before we used otatios ω, B(ω), B(t, ω). Cosider the radom vector π t1 (N ) = N (t 1 ). This is simply the radom variable S t 1 ( ω) X ω) t + (t 1 1 +1( t 1 ) σ σ The secod term i the sum coverges to zero i probability. The first term we rewrite as S (ω) t t 1 σ 1. t 1 ad by CLT it coverges to a ormal N(0, t 1 ) distributio. Similarly, cosider t X m(ω) 1<m t X t (ω) N (t 2) N (t 1) = 2 2 +1 + (t 2 t2 ) σ σ X t 1 +1(ω) (t1 t 1 ) σ Agai by CLT we see that it coverges to ormal N(0, t 2 t 1 ) distributio. Moreover, the joit distributio of (N (t 1 ), N (t 2 ) N (t 1 )) coverges to a joit distributio of two idepedet ormals with zero mea ad variaces t 1, t 2 t 1. Namely, the (N (t 1 ), N (t 2 )) coverges i distributio to 8

(Z 1, Z 1 + Z 2 ), where Z 1, Z 2 are idepedet ormal radom variables with zero mea ad variaces t 1, t 2 t 1. By a similar toke, we see that the distributio of the radom vector π t1,...,t k (N ) = (N (t 1 ), N (t 2 ),..., N (t k )) coverges i distributio to (Z 1, Z 1 + Z 2,..., Z 1 + + Z k ) where Z j, 1 j k are idepedet zero mea ormal radom variables with variaces t 1, t 2 t 1,..., t k t k 1. O the other had the distributio of π t1,...,t k (N ) is P π 1 t 1,...,t k = P i.i.d. ψ 1 πt 1 1,...,t k. Sice P (k) P ad π is cotiuous, the, applyig mappig theorem (Theorem 1) we coclude that P (k) πt 1 P π 1 1,...,t k t 1,...,t k. Combiig, these two facts, we coclude that the probability measure P πt 1 1,...,t k is the probability measure of (Z 1, Z 1 + Z 2,..., Z 1 + + Z k ) (where agai Z j are idepedet ormal, etc...). What does this mea? This meas that whe we select x C[0, T ] accordig to the probability measure P ad look at its projectio π t1,...,t k (x) = (x(t 1 ),..., x(t k )), the probability distributio of this radom vector is the distributio of (Z 1, Z 1 + Z 2,..., Z 1 + + Z k ). This meas that x has idepedet icremets with zero mea Gaussia distributio ad variaces t 1, t 2 t 1,..., t k t k 1. This is precisely the property we eeded to establish for P i order to argue that it is ideed Wieer measure. This cocludes the proof of Propositio 3 the fact that P is the Weier measure. We also eed to show that the covergece P P holds. For this purpose we will show that P is uique. I this case the covergece holds. Ideed, suppose otherwise, there exists a subsequece (k) such that P (k) P. The we ca fid a bouded cotiuous r.v. X, such that E P(k) X E P X. The we ca fid E 0 ad a subsequece (k i ) of (k) such that E P(ki ) X E P X E 0 for all i. By tightess we ca fid a further subsequece (k ij ) of (k i ) which coverges weakly to some limitig probability measure P. But we have see that every such weak limit has to be a Wieer measure which is uique. Namely P = P. This is a cotradictio sice E P X E P X E 0. (k ij ) It remais to show the uiqueess of Wieer measure. This follows agai from the fact that the Kolmogorov σ-field coicides with the Borel σ-field o C[0, T ]. But the properties of Wieer measure (idepedet icremets with variaces give by the legth of the time icremets) uiquely defie probability o geeratig sets obtaied via projectios π t1,...,t k : C[0, T ] R k. This cocludes the proof of uiqueess. 9

4 Applicatios Theorem 4 has applicatios beyod the existece of Wieer measure. Here is oe of them. Theorem 5. The followig covergece holds max 1 S k sup B(t) (5) σ 0 t T where B is the stadard Browia motio. As a result, for every y max 1 S k lim P( y) = 2(1 Φ(y)), (6) σ where Φ is stadard ormal distributio. Proof. The fuctio g(x) = sup 0 t T x(t) is a cotiuous fuctio o C[0, T ] (check this). Sice by Theorem 4, P P the, by Mappig Theorem, g(n ) g(b), where B is a stadard Browia motio radom sample max 1 S from the Wieer measure P. But g(n k ) = sup 0 t T N (t) =. σ We coclude that (5) holds. To prove the secod part we ote that the set A = {x C[0, T ] : sup 0 t T x(t) = y} has P (A) = 0 recall that the maximum sup 0 t T B(t) of a Browia motio has desity. Also ote that A is the boudary of the set {x : sup 0 t T x(t) y}. Therefore by Portmetau theorem, (6) holds. 5 Additioal readig materials Billigsley [1] Chapter 2, Sectio 8. Refereces [1] P. Billigsley, Covergece of probability measures, Wiley-Itersciece publicatio, 1999. 10

MIT OpeCourseWare http://ocw.mit.edu 15.070J / 6.265J Advaced Stochastic Processes Fall 2013 For iformatio about citig these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.