Notes on Probability. University of Chicago, Vivak Patel January 27, 2015

Transcription

1 Notes o Probability Uiversity of Chicago, 2014 Vivak Patel Jauary 27,

2 Cotets I Fudametal Results 4 1 Basic Measure Theory Special Structures o Sets Measures Radom Variables Measure-Theoretic Cosideratios Distributios Defiig the Itegral Properties of Itegrals Expectatio Product Measures ad Fubii s Theorem 22 4 Idepedece Measure-Theoretic Defiitio ad Properties Sufficiet Coditios for Idepedece Idepedet, Distributios ad Expectatio Sums of Idepedet Radom Variables II Laws of Large Numbers 27 5 Weak Laws of Large Numbers L 2 Weak Laws Triagular Arrays Trucatio Strog Laws of Large Numbers Borel-Catelli Lemmas: from Weak to Strog First Borel-Catelli Lemma Secod Borel-Catelli Lemma Etemadi s Proof of SLLN Radom Series Approach III Cetral Limit Theorems 41 7 DeMoivre-Laplace Theorem 41 8 Weak Covergece Defiitio ad Basic Results Sequetial Compactess of Distributio Fuctios Characteristic Fuctios Defiitio ad Properties Iversio Formula ad Weak Covergece

3 10 Cetral Limit Theorem Lidberg s Method 52 IV Coditioal Expectatio & Martigales Coditioal Expectatio Existece ad Uiqueess Basic Properties Regular Coditioal Distributios Martigales Defiitio ad Basic Properties Martigale Trasforms & Covergece Theorems Applicatios Uiform Itegrability Uiform Itegrability ad Martigale Covergece i L Optioal Stoppig Theorems Maximal Iequalities Backwards/Reverse Martigales V Caoical Processes Exchageable Sequeces Reewal Processes Reewal Processes ad Covergece Reewal Equatio

4 Part I Fudametal Results 1 Basic Measure Theory 1.1 Special Structures o Sets 1. π-system, semi-algebra, algebra, mootoe class, λ-system, ad σ-algebra. Defiitio 1.1. Let Ω be a o-empty set. (a) A π-system, P, is a collectio of subset of Ω that is closed uder fiite itersectios ad cotais the empty set. That is, if A, B P the A B P, ad P. (b) A semi-algebra, S, is a collectio of subsets of Ω which satisfy the followig: i. S is a π-system (cotais the empty set ad is closed uder fiite itersectios) ii. For S S, S c is the fiite, disjoit uio of elemets i S. (c) A algebra, A, is a collectio of subsets of Ω which satisfy: i. Ω A ii. A is closed uder complemetatio (i.e. if A A the A c A) iii. A is closed uder fiite uios (i.e. if A, B A the A B A) (d) A mootoe class, M, is a collectio of subsets of Ω which satisfy: i. Closure uder the uio of icreasig ested elemets of M. (i.e. if A i M such that A i A i+1 the i A i M). ii. Closure uder the itersectio of decreasig ested elemets of M. (i.e. if A i M such that A i A i+1 the i A i M). (e) A λ-system, L, is a collectio of subsets of Ω which satisfy: i. Ω L ii. L is closed uder complemetatio iii. L is closed uder coutable disjoit uios (f) A σ-algebra, F, is a collectio of subset of Ω which satisfy: 2. Examples i. Ω F ii. F is closed uder complemetatio iii. F is closed uder coutable uios Example 1.1. Let Ω = {1, 2, 3, 4}. (a) L = {, Ω, {1, 2}, {1, 3}, {1, 4}, {2, 3}, {2, 4}, {3, 4}} is a λ-system which is ot a algebra (b) P = {, {1, 3}}. The P is a π-system. Moreover, the σ-algebra it geerates is strictly cotaied i L. 4

5 (c) M = {{1}, {1, 3}} is a mootoe class which is ot a algebra. Example 1.2. Let S d be the empty set ad subsets of R d of the form: (a 1, b 1 ] (a d, b d ] a i < b i The S d is a semi-algebra but ot a algebra (oe of the complemets are i S d ). Example 1.3. Let Ω = Z ad A be the collectio of subsets of Ω which have fiite cardiality or which have complemets of fiite cardiality. The A is a algebra ad it is ot a σ-algebra sice we ca geerate 2Z by coutable uios, but this set is ot i A. Example 1.4. Let Ω = (0, 1] ad cosider all left-half ope itervals. The though coutable itersectios we ca geerate {1} but this is ot i the collectio of left-half ope itervals. Hece, this collectio is a algebra, but ot a σ-algebra. 3. Basic Properties (a) Itersectios preserve structure Lemma 1.1. Let F i, i I, be a arbitrary collectio σ-algebras or algebras. The their itersectio is also a σ-algebra or algebra (respectively). Proof. Check each coditio. (b) Covertig semi-algebras ito algebras Lemma 1.2. Let S be a semi-algebra. Let A be the collectio of all fiite disjoit uios of sets i S. The A is a algebra. Proof. Sice S ad its complemet, Ω, ca be expressed as the fiite disjoit uio of elemets i S, Ω A. Let A, B A. The there are S 1,..., S m, T 1,..., T S such that A = + m i=1 S i ad B = + j=1 T j. The A B = + m, i=1,j=1 S i T j. S i T j S ad these are all disjoit. Hece, A B A. Fially, A c = m i=1 Sc i. Each Sc i ca be expressed as fiitely may disjoit uios elemets of S. Hece, A c ca be expressed as the fiite disjoit uios of elemets of S. Therefore, A c A. We ca coclude that A is a algebra. (c) The itersectio of mootoe classes ad algebras is σ-algebras. Lemma 1.3. Let F be a mootoe class ad a algebra. The F is a σ-algebra. Proof. Ω F ad if A F the A c F, which follow directly from F beig a algebra. Let A 1,... F. Let B 1 = A 1 ad B = i=1 A i. The B F sice F is a algebra ad B B +1. Sice F is a mootoe class, i A i = i B i F. Hece, F is a σ-algebra. (d) The itersectio of π-systems ad λ-systems is σ-algebras. 5

6 Lemma 1.4. Let F be a π-system ad a λ-system. The F is a σ-algebra. Proof. Ω F ad if A F the A c F follow from F beig a λ-system. Let A 1,... F. Let B 1 F. Let B 1 = A 1 ad B = A A c 1 A c 1 Hece, all B are disjoit ad B F sice it is a π-system. Ad so i A i = i B i F. (e) A λ-system is closed uder proper set differeces Lemma 1.5. Let L be a λ-system ad A, B L such that A B (strict). The B A L. Proof. We wat to show that A c B L. Notice that B c = A c B c, = A B c, A = A B, ad A c B are disjoit. Sice B c, A L, B c A L. Hece, B A c = (B c A) c L. 4. Halmos Mootoe Class Theorem Theorem 1.1. Let A be a algebra ad M be a mootoe class such that A M. The σ(a) M. Proof. Defie m(a) to be the smallest mootoe class cotaiig A. To show that m(a) is closed uder complemetatio, defie m(a) = {A : A c m(a)}. Show that m(a) is a mootoe class cotaiig A. This implies that for ay A m(a) the A c m(a). Now defie M 1 = {A : A B m(a), B m(a)} This is a mootoe class, ad if we ca show that A M 1 the the result follows. So first we cosider the more geeral case of showig A M 2 which is: M 2 = {A : A B m(a), B A} Clearly, A M 2, which is a mootoe class. Hece, m(a) M 2. Now, if A A ad B m(a) M 2 the A B m(a). Hece, A M Dyki s π-λ Theorem Theorem 1.2. Let P be a π-system cotaied i L a λ-system. The σ(p) L. Proof. Let L 0 be the smallest λ-system cotaiig P. We show that L 0 is a π-system. We follow the same strategy as i the Mootoe Class Theorem. Defie: L 1 = {A : A B L 0, B L 0 } 6

7 If A L 1 the A B L 0 for ay B L 0 ad by Lemma 1.5, A c B L 0. This implies that A c L 0. The property of coutable disjoit uios follows from L 0 beig a λ-system, hece L 1 is a λ-system. We wat to show that P L 1. To do this, we cosider the more geeral case: L 2 = {A : A B L 0, B P} Agai, L 2 is a λ-system. Moreover, it cotais P. Hece, L 0 L 2. Now let P P ad A L 0 L 2 the P A L 0 for ay A L 0. Hece, P L 1. Thus, L 0 L 1, so it is a π-system. It follows that L 0 is a σ-algebra. So σ(p) L Geerated Algebras ad σ-algebras (a) Geerated Algebra. Geerated σ-algebra. Defiitio 1.2. Let S be a collectio of subsets of a o-empty set Ω. i. The smallest algebra cotaiig S, a(s), is the algebra geerated by S. ii. The smallest σ-algebra cotaiig S, σ(s), is the σ-algebra geerated by S (b) Existece of a geerated algebra. Lemma 1.6. Let S be a collectio of subsets of Ω. The there exists a smallest algebra geerated by S. Proof. Let M be the collectio of all algebras cotaiig S. This is o-empty sice the power set of Ω is a algebra cotaiig S. Takig the itersectio over all M results i a(s). (c) Existece of a geerated σ-algebra Lemma 1.7. Let S be a collectio of subsets of Ω. The there exists a smallest σ-algebra geerated by S. Proof. Let M be the collectio of all σ-algebras cotaiig S. This is o-empty sice the power set of Ω is a σ-algebra cotaiig S. Takig the itersectio over all M results i σ(s). 7. Borel Sets Defiitio 1.3. The Borel Sets are the σ-algebra geerated by the ope sets of a topology. Specifically, the Borel Sets usually refer to th ope sets of R d ad, depedig o the dimesio, are deoted B d. 8. Measurable Space Defiitio 1.4. A Measurable Space is the double of a o-void set ad a σ-algebra o that set. 7

8 1.2 Measures 1. σ-fiite. Premeasure. Measure. Probability Measure. Measure Space. Probability Space. Defiitio 1.5. Let Ω be a o-void set. (a) A set fuctio µ o subsets of Ω is σ-fiite if o subsets A which satisfy A = Ω also satisfy µ(a ) <. (b) Let A be a algebra o Ω. A pre-measure, µ, is a set fuctio o A that satisfies: i. µ( ) = 0 ii. µ(a) 0 for ay A A iii. If A i A are disjoit AND i A i A the ( ) µ A i = µ(a i ) i i (c) Let F be a σ-algebra o Ω. A (positive) measure is a set fuctio, µ, o F which is o-egative (possibly ifiite) ad is coutable additive. A probability measure is a special case of a measure i which µ(ω) = 1. (d) A measure space or probability space is the triple of a o-void set, a σ-algebra, ad the measure or probability measure (resp.). 2. Caratheodory Extesio Theorem Theorem 1.3. Let Ω be a o-empty set. (a) Let S be a semi-algebra o Ω ad µ be a o-egative set fuctio o S such that µ( ) = 0. Suppose i. Wheever S S such that S = + m i=1 S i where S i S the µ is fiitely additive. ii. Wheever S S such that S = + i=1 S i where S i S the µ is coutably subadditive. The µ has a uique extesio (agrees o S) to µ, a premeasure o a(s). (b) If µ is a pre-measure o a algebra, A, o Ω, the it has a extesio (agrees o A) to σ(a). (c) Moreover, if µ is σ-fiite the the extesio is uique. 3. Costructio of Measures o the Real Lie Example 1.5. First we costruct a Steltjes Measure Fuctio, F, which satisfies: (a) F is o-decreasig (b) F is right-cotiuous 8

9 Next, o the semi-algebra of itervals of the form (a, b], we defie the o-egative set fuctio µ such that µ( ) = 0 ad: µ(a, b] = F (b) F (a) a < b Fially, by the extesio theorem, we have a uique measure o (R, B). 4. Basic Properties Propositio 1.1. Let (Ω, F, µ) be a measure space. (a) µ is mootoic. (b) µ is subadditive (c) O A, A i F such that A i A, µ is cotiuous from below. (d) O A, A i F such that µ(a 1 ) < ad A i A, µ is cotiuous from above. Proof. (a) Suppose A, B F ad A B. The C = A C B F. Hece: µ(a) µ(a) + µ(c) = µ(b) (b) Let A i F. Let B 1 = A 1 ad B i = A i B i 1. The, by coutable additivity: µ( i A i ) = µ( i B i ) = i µ(b i ) i µ(a i ) (c) Let B 1 = A 1 ad B = A A 1. The: µ(a) = µ(b i ) = lim i=1 i=1 µ(b i ) = lim µ(a ) (d) We have A 1 A i A 1 A. Hece, µ(a 1 A i ) µ(a 1 A). Sice µ(a 1 ) <, the followig holds: µ(a 1 ) µ(a i ) µ(a 1 ) µ(a) Which implies: µ(a i ) µ(a) 2 Radom Variables 2.1 Measure-Theoretic Cosideratios 1. Measurable Fuctio. Radom Vector. Radom Variable. σ-algebra Geerated by a Radom Vector Defiitio 2.1. Let (Ω, F) ad (S, S) be measurable spaces. 9

10 (a) A fuctio f : Ω S is measurable if the pre-images of measurable sets are measurable. That is for ay B S, f 1 (B) = {ω : f(ω) B} F. (b) Suppose (S, S) = (R d, B d ). The a measurable fuctio is called a radom vector. If d = 1 the it is called a radom variable. (c) The σ-algebra geerated by a radom vector is the smallest σ-algebra o the domai which makes the radom vector measurable. 2. Checkig Measurability of a a Fuctio Propositio 2.1. Suppose A geerated S. Moreover, suppose A A, f 1 (A) F. The f is measurable. Proof. We eed to show that B S, f 1 (B) F. cosider: B := {B S : f 1 (B) F} To do this, we B is a σ-algebra. Sice A B the S B. Therefore, f is measurable. 3. Useful applicatios of checkig measurability Example 2.1. Suppose X : Ω R. We ca check that X 1 [, q) for all q Q is measurable. Example 2.2. If the co-domai is R d, we ca simply check over all ope rectagles with ratioal ed-poits. 4. Existece of geerated σ-algebra Lemma 2.1. Let f be a fuctio from Ω to (S, S). The, there is a σ(f), the smallest σ-algebra o Ω, which makes f measurable. Proof. Cosider the collectio G := {{ω : f(ω) B} : B S}. We show that G is a σ-algebra by checkig all of the coditios. (a) Ω G sice f 1 (S) = Ω. (b) If {f B} G, the {f B} C = {f B c } G. (c) If f 1 (B i ) G, the i f 1 (B i ) = f 1 ( i B i ) G. Thus, the set of σ-algebras makig f measurable is o-empty. Hece, there is a smallest σ-algebra makig f measurable (through arbitrary itersectios). 5. Compositios of Measurable Fuctios Propositio 2.2. Let (Ω, F) be a measurable space. (a) Let (S, S), (T, T ) be measurable spaces. Suppose f, g are measurable fuctios such that f : Ω S ad g : S T. The g f is measurable. 10

11 (b) Let X 1,..., X be radom variables ad f : R R be a Borel fuctio. The f(x 1,..., X ) is a radom variable. (c) If X 1,..., X are radom variables the X X is a radom variable. Proof. (a) Let B T. The g 1 (B) S ad so f 1 g 1 (B) F. (b) We eed to show that (X 1,..., X ) is measurable sice composig this with F does ot cause ay difficulties. Moreover, this requires that we kow the product σ-algebra B is geerated by crossproducts of Borel sets. Hece, by Prop 2.1, we eed to check measurable rectagles: {ω : (X 1,..., X ) A 1 A } F Sice X i are radom variables, ad this set is equal to i X 1 i (A i ), it is i F. (c) By the previous two poits, we eed oly check that f(x 1,..., x ) = x x is a measurable fuctio. f is cotiuous ad hece the pre-images of ope sets are ope sets. Thus, by Prop 2.1, f is Borel measurable. 6. Limits, Ifimums ad Supremums of Measurable Fuctios Propositio 2.3. If X 1, X 2,... are radom variables the so are: if X sup X lim sup X lim if X Proof. (a) Let q Q. {if X < q} = {X < q}. The right had side is a measurable set. By Prop 2.1, the ifimum is measurable. (b) Let q Q. {sup X < q} = {X < q}. The right had side is a measurable set. By Prop 2.1, the supremum is measurable. (c) We have that for each k N, sup >k X is a radom variable. Takig the ifimum over k of these radom variables is measurable ad is also the limit supremum. (d) We have that for each k N, if >k X is a radom variable. Takig the supremum over k of these radom variables is measurable ad is also the limit ifimum. 11

12 2.2 Distributios 1. Let (Ω, F, P) be a Probability space. Let X be a radom variable o the probability space. 2. Distributio of X. Distributio Fuctio of X. Defiitio 2.2. The set fuctio µ : B [0, 1] defied by µ(b) = P [X B] is the distributio of X. The Distributio Fuctio of X is the fuctio F (x) = P [X x] = µ([, x]). 3. Distributios are probability measures Lemma 2.2. The distributio of X is a probability measure o (R, B). Proof. Let µ be the distributio of X. (a) µ( ) = P [X ] = P [ ] = 0 (b) Let B 1, B 2,... B be disjoit. The {X B 1 }, {X B 2 },... are disjoit. Hece: ( ) [ µ B i = P X ] B i = P [X B i ] = µ(b i ) i i i i 4. Properties of Distributio Fuctios Lemma 2.3. Let F be a distributio fuctio of X. followig properties. The F has the (a) F is o-decreasig (b) lim x F (x) = 1 ad lim x F (x) = 0. (c) F is right cotiuous (d) Deote F (x ) = lim y x F (y). The F (x ) = P [X < x] (e) P [X = x] = F (x) F (x ) Proof. (a) This follows from the mootoicty of measures. (b) Note that (, ] = ad (, ] = R. (c) Let y x. The (, y ] = (, x]. (d) Let y x. The (, y ] = (, x). (e) P [X = x] = P [X x] P [X < x] = F (x) F (x ). 5. Characterizatio of a Distributio Fuctio. Iverse of a Distributio Fuctio (i proof). 12

13 Lemma 2.4. If F is o-decreasig, has limits 0 ad 1, ad is right cotiuous the there is a radom variable for which F is its distributio fuctio. Proof. Cosider the iterval (0, 1) with the usual σ-algebra ad the Lebesgue measure. Defie radom variable X for ω (0, 1) by: X(ω) = sup y : F (y) < ω We eed to show {X x} = {ω F (x)}. (a) If X(ω) x the F (x) ω. Hece {X x} {ω F (x)} (b) If X(ω) > x the F (x) < ω. Hece, {X x} c {ω F (x)} c. Hece, µx x = µω F (x) = F (x). 2.3 Defiig the Itegral 1. First, we defie the itegral for simple fuctios, bouded fuctios, oegative fuctios ad fially all measurable fuctios with respect to a measure space. The, we prove the followig results for each set of fuctios:s Lemma 2.5. Let f, g be measurable fuctios w.r.t (Ω, F, µ). (a) (Positivity) If f 0 the fdµ 0 (b) (Scalar Multiplicatio) If a R the afdµ = a fdµ (c) (Liearity) (f + g)dµ = fdµ + gdµ (d) (Mootoicity) If f g a.e. the fdµ gdµ (e) (a.e-equal) If f = g a.e. the fdµ = gdµ (f) fdµ f dµ 2. The last three items ca be prove from the first three give some extra assumptios: Lemma 2.6. Let f, g be measurable fuctios w.r.t. (Ω, F, µ). (a) If (a) ad (c) from Lemma 2.5 hold the (d) holds. (b) If (a) ad (c) from Lemma 2.5 hold the (e) holds. (c) If (a) ad (c) from Lemma 2.5 hold ad (b) holds for a = 1 the (f) holds. Proof. (a) If (a) ad (c) hold the sice f g 0 a.e. the (f g)dµ 0. Applyig (c) we get the result. (b) By the first part of this lemma, the iequality i both directios implies the equality of the itegrals. 13

14 (c) f f a.e. ad f f. By (b) we have that fdµ f dµ ad fdµ f dµ. 3. Simple Fuctios (a) Simple Fuctios. Itegral of Simple Fuctio. Defiitio 2.3. Let (Ω, F, µ) be a measure space. i. φ is a simple fuctio if φ ca be writte as φ(ω) = i=1 α i1 {A i } where <, A i F are disjoit ad µ(a i ) <. ii. The itegral of a simple fuctio is defied as: (b) Proof of Lemma 2.5 Proof. φdµ = α i µ(a i ) i. If φ 0 the α i 0. Sice µ(a i ) 0, the i=1 α iµ(a i ) 0. ii. Let a R. The: aφdµ = aα i µ(a i ) = a α i µ(a i ) i=1 iii. If φ = i=1 α i1 {A i } ad ψ = m j=1 β j1 {B j } the {A i B j } are disjoit. Therefore φ + ψ =,m i,j (α i + β j )1 {A i B j }. Hece by coutable additivity of µ: i=1 j=1 i=1 m (α i + β j )µ(a i B j ) = i=1 i=1 α i µ(a i ) + j=1 β j µ(b j ) 4. Bouded Fuctios (a) Bouded fuctios with Fiite Measure Support (BFMS). Itegral of Bouded Fuctio. Defiitio 2.4. Let (Ω, F, µ) be a measure space. i. Let f be measurable. f is bouded with fiite-measure support if f is o-zero oly o a set E F such that µ(e) <. ii. Let S be the collectio of all simple fuctios. The itegral of a BFMS fuctio is: { fdµ = sup (b) Equivalet characterizatio of itegral } ψdµ : ψ S, ψ f a.e. 14

15 Lemma 2.7. Let f be a BFMS fuctio. The: { } { } sup ψdµ : ψ S, ψ f a.e. = if φdµ : φ S, φ f a.e. Proof. By assumptio, M < such that f M. Ad let E be the support of f. The defie: { E k, = ω E : k M f(ω) < k + 1 } M ad let ψ = 1 k=0 k M1 {E k,} ad φ = 1 k=0 k+1 M1 {E k,}. The: ψ f φ for all. Moreover: φ ψ = φ ψ = M µ(e) 0 Hece, the result holds. (c) Proof of Lemma 2.5 Proof. i. If f 0, 0 is a simple fuctio hece by the same property for simple fuctios fdµ 0dµ = 0. ii. Let a R. The for a 0: afdµ = sup φ af φ = sup φ f aφ = a fdµ If a < 0 the: afdµ = if φ f aφ = a fdµ iii. Let φ 1 f ad φ 2 g the φ 1 + φ 2 f + g, the: (f + g)dµ φ 1 + Sice this holds for ay φ 1 f ad φ 2 g we have that: (f + g)dµ fdµ + gdµ For the other directio, let ψ 1 f ad ψ 2 g. The: (f + g)dµ ψ 1 + ψ 2 Hece, fdµ + gdµ (f + g)dµ. φ 2 15

16 5. No-egative fuctios. (a) Itegral of o-egative fuctio. Defiitio 2.5. Let (Ω, F, µ) be a measure space. Let f 0 be a measurable fuctio. Let M be the set of all BFMS fuctios. The the itegral of a o-egative fuctio f is defied as: { } fdµ = sup gdµ : g f, g M (b) Particular costructio covergig to the itegral of f 0. Lemma 2.8. Suppose µ is σ-fiite. Let E Ω be such that µ(e ) <. The: f dµ fdµ E Proof. First ote that f 1 {E } f ad is i M. Hece, E f dµ fdµ. To get the iequality i the other directio, let 0 g f, g M, ad g M for some M > 0. The for M: f dµ gdµ = gdµ gdµ E E Moreover as : gdµ Mµ(Ec supp(g)) 0 Hece: E c lim if f dµ E gdµ We ca do this for every fuctio g M such that g f, ad so the result holds. (c) Proof of Lemma 2.5 Proof. i. 0 is a bouded fuctio which is less tha or equal to f ad it has itegral of 0. ii. If a > 0 the from the lemma: af = lim E af = a lim E c E f = a iii. We have that: (f + g) E f + E g E f 16

17 Hece, (f + g) f + g. For the other directio, let h 1, h 2 M such that h 1 f ad h 2 g. The h 1 + h 2 M ad h 1 + h 2 f + g. Hece: h 1 + h 2 (f + g) Sice this holds for all h 1 f ad h 2 g, the the other directio follows. 6. Itegral of a Measurable Fuctios (a) Positive part. Negative part. Defiitio 2.6. Let f be a fuctio. Its positive part is f + = f 0 ad its egative part is f = ( f) 0. (b) Itegrable. Itegral of a measurable fuctio. Defiitio 2.7. A fuctio f is itegrable if f dµ <. If f is itegrable, the itegral of a measurable fuctio is defied as: fdµ = f + dµ f dµ (c) Miimality of Positive ad Negative Parts ad Itegrability. Lemma 2.9. Suppose f 1, f 2 0 such that f = f 1 f 2. The f 1 f + ad f 2 f ad fdµ = f 1 dµ f 2 dµ Proof. To prove the first part. For ay poit ω Ω, we have a few cases: i. f(ω) = 0 the f + (ω) = f (ω) = 0. The, f 1 (ω) f + (ω) ad f 2 (ω) f (ω). ii. f(ω) > 0 the f + (ω) = f(ω) ad f (ω) = 0. For a cotradictio, if f 1 (ω) < f(ω) the f 1 (ω) f 2 (ω) > f 1 (ω) = 0 > f 2 (ω). Hece, f 1 (ω) f + (ω) ad f 2 (ω) f (ω). iii. This holds for the egative case as well. For the secod part, sice we have oly proved Lemma 2.5(b) for a > 0: f + + f 2 = f + f 1 = f + + f 2 = f + f 1 = f + f = f 1 f 2 = f = f 1 f 2 17

18 (d) Proof of Lemma 2.5. Proof. i. This is the o-egative case. ii. Suppose a 0, the this follows from the o-egative case. If a < 0 the (af) + = ( a)f ad (af) = ( a)f +. Hece: af = ( a)f ( a)f + = a f + a f + = a f iii. Note that (f + g) + (f + g) = f + + g + (f + g ). By the previous lemma: (f + g) = f + + g + f g = f + g 2.4 Properties of Itegrals 1. Jese s Iequality Theorem 2.1. Suppose φ is covex. If µ is a probability measure ad f, φ(f) are both itegrable, the: ( ) φ fdµ φ fdµ Proof. Note that there is always at least oe lie passig through ay poit of φ which remais below φ. We use show this ad prove the result based by cosiderig the poit c = fdµ. By covexity: φ(c) φ(x) c x φ(y) φ(c) y c for x < c ad y > c. So the icreasig limit o the left exists ad the decreasig limit o the right exists. Let m be ay value betwee these two limits. The: m(y c) + φ(c) φ(y) ad m(x c) + φ(c) φ(x) Hece, for ay x, we foud a lie below φ passig through φ(c). Lettig f = x, ad itegratig: φ fdµ φ(c)dµ + m (f c)dµ ( ) φ(c) + m fdµ c ( ) φ fdµ 18

19 2. Holder s Iequality. Theorem 2.2. If (p, q) are cojugate expoets the fg L 1 f L p g L q Proof. If the either term o the right had side is ifiite, we are doe. If either term o the right had side if 0, this implies that the correspodig fuctio is 0 almost everywhere ad so the left had side is 0. Suppose both terms o the right are o-zero ad fiite. Let F ad G be f ad g ormalized by the p ad q orms respectively. The by the covexity of the expoetial fuctio: ( 1 F G = exp q log(f q ) + 1 ) p log(gp ) 1 q F q + 1 p Gp Itegratig both sides results i F G Bouded Covergece Theorem Theorem 2.3. Suppose f are BFMS with respect to a set E ad a boud M > 0 such that f f a.e.. The: fdµ = lim f dµ Proof. Let ɛ > 0. Let B = { f f > ɛ}. The: f f dµ = Ω f f dµ + E B f f dµ B ɛµ(e) + 2Mµ(B ) As, µ(b ) 0 ad sice ɛ is arbitrary, we have covergece i L 1 ad we thus have the result. 4. Fatou s Lemma Theorem 2.4. Suppose f 0 the lim if f dµ lim if f dµ. Proof. Let g = if k f k ad let g = lim g. Now we use σ- fiiteess, trucatio ad Lemma 2.8 to coclude: (a) Let E m Ω such that µ(e M ) < (b) The, by bouded covergece theorem, g mdµ = lim E m g mdµ E m g dµ (c) By the referece lemma, we have that: lim g mdµ = m E m gdµ 19

20 5. Mootoe Covergece Theorem Theorem 2.5. If f 0 ad f f a.e. the f dµ fdµ. Proof. By mootoicity, f dµ f +1 fdµ for all 1. By Fatou s lemma, fdµ lim if f dµ. The iequalities together imply the result. 6. Domiated Covergece Theorem Theorem 2.6. If f f a.e., f g a.e. for all ad g is itegrable the: f dµ fdµ ad the f f i L 1. Proof. By assumptio, 0 2g f f 2g. Usig Fatou s lemma: 2g lim if 2g f f 2g lim sup f f dµ Therefore, lim sup f f dµ = 0. So we have L 1 covergece which implies the result. 2.5 Expectatio 1. Expected Value. Existece of Expectatio. Defiitio 2.8. Let X be a radom variable o (Ω, F, P). (a) The Expected Value of X is E [X] = XdP (b) The expected value of X exists if either E [X + ] < ad/or E [X ] <. 2. Chebyshev s Geeral Iequality. Lemma Suppose φ : R R 0. Let A B ad i A = if{φ(y) : y A}. Let X be a radom variable: i A P [X A] E [φ(x)1 {X A}] E [φ(x)] Proof. By defiitio, i A 1 {X A} φ(x)1 {X A} φ(x). Take expectatios. 20

21 3. A importat limit theorem Theorem 2.7. Suppose X X a.s.. Let g, h be cotiuous fuctios such that: (a) g 0 ad g(x) as x (b) h(x) /g(x) 0 as x (c) E [g(x )] K < for all. The E [h(x )] E [h(x)] Proof. Let M > 0 such that P [ X = M] = 0 ad for x > M, h(x) < ɛg(x). (a) Trucate. Defie Ȳ = Y 1 { Y M}. The by bouded covergece theorem, E [ h( X ) ] E [ h( X) ]. Let G = { X > M} ad G = { X > M}. (b) Estimatio. The for sufficietly large : E [h(x)] E [h(x )] E [ h(x) h( X) ] + E [ h(x ) h( X ) ] + E [ h( X ) ] E [ h( X) ] h(x) + h(x ) + ɛ G ɛg(x) + G G ɛg(x ) + ɛ G 2ɛK + ɛ 4. A importat example. Example 2.3. The previous theorem ca be used whe g(x) = x p ad h(x) = x. This implies that if the L p orms are uiformly bouded, ad X X a.s. the E [X ] E [X]. 5. Computig Expectatio - Chage of Variables. Theorem 2.8. Let X be a radom variable from (Ω, F, P) ito (S, S). Let µ be its distributio. Let f : (S, S) (R, B) such that f 0 or E [ f(x) ] <. The: E [f(x)] = f(y)µ(dy) S Proof. Let A S. The: E [1 {X A}] = P [X A] = µ(a) = S 1 {s A} dµ By liearity, this exteds to simple fuctios. By mootoe covergece, this exteds to all o-egative fuctios f. Ad give the itegrability assumptios, we ca use the positive ad egative parts to fiish the coclusio. 21

22 3 Product Measures ad Fubii s Theorem 1. Rectagles. Product σ-algebra. Defiitio 3.1. Let (X, X, µ) ad (Y, Y, ν) be σ-fiite measure spaces. Let Ω = X Y ad S = {A B X Y}. The elemets of the (semialgebra) S are called rectagles ad F = σ(s) is the Product σ-algebra. 2. Existece ad Uiqueess of Product Measure Theorem 3.1. There is a uique measure π o (Ω, F) such that π(a B) = µ(a)ν(b). Proof. This is a applicatio of the Caratheodory Extesio theorem. To satisfy the hypotheses of this result, we must show that π is a pre-measure o (Ω, S). Let π(a B) = µ(a)ν(b) ad suppose: A B = i N (A i B i ) Let I(x) = {i : x A i } ad ote that sice these are rectagles that B = i I(x) B i. The, by mootoe covergece theorem: E [1 {A} (x)ν(b)] = E 1 {A i } (x)ν(b i ) i I(x) [ ] = E 1 {A i } (x)ν(b i ) i N = i N E [1 {A i } (x)] ν(b i ) = i = i µ(a i )ν(b i ) π(a i B i ) Hece, by the extesio theorem a uique measure exists o the algebra geerated by S. Sice both µ ad ν are σ-fiite, the this measure exteds uiquely to F. 3. Fubii s Theorem Theorem 3.2. Let (Ω, F, π) be the product measure space by σ-fiite measure spaces with measure µ ad ν. If f 0 or f dπ < the: Ω f(x, y)dν(y)dµ(x) = f(z)dπ(z) = f(x, y)dµ(x)dν(y) X Y Ω Y X Proof. 22

23 (a) Let E Ω ad defie E x = (Y {x}) E. If E F the E x Y Clearly, this is satisfied for rectagles. Let E = {E : E x Y}. The, this is satisfied for complimets ad coutable uios. Hece, F E. (b) If E F the g(x) = ν(e x ) is X measurable ad g(x)dµ = π(e). X Let L be the collectio for which this holds, ad we ca assume that µ(x) or ν(y ) are fiite ad use σ-fiiteess to get the result. We show that L is a λ-system. i. If E = Ω the E x = Y. Hece: ν(y )dµ(x) = ν(y )µ(x) = π(ω) X ii. If E L the: π(e c ) = X (ν(y ) ν(e x )) dµ(x) = X ν(e c x)dµ(x) iii. If E i L are disjoit, by mootoe covergece: π( i E i ) = π(e i ) = ν (( i E i ) x ) dµ(x) i X Sice S L, F L by π λ theorem. (c) Use stadard machiery to build up to simple fuctios, positive fuctios ad the itegrable fuctios. 4 Idepedece 4.1 Measure-Theoretic Defiitio ad Properties 1. Idepedet σ-algebras, idepedet radom variables, ad idepedet evets. Defiitio 4.1. (a) Let F 1,..., F be σ-algebras. These are idepedet σ-algebras if for ay A i F i ad I {1,..., }: P [ i I A i ] = i I P [A i ] (b) X 1,..., X are idepedet radom variables wheever σ(x 1 ),..., σ(x ) are idepedet σ-algebras. (c) A 1,..., A are idepedet evets wheever σ(a 1 ),..., σ(a ) are idepedet. 2. Equivalet Defiitio Lemma 4.1. I the defiitio of idepedet σ-algebras, the coditio that I {1,..., } is equivalet to I = {1,..., }. 23

24 Proof. Clearly, I = {1,..., } requires I {1,..., }. Now suppose the defiitio requires I = {1,..., }. Choose I {1,..., }. For j {1,..., } \ I, let A j = Ω. The result follows. 4.2 Sufficiet Coditios for Idepedece 1. Idepedet π-systems Lemma 4.2. Let P 1,..., P be idepedet π-systems. The, the σ- algebras they geerate are idepedet. Proof. Let F P 2... P. Let L = {A : P [A F ] = P [A] P [F ], F {. Note that P 1 L. Now, we show that L is a λ-system. (a) Ω L (b) If A L. The P [A c F ] = P [F ] P [A F ] = P [A c ] P [F ] (c) If A i L are disjoit, the [( ) ] P A i F = i i [ ] P [A i ] P [F ] = P [F ] P A i By π λ theorem, we have that σ(p 1 ), P 2,..., P are idepedet. Now we simply repeat the proof for each i with F σ(p 1 ) P i 1 P i+1 P i 2. Fuctios of Idepedet Radom Variables Theorem 4.1. Let F i,j for 1 i ad 1 j m(i) be idepedet. Let G i = σ( j F i,j ). The G 1,..., G are idepedet. Proof. Let S i be the rectagles geerated by F i,j for 1 j m(i). These are π-systems ad are idepedet. By the previous lemma, the σ-algebras they geerate are idepedet. Corollary 4.1. Let X i,j be idepedet radom variables for 1 i ad 1 j m(i). Let f i be measurable fuctios o R m(i). The f i (X i,1,..., X m(i),i ) are idepedet. 4.3 Idepedet, Distributios ad Expectatio 1. Idepedece ad Distributios Theorem 4.2. Let X 1,..., X be idepedet radom variables such that X i µ i. The: (X 1,..., X ) µ 1 µ 24

25 Proof. Let A = A 1 A B. The: P [(X 1,..., X ) A] = P [X 1 A 1,..., X A ] = P [X i A i ] = i=1 µ i (A i ) i=1 = µ 1 µ (A) Hece, this holds for the π-system geerated by the rectagles. By the Extesio Theorem, sice the product measure is uique, P = µ 1 µ. 2. Idepedece ad Itegratio Theorem 4.3. Suppose X, Y are idepedet with distributios µ ad ν. (a) If h : R 2 R is measurable such that h 0 ad E [ h(x, Y ) ] < the: E [h(x, Y )] = h(x, y)µ(dx)ν(dy) (b) I particular, if f, g : R R are measurable such that either f, g 0 or f, g are itegrable the: E [f(x)g(y )] = E [f(x)] E [g(y )] Proof. By chage of variables formula, idepedece, ad Fubii: E [h(x, Y )] = h(x, y)d(µ ν)((x, y)) = h(x, y)dµ(x)dν(y) Ω If h(x, y) = f(x)g(y) the: E [h(x, Y )] = h(x, y)dµ(x)dν(y) = The result follows. g(y) f(x)dµ(x)dν(y) 4.4 Sums of Idepedet Radom Variables 1. Distributio Fuctio of Sums Theorem 4.4. If X, Y are idepedet with distributio fuctios F ad G respectively, the: P [X + Y z] = F (z y)dg(y) y R 25

26 Proof. Let µ ad ν be probability measures such that X µ ad Y ν. The by chage of variables, idepedece ad Fubii s theorem: P [X + Y z] = dp X+Y z = d(µ ν)((x, y)) = = x+y z z y y R y R dµ(x)dν(y) F (z y)dg(y) 2. Simple chage of measure Lemma 4.3. If µ is a probability measure such that for some Lebesgue measurable fuctio f, µ(a) = f(x)dx the for ay measurable fuctio g o-egative or itegrable: A gdµ = g(x)f(x)dx Proof. Let A be a measurable set. The: 1 {A} dµ = µ(a) = 1 {A} f(x)dx Usig liear combiatios, we have simple fuctios. The usig mootoe covergece, we will have o-egative fuctios. Usig positive ad egative parts will give us itegrable fuctios. 3. Desity of Sums Theorem 4.5. Suppose that X has desity f ad Y has distributio fuctio G ad both are idepedet. (a) X + Y has desity h(x) = f(x y)dg(y) (b) If Y admits a desity the X +Y has desity h(x) = f(x y)g(y)dy Proof. We have from Fubii: x H(x) = F (x y)dg(y) = f(z y)dzdg(y) = x f(z y)dg(y)dz Hece, X + Y admits a desity, which is give by f(z y)dg(y). Secodly, if G admits a derivative, the we ca just apply the previous lemma. 26

27 Part II Laws of Large Numbers 5 Weak Laws of Large Numbers 5.1 L 2 Weak Laws 1. Ucorrelated Radom Variables Lemma 5.1. Let X 1,..., X have E [ X i 2] < ad be ucorrelated. The V [ i X i] = i V [X i]. Proof. Assume W.L.O.G. that E [X i ] = 0. The: [ ] V X i = V [X i ] + 2 E [X i X j ] = i i i<j i V [X i ] 2. Covergece i L p implies covergece i measure Lemma 5.2. Fix p > 0. If E [ X p ] 0 the X 0 i P. Proof. Use Chebyshev s iequality with φ(x) = x p. 3. Weak Law i L 2 Theorem 5.1. Let X 1,... be ucorrelated with E [X i ] = 0 ad V [X i ] C <. The S 0 i L2 ad S 0 i P. Proof. E [ S 2] 2 C 0 Covergece i probability follows from Chebyshev s iequality. 4. Polyomial Approximatio Example 5.1. Let f be cotiuous o [0, 1]. Let f (x) = m=0 The sup x [0,1] f (x) f(x) 0. ( ) x m (1 x) m f(m/) m 27

28 Proof. Let X 1,..., X be beroulli with probability p. The: f (p) = = ( ) p m (1 p) m f(m/) m P [S = m] f(m/) m=0 m=0 = E [f(s /)] Now we must show that E [f(s /)] f(p). (a) Let ɛ > 0. The δ > 0 such that (by uiform cotiuity): P [ f(s /) f(p) > ɛ] P [ S / p > δ] The right had side teds to 0 by the L 2 weak covergece theorem. Hece f(s /) f(p) i probability. (b) To get covergece i expectatio (Borel-Catelli would be easier to use), let G = { f(s /) f(p) > ɛ}. The: E [ f(s /) f(p) ] E [ f(s /) f(p) 1 {G }] + ɛ Hece, the result holds for all p. 2 f P [G ] + ɛ 5.2 Triagular Arrays 1. Cotrollig Variace of Sum Lemma 5.3. Let µ = E [X ] ad σ 2 = V [X ]. If σ2 b 2 0 the X µ b 0 P Proof. Chebyshev. [ ] X µ P > ɛ b V [X ] b 2 σ2 b Coupo Collector Example 5.2. Suppose there are distict items, ad we choose oe item, idepedetly uiformly with replacemet. How log does it take to collect a complete set? What is the asymptotic behavior as? 28

29 Solutio. We ca oly choose at distict momets i time, which we idex by t. Let X t be the umber of distict coupos collected by time t. Let N(k) = if{t : X t = k}. The, N() N( 1), N( 1) N( 2),, N(1) are idepedet radom variables with a geometric distributio. The probability of success for N(k) N(k 1) is Hece: Also: E [N()] = (k 1) E [N(i) N(i 1)] = i=1 V [N()] = i=1 V [N(i) N(i 1)] = 2 i=1 Therefore, by the previous lemma: N() k=1 1 k log (i 1) = 1 i 0 P k=1 1 k 2 i=1 That is: N() log 1 P 5.3 Trucatio 1. Geeral Theorem Weak Law for Triagular Arrays Theorem 5.2. For each, let X,k, 1 k be idepedet. Let b > 0 such that b, ad let X,k = 1 {X,k 1 { Y,k b }}. Suppose as : (a) Deviatio from Trucatio Cotrol: P [ X,k b ] 0 k=1 (b) Variace of Trucatio Cotrol: Lettig a = k=1 E [ X,k ]. The: k=1 E [ X2,k ] b 2 0 S a b 0 P 29

30 Proof. There are two parts. First, we show that for T, the partial sums of the trucatios, the result holds. The, we approximate the differece betwee S ad T (a) By the secod assumptio ad Chebyshev s theorem: [ ] [ ] T a P > ɛ k=1 E X2,k 0 b (b) By the first assumptio: P [S T ] = b 2 P [ X,k > b ] 0 k=1 2. I.I.D. Weak Law of Large Numbers Theorem 5.3. Let X 1,... be i.i.d. such that xp [ X i > x] 0 as x Let µ = E [X 1 1 { X 1 }]. The: S µ 0 P Proof. First we must cotrol the deviace from the trucatio. That is: P [ X i ] = P [ X 1 ] 0 i=1 Now, we must cotrol the variace. First ote that: E [ X i 2] = 0 2yP [ X 1 y] dy Let M = sup y 0 yp [ X 1 y]. Let ɛ > 0 ad fid K ɛ such that if y K the: yp [ X 1 y] < ɛ The: E [ X i 2] Kɛ 2 M + 2ɛ( K ɛ ) 0 So to get the cotrol over variace, we ote that: k=1 E [ ] X2 k 2 = E [ ] X 2 1 2K ɛm + 2ɛ( K ɛ ) 2ɛ Sice ɛ is arbitrary, we have to oly apply the Weak Law for Triagular Arrays. 30

31 3. I.I.D. Itegrable Weak Law of Large Numbers Theorem 5.4. Let X 1,... be i.i.d. itegrable radom variables with µ = E [X 1 ]. The: S µ P Proof. We simply must show that xp [ X 1 x] 0 as x. Sice we are o a probability measure, P [ X 1 x] 0 as x. Sice X 1 is itegrable, lettig ɛ > 0 we ca fid a y such that if x > y: xp [ X 1 x] E [ X 1 1 { X 1 x}] < ɛ Hece, we have satisfied the first coditio. To show that µ µ, we have that: [ ] lim µ E [X 11 { X 1 }] = µ E lim X 11 { X 1 } 0 6 Strog Laws of Large Numbers 6.1 Borel-Catelli Lemmas: from Weak to Strog 1. Ifiitely Ofte. Almost Always. Defiitio 6.1. Let A be a sequece of measurable subsets of a probability space. (a) ω occurs ifiitely ofte i A if ω {A i.o} = lim sup A = (b) ω occurs almost always i A if ω {A a.a.} = lim if A = =1 m= =1 m= Note 6.1. We ca iterpret these both as follows. ω occurs ifiitely ofte meas that o matter what N is chose, there is a m N such that ω A m. If ω does ot occur ifiitely ofte, it will occur oly i fiitely may A m. If ω occurs almost always that meas that after some N, ω A m for every m N. A m A m 31

32 6.1.1 First Borel-Catelli Lemma 2. First Borel-Catelli Lemma Lemma 6.1. If k=1 P [A k] < the P [A i.o.] = 0. Proof. This is a cosequece of mootoic limits of measures: P [A i.o.] = lim P lim = 0 m= m A m P [A m ] 3. Couter-Example for Coverse. Example 6.1. Cosider A = [0, 1/] with Lebesgue measure o [0, 1]. Here we have that P [A i.o.] = P [0] = 0. However, =1 P [A ] = 1/ =. =1 4. Applicatios (a) Covergece i probability ad sub(sub)sequece covergece almost everywhere Theorem 6.1. X X P if ad oly if for every subsequece idexed by (m) there is a subsequece for which X (m,k) X a.s. Proof. Suppose X X i probability. Let X (m) be a subsequece. We ca the fid a subsequece idexed by (m, k) such that for ɛ k 0: P [ ] 1 X (m,k) X > ɛ k < 2 k By Borel-Cateilli, the probability that these evets occur ifiitely ofte is 0. Therefore, for almost every ω Ω, X (m,k) (ω) X(ω). For the other directio, suppose X X i probability. The there is a ball about X of radius ɛ ad a subsequece of X idexed by (m) which remais outside of this ball with probability 1. However, we ca extract a subsequece X (m, k) which coverges to X almost surely, which is a cotradictio. (b) Cotiuous fuctios of sequeces covergig i Probability Theorem 6.2. If f is cotiuous ad X X i probability the f(x ) f(x) i probability. Also, if f is bouded the E [f(x )] E [f(x)]. 32

33 Proof. Select ay subsequece of X ad extract a further subsequece which coverges to X almost surely. The by cotiuity, f(x (m,k) ) coverges to f(x) almost surely. Hece, by the previous theorem, we have the first result. For the secod result, by the domiated covergece theorem, E [ f(x (m,k) ) ] E [f(x)]. For a cotradictio, we ca suppose a subsequece of f(x ) does ot coverge i expectatio to f(x), but we ca fid a subsequece which does of this subsequece which does. (c) Fatou s Lemma for Probability Theorem 6.3. Suppose X 0 ad X X i probability. The: lim if E [X ] E [X] Proof. Take sub-subsequeces of X o which almost sure covergece holds. The apply Fatou s lemma. The, argue by cotradictio o subsequeces that the result holds. (d) Domiated Covergece for Probability Theorem 6.4. Suppose X is domiated by a radom variable Y which is itegrable, ad X X i probability. The, E [X ] E [X]. Proof. Go to sub-subsequeces ad use the domiated covergece theorem for almost surely covergig sequeces. The, by cotradictio o sub-subsequeces, we have that E [X ] E [X] (e) Fourth-Momet Boud Strog Law of Large Numbers Theorem 6.5. Let X 1,... be idepedet mea 0 radom variables such that E [ ] Xi 4 C <. The: S 0 a.s. Proof. This will be a cosequece of Chebyshev, ad because there is the fourth momet, we will be able to apply borel catelli. So: [ ] S P > ɛ E [ ] S 4 4 ɛ 4 Note that (x x ) 4 will have: i. terms of the form x 4 i ii. ( ) 2 ways of choosig (i, j) for the term x 2 i x 2 4! j ad 2!2! ways of assigig the expoet. Hece, there are 3( 1) such terms. iii. ( ) 3 ways of choosig (i, j, k) ad 4! 2!1!1! ways of assigig the expoet. 33

34 iv. ( 4) ways of choosig (i, j, k, l) ad 4! ways of assigig the expoet. Sice the latter two terms will be 0 because of idepedece, we have: E [ ] S 4 C + 3( 1)C2 4 ɛ 4 ɛ 4 4 = O( 2 ) Now we ca choose ɛ 0 such that ɛ = 1 δ where 0 < δ < 1. The: [ ] S P > ɛ < =1 By Borel-Catelli, we have that these evets occur ifiitely ofte with probability 0. Hece, we have almost sure covergece Secod Borel-Catelli Lemma 5. Secod Borel-Catelli Lemma Lemma 6.2. Suppose A are idepedet evets ad =1 P [A ] =. The, P [A i.o.] = 1. Proof. Let M > N. The: P [ ( M =NA c ] M = (1 P [A ]) exp =N [ ] Takig the limit over M, we see that P N A the result. ) M P [A ] =N = 1. Thus, we have 6. Applicatio: Existece of Limit Theorem 6.6. If X 1,... are i.i.d. with E [ X i ] = the P [ X i.o] = 1 Also: P [lim S / exists i (, )] = 0 Proof. For the first poit. Note that: E [ X i ] = 0 P [ X i x] dx P [ X i ] Applyig the Secod Borel-Catelli Lemma gives the result. For the secod result, we have that: =0 S S = S ( + 1) X

35 Let C be the set o which the limit coverges. The o C { X i.o}: S S > 2 3 i.o. So we have a cotradictio. 7. Extesio. Theorem 6.7. If A 1,... are pairwise idepedet evets ad =1 P [A ] = the m=1 1 {A m} m=1 P [A 1 a.s. m] 6.2 Etemadi s Proof of SLLN 1. Summatio Boudig Lemma Lemma 6.3. Let y > 0. The 2y k>y 1 k 2 4. Proof. If y > 1 the 2y k>y 1 k 2 2y 2y 2y y 4 k> y y 1 k 2 1 z 2 If 0 < y 1 the Whe y = 0, the etire term is 0. 2y 1 k k 2 k>y k z Boudig Variace Lemma 6.4. Let X 1,... be idetically distributed, itegrable radom variables. The: V [ ] Xk 2 k 2 4E [ X i ] k=1 35

36 Proof. Boudig variace above by the secod momet, ad applyig the previous lemma: k=1 1 k 2 0 2xP [ X 1 x] dx P [ X 1 x] 2x 0 k=1 4 P [ X 1 x] 4E [ X 1 ] 1 k 2 dx 3. Pairwise Idepedet SLLN Theorem 6.8. Let X 1,... be pairwise idepedet idetically distributed radom variables such that E [ X i ] <. Let E [X i ] = 0. The S 0 a.s.. Proof. First, the hypotheses are satisfies by the positive ad egative parts of X i, so we eed oly cosider the positive case. As per usual, we trucate the radom variable. Use covergece covergece i probability with Borel-Catelli to get the covergece of subsequeces a.s.. We the sadwich the remaiig terms betwee the subsequeces. (a) Trucatio. Let Y k = X k 1 { X k k}. The Y k are pairwise idepedet. Moreover: P [X k Y k ] = k=1 k=1 X k k dp E [ X 1 ] Hece, by Borel Catelli, X k Y k for almost every ω at most fiitely may times. Lettig T be the partial sums of Y k, for a.e. ω Ω: Hece, i the limit S T S (ω) T (ω) R(ω) < 0 a.s. Covergece i Probability of Subsequece. Let ɛ > 0 ad α > 1. Let k() = α. We cosider the behavior of the subsequece T k(). P [ T k() E [ ] ] V [ ] T k() T k() > ɛ(k) =1 =1 m=1 m=1 ɛ 2 k() 2 V [Y m ] ɛ 2 k() m 1 α 2 V [Y m ] ɛ 2 1 m α 2 4E [ X 1 ] ɛ 2 (1 α 2 ) 36

37 By Borel Catelli, these evets happe oly fiitely ofte a.s. Sice ɛ is arbitrary, ad by domiated covergece theorem: T k() k() 0 a.s. Sadwichig. Let k() m k( + 1). T k() m T m k( + 1) ad T m k() T k(+1) m a.s. Therefore: T k() k( + 1) T m m T k(+1) k() Sice the limit of T k() /T k(+1) = 1 α, ad α > 1 is arbitrary, the result follows. 4. Ubouded Positive Part SLLN Theorem 6.9. Let X 1,... be i.i.d. such that E [ X + i. The S a.s. ] [ ] = ad E X i < Proof. Let X m i = X + i 1 { X + i m } X i. The, Xm i are i.i.d. ad itegrable. By the SLLN: S m E [Xm 1 ] Sice S Sm ad X m i are mootoically icreasig, the by the mootoe covergece theorem, the result follows. 5. Applicatio to Reewal Theory Theorem Let X 1,... be i.i.d. with 0 < X i <. Let T = i=1 X i ad let N t = sup{ : T t}. If E [X 1 ] = µ, as t : N t t 1 µ a.s. Proof. Let Ω 0 = {T / µ}. Sice T < for all, the as t, N t. Hece: T Nt(ω) N t (ω) µ N t (ω) N t (ω) Fially: T Nt t T N t+1 N t + 1 N t N t N t + 1 N t 37

38 6.3 Radom Series Approach 1. The results above are approached usig a similar set of techiques: (a) Trucatig the radom variable ad cotrollig trucatio error. (b) Applyig Chebyshev s Iequality to get covergece i probability of the sequece or subsequece of S. Achieved by some type of variace cotrol. (c) Applyig Bore-Catelli s First Lemma to a subsequece to demostrate almost sure covergece. (d) Sadwichig the origial sequece betwee these terms to demostrate almost sure covergece. 2. I the followig treatmet, there may be o eed for Trucatio ad the last three steps ca be replaced by cotrollig the series usig a maximal iequality (which is derived usig Chebyshev). 3. Tail σ-algebras (a) Tail σ-algebra Defiitio 6.2. Let F = σ(x, X +1,...). The the tail σ-algebra is T = F (b) Examples Example 6.2. If B B the {X B i.o} T. This holds sice: {X m B m } F m Example 6.3. Let S = X X. The: i. {lim S exists } T. The existece of the limit does ot deped o a fiite umber of iitial radom variables, hece, the evet is i F for all. ii. {lim sup S > 0} T. The sum always depeds o the iitial X 1,..., X 1, hece the evet caot be a tail evet. iii. {lim sup S > ab } T for b. This holds sice for sufficietly large, X 1,..., X 1 will have little effect o the evet sice their cotributio is dimiished by b. (c) Kolmogorov s 0-1 Law Theorem Let X 1,... be idepedet radom variables. If A is a tail evet i the tail σ-algebra geerated by these radom variables the P [A] {0, 1}. Proof. There are two parts i. Let k > 0. The, σ(x 1,..., X k 1 ) is idepedet of σ(x k,..., X k+m ). The, σ(x 1,..., X k 1 ) ad M σ(x k,..., X k+m ) are idepedet π-systems. Therefore, σ(x 1,..., X k 1 ) ad F k are idepedet. 38

39 ii. Let k > 0. The, by the first poit, σ(x 1,..., X k 1 ) is idepedet of T F k. Hece, k σ(x 1,..., X k 1 ) ad T are idepedet π systems. Therefore, σ(x 1,...) is idepedet of T. iii. So if A T the A σ(x 1,...). Therefore: P [A] = P [A A] = P [A] P [A] 4. Kolmogorov s Maximal Iequality Theorem Suppose X 1,..., X are idepedet with mea zero ad fiite variace. The: [ ] P max S k x V [S ] 1 k x 2 Proof. Let A k = {S k x : j < k = S j < x}. The, {max 1 k S k x} = k=1 A k, ad A k are disjoit. We achieve the result by workig with both eds: (a) By Chebyshev s Iequality: x 2 k=1 P [A k] k=1 S 2 k 1 {A k } dp (b) From the other side: S 2 = (S k + S S k ) 2 A k A k = Sk 2 + 2S k (S S k ) + (S S k ) 2 A k = Sk 2 + (S S k ) 2 A k Sk 2 A k x 2 P [A k ] Takig the sum over k, the result follows. 5. Applicatios of Kolmogorov s Maximal Iequality (a) Covergece of Radom Series Theorem Let X 1,... be mea zero, idepedet radom variables. If =1 V [X ] < the =1 X coverges a.s. Proof. Let S be the partial sums. We show that the Cauchy sequece coverges almost surely, which implies the existece of a limitig radom variable. We wat to show that for ay ɛ > 0 as M : [ ] [ ] P sup,m M S S m > 2ɛ 2P sup S S M > ɛ M 0 39

40 Kolmogrov s Maximal Iequality oly applies i the fiite case. Hece, we apply it to the fiite case ad the take a limit: [ ] P max S S M > ɛ 1 M N ɛ 2 V [S N S M ] 1 ɛ 2 N i=m+1 i=m+1 V [X i ] V [X i ] The right had side o loger depeds o N, ad so takig the mootoic limit, we have that the supremum is bouded by the tail sum of variaces from M + 1 to, which teds to 0 as M by the assumptio. Hece, the result holds. (b) Kolmogrov s Three Series Theorem Theorem Let X 1,... be idepedet radom variables. Let A > 0 ad Y i = X i 1 { X i A}. The i=1 X i coverges if ad oly if the followig three hold: i. Cotrol Trucatio Error: P [ X > A] < ii. Cotrol Variace of Trucatio: V [Y ] < iii. Covergece of Trucatio Mea: E [Y ] coverges Proof. We prove the ( ) directio first. By the secod coditio, Y E [Y ] coverge a.s.. By the secod coditio, Y coverges almost surely. By Borel Catelli ad the first coditio, P [X Y ] = 0, which implies the result. For ( ). Suppose the series coverges but the trucatio error cotrol does ot hold. The by the secod Borel-Catelli lemma, P [ X > A i.o.] = 1 which implies X caot coverges. Hece, the first coditio holds. Now suppose the variace goes to ifiity ad defie c = m=1 V [Y m] i. Sice X m coverges a.s. the Y m coverges a.s. ad so T = Ym c 0 d ii. Lettig X,m = Y m E [Y m ] c ad S = m X,m, by Lidberg Feller CLT, S χ d. Therefore, S T χ d but S T is ot radom. Now for the fial coditio, we have that Y coverges a.s. ad Y E [Y ] coverges a.s. 40

41 6. Alterative Proof of SLLN (a) Kroecker s Lemma Lemma 6.5. If a ad x =1 a (b) Strog Law of Large Numbers coverges the m=1 xm a 0. Theorem Let X 1,... be i.i.d. radom variables with E [ X i ] <. Let E [X i ] = 0. The: S 0 a.s. Proof. We will apply Kroecker s Lemma by showig that X =1 coverges a.s.. To do this, we eed oly boud the sum of the variaces, which follows from Lemma 6.4: =1 V [X ] 2 4E [ X 1 ] < Part III Cetral Limit Theorems 7 DeMoivre-Laplace Theorem 1. Properties of Expoetial Fuctio Lemma 7.1. The followig results are i the real field, but exted to complex values. (a) If c j 0, a j, ad a j c j λ the (1 + c j ) aj exp(λ) (b) If max 1 j c j, 0, j=1 c j, λ ad sup j=1 c j, < the j=1 (1 + c j,) exp(λ) Proof. For the first result, we have that: log(1 + c j )/c j 1 as c j 0. Hece: log(1 + c j ) a j log(1 + c j ) = a j c j λ c j For the secod result, we have that; j=1 c j, 1 + c j, c j, λ 41

42 2. Let X i be idepedet Rademacher radom variables ad S be their partial sums 3. DeMoivre-Laplace Theorem Theorem 7.1. If a < b ad m the [ P a S ] b ( ) m 1 x 2 b exp m 2π 2 a Proof. Note that whe is eve, S is eve. Whe is odd, S is odd. Sice S 2+1 {S 2 + 1, S 2 1} it is simpler to cosider the eve case ad the exted to the odd case. (a) For, k Z, we have that: P [S 2 = 2k] = P [ + k : +1, k : 1] ( ) 2 = k For sufficietly large, we ca use Stirlig s Approximatio: P [S 2 = 2k] = (2)! ( + k)!( k)! π ( + k) +k 2π( + k)( k) k 2π( k) 1 ( ) 1/2 ( 1 k2 π k ) k ( 1 k ) k (b) Suppose 2k 2 / x 2 R. The, applyig Lemma 7.1: P [S 2 = 2k] 1 ( ) 1/2 ( 1 x2 /2 1 + x ) x /2 ( 1 x ) x /2 π exp(x 2 /2) exp( x 2 /2) exp( x 2 /2) π 1 π exp( x 2 /2) (c) Therefore: [ P a S 2 / ] 2 b = m [a 2,b 2] 2Z m [a,b] 2/Z P [S 2 = m] ( ) 1/2 1 2 exp( x 2 /2) 2π This is a Riema sum, ad so i the limit, we have the result. 42

43 8 Weak Covergece 8.1 Defiitio ad Basic Results 1. Coverge Weakly. Coverge i Distributio Defiitio 8.1. (a) A sequece of distributio fuctios, F, coverges weakly to a limit F if F (y) F (y) at all cotiuity poits of F. (b) A sequece of radom variables X coverges weakly or coverge i distributio if their distributio fuctios coverge weakly. 2. Examples Example 8.1. The DeMoivre Laplace Theorem which states that Rademacher radom variables coverge weakly to χ, a ormally distributed radom variable, is a example of weak covergece. Example 8.2. Covergece at Discotiuities. Let X F ad X = X + 1. The F (x) = F (x 1/) ad lim F (x) = F (x ) Therefore, if x is a discotiuity, we have a issue if weak covergece required covergece at poits of discotiuity as well. 3. Weak Covergece ad Idetically Distributed Radom Variables Covergig a.s. Theorem 8.1. If F F d, the Y for 1 such that Y F ad Y Y a.s.. Proof. Let F = F. Let Y (ω) = sup{y : F (y) < ω}. Let Ω 0 [0, 1] for which F is cotiuous, the o Ω 0, F 1 exists ad F F 1 (ω) = ω. For ω Ω 0 : (a) Suppose y < F 1 (ω). The F (y) < ω. For sufficietly large, F (y) < ω. Hece, y Y (ω). Lettig y F 1 (ω), Y (ω) lim if Y (ω). (b) Suppose y > F 1 (ω). The F (y) > ω. For sufficietly large, F (y) > ω. Hece, y Y (ω). Lettig y F 1 (ω), Y (ω) lim sup Y (ω). 4. Covergece Results (a) Fatou s Lemma Lemma 8.1. If g 0 ad cotiuous, ad X X d, the lim if E [g(x )] E [g(x )] 43