Probability ad Radom Processes Lecture 5 Probability ad radom variables The law of large umbers Mikael Skoglud, Probability ad radom processes 1/21 Why Measure Theoretic Probability? Stroger limit theorems Coditioal probability/expectatio Proper theory for cotiuous ad mixed radom variables Mikael Skoglud, Probability ad radom processes 2/21
Probability Space A probability space is a measure space Ω, A, P the sample space Ω is the uiverse, i.e. the set of all possible outcomes the evet class A is a σ-algebra of measurable sets called evets the probability measure is a measure o evets i A with the property P Ω = 1 Mikael Skoglud, Probability ad radom processes 3/21 Iterpretatio A radom experimet geerates a outcome ω Ω For each A A either ω A or ω / A A evet A i A occurs if ω A with probability P A sice A is the σ-algebra of measurable sets, we are esured that all reasoable combiatios of evets ad sequeces of evets are measurable, i.e., have probabilities Mikael Skoglud, Probability ad radom processes 4/21
With Probability Oe A evet E A occurs with probability oe if P E = 1 almost everywhere, almost certaily, almost surely,... Mikael Skoglud, Probability ad radom processes 5/21 Idepedece E ad F i A are idepedet if P E F = P EP F The evets i a collectio A 1,..., A are pairwise idepedet if A i ad A j are idepedet for i j mutually idepedet if for ay {i 1, i 2,..., i k } {1, 2,..., } P A i1 A i2 A ik = P A i1 P A i2 P A ik A ifiite collectio is mutually idepedet if ay fiite subset of evets is mutually idepedet mutually pairwise but ot vice versa Mikael Skoglud, Probability ad radom processes 6/21
Evetually ad Ifiitely Ofte A probability space Ω, A, P ad a ifiite sequece of evets {A }, defie lim if A = A k, lim sup A = A k =1 k= =1 k= ω lim if A iff there is a N such that ω A for all > N, that is, the evet lim if A occurs evetually, {A evetually} ω lim sup A iff for ay N there is a > N such that ω A, that is, the evet lim sup A occurs ifiitely ofte {A i.o.} Mikael Skoglud, Probability ad radom processes 7/21 Borel Catelli The Borel Catelli lemma: A probability space Ω, A, P ad a ifiite sequece of evets {A } 1 if P A <, the P {A i.o} = 0 2 if the evets {A } are mutually idepedet ad P A =, the P {A i.o} = 1 Mikael Skoglud, Probability ad radom processes 8/21
Radom Variables A probability space Ω, A, P. A real-valued fuctio Xω o Ω is called a radom variable if it s measurable w.r.t. Ω, A Recall: measurable X 1 O A for ay ope O R X 1 A A for ay A B the Borel sets Notatio: the evet {ω : Xω B} X B P {X A} {X B} P X A, X B, etc. Mikael Skoglud, Probability ad radom processes 9/21 Distributios X is measurable P X B is well-defied for ay B B The distributio of X is the fuctio µ X B = P X B, for B B µ X is a probability measure o R, B The probability distributio fuctio of X is the real-valued fuctio F X x = P {ω : Xω x} = otatio = P X x F X is obviously the distributio fuctio of the fiite measure µ X o R, B, i.e. F X x = µ X, x] Mikael Skoglud, Probability ad radom processes 10/21
Idepedece Two radom variables X ad Y are pairwise idepedet if the evets {X A} ad {Y B} are idepedet for ay A ad B i B A collectio of radom variables X 1,..., X is mutually idepedet if the evets {X i B i } are mutually idepedet for all B i B Mikael Skoglud, Probability ad radom processes 11/21 Expectatio For a radom variable o Ω, A, P, the expectatio of X is defied as E[X] = XωdP ω For ay Borel-measurable real-valued fuctio g E[gX] = gxdf X x = gxdµ X x Ω i particular E[X] = xdµ X x Mikael Skoglud, Probability ad radom processes 12/21
Variace The variace of X, VarX = E[X E[X] 2 ] Chebyshev s iequality: For ay ε > 0, P X E[X] ε VarX ε 2 Kolmogorov s iequality: For mutually idepedet radom variables {X k } with VarX k <, set S j = j X k, 1 j, the for ay ε > 0 P max S j E[S j ] ε VarS j ε 2 = 1 Chebyshev Mikael Skoglud, Probability ad radom processes 13/21 The Law of Large Numbers A sequece {X } is iid if the radom variables X all have the same distributio ad are mutually idepedet For ay iid sequece {X } with µ = E[X ] <, the evet lim occurs with probability oe 1 X k = µ Toward the ed of the course, we will geeralize this result to statioary ad ergodic radom processes... Mikael Skoglud, Probability ad radom processes 14/21
S = 1 X µ with probability oe S µ i probability, i.e., for each ε > 0 lim P { S µ ε} = 0 i geeral i probability does ot imply with probability oe covergece i measure does ot imply covergece a.e. Mikael Skoglud, Probability ad radom processes 15/21 The Law of Large Numbers: Proof Lemma 1: For a oegative radom variable X P X E[X] =1 P X =0 Lemma 2: For mutually idepedet radom variables {X } with VarX < it holds that X E[X ] coverges with probability oe Lemma 3 Kroecker s Lemma: Give a sequece {a } with 0 a 1 a 2 ad lim a =, ad aother sequece {x k } such that lim k x k exists, the lim 1 a a k x k = 0 Mikael Skoglud, Probability ad radom processes 16/21
Assume without loss of geerality why? that µ = 0 Lemma 1 =1 P X = =1 P X 1 < Let E = { X k k i.o.}, Borel Catelli P E = 0 we ca cocetrate o ω E c Let Y = X χ { X <}; if ω E c the there is a N such that Y ω = X ω for N, thus for ω E c lim 1 X k = 0 lim 1 Y k = 0 Note that E[Y ] µ = 0 as Mikael Skoglud, Probability ad radom processes 17/21 Lettig Z = 1 Y, it ca be show that =1 VarZ < requires some work. Hece, accordig to Lemma 2 the limit Z = lim Z k E[Z k ] exists with probability oe. Furthermore, by Lemma 3 1 Y k E[Y k ] = 1 kz k E[Z k ] 0 where also 1 E[Y k ] 0 sice E[Y k ] E[X k ] = E[X 1 ] = 0 Mikael Skoglud, Probability ad radom processes 18/21
Proof of Lemma 2 Assume w.o. loss of geerality that E[X ] = 0, set S = X k For E A with E 1 E 2 it holds that P E = lim P E Therefore, for ay m 0 P { S m+k S m ε} = lim P = lim P { S m+k S m ε} max S m+k S m ε 1 k Mikael Skoglud, Probability ad radom processes 19/21 Let Y k = X m+k ad T k = k Y j = S m+k S m, j=1 the Kolmogorov s iequality implies P max S m+k S m ε 1 k VarS m+ S m ε 2 = 1 ε 2 m+ k=m+1 VarX k Hece P { S m+k S m ε} 1 ε 2 k=m+1 VarX k Mikael Skoglud, Probability ad radom processes 20/21
Sice VarX <, we get lim P m { S m+k S m ε} = 0 Now, let E = {ω : {S ω} does ot coverge}. The ω E iff {S ω} is ot a Cauchy sequece for ay there is a k ad a r such that S +k S r 1. Hece, equivaletly, { E = S +k S 1 } r k r=1 k For F 1 F 2 F 3, P k F k = lim P F k, hece for ay r > 0 { P S +k S 1 } { = lim r P S +k S 1 } r That is, P E = 0 k Mikael Skoglud, Probability ad radom processes 21/21