Topics in Probability Theory and Stochastic Processes Steven R. Dunbar. Almost Sure Events

Transcription

1 Steve R. Dubar Departmet of Mathematics 203 Avery Hall Uiversity of Nebraska-Licol Licol, NE Voice: Fax: Topics i Probability Theory ad Stochastic Processes Steve R. Dubar Almost Sure Evets Ratig Mathematicias Oly: prologed scees of itese rigor. 1

2 Sectio Starter Questio Cosider a ifiite sequece of coi flips. What probability would you assig to the sequece that has a Head o its iitial flip ad the ay sequece of Heads ad Tails o all the subsequet flips? What probability would you assig to the sequece of flips that is alterately Heads ad Tails, cotiuig idefiitely? Key Cocepts 1. A subset A Ω is of fiite type if there exists a = (A) 1 ad a subset A Ω so that A = { ω Ω : ω () A }, where ω () = (ω 1,..., ω ). 2. N Ω is a egligible evet (that is, a evet of probability measure 0) if for every ɛ > 0 there exists a coutable set {A k : k 1} of fiite type evets so that N k 1 A k ad k 1 P [A k] < ɛ. 3. Every coutable uio of egligible sets is egligible. 4. If p 0, 1, every coutable subset of Ω is egligible. 5. The set of evets {A i } i I are idepedet if [ ] P A ik = k=1 P [A ik ] k=1 for every fiite set of distict idexes i 1,..., i I. 6. Evets determied by coordiates with disjoit sets of idexes are idepedet. 7. The set of sequeces that are periodic after a certai idex is egligible. 8. A set E is called a tail evet if E is ot a fiite type evet. 2

3 9. Let B be a sequece of sets i Ω. (i.o. stads for ifiitely ofte. ) {B i.o. } = lim m =mb. 10. The Kolmogorov 0-1 Law says that if B 1, B 2,... are idepedet, ad if E = {B i.o. }, the P [E] = 0 or P [E] = 1. Vocabulary 1. A subset A Ω is of fiite type or is a fiite type evet if there exists a = (A) 1 ad a subset A Ω so that A = { ω Ω : ω () A }, where ω () = (ω 1,..., ω ). 2. N Ω is a egligible evet (that is, a evet of probability measure 0) if for every ɛ > 0 there exists a coutable set {A k : k 1} of fiite type evets so that N k 1 A k ad k 1 P [A k] < ɛ. 3. A evet A Ω is a almost sure evet if A c = Ω \ A is egligible. 4. The set of evets {A i } i I are idepedet if [ ] P A ik = P [A ik ] k=1 k=1 for every fiite set of distict idexes i 1,..., i I. 5. A set E is called a tail evet if E is ot a fiite type evet. 3

4 Mathematical Ideas The Strog Law of Large Numbers says it is certai that S / approaches the probability of success p as the umber of tosses approaches ifiity. To be rigorous, we eed to make the statemet it is certai precise. The problem is that there exist ifiite sequeces of heads ad tails such that the proportio of heads does ot coverge to p, such as the sequece cosistig of all tails. I fact, there exist sequeces of heads ad tails where the proportio of heads does ot coverge at all. However, we ca exclude such evets by defiig the cocept of a almost sure evet. The Strog Law of Large Numbers the tells us that the sequece S / coverges almost surely to p. This fudametal idea is due to E. Borel i 1909, [1]. Fiite Type Evets First we cosider the subset of elemets i Ω defied by a coditio depedig o oly fiitely may coordiates. The realizatio or o-realizatio of such a evet is determied after a fixed ad fiite umber of elemetary trials. Recall (Biomial Distributio) that a composite experimet cosists of repeatig a elemetary Beroulli trial times. The sample space, deoted Ω, is the set of all possible sequeces of 0 s ad 1 s represetig all possible outcomes of the composite experimet. We deote a elemet of Ω as ω = (ω 1,..., ω ), where each ω k = 0 or 1. That is, Ω = {0, 1}. We assig a probability measure P [ ] o Ω by multiplyig the probabilities of each Beroulli trial i the composite experimet accordig to the priciple of idepedece. Thus, for k = 1,...,, P 1 [ω k = 0] = q ad P 1 [ω k = 1] = p ad iductively for each (e 1, e 2,..., e ) {1, 0} P +1 [ω +1 = 1 ad (ω 1,..., ω k ) = (e 1,..., e )] = Defiitio. Cosider the space P 1 [ω +1 = 1] P [(ω 1,..., ω ) = (e 1,..., e )] Ω = {ω = (ω ) =1 : ω = 0, 1 for all }. Defie S := ω ω. A subset A Ω is of fiite type or is a fiite type evet if there exists a = (A) 1 ad a subset A Ω so that 4

5 A = { ω Ω : ω () A }, where ω () = (ω 1,..., ω ) is the projectio of the ifiite sequece ω oto the fiite head of legth. If A is of fiite type, the we defie the probability to be P [A] = P (A ) [A ] = p S(ω) q ( S(ω)). ω () A This probability is well-defied, see the Problems. Defiitio. Note that P [ ] has the property of ivariace uder shiftig, that is, if A is a evet, ad k is a positive iteger, the the evet {ω Ω : (ω k, ω k+1, ω k+2,..., ) A} has the same probability as the evet A. Example. The set A cosistig of the sequeces with a Head o its iitial flip ad the ay sequece of Heads ad Tails o all the subsequet flips is a fiite type evet. Takig a 1 to represet a Head ad a 0 to represet a tail, the iteger (A) = 1 ad the subset A = {1} so A Ω 1 = {0, 1}. The A = { ω Ω : ω (1) A = {1} } Defiitio. A set of subsets of some uiversal set is a field if it is closed uder fiite uios, fiite itersectios, ad complemets. Defie E to be the set of fiite type evets. Note that E, Ω E. The set E is closed uder takig complemets ad also closed uder fiite uios ad fiite itersectios. Thus, we see that E is a field. This follows by direct verificatio, see the Problems. Note that P : E [0, 1] ad P [Ω] = 1 ad P [ ] = 0 ad P [A B] = P [A] + P [B] if A B =. Defiitio. A set of subsets of some uiversal set is a Borel field if it is closed uder complemets, ad coutable itersectios ad uios. For ay set C of subsets of Ω, deote by F(C) the smallest Borel field cotaiig C. Negligible ad Almost Sure Evets Defiitio. We say that N Ω is a egligible evet, that is, a evet of probability measure 0, if for every ɛ > 0 there exists a coutable set {A k : k 1} of fiite type evets so that N k 1 A k ad k 1 P [A k] < ɛ ad we write P [[] A] a.s.. A evet A Ω is a almost sure evet if A c = Ω \ A is egligible. 5

6 Propositio Every subset of Ω that is cotaied i a egligible evet is egligible. 2. Every coutable uio of egligible sets is egligible. 3. If p 0, 1, every coutable subset of Ω is egligible. Proof. 1. Suppose that A is egligible ad B A. The there exist A k Ω, a set of fiite type evets, so that A k 1 A k. The we see that B k 1 A k ad k 1 P [A k] ɛ. Thus, B is egligible. 2. Let {N } =1 be a coutable set of egligible sets. Fix ɛ > 0 ad fid (by defiitio) a set of egligible evets {A,k } k=1 so that N k 1 A,k ad P [A,k ] ɛ. The 2 ad 1 P [A,k ] ɛ. k 1 N = 1 N 1 A,k k 1 3. Suppose that p 0, 1. First prove that the sigleto set {ω} is egligible. Note that {ω} { ω Ω : ω () = ω ()} = A ad P [A ] max {p, (1 p) }. For 0 so that p, (1 p) ɛ, the we see that {ω} A ad P [A ] ɛ. Thus we see that {ω} is egligible ad by part (2), we see that a coutable uio of sigletos is egligible. Defiitio. The set of evets {A i } i I are idepedet if [ ] P A ik = P [A ik ] k=1 for every fiite set of distict idexes i 1,..., i N. Remark. The sets A i are idepedet if ad oly if the sets A c i are idepedet. This follows because k=1 P [ A C 1 A c 2] = P [(A1 A 2 ) c ] = 1 P [A 1 A 2 ] = 1 P [A 1 ] P [A 2 ] + P [A 1 A 2 ] = 1 P [A 1 ] P [A 2 ] + P [A 1 ] P [A 2 ] = (1 P [A 1 ])(1 P [A 2 ]). 6

7 Propositio 2. Let {A i } i I be a family of evets. If for each i I there exists a fiite subset E i N ad a subset A i {0, 1} E i such that E i E j = if i j ad A i = {ω Ω : (ω ) Ei A i}. The the evets {A i } i I are idepedet. Remark. This propositio meas that evets that are determied by coordiates with disjoit sets of idexes are idepedet. Example. Let A i be the evet that the ith coi flip is heads. Let E i = {i} ad A i be the set so that ω i = 1. Certaily, E i E j = for i j. Also, A i = {ω Ω : (ω ) Ei A i} = {ω Ω : ω i = 1}. Note P [A i ] = p ad P [ k=1 A k] = p. Thus, we see that P [ k=1 A k] = k=1 P [A k]. Propositio 3. Let b be a word from the alphabet {0, 1}; i.e., b is a fiite sequece of 0 s ad 1 s. The set is egligible. A = {ω Ω : b is ot foud i ω} Proof. 1. Let b be a word of legth j > 0. For all m 0, let A m be the set of all ω Ω so that (ω mj+1,..., ω (m+1)j ) b. I other words, we are dividig ω ito cosecutive o-overlappig blocks of legth j. 2. Note that P [A 0 ] < 1 ad the property of ivariace uder shiftig tells us that P [A m ] < The A i are idepedet by Propositio 3 ad so [ ] P A k = (P [A 0 ]) m+1. k m 4. This value ca be made arbitrarily small by choosig a large eough m. 5. Let B = {ω Ω : (ω +1,..., ω +j ) b} k m A k. 6. Thus, we see that the probability of B ca be made arbitrarily small. 7

8 7. Note that A B ad so A is egligible. Corollary 1. All possible words almost surely appear i the sequece ω. Proof. (of Corollary 1) The set of all possible words is coutable, so let b i be a eumeratio of the set of possible words. By Propositio 3 the set A bi = {ω Ω : b i is ot foud i ω} is egligible. The cosider A = bi A bi. By Propositio 1, the set A is egligible. Fially cosider ω A C = Ω \ A ad a arbitrary word b. The word b must appear i A C, ad A C is a almost sure evet. Corollary 2. The set of sequeces that are periodic after a certai idex is egligible. Proof. (of Corollary 2) 1. Let P i,j be the set of all ω that begi periodicity at idex i ad the period has legth j. 2. First, show that the set P 1,j of purely periodic sequeces is egligible. Let b j = 0 j be the word of legth j with all zeros. There is oly sequece ω bj i P 1,j which starts with the word b ad so we see that P i,j = { ω bj } {ω : ot cotaiig bj }. Note that the first set is egligible sice it is a sigleto ad the secod set is a subset of a egligible set by Propositio 3. So P 1,j is egligible. 3. Next, for i 2 the set P i,j is egligible. For = 0,..., 2 i 1 1 let b be the biary represetatio with legth i 1 of the umber. 4. For = 0,..., 2 i 1 1 let P b,j be the set of sequeces which start with b ad are periodic with period j startig at idex i. By the property of ivariace uder shiftig, P [P b,j] = P [P 1,j ] ad so P bi,j is egligible. Fially, ad so P i,j is egligible. P i,j = 2 i 1 1 =0 P b,j 8

9 Tail Evets Defiitio. A set E is called a tail evet if E is ot a fiite type evet. That is, whether or ot ω E does ot deped o the first coordiates of ω o matter how large is. Some authors refer to a tail evet as a remote evet. Example. Cosider the set E of coi-tossig sequeces such that S does ot coverge to p. This set caot deped o the first coordiates of ω o matter how large is, hece it is a tail evet. Remark. E F(Ω) is a tail evet if E F(X, X +1,... ) for all. Here F(Ω) is the Borel field of Ω ad F(X, X +1,... ) is the smallest Borel field cotaiig all possible sequeces begiig at the th idex. Remark. This defiitio may seem quite abstract, but it captures i a formal way the sese i which certai evets do ot deped o ay fiite umber of their coordiates. For example, cosider the set E of coi-tossig sequeces such that S does ot coverge to p. That is, E = { ω : X X } p. The for ay k 1, E = { ω : X k + + X } p. so that E F(X, X +1,... ) for all k 1, E is a tail evet. Defiitio. Let B be a sequece of sets i Ω. (i.o. stads for ifiitely ofte. ) {B i.o.} = lim m =mb. So ω {B i.o.} if ad oly if ω B for ifiitely may. Propositio 4. Give {a } =1 with lim a = +, the S lim sup a p = + almost surely. 9

10 Remark. If a a, the [ lim P a ] S p < c = lim P = lim P = lim P = c a p(1 p) c a p(1 p) [ a ] S p < c [ a ] S p p(1 p) < c p(1 p) [ ] S p c < p(1 p) a p(1 p) 1 2π e x2 /2 dx by the Cetral Limit Theorem ad this is a fiite value. O the other had, the Moderate Deviatios Theorem says that if the 1. (a ) is a sequece of real umbers, 2. a as ad 3. lim a 1/6 = 0. [ S P p p(1 p) a ] [ = P a 1 a 2π e a2 /2. ( ) S p ] p(1 p) This pair of previous results, together with the Propositio show how fiely balaced aroud p is the sequece ( S p ). Proof. 1. Set A m = { ω : lim sup a S p } < m for all m > 0. For ay ω A m is a N (possibly depedig o ω ) such that for every > N we have a S p < m. { } 2. Let A k,m = ω : a k k S k (ω) p k < m. For ω A m there exists k so that ω A k,m. 10

11 3. Note that A m k A k,m. 4. P [A k,m ] m a k p 1 p m a k p 1 p e x2 /2 2π dx 5. Give ɛ, because a 2 j k there exists k j so that P [A k,m ] < ɛ ad we 2 j ca choose the k j as a icreasig sequece. Thus, P [ A kj,m] < ɛ k=1 ad A m j=1a kj,m. Thus, A m is egligible. This is true for ay value m ad so [ ] S P lim sup a p = a.s. Sources This sectio is adapted from: Heads or Tails, by Emmauel Lesige, Studet Mathematical Library Volume 28, America Mathematical Society, Providece, 2005, Chapter 11.1, pages [3]. Some of the remarks ad examples are also adapted from Probability by Leo Breima, Addiso-Wesley, Readig MA, 1968, Chapters 1 ad 3. [2] 11

12 Algorithms, Scripts, Simulatios Algorithm AlmostSureEvets-Simulatio Commet Post: Observatio that ay coi flip sequece selected by a pseudo-radom-umber-geerator has scaled deviatios which exceed a sequece of itegers, suggestig the almost sure result of 4 Commet Post: Idexes ad values of a millio flip coi-flip sequece where scaled deviatios from p exceed the coutig sequece. 1 Set probability of success p 2 Set legth of coi-flip sequece 3 Iitialize ad fill legth array of coi flips 4 Use vectorizatio to sum to get the cumulative sequece S of heads 5 Use vectorizatio to compute the scalig factor array a 6 Use vectorizatio to compute the array of scaled deviatios a S / p 7 Iitialize the subsequece idex k to 1 8 while ay scaled deviatios exceed k 9 Set N k as the first idex where the scaled deviatios exceed k 10 Prit k, S Nk ad scaled deviatio a Nk Nk S Nk /N k p. Scripts Scripts R R script for Almost Sure Evets. 1 p < <- 1e+6 3 coiflips <- ( ruif () <= p) 4 S <- cumsum ( coiflips ) 5 6 a <- (1: ) ^(1 /6) 7 deviatios <- a* sqrt (1: )*( abs ( S/ (1: ) - p )) 8 9 k < while ( legth ( which ( deviatios > k)) > 0 ) { 11 Nk <- mi ( which ( deviatios > k)) 12

13 12 cat (" Idex :", Nk, " Sum :", S[Nk], " Scaled Deviatio :", deviatios [Nk], "\") 13 k <- k+1 14 } Octave Octave script for Almost Sure Evets. 1 p = 0. 5; 2 = 1e +6; 3 coiflips = rad (1,) <= p; 4 S = cumsum ( coiflips ); 5 6 a = (1: ).^(1/6) ; 7 deviatios = a.* sqrt (1: ).*( abs ( S./(1: ) - p* oes (1,)) ); 8 9 k = 1; 10 while ( ay ( deviatios > k) ) 11 Nk = fid ( deviatios > k) (1) ; 12 pritf (" Idex : %i, Sum : %i, Scaled deviatio : %f \", 13 Nk, S(Nk), deviatios (Nk)) 14 k = k +1; 15 edwhile Perl Perl PDL script for Almost Sure Evets. 1 $p = 0. 5; 2 $ = 1e +6; 3 $coiflips = radom ( $) <= $p; 4 $S = cumusumover ( $coiflips ); 5 6 $array = zeros ($) -> xlivals ( 1, $ ); 7 $a = $array **( 1 / 6 ); 8 $deviatios = $a * sqrt ( $array ) * ( abs ( $S / ( $array ) - $p ) ); 9 10 $k = 1; 11 while ( ay $deviatios > $k ) { 12 $Nk = ( ( which $deviatios > $k ) - > rage (0) ); 13 prit " Idex :", $Nk, " Sum :", $S -> rage ([ $Nk ]), " Scaled Deviatio :", 13

14 14 $deviatios -> rage ([ $Nk ]), "\"; 15 $k = $k + 1; 16 } SciPy Scietific Pytho script for Almost Sure Evets. 1 import scipy 2 3 p = = 1e+6 5 coiflips = scipy. radom. radom ( ) <= p 6 7 # Note Booleas True for Heads ad False for Tails S = scipy. cumsum ( coiflips, axis =0) 11 # Note how Booleas act as 0 ( False ) ad 1 ( True ) array = scipy. arage (1, + 1, dtype = float ) 14 a = array ** (1. / 6.) 15 deviatios = a * scipy. sqrt ( array ) * scipy. absolute ( S / array - p) k = 1 18 while scipy. ay ( deviatios > k): 19 Nk = scipy. ozero ( deviatios > k) [0][0] 20 prit Idex :, Nk, Sum :, S[Nk], Scaled Deviatio :, \ 21 deviatios [ Nk] 22 k = k

15 Problems to Work for Uderstadig 1. Show that the defiitio that if A is of fiite type, the we defie the probability to be P [A] = P (A ) [A ] = ω () A p S(ω) q ( S(ω)) is well-defied. That is, if A Ω (A ) ad A Ω (A ) are two evets such that A = { ω Ω : ω () A } ad A = { ω Ω : ω () A }, the P [A] = P (A ) [A ] = P (A ) [A ]. 2. Show that: (a) E. (b) Ω E. (c) The set E is closed uder takig complemets. (d) The set E is closed uder fiite uios ad fiite itersectios. 3. Show that P [ ] has the property of ivariace uder shiftig. 4. Show that P [ ] is mootoe, that is, if A B the P [A] P [B]. 5. Is a sigleto set {ω} is a tail evet? 6. Is the set of tail evets is coutable? Readig Suggestio: Refereces [1] E. Borel. Sur les probabilités démombrables et leurs applicatios arithmétiques. Redicoti del Circolo Math. di Palermo, 26: ,

16 [2] Leo Breima. Probability. Addiso Wesley, [3] Emmauel Lesige. Heads or Tails: A Itroductio to Limit Theorems i Probability, volume 28 of Studet Mathematical Library. America Mathematical Society, Outside Readigs ad Liks: I check all the iformatio o each page for correctess ad typographical errors. Nevertheless, some errors may occur ad I would be grateful if you would alert me to such errors. I make every reasoable effort to preset curret ad accurate iformatio for public use, however I do ot guaratee the accuracy or timeliess of iformatio o this website. Your use of the iformatio from this website is strictly volutary ad at your risk. I have checked the liks to exteral sites for usefuless. Liks to exteral websites are provided as a coveiece. I do ot edorse, cotrol, moitor, or guaratee the iformatio cotaied i ay exteral website. I do t guaratee that the liks are active at all times. Use the liks here with the same cautio as you would all iformatio o the Iteret. This website reflects the thoughts, iterests ad opiios of its author. They do ot explicitly represet official positios or policies of my employer. Iformatio o this website is subject to chage without otice. Steve Dubar s Home Page, to Steve Dubar, sdubar1 at ul dot edu Last modified: Processed from L A TEX source o September 25,