Indian Institute of Tehnology Bombay Department of Eletrial Engineering Handout 5 EE 325 Probability and Random Proesses Leture Notes 3 July 28, 2014 1 Axiomati Probability We have learned some paradoxes assoiated with traditional probability theory, in partiular the so alled Bertrand s paradox. The problem there was an inaurate or inomplete speifiation of what the term random means. In fat, formulating a random experiment needs to be more onstrutive, learly outlining the way probabilities of various events are generated. It is here that the axiomati framework or modern framework 1 takes over. The modern notion defines a probability spae, sometimes referred to as probability trinity. Muh like the Indian trinity of srushti(brahma), sthithi(vishnu), samhara(maheshwara) or western father, son, holy soul, the probability spae is about how outomes, events and probabilities are onstruted unambiguously. Indeed a probabilisti universe in ation. However, do no read further into this mythologial onnetion, our trinity has more reation, maintenane and lending out, its Mumbai after all. We will define a few axioms on events and their probability assignment, and further use this to onstrut suitable probability spaes. 2 Probability Spae (Ω, F, P ) The probability spae has three onstituents, Ω, F and P. 1. The first entity Ω (borrowed from lassial probability) denotes the sample-spae, where eah outome of the experiment is a sample-point. 2. The seond entity F, also known as event-spae, is a lass of subsets of Ω, having ertain extra qualifiations than its lassial ounterpart. In partiular, the eventspae F is a σ field (pronouned as sigma field). 3. The third entity is the probability measure, whih assoiates a value in [0, 1] for any event A F. In fat, the three entities appear pretty muh the way they were introdued in lassial probability. However, we will refine our definitions and list a set of axioms that governs the onstrution of the last two entities. We already had several examples about the first entity. So let us start with the seond one, the so alled event-spae F. 3 Event-Spae F Given several events, event-management is a neessity (imagine Olympis or Tehfest). Event-management allows us to do operations on events in F. Sine events themselves are subsets of Ω, the permitted operations are the well-known set operations. In order to do event-management, the spae F should be stable with respet to set operations. The word
Events F Set Manipulator Events F Figure 1: A familiar linear systems view of stability stable here is very important. It emphasizes that if you perform permissible operations on event(s) in F, what results is an event (possibly different) in F, see Figure 1. Now this resembles very muh of a say in riket, what happens in a riket-field stays there. In fat a riket-field provides a level-playing ground for various ations and events to our, and ideally no event boils out of the field, that is the ground, stadium and partiipants 2. The word field applies to a more general ontext than a riket field, feel free to think about a riket field and the above proverb. Those who are used to set-theory may guess immediately that the property stable is similar in spirit to a lass of subsets being losed with respet to usual set-operations. However, there are subtle differenes between the terms losed and stable, at least in the onventional use of the first. Conventional set operations on event(s) are 1. Complement, denoted as A or Ā. 2. Union: A 1 A 2. 3. Intersetion: A 1 A 2. 4. Set differene: A 1 A 2, whih is A 1 A 2. 5. Symmetri differene: A 1 A 2 whih is (A 1 A 2 ) (A 2 A 1 ). 3.1 Stability and Complements Events of interests in an experiment are usually assoiated with a YES/NO question. For example, has A happened?, or is it HEAD?. An event A has not happened will mean that the event A has happened. Thus, has A not happened an be asked equivalently in terms of A happening. As a result, we want both parties to be members of F for it to be stable. A stable lass needs to be losed under omplementing, i.e. 3.2 Stability and Unions A F A F. A lass F is losed under set-unions if the union of any two elements in the lass will give rise to another element in the lass. By repetition, we an say that if A i F, 1 i n, then n i=1 A i F. In other words, F is losed under finite unions. Definition 1 A olletion of subsets whih inludes Ω and losed with respet to (i)omplements and (ii) finite unions is known as a field or an algebra. 1 attributed to A. N. Kolmogorov, the Russian mathematiian 2 let us forget the idiot box and sidduisms 2
On the other hand, if F is stable, then it needs to be losed under ountable unions, i.e. i=1 A i F. Thus ountable unions annot result in anything outside the event-spae, pointing to the stability or robustness of our spae. The rest of the set-operations an be written in terms of unions and omplements, and thus the definition of stability an be learly stated in the first two operations. Notie that stability with respet to ountably infinite unions is an extremely important property, whih is neessary in many irumstanes. We provide a simple example to differentiate between finite and infinite (ountable) unions. Example 1 Let Ω = [0, 1) and onsider intervals in R of the form A i = [0 + 1 i+2, 1), i N. For any finite n, we know that i n A i = [ 1 n+2, 1), whih is losed on the left. However, i 2 A i = (0, 1), whih is open at the left, and thus not inluded in the set of subintervals of the form [a, b). For our purposes, we demand that the field be stable with respet to ountably infinite unions too. So far we have used the terminology stable to onvey a more intuitive notion familiar from our eletrial engineering bakground. We will now give a proper definition to the event spae that we are interested and drop the word stable from further usage. Definition 2 A olletion of subsets whih inludes Ω and losed with respet to (i)omplements and (ii) ountable unions is known as a sigma- field or a sigma-algebra. From now onward, the meaning of F is a σ-field should be lear, and we may, at times, will simply say F, that it is a σ field will be lear from the ontext. 3.3 Event-Axioms We will summarize the natural desired properties of the event-spae as event-axioms. While this is rephrasing some of the statements in the previous sub-setion, their importane warrants a repetition. 1. Axiom-I: Ω F. 2. Axiom-II: A F A F. 3. Axiom-III: For any arbitrary ountable index set T, A i F, i T A i F. In literature you may find a different set of axioms defining the event-spae. This should not be ausing onfusion, as the whole spae an be onstruted based on either of the axioms. For example, many text-books replae axiom-i by F. This is ertainly true in the light of axioms I and II above, as Ω F and = Ω F ( will be alled the null-set or emptyset). To emphasize again, the set T ontains arbitrary indies, but ountable. Thus T an be the set of all even numbers, or the olletion of primes (is this ountably infinite?) or finite sets as in T = {i N 10 i 10000}, so on and so forth. 3
3.4 De Morgan s Laws Let us start with a simple exerise. Exerise 1 Simplify the expressions below. (i)a, (ii)a, (iii) A A, (iv) A A, (v) A (B A), (vi) A (B A). Familiarity with most of these operations will be assumed in this ourse 3. A set of laws, though familiar, we will devote some time are the De Morgan s laws. Law I: Law II: (A B) = A B (A B) = A B We are in fat interested in the extension of these laws to operations over arbitrary ountable set of indexes. Extension Laws: Proof: ( A i ) ( A i ) x ( A i ) x A i, i T x A i, i T = A i (1) = A i (2) x A i. Exerise 2 Prove the seond extension stated above. 3.5 Examples of σ fields 1. Trivial σ field : {Ω, }. 2. Gross σ field : P(Ω), the power-set of Ω. This is the most useful σ-field when Ω is finite or ountable. However, keeping trak of the events in P(Ω) for real-valued Ω an turn to be a mundane, if not impossible, task(this is even true when Ω = N). 3. A non-trivial example: {Ω, A, A, } 4. Borel σ-field : B, this is the single most important σ field when Ω is a real interval or retangle (higher dimensional) in Cartesian oordinates. Exerise 3 Verify that the third example is indeed a σ field. 3 otherwise it is time that you grab the set-theory books 4
3.6 Reipe for Finite Sigma Fields There are indeed many sigma fields ontaining a lass of events, let us say, E. The larity of understanding an be helped by onstruting a reipe by whih one an onstrut sigmafields. More preisely, what we onstrut are more aptly fields, finite sets whih are losed with respet to omplements and unions. For example, we have already onstruted a field of size 4 above. Let us onstrut a field of size 8 now. Example 2 Let A and B are disjoint subsets of Ω and onsider C = (A B). Let us enumerate a 3-bit binary numbering system, and hoose the subset in F using these bits as shown below. 0 0 0-0 0 1 - A 0 1 0 - B 0 1 1 - A B 1 0 0 - C 1 0 1 - A C 1 1 0 - B C 1 1 1 - Ω It is easy to verify that this is a sigma-field and one is free to hoose the disjoint sets A and B at onveniene. Exerise 4 Construt a sigma-field whih has 16 events in it. 3.7 Mutually Exlusive Events The events A 1,, A n are mutually exlusive (or pairwise disjoint 4 ) if at most one of them is true. i.e. no two share a ommon outome of the experiment. 4 Probability Axioms A i happened A j, j i did NOT happen. In addition to the three event axioms that we had desribed, we will also outline a set of axioms on the third entity of the probability spae, namely the probability measure P ( ). Axiom-I: Axiom-II: Axiom-III: For any event A, P (A) 0. For any ountable index set T, and a olletion A i, i T of mutually exlusive events, P ( A i ) = P (A i ). The sample-spae Ω F is ertain, i.e. P (Ω) = 1. We will inlude some disussion on ountable sets and reals in the oming leture notes. To be frank, there is not muh need to know anything more than the distintion between ountable and unountable, throughout the ourse, one we are profiient enough to avoid any pit-falls. So feel free to onentrate on the rest of the aspets if this ountable disussion is not appealing to you 4 pairwise disjoint is a stronger notion than disjoint. The latter means the intersetion of the given sets is, whereas the former says that the intersetion of any two sets is. 5