ECE353: Probability and Random Processes Lecture 2 - Set Theory Xiao Fu School of Electrical Engineering and Computer Science Oregon State University E-mail: xiao.fu@oregonstate.edu January 10, 2018
Set Theory Set theory is the mathematical basis of proabability. A set contains elements, e.g., A = {1, 2, 3}, B = {h, t}. In an experiment, S is a set of elementary outcomes. A subset of S, i.e., A S, is an event, where A S reads A is a subset of or equal to S. We use lower-case letters to denote the elements in sets, e.g., x A means x belongs to A. ECE353 Probability and Random Processes X. Fu, School of EECS, Oregon State University 1
Union Union of A and B is A B = {x S x A or x B}, where A and B are events and S is the entire sample space. x A B if and only if x A or x B. ECE353 Probability and Random Processes X. Fu, School of EECS, Oregon State University 2
Intersection Intersection of A and B is A B = {x S x A and x B}, where A and B are events and S is the entire sample space. x A B if and only if x A and x B. ECE353 Probability and Random Processes X. Fu, School of EECS, Oregon State University 3
Complement of A is denoted as Complement A c = {x S x / A}. note that (A c ) c = A. ECE353 Probability and Random Processes X. Fu, School of EECS, Oregon State University 4
Empty Set Empty set, denoted as, is defined through its properties 1. A = A. 2. A =. 3. c = S. ECE353 Probability and Random Processes X. Fu, School of EECS, Oregon State University 5
Set Difference Set difference of A and B is A B = {x S x A and x / B } = {x S x A and x B c } Note that A B B A A symmetrical set difference notion of A and B is A\B = {x S (x A or x B) and (x / A B)} Note that A\B = B\A ECE353 Probability and Random Processes X. Fu, School of EECS, Oregon State University 6
Theorem: De Morgan s Law De Morgan s Law (A B) c = A c B c. a Theorem is something that we need to prove from Axioms and Definations. Proof: the basic idea is to show (A B) c A c B c and A c B c (A B) c hold simultaneously. Assume x (A B) c. This means that x / A B, which means x / A and x / B. Together, this means that x A c and x B c, which leads to x A c B c. Assume x A c B c. Then we know that x A c and x B c. This is equivalent to x / A and x / B. By definition it means that x / A B, which means that x (A B) c. ECE353 Probability and Random Processes X. Fu, School of EECS, Oregon State University 7
Disjoint Sets Sample Space, S, is the set of all elementary outcomes. Event, E, is a subset of S. Definition: Two sets are disjoint if and only if the intersection is empty. Definition: Sets A 1,..., A n are mutually disjoint if and only if A i A j =, for all i, j. Definition: If A 1,..., A N are collectively exhaustive if their union is S, i.e., A 1... A n = S or n i=1 A i = S. The above definition of collectively exhaustive can be generalized to countably infinite number of A i s, i.e., using i=1a i = S. ECE353 Probability and Random Processes X. Fu, School of EECS, Oregon State University 8
Partition of Sample Space If {A i } i=1 are both mutually disjoint and collectively exhaustive, then we call them a partition of S. A puzzle pattern, or a mosaic of the sample space S. ECE353 Probability and Random Processes X. Fu, School of EECS, Oregon State University 9
Remarks The book calls a partition an event space. We will not follow this terminology for several reasons. We will stick with partition. Why do we care about partition? Basic idea is that using partition we can express any event as a union of disjoint events, which may make the calculation of probability much easier. ECE353 Probability and Random Processes X. Fu, School of EECS, Oregon State University 10
Axioms of Probability Theory mathematics cannot tell you what it is, but what would be if Probability Theory is based on several Axioms. Axiom 1: Probability of any event A S is greater than or equal to 0, i.e., P [A] 0, A S. Axiom 2: P [S] = 1. ( something will happen ). Axiom 3: For any countable collection of mutually disjoint events A 1, A 2,..., we have P [A 1 A 2...] = P [A 1 ] + P [A 2 ] +... Theorems, propositions, lemmas, and corollaries need proof. ECE353 Probability and Random Processes X. Fu, School of EECS, Oregon State University 11
Basic Theorems of Probability Theory Theorem: P [ ] = 0. Proof: Since P [S ] = P [S]. Using Axiom 3, we know that P [S] + P [ ] = P [S] = P [ ] = 0. Theorem: P [A c ] = 1 P [A]. Proof: Note that A and A c are disjoint and that A A c = S. Hence, we have 1 = P [S] = P [A A c ] = P [A] + P [A c ] ECE353 Probability and Random Processes X. Fu, School of EECS, Oregon State University 12
Basic Theorems of Probability Theory Theorem: A B = P [A] P [B]. Proof: First note that B = (B A) A. Then, we have, by Axiom 3, that P [B] = P [B A] + P [A]. Also note that by Axiom 1 we have P [B A] 0. Therefore, it is obvious that P [B] P [A]. ECE353 Probability and Random Processes X. Fu, School of EECS, Oregon State University 13
Basic Theorems of Probability Theory Theorem: For every partition {B 1, B 2, B 3,...} of S, it holds that P [A] = i P [A B i ], A S. This is called Law of Total Probability extremely useful in engineering. ECE353 Probability and Random Processes X. Fu, School of EECS, Oregon State University 14
Conditional Probability Example: somebody is rolling a fair die behind a curtain. Sample Space S = {1, 2, 3, 4, 5, 6} P [{1}] = P [{2}] =... = P [{6}] = 1/6 is now the probability model. Q: what is the probability of P [{2, 3}] (or, equivalently, P [{2} {3}])? ECE353 Probability and Random Processes X. Fu, School of EECS, Oregon State University 15
Conditional Probability In practice, we sometimes can get side information from somewhere, which could change our probability model. Suppose that there is a person who tells you that this time you have an even number. Then, what is the probability of getting 3? What is the probability of getting 2? P [{3} Even] = P [{3} {2, 4, 6}] = 0. P [{2} Even] = P [{2} {2, 4, 6}] = 1/3. ECE353 Probability and Random Processes X. Fu, School of EECS, Oregon State University 16
Consider two events A and B. More on Conditional Probability Assume that we have the side information that B has already happened, we would like to know what is the probability of A conditioned on that B has happened, i.e., P [A B]. ECE353 Probability and Random Processes X. Fu, School of EECS, Oregon State University 17
Consider two events A and B. More on Conditional Probability Assume that we have the side information that B has already happened, we would like to know what is the probability of A conditioned on that B has happened, i.e., P [A B]. Conjecture: P [A B] = P [A B]? ECE353 Probability and Random Processes X. Fu, School of EECS, Oregon State University 18
More on Conditional Probability P [A B] = P [A B] is not correct for a couple of reasons: P [A B] should be larger than P [A B] by intuition. When A is a superset of B, P [A B] = 1 but P [A B] = P [B]. Intuition: we should scale (normalize) P [A B] to represent P [A B]. Definition: the probability of A conditioned on B is P [A B] = P [A B] P [B]. ECE353 Probability and Random Processes X. Fu, School of EECS, Oregon State University 19
First, 0 < P [B] 1. Then, More on Conditional Probability P [A B] = i.e., the first bug is fixed. Second, when B A, then P [A B] = The second bug is fixed. P [A B] P [B] P [A B] P [B] P [A B]; = P [B] P [B] = 1. Example: Let s go back to the rolling dice example. P [{4} Even] = P [{4 Even}] P [Even] = 1/6 1/2 = 1/3. ECE353 Probability and Random Processes X. Fu, School of EECS, Oregon State University 20
Remarks on Conditional Probability Remark 1: The definition of conditional probability automatically assumes that P [B] > 0. Remark 2: P [A B] itself is a respectable probability measure, i.e., P [A B] satisifes Axioms 1-3 of Probability Theory. Remark 3: If A B =, then we have P [A B] = 0. ECE353 Probability and Random Processes X. Fu, School of EECS, Oregon State University 21