ECE5: Stochastic Signals and Systems Fall 8 Lecture - September 6, 8 Prof. Salim El Rouayheb Scribe: Peiwen Tian, Lu Liu, Ghadir Ayache Chapter : Introduction to Probability Theory Axioms of Probability Definition (Probability Space). The Probability space is defined by a triplet (Ω, F, P), where: Ω is the Sample space F is the set of Events P is the Probability function Definition (Sample space). The Sample space, Ω, is the set of all possible outcomes of a random experiment. Example. When we toss a coin, all the possible outcomes are Heads or Tails. Therefore, the sample space of a coin tossing is Ω = {Head, Tail}. Example. When we toss a die, one of the 6 faces is going to come up. Therefore, the sample space of a die tossing is Ω = {,,,, 5, 6}. Example. Suppose that we want to measure the temperature a Thursday afternoon in September. Then Ω = R. Definition (Event). An event E is a subset of the sample space, i.e., E Ω. Example. Coin Tossing: The Event of getting a Head is E = {H} Example 5. Die Tossing: The Event of getting a multiple of is E = {, 6} Example 6. Temperature measurement: The Event of getting a temperature between 7 o F and 9 o F is E = [7, 9] Example 7. If we toss a fair coin twice, then the sample space is Ω = {HH, HT, T H, T T }. Consider the event A at least one Head occurs ; then, the event is A = {HH, HT, T H}. Let B be the event of tossing the coin repeatedly until a Head occurs. Then, B = {H, T H, T T H,... }. Let C be the event of tossing the coin an even number of times until a Head occurs. Then, C = {T H, T T T H,... }.
Definition (F). F is the set of all events Typically when Ω is countable, F is the set of all subsets of Ω, i.e., the power set of Ω. But this is not the case for Ω uncountable, where F is going to be too large and there will often be sets to which it will be impossible to assign a unique measure like in Ω = R. (Check Definition 5. and Remark.) Remark. F must be a σ algebra such that. Ω F,. If A F, then its complement set A C F,. if A i F for all i =,..., then i= A i F Corollary. From De Morgan s Law and the previous property we get that if A i F for all i =,..., then i= A i F Example 8. Temperature measurement How to prove that (a, b) is an Event when [a, b] F? If E = [a, b] F, then a & b F. T hen [a, b] {ā} { b} F which is (a, b). Definition 5 (Borel Set). Borel sets are the sets that can be constructed from open or closed sets by repeatedly taking countable unions and intersections. Formally, Borel algebra is the smallest σ algebra that makes all open sets measurable. Remark. Not all subsets of Ω are events. You can define sets that have no probability. In such a case, we have to use the smallest σ algebra called Borel algebra that contains all closed intervals For this class, any subset of Ω is an event. Definition 6 (Axioms of probability). A probability measure P on Ω is a function such that it satisfies the following properties: P : Ω [, ], E P (E), () P ( ) =. () P (Ω) =. () If A, A, A... are disjoint subsets of Ω, ( ) P A i = i= i P (A i ). Here, Ω is the power set of Ω. Lemma. Let A and B be two subsets of Ω. We define have: Ā to be the complement of A in Ω, we
(a) P (Ā) = P (A). (b) If A B, then P (A) P (B). (c) P (A B) = P (A) + P (B) P (A B). Proof. For part(a), For part(b), P (A Ā) = P (Ω) = and A, Ā are disjoint P (A) + P (Ā) = P (Ā) = p(a) B = A (B \ A) P (B) = P (A) + P (B \ A) P (A) For part(c), Now, P (A B) = P (A) + P (B \ A) = P (A) + P (B \ A B). (A B) B P (B \ A B) = P (B) P (A B). Lemma (Union bound). Let A and B be two subsets of Ω, then In general, P (A B) P (A) + P (B). ( n ) P A i i= n P (A i ). Example 9 (Tossing a Die (a)). A : The result number is a multiple of. A : The result number is a multiple of. A = {,, 6}, P (A ) =. A = {, 6}, P (A ) =. i= P (A A ) P (A ) + P (A ) = + = 5 6 In fact, A A = {,,, 6} and P (A A ) =. Example (Tossing a Die (b)). A : The result number is less than or equal to. A : The result number is prime. A = {,, 5, 6}, P (A ) =. A = {,, 5}, P (A ) =. P (A A ) P (A ) + P (A ) = + = 7 6 > In fact, A A = {,,, 5, 6} and P (A A ) = 5 6.
Figure : Graph connection. Example (Random Graphs). Consider the graph G = (V, E) over vertices, given in Figure, where V = {,,, } is the vertex set and E = {{, }, {, }, {, }} is the edge set. A random graph G defined over the vertex set V is a graph where an edge between any two vertices exists with a probability p. If we take a graph on n vertices and the edge exists between vertices n(n ) with probability=p=.5. Then the number of subsets of V of size = = ( n ). The number of subsets of V of size k = ( n k). If we have vertices in a graph.what is the probability that vertex is connected to k other nodes? (a) vertices,, are connected to each other (b) vertices, are connected to each other (c) vertices, are connected to each other (d) vertices,,, connected to each other Figure : Graph connection for vertices Let N be the neighbors of vertex, N = φ in fig(a), N = {, } in fig(b), N = {, } in fig(c), N = {,, } in fig(d). Then we define the event A N is that the vertex is connected to the vertices in N We say vertex is connected to k other vertices, if k =, all the possible graph are as (Figure ).
(a): Node connected to and (b): Node connected to and. (c): Node connected to and. Figure : vertex is connected to two vertices Define event A vertex is connected to other vertices,therefore: The probability of this event A is A = A {,} A {,} A {,}. P (A) = P (A {,} ) + P (A {,} ) + P (A {,} ). The probability of vertex is connected to vertex and is therefore, P (A {,} ) = ( ) = p ( p) = P (A {,} ) = P (A {,} ), P (A) = p ( p). In general, the probability vertex is connected to k specific vertices is P (A N ) = p k ( p) n k. The probability vertex is connected to k other vertices is P (A) = ΣP (A N ), ( ) n = p k ( p) n k. k. Conditional Probability Example. Let A be the event of tossing two fair dice such that the total exceeds 6. (a) Find P (A). The set of all events Ω is given by the following set: (, ), (, ),... (, ), (, ), (, ),... (, 6), Ω =....... (6, ), (6, ),... (6, 6). 5
Now, we need to find P (A). All the possible outcomes of A (total exceeds 6) are: A = {(, 6) (, 5), (, 6) (, ), (, 5), (, 6) (, ), (, ), (, 5), (, 6) (5, ), (5, ), (5, ), (5, 5), (5, 6) (6, ), (6, ), (6, ), (6, ), (6, 5), (6, 6)}. Therefore P (A) = e A P (e) fair dice = 6 (b) Let B the event that the first dice is. Find P (B). All the possible outcomes of event B are: B = {(, ), (, ), (, ), (, ), (, 5), (, 6)}. () Then all the possible outcomes of event A given B are the events in equation () satisfying A (total exceeds 6), hence (A B) = {(, ), (, 5), (, 6)}. (c) What is the probability of Total exceeds 6 given that the first dice is we can find that by P (A B) = 6, P (A B) = P (A B). P (B) Definition 7 (Conditional probability). We define the conditional probability of an event A given that event B happened (with P (B) > ) by: P (A B) = P (A B). P (B) Definition 8 (Independent events). Two events A and B are independent iff In general, P (A B) = P (A)P (B). We can also say that the events A and B are independent iff P (A B) = P (A)P (B A) () = P (B)P (A B). () P (A B) = P (A), (P (B) ) P (B A) = P (B), (P (A) ). 6
T ransmitted bits X -ε Received bits Y ε ε -ε Figure : Binary Symmetric Channel (BSC) with probability of error P e = ε. Example (Binary symmetric channel). In the BSC of Fig. 8 the bits are flipped with probability ε (ε is called crossover probability), we can write ε = P (Y = X = ) = P (Y = X = ). Suppose the bits and are equal likely to be sent, i.e., P (X = ) = P (X = ) =.5, Q. Find the probability of sending a and receiving a. Ans. P (X =, Y = ) = P (X = )P (Y = X = ) =.5( ε).. Total Law of Probability Theorem. Let A, A,..., A n be n mutually disjoint events such that Ω = n A i (P (A i ) ), () i= then for any event B Ω we have P (B) = P (A )P (B A ) + P (A )P (B A ) +... + P (A n )P (B A n ). B... A A A n Figure 5: Total law of probability. 7
Proof. For n= B = (B A ) (B A ), (5) P (B) = P (B A ) + P (B A ), (6) = P (A )P (B A ) + P (A )P (B A ). (7) Example. (BSC) Consider a BSC in Fig. 6 with crossover probability ε =.. The probability of sending is. and the probability of sending is.6. Q. Find the probability of receiving a. X.9 Y...9 Figure 6: Binary Symmetric Channel with probability of error P e =.. Ans. The probability of sending is P (X = ) =.6, and the probability of sending is P (X = ) =.. Then if we want to know the probability of receiving, we can use the total law of probability to calculate P (Y = ), P (Y = ) = P (X = )P (Y = X = ) + P (X = )P (Y = X = ), = (.) (.9) + (.6) (.) =... Birthday paradox Question:What is the probability that at least students in class have the same birthday. E:at least students have the same birthday. Number of days per year is n, number of students in class is m. Ē: each student has distinct birthday. Answer: We know that P (Ē) = ( n ) ( n ) ( m n ). k n e k n, k n. 8
Then, P (Ē) = e n e n e m n, = exp( ( + + + m )), n ) m(m = e n, e m n. Now we have student m = 5, and number of birthdays n = 65. P (E) e 65, 5 96.7%. Question:How big the class should be if the probability of students have same birthday is larger than 5%? Answer: Then P (E) =. e m n =, so m n = ln, m = ln n,. So we need approximately students in same class to make the probability that at least students have the same birthday is larger than. Theorem (Baye s Theorem). P (A i B) = P (B A i )P (A i ) n i= P (B A i)p (A i ). (8) Example 5 (BSC). In this case we have P (X = ) = P (X = ) = transmitted) Suppose we observe Y =. What value of X should we decode? P (X = Y = ) = P (X =, Y = ). P (Y = ) (s and s are equal likely 9
B... A A A n Figure 7: Baye s theorem. T ransmitted bits X -ε Received bits Y ε ε -ε Figure 8: Binary Symmetric Channel (BSC) with probability of error P e = ɛ. According to the Baye s theorem P (X = Y = ) = P (X = )P (Y = X = ) P (X = )P (Y = X = ) + P (X = )P (Y = X = ), =.5( ε).5ε +.5( ε), = ε.