Random Signals and Systems Chapter 1 Jitendra K Tugnait James B Davis Professor Department of Electrical & Computer Engineering Auburn University
2
3
Descriptions of Probability Relative frequency approach» Physical approach» Intuitive» Limited to relatively simple problems Axiomatic approach» Mathematical approach» Theoretical» Can handle complicated problems» ELEC 3800 is based on this approach 4
Elementary Set Theory A set is a collection of elements Ex: A={1,2,3,4,5,6} set B is a subset of A if all elements of B are in A Ex: B={1,2,3} We denote this as A B A B empty set: equality: A=B iff and A B B A 5
Elementary Set Theory cont d union» sum» Logical OR A B A B A B intersection» product» Logical AND A B mutually exclusive if A B A B 6
Elementary Set Theory cont d complement A B AB A B AB A A difference A B AB A AB A B 7
Axiomatic Approach A probability space outcomes of an experiment S is the set of all possible Ex: rolling a six-sided die S= {1,2,3,4,5,6} An event is a subset of the space S is the certain event is the impossible event S an event consisting of a single element is called an elementary event 8
Axiomatic Approach cont d To each event, we assign a probability P ( A ) Axioms of Probability 1) PA ( ) 0 2) P S 1 3) If A B,then P AB P A P B All probability theory can be derived from these axioms. 9
Axiomatic Approach - Example The third axiom describes how to compute the probability of the sum of mutually exclusive events: P AB P A P B How do we add non-mutually exclusive events? 10
Axiomatic Approach - Example Suppose A and B are not mutually exclusive P AB P A AB A P A P AB also B AB AB P B P A B P A B P A B P B P A B PA B P A P B P AB B 11
Axiomatic Approach cont d Ex: six-sided die S= {1,2,3,4,5,6} i 1 6 P R i=1,,6 A {1, 3} {1} {3} (mutually exclusive) 1 1 1 PA P{1} P{3} 6 6 3 12
Axiomatic Approach cont d Ex: six-sided die S= {1,2,3,4,5,6} A {1, 3} B {3,5} A B {3} AB {1,3,5} PA B PA P B P AB 1 1 1 1 3 3 6 2 or P AB P{1} P{3} P{5} 1 1 1 1 6 6 6 2 13
Axiomatic Approach cont d Ex: A dodecahedron is a solid with twelve sides and is often used to display the twelve months of the year. When this object is rolled, let the outcome be taken as the month appearing on the upper face. Also let A={January}, B={Any month with exactly 30 days}, and C={Any month with exactly 31 days}. Find P A C P A C P B C P B C Months with 31 days: January, March, May, July, August, October, December (7) Months with 30 days: April, June, September, November (4) 14
Axiomatic Approach cont d Ex: A dodecahedron is a solid with twelve sides and is often used to display the twelve months of the year. When this object is rolled, let the outcome be taken as the month appearing on the upper face. Also let A={January}, B={Any month with 30 days}, and C={Any month with 31 days}. Find 1 P AC P(January) 12 P AC PA ( ) PC ( ) PA ( C ) 1 7 1 12 12 12 7 12 15
Axiomatic Approach cont d Ex: A dodecahedron is a solid with twelve sides and is often used to display the twelve months of the year. When this object is rolled, let the outcome be taken as the month appearing on the upper face. Also let A={January}, B={Any month with 30 days}, and C={Any month with 31 days}. Find PB C P( ) 0 P BC P( B) P( C) P( BC ) 4 7 12 12 11 12 0 16
Axiomatic Approach cont d Three non-mutually exclusive events A C PA B C P A P BP C P AB P AC P BC P AB C B 17
Probability of a Complement PB ( )? S BB PS ( ) PB ( B ) 1 PB ( ) PB ( ) PB ( ) 1 PB ( ) 18
1-7.1 1-7.2 1-7.5 Homework 19
Conditional Probability P A B PB 0 PB P A B We can show that a conditional probability is a really probability satisfying: S P A B 0 P B 1 If AC, P AC B P A B P C B 20
Conditional Probability cont d Ex: six-sided die A {1, 2} B {2, 4,6} find the probability of rolling a 1 or 2 given that the outcome is an even number / P A B PB PB P A B P {2} 1 6 1 12 3 Similarly, probability of even number given 1 or 2 / P B A PA P A B 16 1 13 2 Note that, except in special cases, P(A B) P(B A) 21
Example Ex: A manufacturer of electronic equipment purchases 1000 ICs from supplier A, 2000 ICs from supplier B, and 3000 ICs from supplier C. Testing reveals that the conditional probability of an IC failing during burn-in is, for devices from each of the suppliers PF A 0.1 P F B 0.05 P F C 0.08 The ICs from all suppliers are mixed together and one device is selected at random. a) What is the probability that it will fail during burn-in? b) Given that the device fails, what is the probability that the device came from supplier A? 22
Total Probability Suppose we have n mutually exclusive events A1 An that span the probability space S A1A 2 An S Consider S an event B S 1 2 n B B A B A B A P B P B A P B A P B A 1 2 PB A i PB / AiP Ai (from conditional probability) n Then, PB PB A1P A1 P B A2 P A2 P B An P An (total probability) 23
Example A candy machine has 10 buttons» 1 button never works» 2 buttons work half the time» 7 buttons work all the time A coin is inserted and a button is pushed at random What is the probability no candy is received? No candy (NC) can happen two ways:» Push button that never works (100%)» Push one of the buttons that work half the time (50%) We must weight each possibility by the chance it occurs» Button that never works (10%)» One of the buttons that work half the time (20%) 24
Example P ( B never ) 0.10 NC PB ( ) 0.20 B never half P ( NC B ) 1.00 never PNC ( B ) 0.50 half B half B good P ( NC ) PNC ( B ) PB ( ) PNC ( B ) PB ( ) never never half half (1.00)(0.10) (0.50)(0.20) 0.20 25
Example Ex: A manufacturer of electronic equipment purchases 1000 ICs from supplier A, 2000 ICs from supplier B, and 3000 ICs from supplier C. Testing reveals that the conditional probability of an IC failing during burn-in is, for devices from each of the suppliers PF A 0.1 P F B 0.05 P F C 0.08 The ICs from all suppliers are mixed together and one device is selected at random. a) What is the probability that it will fail during burn-in? b) Given that the device fails, what is the probability that the device came from supplier A? 26
Total Probability cont d F A B C P A P B P C are known a priori probabilities because they are known before the experiment is performed PA F is called an a posteriori probability because it is applied after the experiment is performed 27
Total Probability cont d We can relate the a priori probability to the a posteriori probability by: / / P A B P A B P B P B A P A i i i i / P A B i / i i PB P B A P A P B P B/ A1 P A1 P B/ A2 P A2 P B/ An P An (from total probability) / P A B i P B/ AiP Ai / / / P B A P A P B A P A P B A P A 1 1 2 2 This is called Bayes Theorem. n n 28
Total Probability cont d Let s return to the problem: A manufacturer of electronic equipment purchases 1000 ICs from supplier A, 2000 ICs from supplier B, and 3000 ICs from supplier C. Testing reveals that the conditional probability of an IC failing during burn-in is, for devices from each of the suppliers PF / A 0.1 PF / B 0.05 PF / C 0.08 The ICs from all suppliers are mixed together and one device is selected at random. a) What is the probability that it will fail during burn-in? b) Given that the device fails, what is the probability that the device came from supplier A? 1 6 a) PA 3 P C 6 2 P B 6 P F P F/ A P A P F/ B P B P F/ C P C 1 2 3 0.1 0.05 0.08 0.0733 6 6 6 b) PA / F / PF P F A P A 1 0.1 6 0.2274 0.0733 29
Example A test for cancer has the following properties» If a person has cancer, the test will state they have cancer 95% of the time. Medical literature: sensitivity=95%» If a person does not have cancer, the test will state they do not have cancer 95% of the time. Medical literature specificity=95% The cancer rate in 20-29 year olds is 0.2076%*» P ( Ctest C) 0.95 PC ( test C) 0.95 P ( C ) 0.002076 * All cancers, NIH NCI SEER Cancer Statistics Review 30
Example cont d QUESTION: Should this test be used to screen 20 year olds for cancer? Really what we are interested in is» If the test says I have cancer, what is the probability I really have cancer? Positive predictive value (PPV)» If the test says I do not have cancer, what is the probability I really do not have cancer? Negative predictive value (NPV) P ( C Ctest ) PC ( Ctest ) 31
Example cont d PC ( C ) test P ( Ctest C) P( C) PC ( ) test PC ( CPC ) ( ) test PC ( CPC ) ( ) PC ( CPC ) ( ) test test 0.95 0.002076 0.95 0.002076 0.05 (1 0.002076) 0.038 If a 20-29 year old tests positive for cancer, there is only a 3.8% chance they actually have cancer. CONCLUSION: Do not use test for screening 20-29 year olds. 32
Example cont d Age Group PPV (%) 20-29 3.8023 30-39 9.2326 40-49 19.2343 50-59 36.8524 60-69 56.8987 70-79 69.1181 80+ 69.2864 33
Example cont d 10,000 people P(cancer)=0.002 (2%)» 20 have cancer» 9980 do not have cancer Ctest Ctest C C 19 PC ( Ctest ) 0.037 (3.7%) 518 19 499 518 1 9481 9482 20 9980 34
1-8.1 1-8.2 Homework 35
Independence Example: Flip a coin two times» What is the probability of two heads?» P(H,H)=P(H)P(H)=(0.5)(0.5)=0.25 Previously, we found that if two events are independent, their joint probability is the product of their marginal probabilities. 36
Independence Two events A and B are independent if and only if (iff) P AB P AP B In some cases, independence is assumed or determined by the physics of the situation In other cases, independence can be established mathematically by showing that the above equation holds. 37
Independence Three events, A 1, A 2, A 3, are independent iff 1 2 1 2 1 3 1 3 2 3 2 3 P A A P A P A P A A P A P A P A A P A P A P A A A P A P A P A 1 2 3 1 2 3 pair wise comparison is not sufficient. For n n events, 2 n 1 comparisons are required 38
Independence cont d Ex: rolling a pair of dice: event A: getting a total of 7 event B: getting a total of 11 Are A and B independent? NO! - mutually exclusive events are never independent! (if one occurs, the other can not). 39
Independence cont d Ex: rolling a pair of dice event A: getting a total of odd number event B: getting a total of 11 (these events are not mutually exclusive) since B A, A B B 2 1 36 18 P A P A B P B 1 2 P AB 1 1 1 P A P B 18 2 18 Thus, A and B are not statistically independent. 40
Independence cont d Ex: rolling single die A {1, 2,3} B {3, 4} P A P B 1 2 1 3 1 1 1 6 2 3 {3} P AB P P A P B A and B are independent, although the physical significance of this is not intuitively clear. 41
Independence cont d Ex: A card is selected at random from a standard deck of 52 cards. Let A be the event of selecting an ace, and let B the event of selecting a spade. Are these events statistically independent? 4 52 13 52 PA PB P A B 1 52 P A P B 4 13 1 P AB 52 52 52 A and B are independent. 42
Independence Complements» If A and B are independent, then so are A and B A and B A and B» Proof: P ( A) P([ AB] [ AB ]) P ( A) P( AB) P( AB) PA ( B ) PA ( ) PA ( B ) PA ( ) PAPB ( ) ( ) PA ( )[1 PB ( )] PAPB ( ) ( ) 43
Probability of a Difference For any events A and B, PA ( B) PA ( B ) PA ( ) PA ( B ) Note: PA ( ) PA ( B ) PA ( B ) PA ( B ) PA ( ) PA ( B ) 44
Independence cont d Ex: In the switching circuit shown below, the switches are assumed to operate randomly and independently. B C Path 2 A D Path 1 If each switch has a probability of 0.2 of being closed, find the probability that there is a complete path through the circuit. (Path 1: A and D are closed Path 2: A, B and C are closed) 45
Independence cont d event P1: path 1 complete event P2: path 2 complete P P1P 2 P P1 P P2 P P1P 2 1 P P 2 P P 1 1 1 5 5 25 1 1 1 1 5 5 5 125 independence used 1 1 1 1 1 PP 1P 2 5 5 5 5 625 1 1 1 PP 1 P 2 0.0464 25 125 625 46
1-9.2 Homework 47
Combined Experiments So far, the probability space, S, has been associated with a single experiment. Now consider two experiments:» Rolling a die and flipping a coin» Rolling a die 2 times We call these experiments combined experiments 48
Combined Experiments Suppose that there are two experiments: S S 1 2 {,, } 1 2 {,, } 1 2 n m The probability space of the combined experiment is the Cartesian product of the two spaces: S= S 1 S 2 The elements of S are the ordered pairs,,,,, 1 1 2 2 n m 49
Combined Experiments cont d Ex: Let s take the experiments of rolling a die and tossing a coin S S 1 2 {1,2,3,4,5,6} {H,T} S= S 1, H, 1, T, 2, H, 6, T 1 S 2 Same things hold for subsets (events): If subset A is an event in 1 S 1 If subset is an event in A 2 S 2 then is an event in 1 2 A A A S 50
Combined Experiments cont d Ex: A A 1 2 {1, 3} {H} A A A 1 2 1, H, 3, H If the two experiments are independent, then: P A P A1A2 P A1 P A2 1 PA 1 3 1 P A 2 2 1 2 1 1 1 3 2 6 P A P A P A 51
Bernoulli Trials The same experiment is repeated n times and it is desired to find the probability that a particular event occurs exactly k of these times. P(exactly two 4 s in any order in 3 rolls)=? Let A rolling a 4 A not rolling a 4 P A P A 1 6 5 6 AAA Total of 8 possible outcomes, but only 3 of them have exactly two 4 s 1 1 5 prob AAA 6 6 6 prob 1 5 1 P(exactly two 4 s in any order in 3 rolls) AAA 6 6 6 prob 2 1 1 5 3 6 6 (the three situations are mutually exclusive so we can sum them) 5 1 1 6 6 6 53
Bernoulli Trials cont d In general, probability event A occurs k times in n trials can be given as: n Pn k p q k k n k where p P A q PA 1 p Assumptions: 1) independent events 2) p and q are same for each event n n! k k! n k! 54
Binomial Coefficients ( a b) n 1 1 1 1 2 1 1 3 3 1 1 4 6 4 1 1 5 10 10 5 1 Pascal s Triangle 56
Bernoulli Trials cont d Ex: 32-bit words in memory 3 10 probability of an incorrect bit p q 0.001 0.999 32 1 31 P(1 error in a word) P 32 0.031 1 0.001 0.999 1 32 0 32 P(no error) P 32 0.9685 0 0.001 0.999 0 57
Bernoulli Trials cont d Ex: toss a coin 4 times and what is P(at least two heads)? P(at least two heads) P4 2 P4 3P44 P 4 P 4 P 4 2 3 4 2 2 4 1 1 3 2 2 2 8 3 1 4 1 1 1 3 2 2 4 4 0 4 1 1 1 4 2 2 16 P(at least two heads) 3 1 1 11 8 4 16 16 58
Bernoulli Trials cont d Ex: toss a pair of dice 8 times a) Probability of a 7 exactly 4 times A: rolling 7 A={(1,6),(6,1),(2,5),(5,2),(3,4),(4,3)} b) An 11 occurs 2 times B: rolling 11 B={(5,6),(6,5)} 8 6 1 PA ( ) p 8 36 6 1 5 1 6 6 P A q P 2 1 36 18 1 17 1 18 18 P B p P B q P 4 4 8 1 5 4 0.02605 4 6 6 2 6 8 1 17 2 0.0613 2 18 18 59
Bernoulli Trials cont d c) Probability of a 12 more than once C: rolling 12 B={(6,6)} 1 PC p 36 1 35 q 1 36 36 P(more than one 12) P 8 P 8 8 35 0 0.7982 36 1 P8 0 P8 1 1 7 8 1 35 1 0.1825 1 36 36 P(more than one 12) 1 0.7982 0.1825 0.0193 60
Example Suppose you are a design engineer for a Mars spacecraft.» Need to communicate with Earth» Use 12-bit words» The probability of a single bit error is 0.001 Can use two types of codes» No code Send 12 data bits in a 12-bit codeword» Extended Golay code Encode 12 data bits in a 24-bit codeword Can correct errors of 3 bits or less Which code is more reliable? 61
Example cont d No code (12 data bits in 12 code bits)» Let A = probability of a one-bit error in transmission p = 0.001 q = 0.999 Probability of a transmission error 12 0 12 P(error) 1 p12 (0) 1 p q 0 1 (0.001) (0.999) 1 0.9881 0.012 0 12 62
Example cont d Extended Golay code (12 data bits in 24 code bits, can correct 3-bit errors) Probability of a transmission error P(error) 1 [ p (0) p (1) p (2) p (3)] 24 24 24 24 24 24 24 24 p (0) pq p (1) pq p (2) pq p (3) pq 0 1 2 3 0.9763 0.0235 =2.70 10 =1.98 10 0 24 1 23 2 22 3 21 24 24 24 24-4 -6 P(error) 1.04 10 8 Use Extended Golay code! This code was used by the Voyager spacecraft to send back images of Saturn and Jupiter. 63
DeMoivre-Laplace Theorem When n gets large, it is difficult to compute P n (k). Use the DeMoivre-Laplace approximation If 1) npq 1 2) k np is on the order of or less than npq n k n k 1 Pn k p q e k 2 npq 2 2 k np npq This theorem is used to simplify the evaluation of binomial coefficients and the large powers of p and q by approximating them. 64
DeMoivre-Laplace Theorem cont d Ex: 8000 character file transfer 0.001 chance of one character error a) P(no error) 8000 4 1 0.001 3.34 10 n 8000 p 0.001 q 0.999 8000 0.001 0.999 10 b) P(exactly 10 errors) 1 10 7990 2 8000 0.001 0.999 0.1099 c) What should be p such that P(no error) 0.99 p 8000 e 10 8 2 16 0.999 1 0.99 p 8000 log 1 log 0.99 1 p 10 log 0.99 8000 p 1 10 1.256 10 log 0.99 8000 6 65