Chapter 3 Conditional Probability and Independence. Wen-Guey Tzeng Computer Science Department National Chiao Tung University

Transcription

1 Chapter 3 Conditional Probability and Independence Wen-Guey Tzeng Computer Science Department National Chiao Tung University

2 Conditional probability P(A B) = the probability of event A given the occurrence of event B Knowing that B has occurred changes the probability of A s occurrence D A Definition. If P(B)>0, the conditional probability of A given B, P( AB) P( A B) P( B) 2016 Fall 2 F S

3 Examples Pr(Sunny day) Pr(Sunny day summer) Pr(Sunny day winter) Pr(Pass the course) Pr(Pass the course study hard) Pr(Pass the course cut class) 2016 Fall 3

4 S (each sample has two attributes: number, color) Red P(Odd Red) = P(Red Odd) = P(>2) = P(>2 Red) = P(>2 Odd) = 2016 Fall 4

5 Example: A class of 80 students. 50 of them take Linear Algebra, 60 of them take Calculus, and 30 of them take both. Randomly choose a student and find that the student takes Calculus. What is the probability that the student takes Linear Algebra also? Sol: Linear Algebra Calculus S1 X 0 S2 0 0 S3 X 0 S4 0 X Let B be event that a selected student takes Calculus. You only consider these 60 students. Let A be the event that a selected student takes Linear Algebra. P(A B )= P(AB)/P(B) = (30/80) / (60/80) 2016 Fall 5

6 Example: Draw 8 cards from a deck of 52 cards. Given 3 of them are spades, what is the probability that the remaining 5 are spades? We want to compute P(the remaining 5 are also spades 3 are spades)=? Sol: S={2S 4H JH KD 5C 4C 8S AD, } permuations out of 52 cards --- all outcomes are equally likely B: the event that at least 3 of them are spades A: the event that the remaining 5 are spades P(A B) = P(AB)/P(B) = 5.44 x 10-6 P(AB)=C(13,8)x8!/(C(52,8)x8!) P(B)=[C(13,3)C(39,5)+C(13,4)C(39,4)+ +C(13,8)C(39,0)]x8! / (C(52,8)x8!) 2016 Fall 6

7 2016 Fall 7

8 Example: Box 1: 2 red balls and 3 green balls Box 2: 4 red balls and 1 green balls Choose a box with P(Box 1)=0.6, P(Box 2)=0.4, and then randomly choose two balls from the box. What is the sample space? What is the probability of the event that Box 1 is chosen and two balls are both red? the event that two red balls are chosen under the condition that Box 2 is chosen? the event that one red and one green are chosen? the event that Box 1 is chosen under the condition that two balls are red and green? 2016 Fall 8

9 Contingency table P(BiDj) D1: balls are RR D2: balls are RG D3: balls are GG B1: Box 1 is chosen B2: Box 2 is chosen 2016 Fall 9

10 Conditional probability = reduction of sample space A S C P(. B) S = B: New sample space New probabilities for events 2016 Fall 10

11 S (each sample has two attributes: number, color) Red Adjusted probability Fall 11 5

12 All probability theorems hold under conditional probability, for P(B) 0, P(A c B) = 1-P(A B) P(E F B) = P(E B)+P(F B)-P(EF B) IF C A, P(C B) P(A B) Inclusion-exclusion principle 2016 Fall 12

13 Example: (revisited) Draw 8 cards from a deck of 52 cards. Given 3 of them are spades, what is the probability that the remaining 5 are spades? We want to compute P(the remaining 5 are also spades 3 are spades)=? Sol: 2016 Fall 13

14 Example: A child mixes 10 good and 3 dead batteries. To find the dead battery, the father tests batteries one by one and without replacement. If the first 4 batteries are good, what is the probability that the fifth is dead? Sol: Old sample space S = {all sequences of 10 good and 3 dead batteries} P(the fifth is dead the first 4 are good) New sample space S = {all sequences of 6 good and 3 dead batteries} P(the first is dead) -- in the new sample space S =3/ Fall 14

15 A mind game: There are three boxes. One contains a 100-dollar bill and the other two are empty. You select one of them randomly. What is the probability that you win 100 dollars? I open an empty box that is not selected by you. Will you switch to the other un-opened box? Yes, always switch. Why? No, never switch. Why? Randomly, why? Fall 15

16 Sol: Assume that the bill is in box 1. reduce the sample space Let outcome = (x, y, z) x: your first choice y: the opened box z: the final choice (switched to) No: never switch The reduced sample space S={(1,2,1), (1,3,1), (2,3,2), (3,2,3)} P({(2,3,2)}) = P({(3,2,3)}) =1/3, P({(1,2,1)}=P({(1,3,1)}) = 1/6 P(you win) = P({(1,2,1), (1,3,1)} = 1/3 Yes: always switch The reduced sample space S={ (1, 2, 3), (1, 3, 2), (3, 2, 1), (2, 3, 1)} P({(3,2,1)}) = P({(2,3,1)}) = 1/3, P({(1,2,3)})=P({(1,3,2)}) = 1/6 P(you win)=p({(3,2,1), (2,3,1)})=2/ Fall 16

17 Law of multiplication P( AB) P( B) P( A B) P( A) P( B A) Think A and B in temporal or logical order Extension: P(A A A ) P(A )P(A A )P( A3 A1 A2) 2016 Fall 17

18 Example: Suppose 5 good fuses and 2 defective ones have been mixed up. To find the defective fuses, we test them 1-by-1, at random and without replacement. What is the probability that we are lucky and find both of the defective fuses in the first two tests? Sol: Let D 1 and D 2 be the events of finding a defective fuse in the 1 st and 2 nd tests, respectively. P(D 1 D 2 ) = P(D 1 )P(D 2 D 1 ) = 2/7x1/6 = 1/21 P(find both defective fuses in exactly 3 tests) =? 2016 Fall 18

19 Law of total probability Theorem (Law of total probability) Let P(B)>0, and P(B c )>0, then P(A) = P(A B)P(B) + P(A B c )P(B c ) Proof: P(A) = P(AB) + P(AB c ) and use the law of multiplication Fall 19

20 Example: An insurance company rents 35% of the cars for its customers from agency I and 65% from agency II. If 8% of the cars of agency I and 5% of the cars of agency II break down during the rental periods, what is the probability that a car rented by this insurance company breaks down? Sol: P(A) = P(A I)P(I)+ P(A II)P(II) = (0.08)(0.35)+(0.05)(0.65) = Tree diagram method? 2016 Fall 20

21 Gambler s ruin problem 2 gamblers play the game of heads or tails Each time a fair coin lands heads up, player A wins $1 from B, and each time it lands tails up, player B wins $1 from A. Suppose that the player A initially has a dollars and player B has b dollars. If they continue to play this game successively until one is ruined. What is the probability that A will be ruined? What is the probability that the game goes forever with nobody winning? 2016 Fall 21

22 Sol: Sample space? a+b a Fall 22

23 Let E i be the event that A will be ruined if he starts with i dollars, and let p i =P(E i ). Note: total amount is a+b. Our aim is to calculate p a. Initially, p 0 =1, p a+b =0 For 0< i < a+b Let F be the event that A wins the 1 st game. Then P(E i ) = P(E i F) + P(E i F c ) = P(E i F) P(F) + P(E i F c ) P(F c ) = P(E i+1 ) P(F) + P(E i-1 ) P(F c ) => p i = p i+1 (1/2)+p i-1 (1/2) => p i+1 p i = p i p i Fall 23

24 p 0 =1, p a+b =0, and let p 1 - p 0 = x p i+1 p i = p i p i-1 = = p 2 - p 1 = p 1 - p 0 = x Thus p 1 = p 0 + x p 2 = p 0 + 2x p i = p 0 + ix and p a+b = p 0 + (a+b)x => x = -1/(a+b) So p i = 1 i/(a+b). In particular p a = b/(a+b) 2016 Fall 24

25 The same method can be used with obvious modifications to calculate q i, the probability that B is ruined if he starts with i dollars. q i = 1 i/(a+b) Since B starts with b dollars, he will be ruined with probability q b = a/(a+b). Thus the probability that the game goes on forever with nobody winning is 1-(q b +p a )= Fall 25

26 Casino s paradox: (unfair game) A gambler goes to a casino for playing roulette with $1000 in pocket. He wins with probability 18/37 by betting $1 on red/black His strategy is to go home if he wins $100 or go broke. What is the probability that he goes home happily? What would happen if he starts with $1,000,000 in pocket? 2016 Fall 26

27 Sol: Sample space? Similar with a=1000, b=100 But with different probability function for samples. Recurrence equation: Let the gambler start with a dollars. He goes home after winning b dollars or going broke. p i = the probability that he goes home sadly, with i dollars in pocket in the beginning. p 0 = 1, p a+b = 0 p i = P(E i F) P(F) + P(E i F c ) P(F c ) = p i+1 (18/37)+p i-1 (19/37) Let r=18/37. p i = r p i+1 + (1-r) p i Fall 27

28 Equation solution: 1 < i < a+b p i = rp i r p i 1 r p i+1 p i = 1 r (p i p i 1 ) p i = 1 1 r 1 r r 1 r a+b i a+b p a = 1 r b 1 r r a+b > r r 1 r b 2016 Fall 28

29 The more money (b) you want to win, the higher probability you will be ruined. Surprisingly, no matter how much money the gambler carries, the probability of going home happily is no more than (18/19) Why? 2016 Fall 29

30 Extension If {B 1, B 2,, B n } is a partition of the sample space S of an experiment and P(B i )>0 for all i. Then, for any event A of S, P(A) = P(A B 1 )P(B 1 ) + P(A B 2 )P(B 2 ) + + P(A B n )P(B n ) 2016 Fall 30

31 Example: Suppose that 80% of the seniors, 70% of the juniors, 50% of the sophomores, and 30% of the freshmen of a college use the library of their campus frequently. If 30% of all students are freshmen, 25% are sophomores, 25% are juniors, and 20% are seniors, what percent of all students use the library frequently? Sol: P(A)= P(A F)P(F) + P(A O)P(O) + P(A J)P(J) + P(A E)P(E) = (0.30)(0.30) + (0.50)(0.25) + (0.70)(0.25) + (0.80)(0.20) = Fall 31

32 Bayes formula P B A = P(A B)P B P(A) = P(A B)P B P(A B)P B +P(A B c )P(B c ) A conditional probability formula with very important applications on Statistics. Prior probability: P(B), P(B c ) the probability of event (hypothesis) B is believed originally Likelihoods (observation probability) : P(A B), P(A B c ) Posterior probability: P(B A) the probability of B under that A is observed The later information (event A occurs) help correct the prior probability Fall 32

33 Application 1: determine fake coin We have a bag of 100 coins. 60 of them are genuine and 40 of them are fake. We randomly pick one. Prior probability Event B: the picked one is genuine. P(B) = 0.6 and P(B c ) = 0.4 Assume: the toss of a genuine coin is fair. The toss of a fake coin is unfair with head probability 0.4. We toss the picked coin 5 times and get HHTTT. This is the event A. Posterior probability P(B A) and P(B c A) 2016 Fall 33

34 Application 2: election poll In some area, by the record of past elections, Republican supporters are 55% and Democrat supporters are 45%. From experience, a Republican votes for Democrat is 10% and a Democrat votes for Republican is 12%. In the upcoming election, some poll company is doing a poll on the area for a party. The poll company can use one of the following methods to write the poll report Fall 34

35 Method 1: Ask the opinions of 1000 people in the area. Report: the supporting rate by the ratio from the answers. For example, if 60% of the answers are in favor of Democratic. The supporting rate of Democratic at the area in this election is 60%. Method 2: Let B the event that a person is a Republican supporter. From the past, Let P(B) = 55% and P(B c ) = 45%. P(Vote for Dem B) = 10% and P(Vote for Rep B c )= 12%. Ask the opinions of 1000 people in the area. Let the result be event A. Report: the supporting rate for Republican is P(B A) and the supporting rate for Democratic is P(B c A). Which one is better? 2015 Fall 35

36 Application 3: economic prediction In the past 50 years. 30 years have growing economies and 20 years have recession economies. By the statistics, in growing year, the probability that stock market rises is 80%. In recession year, the probability that stock market rises is 30%. We want to predict whether the economy grows in this year by looking at the stock market. What is the probability that the economy grows in this year after seeing a down stock market? 2016 Fall 36

37 80% Stock up 60% Grow 20% Stock down 40% Recess 30% Stock up 70% Stock down 2016 Fall 37

38 Example: In a bolt factory, 30% and 70% of production is manufactured by machines I and II, respectively. 4% and 6% of the output of machines I and II are defective. What is the probability that a randomly selected bolt that is found to be defective is manufactured by machine I? probability A: defective A c : non-defective B: manufactured by I B c : manufactured by II We want to compute P(B A)? Prior probability P(B)=0.3 Likelihoods: P(A B)=0.04, P(A B c )=0.06 Posterior probability: P(B A)=0.04*0.3/(0.04* *0.7)= Fall 38

39 Tree diagram method? 2016 Fall 39

40 Example: In a double homicide, a suspect, John, is caught. The jury believes that John is guilty with 15% Later, a DNA sample is found and match John s DNA By forensic estimation, the probability that the DNA does not come from John is How certain should the jury be about that John is guilty? Sol: G: the event that John is guilty, P(G) = 0.15 I: the event that John is innocent, P(I)=0.85 D: the event that the DNA matches John s DNA P(D G)=1, P(D I)=10-9 Compute P(G D)= 2016 Fall 40

41 Example: Cathy goes to hospital for a rare cancer test. The accuracy rate of the test is 99%. If no cancer, the probability of testing negative is 99%. There is still 1% of error. Cathy is tested positive for the cancer. Is this the end of the world for Cathy? Sol: By statistics, P(a person has the cancer) = 0.1% Accuracy rate: P(test positive no cancer) = 1%. P(test positive have cancer) = 100% Thus, P(have cancer test positive) = 9.1% 2016 Fall 41

42 Example: Spam detector A spam mail: Subject: Can You Remaimber The Last Time You Were In UNBELIEVABLE Pain Content: Hi Guys, As a college football player, I've had a ton of injuries over the years. This led to me studying all the best and most effective pain relief methods people have used for thousands of years before the invention of prescription drugs Fall 42

43 Statistics from the past s: B: the event that an is spam. G: the event that an is normal. Word list: w 1, w 2,, w N Wi: the event that word w i appears in an . P(Wi B): the prob. that a spam contains word w i P(Wi G): the prob. that a normal contains word w i Detection: A received M contains words w 1, w 5, w 10. We compute p1=p(b W1, W5, W10) p2=p(g W1, W5, W10) If p1>p2, then say M is spam, else M is normal Fall 43

44 Extension: B 1, B 2,, B n partition S. P(B k A) = P A B k P B k P A B 1 P B P A B n P(B n ) Prior probability: P(B k ), 1 k n Likelihoods: P(A B i ), 1 i n Posterior probability: P(B k A ) 2016 Fall 44

45 A box contains 7 red and 13 blue balls. 2 balls are selected at random and are discarded without their colors being seen. A 3 rd ball is drawn randomly and observed to be red. What is the probability that both of the discarded balls were blue? Sol: Prior probability B 1 : discarded balls are RR, P(B 1 ) = B 2 : discarded balls are RB, P(B 2 ) = B 3 : discarded balls are BB, P(B 3 ) = A: the third ball is R the observation Likelihood P(A B 1 )= P(A B 2 )= P(A B 3 )= Want to compute P(B 3 A) 2016 Fall 45

46 Independence Two events A and B are independent if P(B A ) = P(B): Explanation: the occurrence of event A does not affect the occurrence probability of event B Example 1: toss a coin twice A: the first result is head B: the second result is head A and B are independent. Example 2: toss a coin three times A: the first 2 results are both head B: the last2 results are both tail A and B are not independent Fall 46

47 Theorem:A and B are independent if either one of the following holds P(AB) = P(A)P(B) --- independence is a very strict condition P(A B) = P(A) P(B A) = P(B) Theorem: If A and B are independent, so are A and B c A c and B A c and B c 2016 Fall 47

48 P(AB)=P(A)P(B) A B A B A B 2016 Fall 48

49 Remark: If A and B are disjoint (mutually exclusive), then A and B are dependent. If A B, then A and B are dependent 2016 Fall 49

50 Extension Events A, B, and C are independent if the following all hold: P(AB) = P(A)P(B) P(AC) = P(A)P(C) P(BC) = P(B)P(C) P(ABC) = P(A)P(B)P(C) 2016 Fall 50

51 What is good for independence? easy to compute To compute P(AB) If A and B are not independent, P(AB) = P(A)+P(B) P(AUB) If A and B are independent, P(AB)=P(A)P(B) To compute P(B A) If A and B are not independent, P(B A) = P(AB)/P(A) If A and B are independent, P(B A) = P(B) 2016 Fall 51

52 Example: A card is drawn from a deck of 52 cards and placed back. Then, a second card is drawn. A: the event that the first card is an ace B: the event that the second card is also an ace Are A and B independent? P(A) = P(B)= P(AB) = If the first drawn card is not placed back, then A and B are not independent Fall 52

53 Jailer s paradox Three prisoners Alex, Bill, and Tim. One of them is condemned to death. The other two will be freed. The jailor and the judge know who is condemned to death. Alex has written a letter to his fiancée and wants to give it to either Bill or Tim, whoever goes free, to deliver. Alex asks the jailer to tell him which of the two will be freed. The jailer refuses to give that information to Alex, explaining that, if he does, the probability of Alex dying increases from 1/3 to 1/2. Really? 2016 Fall 53

54 Sol-1: Let A, B, and T be the events that Alex dies, Bill dies, and Tim dies. Let w 1 =(T, the jailer tells Alex that Bill goes free) w 2 =(B, the jailer tells Alex that Tim goes free) w 3 =(A, the jailer tells Alex that Bill goes free) w 4 =(A, the jailer tells Alex that Tim goes free) The sample space S = {w 1, w 2, w 3, w 4 }. P(w 1 )=P(w 2 )=1/3, P(w 3 )=P(w 4 )=1/6 Let J be the event that the jailer tells Alex that Tim goes free Then, P( A J ) P( AJ ) P( J ) 1 P( w4 ) 1 6 P( w2 ) P( w4 ) Fall

55 Sol-2: Zweifel, in June 1986 issue of Mathematics Magazine, page 156 He analyzes this paradox by using Bayes formula: ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( T P T J P B P B J P A P A J P A P A J P J A P 2016 Fall 55

56 Example: We draw cards, one at a time, at random and successively from an ordinary deck of 52 cards with replacement. What is the probability that an ace appears before a face card? Sol: Let E be the event of an ace appearing before a face card. Let A, F, and N be the events of ace, face card, and neither in the first experiment, respectively. Then P(E) = P(E A)P(A) + P(E F)P(F) + P(E N)P(N) = 1x(4/52) + 0x(12/52)+ P(E N)x(36/52) Since E and N are independent (think!), P(E N) = P(E). P(E)=4/52+P(E)(36/52) So, P(E)=1/ Fall 56