Module 2 : Conditional Probability

Transcription

1 Module 2 : Conditional Probability Ruben Zamar Department of Statistics UBC January 16, 2017 Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

2 MOTIVATION The outcome could be any element in the Sample Space, Ω. Sometimes the range of possibilities is restricted because of partial information Examples number of shots: partial info: we know it wasn t an ace ELEC 321 final grade: partial info: we know it is at least a B Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

3 CONDITIONING EVENT The event B representing the partial information is called conditioning event Denote by A the event of interest Example (Number of Shots) B = {2, 3,...} = {not an ace } A = {1, 3, 5,...} = {server wins} Example (Final Grade) B = [70, 100] = {at least a B } (conditioning event) (event of interest) (conditioning event) A = [80, 100] = {an A } (event of interest) Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

4 DEFINITION OF CONDITIONAL PROBABILITY Suppose that P (B) > 0 P (A B) = P (A B) P (B) The left hand side is read as probability of A given B Useful formulas: P (A B) = P (B) P (A B) = P (A) P (B A) Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

5 CONDITIONAL PROBABILITY P (A B), as a function of A (and for B fixed) satisfies all the probability axioms: - P (Ω B) = P (Ω B) /P (B) = P (B) /P (B) = 1 - P (A B) 0 - If {A i } are disjoint then P ( A i B) = P [( A i ) B] P (B) = P [ (A i B)] P (B) = P (A i B) P (B) = P (A i B) Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

6 EXAMPLE: NUMBER OF SHOTS For simplicity, suppose that points are decided in at most 8 shots, with probabilities: Shots Prob Using the table above: P (Sever wins Not an ace) = P ({3, 5, 7}) P ({2, 3, 4, 5, 6, 7, 8}) = = Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

7 EXAMPLE: FINAL GRADE Suppose that P (Grade is larger than x) = 100 x 100 = 1 x 100 Using the formula above: P (To get an A To get at least a B ) = = P ([80, 100]) P ([70, 100]) = = Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

8 SCREENING TESTS Items are submitted to a screening test before shipment The screening test can result in either POSITIVE NEGATIVE (indicating that the item may have a defect) (indicating that the item doesn t have a defect) Screening tests face two types of errors FALSE POSITIVE FALSE NEGATIVE Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

9 SCREENING TESTS (continued) For each item we have 4 possible events Item true status: D = {item is defective} D c = {item is not defective} Test result: B = {test is positive} B c = {test is negative} Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

10 SCREENING TESTS (continued) The following conditional probabilities are normally known Sensitivity of the test: P (B D) = 0.95 (say) which implies Specificity of the test: P (B c D c ) = 0.99 (say) P (B c D) = 0.05 and P (B D c ) = 0.01 The proportion of defective items is also normally known P (D) = 0.02 (say) Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

11 TEST PERFORMANCE The following questions may be of interest: What is the probability that a randomly chosen item tests positive? What is the probability of defective given that the test resulted negative? What is the probability of defective given that the test resulted positive? What is the probability of screening error? We will compute these probabilities Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

12 PROBABILITY OF TESTING POSITIVE P (B) = P (B D) + P (B D c ) = P (D) P (B D) + P (D c ) P (B D c ) = (1 0.02) 0.01 = Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

13 PROB OF DEFECTIVE GIVEN A POSITIVE TEST P (D B) = = = P (D B) P (B) P (D) P (B D) P (B) = Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

14 PROB OF DEFECTIVE GIVEN A NEGATIVE TEST P (D B c ) = P (D Bc ) P (B c ) = P (D) P (Bc D) 1 P (B) = = 0.01 Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

15 SCREENING ERROR P (Error) = P (D B c ) + P (D c B) = P (D) P (B c D) + P (D c ) P (B D c ) = 0.02 (1 0.95) + (1 0.02) 0.01 = Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

16 BAYES FORMULA The formula P (D B) = P (D B) P (B) = P (B D) P (D) P (B D) P (D) + P (B D c ) P (D c ) is the simple form of Bayes formula. This has been used in the "Screening Example presented before. Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

17 BAYES FORMULA (continued) The general form of Bayes Formula is given by P (D i B) = P (D i B) P (B) = P (B D i ) P (D i ) k j=1 P (B D j ) P (D j ) where D 1, D 2,..., D k is a partition of the sample space Ω: Ω = D 1 D 2 D k D i D j = φ, for i = j Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

18 EXAMPLE: THREE PRISONERS Prisoners A, B and C are to be executed The governor has selected one of them at random to be pardoned The warden knows who is pardoned, but is not allowed to tell Prisoner A begs the warden to let him know which one of the other two prisoners is not pardoned Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

19 Prisoner A tells the warden: Since I already know that one of the other two prisioners is not pardoned, you could just tell me who is that Prisoner A adds: If B is pardoned, you could give me C s name. If C is pardoned, you could give me B s name. And if I m pardoned, you could flip a coin to decide whether to name B or C. Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

20 The warden is convinced by prisoner A s arguments and tells him: not pardoned B is Result: Given the information provided by the Warden, C is now twice more likely to be pardoned than A! Why? Check the derivations below: Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

21 NOTATION: A = {A is pardoned} B = {B is pardoned} C = {C is pardoned} b = {The warden says B is not pardoned } Clearly P (A) = P (B) = P (C ) = 1 3 P (b B) = 0 (warden never lies) P (b A) = 1/2 (warden flips a coin) P (b C ) = 1 (warden cannot name A) Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

22 By the Bayes formula: P (A b) = P (b A) P (A) P (b A) P (A) + P (b B) P (B) + P (b C ) P (C ) Hence = = 1 3 P (C b) = 1 P (A b) = = 2 3 Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

23 SCREENING EXAMPLE II The tested items have two components: c 1 and c 2 Suppose D 1 = {Only component c 1 is defective }, P (D 1 ) = 0.01 D 2 = {Only component c 2 is defective }, P (D 2 ) = D 3 = {Both components are defective }, P (D 3 ) = D 4 = {Both components are non defective }, P (D 4 ) = 0.98 Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

24 SCREENING EXAMPLE II (continued) Let Suppose B = {Screening test is positive} P (B D 1 ) = 0.95 P (B D 2 ) = 0.96 P (B D 3 ) = 0.99 P (B D 4 ) = 0.01 Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

25 SOME QUESTIONS OF INTEREST The following questions may be of interest: What is the probability of testing positive? What is the probability that component c i (i = 1, 2) is defective when the test resulted positive? What is the probability that the item is defective when the test resulted negative? What is the probability both components are defective when the test resulted positive? What is the probability of testing error? We will compute these probabilities Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

26 PROB OF TESTING POSITIVE P (B) = P (B D 1 ) + P (B D 2 ) + P (B D 3 ) + P (B D 4 ) = = Notice that the probability of defective is P (D) = = 0.02 Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

27 POSITIVE TEST P (D 1 B) = P (D 2 B) = P (D 3 B) = P (D 4 B) = = = = = Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

28 PROB OF TESTING NEGATIVE P (B c ) = P (B c D 1 ) + P (B c D 2 ) + P (B c D 3 ) + P (B c D 4 ) = = Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

29 A NEGATIVE TEST P (D 1 B c ) = P (D 2 B c ) = P (D 3 B c ) = P (D 4 B c ) = = = = = P (defective B c ) = = Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

30 INDEPENDENCE DEFINITION: Events A and B are independent if P (A B) = P (A) P (B) If A and B are independent then P (A B) = P (A B) P (B) = P (A) P (B) P (B) = P (A) and P (B A) = P (A B) P (A) = P (A) P (B) P (A) = P (B) Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

31 DISCUSSION If P (A) = 1, then A is independent of all B. {}}{ P (A B) = P (A B) + P (A c B) = P (B) =0 P (A B) = =1 {}}{ P (A)P (B) Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

32 DISCUSSION (Cont) Suppose that A and B are non-trivial events ( 0 < P (A) < 1 and 0 < P (B) < 1 ) If A and B are mutually exclusive ( A B = φ ) then they cannot be independent because P (A B) = 0 < P (A) If A B then they cannot be independent because P (A B) = P (A B) P (B) = P (A) P (B) > P (A) Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

33 DISCUSSION (Cont) Suppose Ω = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10} and the numbers are equally likely. A = {1, 2, 3, 4, 5} and B = {2, 4, 6, 8} P (A B) = P ({2, 4}) = 0.20, P (A) P (B) = = 0.20 Hence, A and B are independent In terms of probabilities A is half of Ω. half of B. On the other hand A B is Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

34 DISCUSSION (continued) What happens if P (i) = i 55? P (A B) = P ({2, 4}) = 6/55 = P (A) P (B) = (15/55) (20/55) = Hence, A and B are not independent in this case. Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

35 MORE THAN TWO EVENTS Definition: We say that the events A 1, A 2,..., A n are independent if P (A i1 A i2 A ik ) = P (A i1 ) P (A i2 ) P (A ik ) for all 1 i 1 < i 2 < < i k n, and all 1 k n. Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

36 For example, if n = 3, then P (A 1 A 2 ) = P (A 1 ) P (A 2 ) P (A 1 A 3 ) = P (A 1 ) P (A 3 ) P (A 2 A 3 ) = P (A 2 ) P (A 3 ) P (A 1 A 2 A 3 ) = P (A 1 ) P (A 2 ) P (A 3 ) Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

37 SYSTEM OF INDEPENDENT COMPONENTS In series a b c In parallel a b c Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

38 NOTATION A = {Component a works} B = {Component b works} C = {Component c works} Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

39 INDEPENDENT COMPONENTS We assume that A, B and C are independent, that is P (A B C ) = P (A) P (B) P (C ) P (A B) = P (A) P (B), P (B C ) = P (B) P (C ), P (A C ) = P (A) P (C ) Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

40 RELIABILITY CALCULATION Problem 1: Suppose that P (A) = P (B) = P (C ) = Calculate the reliability of the system a b c Solution: P (System Works) = P (A B C ) = P (A) P (B) P (C ) = = Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

41 PRACTICE Problem 2: Suppose that P (A) = P (B) = P (C ) = Calculate the reliability of the system a b c Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

42 PROBLEM 2 (Solution) P (System works) = 1 P (System fails) = 1 P (A c B c C c ) = 1 P (A c ) P (B c ) P (C c ) = 1 (1 P (A)) (1 P (B)) (1 P (C )) = = Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

43 PRACTICE Problem 3: Suppose that P (A) = P (B) = P (C ) = P (D) = Calculate the reliability of the system subsys I subsys II a c b d Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

44 PROBLEM 3 (Solution) P (System works) = P (subsys I works subsys II works) = P (subsys I works ) P (subsys II works) = [1 P (subsys I fails )] [1 P (subsys II fails )] = [1 P (A c B c )] [1 P (C c D c )] = [1 P (A c ) P (B c )] [1 P (C c ) P (D c )] = [1 (1 P (A)) (1 P (B))] [1 (1 P (C )) (1 P (D))] = ( ) 2 = Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

45 CONDITIONAL INDEPENDENCE Definition: We say that the events T 1, T 2,..., T n are conditionally independent given the event B if P (T i1 T i2 T ik B) = P (T i1 B) P (T i2 B) P (T ik B) for all 1 i 1 < i 2 < < i k n, and all 1 k n. Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

46 For example, if n = 3, then P (T 1 T 2 B) = P (T 1 B) P (T 2 B) P (T 1 T 3 B) = P (T 1 B) P (T 3 B) P (T 2 T 3 B) = P (T 2 B) P (T 3 B) P (T 1 T 2 T 3 B) = P (T 1 B) P (T 2 B) P (T 3 B) Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

47 Notes Conditional independence doesn t imply unconditional independence and vice versa Conditional independence given B doesn t imply conditional independence given B c However, usually both conditional independences are assumed together in applications We will apply this concept in Bayesian probability updating Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

48 Sequential Bayes Formula Let S i be the outcome of the i th test. For instance S 1 = S 2 = S 3 = { The 1 th { The 2 th { The 3 th } test is positive } test is negative } test is negative and so on Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

49 The outcomes S i (i = 1, 2,..., n) are available in a sequential fashion. Let I k = S 1 S 2 S k (data available at step k) and set π 0 = P (E ) π 1 = P (E I 1 ) = P (E S 1 ) π 2 = P (E I 2 ) = P (E S 1 S 2 ) π 3 = P (E I 3 ) = P (E S 1 S 2 S 3 ) and so on Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

50 Conditional Independence Assumption Assume that the S i (i = 1, 2,..., n) are independent given E and also given E c. Then, for k = 1, 2,..., n π k = P (S k E ) π k 1 P (S k E ) π k 1 + P (S k E c ) (1 π k 1 ) Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

51 Proof π k = P (I k E ) π 0 P (I k E ) π 0 + P (I k E c ) (1 π 0 ) = P (I k 1 S k E ) π 0 P (I k 1 S k E ) π 0 + P (I k 1 S k E c ) (1 π 0 ) = P (S k E ) P (I k 1 E ) π 0 P (S k E ) P (I k 1 E ) π 0 + P (S k E c ) P (I k 1 E c ) (1 π 0 ) Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

52 Proof π k = P (S k E ) P (I k 1 E ) P (S k E ) P (I k 1 E ) + P (S k E c ) P (I k 1 E c ) = P (S k E ) P (I k 1 E ) /P (I k 1 ) [P (S k E ) P (I k 1 E ) + P (S k E c ) P (I k 1 E c )] /P (I k 1 ) = P (S k E ) π k 1 P (S k E ) π k 1 + P (S k E c ) (1 π k 1 ), π k 1 = P (E I k 1 ) Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

53 Pseudo Code Input: (S 1, S 2, S 3,..., S n ) = (1, 0, 1,..., 0) (outcomes for the n tests) π = P (E ) defective") (prob of event of interest, for instance E= "the part is p k = P (S k = + E ) k = 1, 2,..., n (Sensitivity of k th test) q k = P (S k = E c ) k = 1, 2,..., n (Specificity of k th test) Output π k = P (E S 1 S 2 S k ), k = 1, 2,..., n Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

54 Pseudo Code Example of Input: n = 4, π = 0.05 Test Results = (1, 1, 0, 1) k p k = P (1 Defective) 1 q k = P (1 Non Defective) 1 p 1 = q 1 = p 2 = q 2 = p 3 = q 3 = p 4 = q 4 = 0.15 Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

55 Pseudo Code - Computation Computation of π k 1) Initialization: Set π 0 = π 2) k-step: If S k = 1, set a = p k and b = 1 q k If S k = 0, set a = 1 p k and b = q k 3) Computing π k : π k = aπ k 1 aπ k 1 + b (1 π k 1 ) Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

56 Spam Detection When you receive an , your spam fillter uses Bayes rule to decide whether it is spam or not. Basic spam filters check whether some pre-specified words appear in the ; e.g. {diplomat,lottery,money,inheritance,president,sincerely,huge,...}. We consider n events W i telling us whether the i th pre-specified word is in the message Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

57 Let E = { is spam } W i = { word i is in the message }, i = 1, 2,..., n Assume that W 1, W 2,..., W n are conditionally independent given E and also E c. Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

58 Human examination of a large number of messages is used estimate π 0 = P (E ) The training data is also used to estimate p i = P (W i E ) and 1 q i = P (W i E c ) Let I n = S 1 S 2 S n, where S i is either W i or W c i. The spam filter assumes that the W i are conditionally independent (given E and given E c ) to compute P (E I n ) = P (I n E ) P (E ) P (I n E ) P (E ) + P (I n E c ) P (E c ) Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

59 Sequential Updating The posterior probs π k = P (E I k ) (k = 1, 2,..., n 1) can be computed sequentially using the formula π k = P (E I k ) = P (S k E ) P (E I k 1 ) P (S k E ) P (E I k 1 ) + P (S k E c ) P (E c I k 1 ) = P (S k E ) π k 1 P (S k E ) π k 1 + P (S k E c ) (1 π k 1 ) An early decision to classify the as spam can be made if P (E I k ) becomes too large (or too small). Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

60 Numerical Example For a simple numerical example consider a case with and conditional probabilities n = 8 words, P (Spam) = 0.10 Word P(Word Spam) P(Word No Spam) W W W W W W W W Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61

61 Word Word Status P(Spam I k ) W W W W W W W W Ruben Zamar Department of Statistics UBC Module () 2 January 16, / 61