Will Monroe July 5, with materials by Mehran Sahami and Chris Piech. image: Therightclicks. Independence

Transcription

1 Will Monroe July 5, 2017 with materials by Mehran Sahami and Chris Piech image: Therightclicks Independence

2 Announcements: Problem Set 1 due! Solutions to be posted next Wednesday (no submissions allowed >1 week late, even if you're willing to take lots of late penalties!)

3 Announcements: Problem Set 2 out (Cell phone location sensing) Due next Wednesday, 7/12, at 12:30pm (before class). 13 problems 1 coding problem (this time with a bit of data to work with!)

4 Review: Conditional probability The conditional probability P(E F) is the probability that E happens, given that F has happened. F is the new sample space. P( EF) P(E F )= P( F) S E EF F

5 Conditional probability: A cautionary tale xkcd by Randall Munroe

6 Review: Chain rule of probability The probability of all events happening is the probability of the first happening times the prob. of the second given the first times the prob. of the third given the first two...etc. P(EFG )=P( E) P(F E) P(G EF) S S E EF F = F x EF F

7 Review: Law of total probability You can compute an overall probability by summing over mutually exclusive and exhaustive sub-cases. P(F )= P(E i F ) i P(F )= P(E i ) P(F E i ) S E1 E1F E2F E4F E3F E4 i E2 E3 F

8 Review: Bayes' theorem You can flip a conditional probability if you multiply by the probability of the hypothesis and divide by the probability of the observation. P(F E) P (E) P(E F )= P(F )

9 Independence Two events are independent if you can multiply their probabilities to get the probability of both happening. P(EF )=P (E) P(F ) E F ( independent of ) E S EF F

10 Rolling two dice 1 P( D 1=1, D 2=1)= 36 E D1 D2 P(EF) = 1/6 1/6 F P(E) = 1/6 S = {(1, 1), (1, 2), (1, 3), (1, 4), (1, 5), (1, 6), S = {(2, 1), (2, 2), (2, 3), (2, 4), (2, 5), (2, 6), S = {(3, 1), (3, 2), (3, 3), (3, 4), (3, 5), (3, 6), P(F) = 1/6 S = {(4, 1), (4, 2), (4, 3), (4, 4), (4, 5), (4, 6), S = {(5, 1), (5, 2), (5, 3), (5, 4), (5, 5), (5, 6), S = {(6, 1), (6, 2), (6, 3), (6, 4), (6, 5), (6, 6)}

11 Rolling two dice E G P( D 1=1, D 1 + D 2=4)=? D1 D2 P(EG) = 1/36 1/6 3/36 P(E) = 1/6 S = {(1, 1), (1, 2), (1, 3), (1, 4), (1, 5), (1, 6), S = {(2, 1), (2, 2), (2, 3), (2, 4), (2, 5), (2, 6), S = {(3, 1), (3, 2), (3, 3), (3, 4), (3, 5), (3, 6), P(G) = 3/36 S = {(4, 1), (4, 2), (4, 3), (4, 4), (4, 5), (4, 6), S = {(5, 1), (5, 2), (5, 3), (5, 4), (5, 5), (5, 6), S = {(6, 1), (6, 2), (6, 3), (6, 4), (6, 5), (6, 6)}

12 Independence Two events are independent if you can multiply their probabilities to get the probability of both happening. P(EF )=P (E) P(F ) E F ( independent of ) E S EF F

13 Independence as conditional P( E F)=P( E) (if and only if E, F are independent)

14 Conditioning on the complement If E, F are independent: C P( E F)=P( E F )

15 Three events E, F, G are independent if: P( EFG)=P ( E) P (F) P(G) and P (EF)=P( E) P (F ) and P (EG)=P( E) P (G) and P (FG)=P( F ) P(G)

16 Rolling two dice again E: event that D1 = 1 D1 F: event that D2 = 6 D2 G: event that D1 + D2 = 7 P(E) = 1/6 P(F) = 1/6 P(EF) = 1/36 P(EF) = 1/6 1/6 Are E and F independent? A) Yes Room: CS109SUMMER17

17 Independence and causation If two events don't affect each other, and also have no unknown factors in common that affect both, then they're likely to be independent. However: If two events are independent, it doesn't necessarily mean they don't affect each other! What it means is that knowing one doesn't give you information about the other.

18 Rolling two dice again E: event that D1 = 1 D1 F: event that D2 = 6 D2 P(E) = 1/6 P(F) = 1/6 P(EF) = 1/36 P(EF) = 1/6 1/6 G: event that D1 + D2 = 7 P(G) = 6/36 = 1/6 P(EG) = 1/36 P(EG) = 1/6 1/6 Are E and G independent? A) Yes Room: CS109SUMMER17

19 Rolling two dice again E: event that D1 = 1 D1 F: event that D2 = 6 D2 P(E) = 1/6 P(F) = 1/6 P(EF) = 1/36 P(EF) = 1/6 1/6 G: event that D1 + D2 = 7 P(G) = 6/36 = 1/6 P(EG) = 1/36 P(EG) = 1/6 1/6 P(FG) = 1/36 P(FG) = 1/6 1/6 Are F and G independent? A) Yes Room: CS109SUMMER17

20 Rolling two dice again E: event that D1 = 1 D1 F: event that D2 = 6 D2 P(E) = 1/6 P(F) = 1/6 P(EF) = 1/36 P(EF) = 1/6 1/6 G: event that D1 + D2 = 7 P(G) = 6/36 = 1/6 P(EG) = 1/36 P(EG) = 1/6 1/6 P(FG) = 1/36 P(FG) = 1/6 1/6 P(EFG) = 1/36 1/6 1/6 1/6 Are E, F and G independent? B) No (!) Room: CS109SUMMER17

21 Many events E 1, E 2,, E n are independent if for every subset E i, E i,, E i : 1 2 r P (E i E i E i )=P( E i ) P (Ei ) P ( E i ) 1 2 r 1 2 r

22 Generating random bits (1 p) p p p n P(first n are 1's) = p p... = pn Each bit is a 1 with probability p, 0 with probability (1 p). E: generate n 1's, followed by a single 0 P(E) = pn (1 - p)

23 (Biased) coin flipping n flips, each flip heads with probability p, tails with probability (1 p) E: n heads P(E) = pn F: n tails P(F) = (1 p)n G: k heads, then n k tails P(G) = pk(1 p)n-k H: exactly k heads n pk (1 p)n k P(H) = k ()

24 Break time!

25 String hashing m strings Roosevelt Adams Nixon n buckets Taft Harrison Washington All buckets equally likely, all independent E: at least 1 hashed to first bucket Fi: string i hashed to first bucket P( E)=P (F 1 F 2 F m )

26 Review: Getting rid of ORs Finding the probability of an OR of events can be nasty. Try using De Morgan's laws to turn it into an AND! c c c P( A B Z )=1 P( A B Z ) S E F

27 String hashing m strings Roosevelt Adams All buckets equally likely, all independent Nixon E: at least 1 hashed to first bucket n buckets Fi: string i hashed to first bucket P( E)=P (F 1 F 2 F m ) Taft C C C =1 P(F 1 F 2 F m ) C C C Harrison =1 P(F 1 ) P (F 2 ) P(F m ) Washington n 1 =1 n m ( )

28 String hashing n buckets m strings Roosevelt Adams All independent, not equally likely. (Each string bucket i with prob. pi) Nixon E: at least 1 string hashed to any of the first k buckets Fi: bucket i gets at least one string Taft P (E)=P ( F 1 F 2 F k ) C C C Harrison =1 P(F 1 F 2 F k ) Washington m =1 ( 1 p1 p2 p k )

29 String hashing n buckets m strings Roosevelt Adams All independent, not equally likely. (Each string bucket i with prob. pi) Nixon E: at least 1 string hashed to each of the first k buckets Fi: bucket i gets at least one string Taft P ( E)=P (F 1 F 2 F k ) s] n o ti p m u s =P( F 1 ) Pyo(F Harrison ur a2s) P( F k ) Washington k c e h [c

30 String hashing n buckets m strings Roosevelt Adams All independent, not equally likely. (Each string bucket i with prob. pi) Nixon E: at least 1 string hashed to each of the first k buckets Fi: bucket i gets at least one string Taft P ( E)=P (F 1 F 2 F k ) C C C Harrison =1 P( F 1 F 2 F k ) k Washington (r+ 1) =1 ( 1) r=1 k (r+1) =1 ( 1) r=1 r ( ) Fi C P (1 p i p i p i ) i 1< < i r i1 < <i r j=1 j m 1 2 r

31 Network reliability A p1 B p2 p3 p4 n independent routers, each works with prob. pi E: path exists from A to B P(E) =? n P ( E)=1 P (all routers fail) =1 (1 p1 )(1 p2 ) (1 pn )=1 (1 pi ) i=1

32 Simplified craps $

33 Simplified craps 9 7

34 Simplified craps 9 4 $ 5 E: roll 5 before rolling 7 Fn: n 1 rolls without a 5 or 7, then a 5 P (E)=P ( F 1 F 2 ) To Infinity! (and beyond?) = P (F n ) n=1 n = (1 ) ( ) n=1 n =( ) ( ) 36 n=1 36 P(5)=4 /36 P (7)=6/36

35 Geometric series n n x + x + x + + x = x i=0 i n+1 1 x = 1 x If x < 1, then as n : n 1 x 1 x i=0 i See Calculation Reference for more super-useful sum and product identities!

36 Simplified craps 9 4 $ 5 E: roll 5 before rolling 7 Fn: n 1 rolls without a 5 or 7, then a 5 P (E)=P ( F 1 F 2 ) To Infinity! (and beyond?) = P (F n ) n=1 P(5)=4 /36 P (7)=6/36 n = (1 ) ( ) n=1 n 4 26 =( ) ( ) 36 n= =( ) = 26 5 (1 ) 36

37 Conditional independence Two events are conditionally independent if you can multiply their conditional probabilities to get the conditional probability of both happening. P(EF G)=P(E G) P(F G) (E F ) G E G EF F

38 Conditional independence In general, (E F ) does not imply (E F ) G or vice versa!

39 Rolling two dice again E: event that D1 = 1 D1 D2 P(E) = 1/6 P(F) = 1/6 P(EF) = 1/36 P(EF) = 1/6 1/6 F: event that D2 = 6 G: event that D1 + D2 = 7 P(E G) = 1/6 P(F G) = 1/6 P(EF G) = 1/6 P(EF) 1/6 1/6

40 Faculty night At faculty night: 44 students 30 straight A's P(A F) CS majors 6 CS w/ straight A's P(A CS, F) = 0.30 Survey the whole dorm: 100 students 30 straight A's P(A) = CS majors 6 CS w/ straight A's P(A CS) = 0.30 Independent! What gives?

41 Faculty night At faculty night: 44 students 30 straight A's P(A F) CS majors 6 CS w/ straight A's P(A CS, F) = 0.30 All the A students came to faculty night. So did all the CS majors. A, CS conditionally dependent, conditioned on faculty night. Survey the whole dorm: 100 students 30 straight A's P(A) = CS majors 6 CS w/ straight A's P(A CS) = 0.30

42 Watering the lawn E: event that it rained today F: event that the sprinkler was on Say E and F are independent. G: event that the grass is wet You observe the grass is wet. Probability of both rain and sprinkler goes up! P(E G) > P(E) P(F G) > P(F) Now you find out the sprinklers were on. Does your belief that it rained change? P(E FG) < P(E G)

43 A graphical representation A, B independent! A A, B not independent? B C C is unknown? C A A B C B Condition on C C A, B not conditionally independent A B A, B conditionally independent!

44 For more on this: the book Daphne Koller Designed & taught earlier versions, wrote the book (literally & figuratively) a12 X1 Stefano Ermon Taught Winter 2017 X3 X2 a21 b12 b11 a23 b22 b31 b21 b32 b33 b14 b24 b13 y1 y2 b34 y3 y4 CS228: Probabilistic Graphical Models Tdunning/Razorbliss

45 Independence Two events are independent if you can multiply their probabilities to get the probability of both happening. P(EF )=P (E) P(F ) E F ( independent of ) E S EF F

46 Conditional independence Two events are conditionally independent if you can multiply their conditional probabilities to get the conditional probability of both happening. P(EF G)=P(E G) P(F G) (E F ) G E G EF F

47 Reminder: Python tutorial Right now! Follow Yuling to Gates B03