Conditional Probability: Review

Transcription

1 CS7: Jean Walrand: Lecture 7 ayes Rule, Mutual Independence, Collisions and Collecting Conditional Probability 2 Independence 3 ayes Rule 4 alls and ins 5 Coupons Conditional Probability: Review Recall: Pr[ ] = Pr[ ] Pr[] Pr[ ] = Pr[]Pr[ ] = Pr[]Pr[ ] and are positively correlated if Pr[ ] > Pr[], ie, if Pr[ ] > Pr[]Pr[] and are negatively correlated if Pr[ ] < Pr[], ie, if Pr[ ] < Pr[]Pr[] and are independent if Pr[ ] = Pr[], ie, if Pr[ ] = Pr[]Pr[] Note: and are positively correlated (Pr[ ] = > Pr[]) Note: = / and are negatively correlated (Pr[ ] = < Pr[]) Conditional Probability: Pictures Illustrations: Pick a point uniformly in the unit square b b2 b b b2 Left: and are independent Pr[] = b;pr[ ] = b Middle: and are positively correlated Pr[ ] = b > Pr[ Ā] = b 2 Note: Pr[] (b 2,b ) Right: and are negatively correlated Pr[ ] = b < Pr[ Ā] = b 2 Note: Pr[] (b,b 2 ) ayes and iased Coin ayes: General Case ayes Rule nother picture: Pick a point uniformly at random in the unit square Then Pr[] = 5;Pr[Ā] = 5 Pr[ ] = 5;Pr[ Ā] = 6;Pr[ ] = 5 5 Pr[] = = Pr[]Pr[ ] + Pr[Ā]Pr[ Ā] 5 5 Pr[ ] = = Pr[]Pr[ ] Pr[]Pr[ ] + Pr[Ā]Pr[ Ā] 46 = fraction of that is inside Pick a point uniformly at random in the unit square Then Pr[ m ] = p m,m =,,M Pr[ m ] = q m,m =,,M;Pr[ m ] = p m q m Pr[] = p q + p M q M p m q m Pr[ m ] = = fraction of inside m p q + p M q M Pr[ n ] = p nq n m p m q m

2 Why do you have a fever? Why do you have a fever? Our ayes Square picture: 5 8 Flu Ebola Why do you have a fever? We found Pr[Flu High Fever] 58, Pr[Ebola High Fever] 5 8, Pr[Other High Fever] 42 Using ayes rule, we find 5 8 Pr[Flu High Fever] = Pr[Ebola High Fever] = Pr[Other High Fever] = The values 58,5 8,42 are the posterior probabilities Other 85 Green = Fever 58% of Fever = Flu % of Fever = Ebola 42% of Fever = Other Note that even though Pr[Fever Ebola] =, one has Pr[Ebola Fever] This example shows the importance of the prior probabilities One says that Flu is the Most Likely a Posteriori (MP) cause of the high fever Ebola is the Maximum Likelihood Estimate (MLE) of the cause: it causes the fever with the largest probability Recall that Thus, p mq m p m = Pr[ m],q m = Pr[ m],pr[ m ] = p q + + p M q M MP = value of m that maximizes p mq m MLE = value of m that maximizes q m ayes Rule Operations Thomas ayes Thomas ayes ayes Rule is the canonical example of how information changes our opinions ayesian picture of Thomas ayes Source: Wikipedia

3 Independence Recall : Consider the example below: and are independent 2 3 Pr[ ] = Pr[]Pr[] Pr[ ] = Pr[] 25 5 ( 2,) are independent: Pr[ 2 ] = 5 = Pr[ 2 ] ( 2, ) are independent: Pr[ 2 ] = 5 = Pr[ 2 ] (,) are not independent: Pr[ ] = 5 = 2 Pr[ ] = 25 Mutual Independence 5 25 Pairwise Independence Flip two fair coins Let = first coin is H = {HT,HH}; = second coin is H = {TH,HH}; C = the two coins are different = {TH,HT }, C are independent;, C are independent;,c are not independent (Pr[ C] = Pr[ ]Pr[C]) If did not say anything about C and did not say anything about C, then would not say anything about C Mutual Independence Example 2 Flip a fair coin 5 times Let n = coin n is H, for n =,,5 Then, m, n are independent for all m n lso, and 3 5 are independent Indeed, Pr[ ( 3 5 )] = 8 = Pr[ ]Pr[ 3 5 ] Similarly, 2 and are independent This leads to a definition alls in bins Definition Mutual Independence (a) The events,, 5 are mutually independent if Pr[ k K k ] = Π k K Pr[ k ], for all K {,,5} (b) More generally, the events { j,j J} are mutually independent if Pr[ k K k ] = Π k K Pr[ k ], for all finitek J Example: Flip a fair coin forever Let n = coin n is H Then the events n are mutually independent Theorem (a) If the events { j,j J} are mutually independent and if K and K 2 are disjoint finite subsets of J, then k K k and k K2 k are independent (b) More generally, if the K n are pairwise disjoint finite subsets of J, then the events k Kn k are mutually independent (c) lso, the same is true if we replace some of the k by Āk Proof: See Notes 25, 27 One throws m balls into n > m bins

4 alls in bins One throws m balls into n > m bins alls in bins }, for large enough n alls in bins }, for large enough n In particular, Pr[no collision] /2 for m 2 /() ln(2), ie, m 2ln(2)n 2 n Eg, Roughly, Pr[collision] /2 for m = n (e 5 6) }, for large enough n The Calculation i = no collision when ith ball is placed in a bin Pr[ i i ] = ( i n ) no collision = m Product rule: Pr[ m ] = Pr[ ]Pr[ 2 ] Pr[ m m ] ( Pr[no collision] = ) ( m ) n n ln(pr[no collision]) = m k= = n ( ) We used ln( ε) ε for ε ( ) m = (m )m/2 ln( k m n ) ( k n ) ( ) k= ( ) m(m ) m2 2 pproximation exp{ x} = x + 2! x 2 + x, for x x ln( x) for x Today s your birthday, it s my birthday too Probability that m people all have different birthdays? With n = 365, one finds Pr[collision] /2 if m If m = 6, we find that 62 } = exp{ } 7 If m = 366, then Pr[no collision] = (No approximation here!)

5 Checksums! Coupon Collector Problem Coupon Collector Problem: nalysis Consider a set of m files Each file has a checksum of b bits How large should b be for Pr[share a checksum] 3? Claim: b 29ln(m) + 9 Proof: Let n = 2 b be the number of checksums We know Pr[no collision] exp{ m 2 /()} m 2 /() Pr[no collision] 3 m 2 /() 3 m b+ m 2 2 b + + 2log 2 (m) + 29ln(m) Note: log 2 (x) = log 2 (e)ln(x) 44ln(x) There are n different baseball cards (rian Wilson, Jackie Robinson, Roger Hornsby, ) One random baseball card in each cereal box If you buy m boxes, (a) Pr[miss one specific item] e m n (b) Pr[miss any one of the items] ne m n Event m = fail to get rian Wilson in m cereal boxes Fail the first time: ( n ) Fail the second time: ( n ) nd so on for m times Pr[ m ] = ( n ) ( n ) = ( n )m ln(pr[ m ]) = m ln( n ) m ( n ) Pr[ m ] exp{ m n } For p m = 2, we need around n ln2 69n boxes Collect all cards? Collect all cards? Summary Experiment: Choose m cards at random with replacement Events: E k = fail to get player k, for k =,, n Probability of failing to get at least one of these n players: p := Pr[E E 2 E n ] How does one estimate p? Union ound: p = Pr[E E 2 E n ] Pr[E ] + Pr[E 2 ] Pr[E n ] Pr[E k ] e m n,k =,,n Plug in and get p ne m n Thus, Pr[missing at least one card] ne m n Pr[missing at least one card] p when m n ln( n p ) To get p = /2, set m = n ln() Eg, n = 2 m = 53;n = 3 m = 76 ayes Rule, Mutual Independence, Collisions and Collecting Main results: ayes Rule: Pr[ m ] = p m q m /(p q + + p M q M ) Product Rule: Pr[ n ] = Pr[ ]Pr[ 2 ] Pr[ n n ] alls in bins: m balls into n > m bins Pr[no collisions] exp{ m2 } Coupon Collection: n items uy m cereal boxes Pr[miss one specific item] e m n ; Pr[miss any one of the items] ne m n Key Mathematical Fact: ln( ε) ε