16 Independence Definitions Potential Pitfall Alternative Formulation. mcs-ftl 2010/9/8 0:40 page 431 #437

Transcription

1 cs-ftl 010/9/8 0:40 page 431 # Independence 16.1 efinitions Suppose that we flip two fair coins siultaneously on opposite sides of a roo. Intuitively, the way one coin lands does not affect the way the other coin lands. The atheatical concept that captures this intuition is called independence: efinition Events A and B are independent if PrŒB 0 or if Pr A j B PrŒA : (16.1) In other words, A and B are independent if knowing that B happens does not alter the probability that A happens, as is the case with flipping two coins on opposite sides of a roo Potential Pitfall Students soeties get the idea that disjoint events are independent. The opposite is true: if A \ B ;, then knowing that A happens eans you know that B does not happen. So disjoint events are never independent unless one of the has probability zero Alternative Forulation Soeties it is useful to express independence in an alternate for: Theore A and B are independent if and only if PrŒA \ B PrŒA PrŒB : (16.) Proof. There are two cases to consider depending on whether or not PrŒB 0. Case 1.PrŒB 0/: If PrŒB 0, A and B are independent by efinition In addition, Equation 16. holds since both sides are 0. Hence, the theore is true in this case. Case.PrŒB > 0/: By efinition , PrŒA \ B Pr A j B PrŒB : 1

2 cs-ftl 010/9/8 0:40 page 43 #438 Chapter 16 Independence So Equation 16. holds if Pr A j B PrŒA ; which, by efinition , is true iff A and B are independent. Hence, the theore is true in this case as well. 16. Independence Is an Assuption Generally, independence is soething that you assue in odeling a phenoenon. For exaple, consider the experient of flipping two fair coins. Let A be the event that the first coin coes up heads, and let B be the event that the second coin is heads. If we assue that A and B are independent, then the probability that both coins coe up heads is: PrŒA \ B PrŒA PrŒB : In this exaple, the assuption of independence is reasonable. The result of one coin toss should have negligible ipact on the outcoe of the other coin toss. And if we were to repeat the experient any ties, we would be likely to have A \ B about 1/4 of the tie. There are, of course, any exaples of events where assuing independence is not justified, For exaple, let C be the event that toorrow is cloudy and R be the event that toorrow is rainy. Perhaps PrŒC 1=5 and PrŒR 1=10 in Boston. If these events were independent, then we could conclude that the probability of a rainy, cloudy day was quite sall: PrŒR \ C PrŒR PrŒC : Unfortunately, these events are definitely not independent; in particular, every rainy day is cloudy. Thus, the probability of a rainy, cloudy day is actually 1=10. eciding when to assue that events are independent is a tricky business. In practice, there are strong otivations to assue independence since any useful forulas (such as Equation 16.) only hold if the events are independent. But you need to be careful lest you end up deriving false conclusions. We ll see several faous exaples where (false) assuptions of independence led to trouble over the next several chapters. This proble gets even trickier when there are ore than two events in play.

3 cs-ftl 010/9/8 0:40 page 433 # Mutual Independence 16.3 Mutual Independence efinition We have defined what it eans for two events to be independent. What if there are ore than two events? For exaple, how can we say that the flips of n coins are all independent of one another? Events E 1 ; : : : ; E n are said to be utually independent if and only if the probability of any event E i is unaffected by knowledge of the other events. More forally: efinition A set of events E 1 ; E ; : : : ; E n, is utually independent if 8i Œ1; n and 8S Œ1; n fig, either 3 3 Pr 4 \ \ E j 5 0 or PrŒE i Pr 4E i ˇˇˇ 5 : j S In other words, no atter which other events are known to occur, the probability that E i occurs is unchanged for any i. For exaple, if we toss 100 fair coins at different ties, we ight reasonably assue that the tosses are utually independent since the probability that the i th coin is heads should be 1=, no atter which other coin tosses cae out heads Alternative Forulation Just as Theore provided an alternative definition of independence for two events, there is an alternative definition for utual independence. Theore A set of events E 1 ; E ; : : : ; E n is utually independent iff 8S Œ1; n, 3 Pr 4 \ E j 5 Y PrŒE j : j S j S j S E j The proof of Theore uses induction and reasoning siilar to the proof of Theore We will not include the details here. Theore says that E 1 ; E ; : : : ; E n are utually independent if and only 3

4 cs-ftl 010/9/8 0:40 page 434 #440 Chapter 16 Independence if all of the following equations hold for all distinct i, j, k, and l: PrŒE i \ E j PrŒE i PrŒE j PrŒE i \ E j \ E k PrŒE i PrŒE j PrŒE k PrŒE i \ E j \ E k \ E l PrŒE i PrŒE j PrŒE k PrŒE l PrŒE 1 \ \ E n PrŒE 1 PrŒE n : : For exaple, if we toss n fair coins, the tosses are utually independent iff for all Œ1; n and every subset of coins, the probability that every coin in the subset coes up heads is A Testing Assuptions about independence are routinely ade in practice. Frequently, such assuptions are quite reasonable. Soeties, however, the reasonableness of an independence assuption is not so clear, and the consequences of a faulty assuption can be severe. For exaple, consider the following testiony fro the O. J. Sipson urder trial on May 15, 1995: Mr. Clarke: When you ake these estiations of frequency and I believe you touched a little bit on a concept called independence? r. Cotton: Yes, I did. Mr. Clarke: And what is that again? r. Cotton: It eans whether or not you inherit one allele that you have is not does not affect the second allele that you ight get. That is, if you inherit a band at 5,000 base pairs, that doesn t ean you ll autoatically or with soe probability inherit one at 6,000. What you inherit fro one parent is what you inherit fro the other. Mr. Clarke: Why is that iportant? r. Cotton: Matheatically that s iportant because if that were not the case, it would be iproper to ultiply the frequencies between the different genetic locations. Mr. Clarke: How do you well, first of all, are these arkers independent that you ve described in your testing in this case? 4

5 cs-ftl 010/9/8 0:40 page 435 # Pairwise Independence Presuably, this dialogue was as confusing to you as it was for the jury. Essentially, the jury was told that genetic arkers in blood found at the crie scene atched Sipson s. Furtherore, they were told that the probability that the arkers would be found in a randoly-selected person was at ost 1 in 170 illion. This astronoical figure was derived fro statistics such as: 1 person in 100 has arker A. 1 person in 50 arker B. 1 person in 40 has arker C. 1 person in 5 has arker. 1 person in 170 has arker E. Then these nubers were ultiplied to give the probability that a randoly-selected person would have all five arkers: PrŒA \ B \ C \ \ E PrŒA PrŒB PrŒC PrŒ PrŒE ;000;000 : The defense pointed out that this assues that the arkers appear utually independently. Furtherore, all the statistics were based on just a few hundred blood saples. After the trial, the jury was widely ocked for failing to understand the A evidence. If you were a juror, would you accept the 1 in 170 illion calculation? 16.4 Pairwise Independence The definition of utual independence sees awfully coplicated there are so any subsets of events to consider! Here s an exaple that illustrates the subtlety of independence when ore than two events are involved. Suppose that we flip three fair, utually-independent coins. efine the following events: A 1 is the event that coin 1 atches coin. A is the event that coin atches coin 3. 5

6 cs-ftl 010/9/8 0:40 page 436 #44 Chapter 16 Independence A 3 is the event that coin 3 atches coin 1. Are A 1, A, A 3 utually independent? The saple space for this experient is: fhhh; HH T; H TH; H T T; THH; TH T; T TH; T T T g: Every outcoe has probability.1=/ 3 1=8 by our assuption that the coins are utually independent. To see if events A 1, A, and A 3 are utually independent, we ust check a sequence of equalities. It will be helpful first to copute the probability of each event A i : PrŒA 1 PrŒHHH C PrŒHH T C PrŒT TH C PrŒT T T 1 8 C 1 8 C 1 8 C : By syetry, PrŒA PrŒA 3 1= as well. ow we can begin checking all the equalities required for utual independence in Theore 16.3.: PrŒA 1 \ A PrŒHHH C PrŒT T T 1 8 C PrŒA 1 PrŒA : By syetry, PrŒA 1 \ A 3 PrŒA 1 PrŒA 3 and PrŒA \ A 3 PrŒA PrŒA 3 ust hold also. Finally, we ust check one last condition: PrŒA 1 \ A \ A 3 PrŒHHH C PrŒT T T 1 8 C PrŒA 1 PrŒA PrŒA : 6

7 cs-ftl 010/9/8 0:40 page 437 # Pairwise Independence The three events A 1, A, and A 3 are not utually independent even though any two of the are independent! This not-quite utual independence sees weird at first, but it happens. It even generalizes: efinition A set A 1, A,..., of events is k-way independent iff every set of k of these events is utually independent. The set is pairwise independent iff it is -way independent. So the sets A 1, A, A 3 above are pairwise independent, but not utually independent. Pairwise independence is a uch weaker property than utual independence. For exaple, suppose that the prosecutors in the O. J. Sipson trial were wrong and arkers A, B, C,, and E appear only pairwise independently. Then the probability that a randoly-selected person has all five arkers is no ore than: PrŒA \ B \ C \ \ E PrŒA \ E PrŒA PrŒE ;000 : The first line uses the fact that A\B \C \ \E is a subset of A\E. (We picked out the A and E arkers because they re the rarest.) We use pairwise independence on the second line. ow the probability of a rando atch is 1 in 17,000 a far cry fro 1 in 170 illion! And this is the strongest conclusion we can reach assuing only pairwise independence. On the other hand, the 1 in 17,000 bound that we get by assuing pairwise independence is a lot better than the bound that we would have if there were no independence at all. For exaple, if the arkers are dependent, then it is possible that everyone with arker E has arker A, everyone with arker A has arker B, everyone with arker B has arker C, and everyone with arker C has arker. In such a scenario, the probability of a atch is PrŒE 1=170: 7

8 cs-ftl 010/9/8 0:40 page 438 #444 Chapter 16 Independence So a stronger independence assuption leads to a saller bound on the probability of a atch. The trick is to figure out what independence assuption is reasonable. Assuing that the arkers are utually independent ay well not be reasonable unless you have exained hundreds of illions of blood saples. Otherwise, how would you know that arker does not show up ore frequently whenever the other four arkers are siultaneously present? We will conclude our discussion of independence with a useful, and soewhat faous, exaple known as the Birthday Paradox The Birthday Paradox Suppose that there are 100 students in a class. What is the probability that soe birthday is shared by two people? Coparing 100 students to the 365 possible birthdays, you ight guess the probability lies soewhere around 1=3 but you d be wrong: the probability that there will be two people in the class with atching birthdays is actually 0: : : :. In other words, the probability that all 100 birthdays are different is less than 1 in 3,000,000. Why is this probability so sall? The answer involves a phenoenon known as the Birthday Paradox (or the Birthday Principle), which is surprisingly iportant in coputer science, as we ll see later. Before delving into the analysis, we ll need to ake soe odeling assuptions: For each student, all possible birthdays are equally likely. The idea underlying this assuption is that each student s birthday is deterined by a rando process involving parents, fate, and, u, soe issues that we discussed earlier in the context of graph theory. The assuption is not copletely accurate, however; a disproportionate nuber of babies are born in August and Septeber, for exaple. Birthdays are utually independent. This isn t perfectly accurate either. For exaple, if there are twins in the class, then their birthdays are surely not independent. We ll stick with these assuptions, despite their liitations. Part of the reason is to siplify the analysis. But the bigger reason is that our conclusions will apply to any situations in coputer science where twins, leap days, and roantic holidays are not considerations. After all, whether or not two ites collide in a hash table really has nothing to do with huan reproductive preferences. Also, in pursuit of 8

9 cs-ftl 010/9/8 0:40 page 439 # The Birthday Paradox generality, let s switch fro specific nubers to variables. Let be the nuber of people in the roo, and let be the nuber of days in a year. We can solve this proble using the standard four-step ethod. However, a tree diagra will be of little value because the saple space is so enorous. This tie we ll have to proceed without the visual aid! Step 1: Find the Saple Space Let s nuber the people in the roo fro 1 to. An outcoe of the experient is a sequence.b 1 ; : : : ; b / where b i is the birthday of the ith person. The saple space is the set of all such sequences: S f.b 1 ; : : : ; b / j b i f1; : : : g g: Step : efine Events of Interest Our goal is to deterine the probability of the event A in which soe pair of people have the sae birthday. This event is a little awkward to study directly, however. So we ll use a coon trick, which is to analyze the copleentary event A, in which all people have different birthdays: A f.b 1 ; : : : ; b / S j all b i are distinct g: If we can copute PrŒA, then we can copute what really want, PrŒA, using the identity PrŒA C PrŒA 1: Step 3: Assign Outcoe Probabilities We need to copute the probability that people have a particular cobination of birthdays.b 1 ; : : : ; b /. There are possible birthdays and all of the are equally likely for each student. Therefore, the probability that the ith person was born on day b i is 1=. Since we re assuing that birthdays are utually independent, we can ultiply probabilities. Therefore, the probability that the first person was born on day b 1, the second on b, and so forth is.1= /. This is the probability of every outcoe in the saple space, which eans that the saple space is unifor. That s good news, because, as we have seen, it eans that the analysis will be sipler. Step 4: Copute Event Probabilities We re interested in the probability of the event A in which everyone has a different birthday: A f.b 1 ; : : : ; b n / j all b i are distinct g: 9

10 cs-ftl 010/9/8 0:40 page 440 #446 Chapter 16 Independence This is a gigantic set. In fact, there are choices for b i, so forth. Therefore, by the Generalized Product Rule, 1 choices for b, and jaj Š. 1/. /. C 1/:. /Š Since the saple space is unifor, we can conclude that PrŒA jaj Š. /Š : (16.3) We re done! Or are we? While correct, it would certainly be nicer to have a closed-for expression for Equation That eans finding an approxiation for Š and. /Š. But this is what we learned how to do in Section 9.6. In fact, since and are each at least 100, we know fro Corollary 9.6. that p e and p. / e are excellent approxiations (accurate to within.09%) of Š and. /Š, respectively. Plugging these values into Equation 16.3 eans that (to within.%) 1 PrŒA p e p. / r r r e. e e ln. / e ln. / e. / ln. /. / e. / ln. /. / ln. / / ln. e. / C 1 / ln. / : (16.4) 1 If there are two ters that can be off by.09%, then the ratio can be off by at ost a factor of.1:0009/ < 1:00. 10

11 cs-ftl 010/9/8 0:40 page 441 # The Birthday Paradox We can now evaluate Equation 16.4 for 100 and 365 to find that the probability that all 100 birthdays are different is 3:07 : : : 10 7 : We can also plug in other values of to find the nuber of people so that the probability of a atching birthday will be about 1=. In particular, for 3 and 365, Equation 16.4 reveals that the probability that all the birthdays differ is So if you are in a roo with 3 other people, the probability that soe pair of people share a birthday will be a little better than 1=. It is because 3 sees like such a sall nuber of people for a atch that the phenoenon is called the Birthday Paradox Applications to Hashing Hashing is frequently used in coputer science to ap large strings of data into short strings of data. In a typical scenario, you have a set of ites and you would like to assign each ite to a nuber fro 1 to where no pair of ites is assigned to the sae nuber and is as sall as possible. For exaple, the ites ight be essages, addresses, or variables. The nubers ight represent storage locations, devices, indices, or digital signatures. If two ites are assigned to the sae nuber, then a collision is said to occur. Collisions are generally bad. For exaple, collisions can correspond to two variables being stored in the sae place or two essages being assigned the sae digital signature. Just iagine if you were doing electronic banking and your digital signature for a $10 check were the sae as your signature for a $10 illion dollar check. In fact, finding collisions is a coon technique in breaking cryptographic codes. 3 In practice, the assignent of a nuber to an ite is done using a hash function h W S! Œ1; ; where S is the set of ites and jsj. Typically, the values of h.s/ are assigned randoly and are assued to be equally likely in Œ1; and utually independent. For efficiency purposes, it is generally desirable to ake as sall as necessary to accoodate the hashing of ites without collisions. Ideally, would be only a little larger than. Unfortunately, this is not possible for rando hash functions. To see why, let s take a closer look at Equation The possible.% error is so sall that it is lost in the... after Such techniques are often referred to as birthday attacks because of the association of such attacks with the Birthday Paradox. 11

12 cs-ftl 010/9/8 0:40 page 44 #448 Chapter 16 Independence By Theore and the derivation of Equation 16.4, we know that the probability that there are no collisions for a rando hash function is e. C 1 / ln. / : (16.5) For any, we now need to find a value of for which this expression is at least 1/. That will tell us how big the hash table needs to be in order to have at least a 50% chance of avoiding collisions. This eans that we need to find a value of for which C 1 ln To siplify Equation 16.6, we need to get rid of the ln this by using the Taylor Series expansion for 1 ln : (16.6) ter. We can do ln.1 x/ x x x 3 3 to find that 4 ln ln ln C C C : 4 This ay not look like a siplification, but stick with us here. 1

13 cs-ftl 010/9/8 0:40 page 443 # The Birthday Paradox Hence, C 1 ln C 1 C C C 3 3 C C C C 3 C C C 1 C C C C 3 6 C C C 1 C C C : (16.7) If grows faster than, then the value in Equation 16.7 tends to 0 and Equation 16.6 cannot be satisfied. If grows ore slowly than, then the value in Equation 16.7 diverges to negative infinity, and, once again, Equation 16.6 cannot be satisfied. This suggests that we should focus on the case where. /, when Equation 16.7 siplifies to and Equation 16.6 becoes ln 1 : (16.8) Equation 16.8 is satisfied when ln./ : (16.9) In other words, needs to grow quadratically with in order to avoid collisions. This unfortunate fact is known as the Birthday Principle and it liits the efficiency of hashing in practice either is quadratic in the nuber of ites being hashed or you need to be able to deal with collisions. 13

14 cs-ftl 010/9/8 0:40 page 444 #450 14

15 MIT OpenCourseWare J / 18.06J Matheatics for Coputer Science Fall 010 For inforation about citing these aterials or our Ters of Use, visit: