The first bound is the strongest, the other two bounds are often easier to state and compute. Proof: Applying Markov's inequality, for any >0 we have

Transcription

1 The first bound is the strongest, the other two bounds are often easier to state and compute Proof: Applying Markov's inequality, for any >0 we have Pr (1 + ) = Pr For any >0, we can set = ln 1+ (4.4.1): Pr (1+ )< (1+) ) >0to get. MAT RandAl, Spring Feb For (4.4.2) we need to show that, for 0 < 1, (1 + ) ). Taking the logarithm of both sides, we obtain the equivalent condition 1 + ln Computing the derivatives of, we have: )=1 1+ ln ln MAT RandAl, Spring Feb

2 We see that < 0for <1/2 and >0 for 12 Hence first decreases and then increases over the interval [0,1] Since (0)=0and 1 <0, we can conclude that in the interval [0,1] Since (0)=0, it follows that 0in that interval, proving (4.4.2). MAT RandAl, Spring Feb To prove (4.4.3), let = (1 +. Then, for, 5. Hence, using (4.4.1), Pr (1+ 1+ (1 + ) ) 6 2 MAT RandAl, Spring Feb

3 We obtain similar results bounding the deviation below the mean Theorem 4.5: Let,, be independent Poisson trials s.t. Pr =1 =. Let = and ]. Then for 0<<1: Pr (1 (1 + ) ) Pr (1. Again, the first bound is stronger, but the latter is generally easier to use and sufficient in most applications ; MAT RandAl, Spring Feb Often the following form of the Chernoff bound is used Corollary 4.6: Let,, be independent Poisson trials s.t. Pr =1 =. Let = and ]. For 0<<1: Pr( In practice we often do not have the exact value of ] Instead we can use in Theorem 4.4 and []in Theorem 4.5. MAT RandAl, Spring Feb

4 Example: Coin Flips Let be the number of heads in a sequence of independent fair coin flips Applying the Chernoff bound (4.6), we have Pr ln 2 exp = 2 6ln Thus, the concentration around the mean /2 is very tight; most of the time, the deviations from the mean are of the order of ( ln ) MAT RandAl, Spring Feb Consider the Pr of /4 or /4 heads in a sequence of independent fair coin flips Chebyshev s inequality showed that Pr 2 4 Already, this is worse than the Chernoff bound just calculated for a significantly larger event! Using the Chernoff bound, we find that Pr exp 1 1 = Thus, Chernoff's technique gives a bound that is exponentially smaller than that obtained using Chebyshev's inequality MAT RandAl, Spring Feb

5 Application: Estimating a Parameter Evaluate the probability that a particular gene mutation occurs in the population An expensive lab test determines if a DNA sample carries the mutation We would like to obtain a relatively reliable estimate from a small number of samples Let be the unknown value that we are trying to estimate Assume that we have samples and that = of these samples have the mutation MAT RandAl, Spring Feb Given a sufficiently large number of samples, we expect to be close to the sampled value Definition 4.2: A confidence interval for a parameter is an interval ], s.t. Pr( Instead of predicting a single value for the parameter, we give an interval that is likely to contain the parameter If can take on any real value, it may not make sense to try to pin down its exact value from a finite sample, but it does make sense to estimate it within some small range MAT RandAl, Spring Feb

6 We want both the interval size and the error probability to be as small as possible We derive a trade-off between these two parameters and the number of samples In particular, given that among samples (chosen uniformly at random from the entire population) we find the mutation in exactly samples, we need to find values of and for which Pr( ) = Pr( ), )) MAT RandAl, Spring Feb Now ), so ] = If then we have one of the following two events: 1. if, then = ](1 + ) ; 2. if, then = ](1 ) We can apply the Chernoff bounds of Thms 4.4 and 4.5 to compute Pr = Pr + Pr 1+ MAT RandAl, Spring Feb

7 This bound is not useful because the value of is unknown A simple solution is to use the fact that 1, yielding Pr Setting, we obtain a tradeoff between,, and the error probability MAT RandAl, Spring Feb Better Bounds for Some Special Cases We can obtain stronger bounds using a simpler proof technique for some special cases of symmetric RVs Theorem 4.7: Let,, be independent RVs with Let = Pr = 1 = Pr 1 = For any >0, Pr =. MAT RandAl, Spring Feb

8 Proof: For any > 0, To estimate and = , we observe that =1++! +! =1+ + 1!! using the Taylor series expansion for. Thus, =! 2! MAT RandAl, Spring Feb Using this estimate yields and = Pr = Pr Setting, we obtain Pr By symmetry we also have Pr MAT RandAl, Spring Feb

9 Corollary 4.8: Let,, be independent RVs, Let = Pr = 1 = Pr 1 = For any >0, Pr = 2. Apply transformation = ( + 1)/2 to prove Corollary 4.9: Let,, be independent RVs, Pr = 1 = Pr = 0 = 1 2. Let = and For any >0,Pr. 2. For any > 0, Pr (1 + ) =. MAT RandAl, Spring Feb Application: Set Balancing Given an matrix with entries in {0,1}, let = We are looking for a vector with entries in 1,1} that minimizes = max,, MAT RandAl, Spring Feb

10 This problem arises in designing statistical experiments Each column of the matrix represents a subject in the experiment and each row represents a feature The vector partitions the subjects into two disjoint groups, so that each feature is roughly as balanced as possible between the two groups One of the groups serves as a control group for an experiment that is run on the other group MAT RandAl, Spring Feb We randomly choose the entries of, with Pr = 1 = Pr 1 = 1 2 The choices for different entries are independent Surprisingly, although this algorithm ignores the entries of the matrix, is likely to be only ln This bound is fairly tight: When, there exists a matrix for which is for any choice of MAT RandAl, Spring Feb

11 Theorem 4.11: For a random vector with entries chosen independently and with equal probability from the set 1,1}, Pr ln 2. Proof: Consider the th row,,, and let be the number of 1s in that row. If ln, then clearly = ln. On the other hand, if > ln then we note that the nonzero terms in the sum MAT RandAl, Spring Feb = are independent random variables, each with probability 1/2 of being either +1 or 1. Now using the Chernoff bound of Corollary 4.8 and the fact that, Pr > ln 2. By the union bound, the probability that the bound fails for any row is at most 2/. MAT RandAl, Spring Feb

12 5 Balls, Bins, and Random Graphs Let us throw balls randomly into bins, each ball lands in a bin chosen independently and uniformly at random (I+U@R) We use the techniques we have developed previously to analyze this process and develop a new approach based on what is known as the Poisson approximation MAT RandAl, Spring Feb Example: The Birthday Paradox Is it more likely that some two people in the room share the same birthday or that no two people in the room share the same birthday? We assume that the birthday of each person is a random day from a 365-day year, each chosen I+U@R We assume that a person's birthday is equally likely to be any day of the year, we avoid leap years, and we ignore the possibility of twins MAT RandAl, Spring Feb

13 Let there be 30 people Thirty days must be chosen from the 365; there ways to do this are These 30 days can be assigned to the people in any of the 30! possible orders Hence there are 30! configurations where no two people share the same birthday, out of the 365 ways the birthdays could occur Thus, the probability is 30! 365 MAT RandAl, Spring Feb We can also consider one person at a time The first person has a birthday The probability that the second person has a different birthday is (1 1/365) The probability that the third person then has a birthday different from the first two, given that they have different birthdays, is (1 2/365) Continuing on, the probability that the th person has a different birthday than the first 1, assuming that they have different birthdays, is (1 ( 1)/365) MAT RandAl, Spring Feb

14 So the probability that 30 people all have different birthdays is the product of these terms: This product is , so when 30 people are in the room there is more than a 70% chance that two share the same birthday A similar calculation shows that only 23 people need to be in the room before it is more likely than not that two people share a birthday MAT RandAl, Spring Feb More generally, if there are people and possible birthdays then the probability that all have different birthdays is Using that when is small compared to, we see that if is small compared to then = exp MAT RandAl, Spring Feb

15 Hence the value for at which the probability that people all have different birthdays is 1/2 is approximately given by the equation =ln2, or = ln 2 For = 365, this approximation gives = 22.49, matching the exact calculation quite well Mars has = 687 days, need = aliens MAT RandAl, Spring Feb The following simple arguments give loose bounds and good intuition Let us consider each person one at a time, and let be the event that the th person's birthday does not match any of the birthdays of the first 1 people Then the probability that the first people fail to have distinct birthdays is Pr Pr 1 1) If this Pr is < 1/2, so with people the Pr is 1/2 that all birthdays will be distinct MAT RandAl, Spring Feb

16 Now assume that the first distinct birthdays people all have Each person after that has probability at least =1 of having the same birthday as one of these first people Hence the Pr that the next people all have different birthdays than the first 1 < 1 <1 2 Hence, once there are 2 people, the Pr is at most 1/ that all birthdays will be distinct MAT RandAl, Spring Feb Balls into Bins balls are thrown into bins, with the location of each ball chosen I+U@R from the possibilities The question behind the birthday paradox is whether or not there is a bin with two balls How many of the bins are empty? How many balls are in the fullest bin? Many of these questions have applications to the design and analysis of algorithms MAT RandAl, Spring Feb

17 Birthday paradox: place balls randomly into bins then, for some, at least one of the bins is likely to have more than one ball in it Another interesting question concerns the max number of balls in a bin, or the maximum load Let us consider the case where, so that the number of balls equals the number of bins and the average load is 1 Of course the maximum load is, but it is very unlikely that all balls land in the same bin We seek an upper bound that holds with probability tending to 1 as grows large MAT RandAl, Spring Feb We can show that the maximum load is not more than 3ln/ ln ln with probability at most for sufficiently large via a direct calculation and a union bound This is a very loose bound; although the maximum load is in fact (ln/ ln ln ) with probability close to 1, the constant factor 3 is chosen to simplify the argument and could be reduced with more care Lemma 5.1: When balls are thrown I+U@R into bins, the probability that the maximum load is more than 3ln/ ln ln is at most for sufficiently large MAT RandAl, Spring Feb

18 Proof: The probability that bin 1 receives at least balls is at most This follows from a union bound; there are distinct sets of balls, and for any set of balls the probability that all land in bin 1 is. 1 We now use the inequalities 1. 1! MAT RandAl, Spring Feb The second inequality is a consequence of the following general bound on factorials: since we have! <!>! Applying a union bound again allows us to find that, for 3ln/lnln, the probability that any bin receives at least balls is bounded above by lnln 3ln MAT RandAl, Spring Feb

19 lnln ln 1 for sufficiently large. MAT RandAl, Spring Feb