Tail Inequalities Randomized Algorithms. Sariel Har-Peled. December 20, 2002

Transcription

1 Tail Inequalities Randomized Algorithms Sariel Har-Peled December 0, 00 Wir mssen wissen, wir werden wissen (We must know, we shall know) David Hilbert 1 Tail Inequalities 1.1 The Chernoff Bound Special Case Theorem 1.1 Let X 1,..., X n be n independent random varaibles, such that PrX i 1 PrX i 1 1, for i 1,..., n. Let Y n i1 X i. Then, for any > 0, we have PrY e /n. Proof: Clearly, for an arbitrary t, to specified shortly, we have PrY Prexp(tY ) exp(t ) E exp(ty ) exp(t ), the first part follows by the fact that exp( ) preserve ordering, and the second part follows by the Markov inequality. Observe that Eexp(tX i ) 1 et + 1 e t et + e t 1 ) t (1 + + t1!! + t3 3! ) t (1 + t1!! t3 3! + ) (1 + + t! tk (k)! +, by the Taylor expansion of exp( ). Note, that (k)! (k!) k, and thus Eexp(tX i ) i0 t i (i)! i0 t i i (i!) i0 1 i! ( ) t i exp ( t / ), 1

2 again, by the Taylor expansion of exp( ). Next, by the independence of the X i s, we have ( ) n Eexp(tY ) E exp tx i E exp(tx i ) Eexp(tX i ) We have n e t / e nt /. i1 PrY exp(nt /) exp(t ) i i i1 exp ( nt / t ). Next, by minimizing the above quantity for t, we set t /n. We conclude, ( ( ) ) n PrY exp ) n n exp (. n By the symmetry of Y, we get the following: Corollary 1. Let X 1,..., X n be n independent random variables, such that PrX i 1 PrX i 1 1, for i 1,..., n. Let Y n i1 X i. Then, for any > 0, we have Pr Y e /n. Corollary 1.3 Let X 1,..., X n be n independent coin flips, such that PrX i 0 PrX i 1 1, for i 1,..., n. Let Y n i1 X i. Then, for any > 0, we have Y n Pr e /n. 1. The Chernoff Bound General Case Here we present the Chernoff bound in a more general settings. Question 1.4 Let 1. X 1,..., X n - n independent Bernoulli trials, where Each X i is known as a Poisson trials.. X b i1 X i. µ EX i p i. Question: Probability that X > (1 + δ)µ? PrX i 1 p i, and PrX i 0 q i 1 p i.

3 Theorem 1.5 For any δ > 0, ( ) e δ µ PrX > (1 + δ)µ <. (1 + δ) 1+δ Or in a more simplified form, for any δ e 1, PrX > (1 + δ)µ < exp ( µδ /4 ), (1) and for δ e 1. PrX > (1 + δ)µ < µ(1+δ), () Remark 1.6 Before going any further, it is maybe instrumental to understand what this inequality implies. Set all probabilities to be p i 1/, and set δ t/ µ. ( µ is approximately the standard deviation of X if p i 1/) Using very fluffy math, in particular e δ 1 + δ), we get the following: Pr X µ > tσ X PrX > (1 + δ)µ < ( ) e δ µ ( 1 + δ (1 + δ) 1+δ (1 + δ) 1+δ ( ) (t/ n)n/ 1 ( e δ) (t/ n)n/ 1 + δ e (t /n)n/ e t. Thus, Chernoff inequality implies exponential decay with the standard deviation, instead of just polynomial (like the Cheby s inequality). We emphasize again that above calculation is incorrect, and should only be interpreted as an intuition of what is going on. Proof: (of Theorem 1.5) By Markov inequality, we have: On the other hand, Ee tx E PrX > (1 + δ)µ Pr Pr X > (1 + δ)µ < e tx > e t(1+δ)µ. E e tx e t(1+δ)µ e t(x 1+X...+X n) E e tx 1 E e txn. ) n/ Namely, Pr X > (1 + δ)µ < n i1 E e tx i e t(1+δ)µ n i1 ((1 p i)e 0 + p i e t ) e t(1+δ)µ n i1 (1 + p i(e t 1)) e t(1+δ)µ. 3

4 Let y p i (e t 1). We know that 1 + y < e y (since y > 0). Thus, n i1 Pr X > (1 + δ)µ < exp(p i(e t 1)). exp( n i1 p i(e t 1)) e t(1+δ)µ e t(1+δ)µ exp((et 1) n i1 p ( i) exp((et 1)µ) exp(e t 1) e t(1+δ)µ e t(1+δ)µ e ( ) t(1+δ) µ exp(δ), (1 + δ) (1+δ) if we set t log(1 + δ). For the proof of the simplified form, see Section 1.3. Definition 1.7 F + e µ. (µ, δ) δ (1+δ) (1+δ) Example 1.8 Arkansas Aardvarks win a game with probability 1/3. What is their probability to have a winning season with n games. By Chernoff inequality, this probability is smaller than e F + 1/ n/3 (n/3, 1/) ( ) n/ n For n 40, this probability is smaller than For n 100 this is less than For n 1000, this is smaller than (which is pretty slim and shady). Namely, as the number of experiments is increases, the distribution converges to its expectation, and this converge is exponential. Exercise 1.9 Prove that for δ > e 1, we have F + (µ, δ) < (1+δ)µ e (1+δ)µ. 1 + δ ) µ Theorem 1.10 Under the same assumptions as Theorem 1.5, we have: PrX < (1 δ)µ < e µδ /. Definition 1.11 F (µ, δ) e µδ /. (µ, ε) - what should be the value of δ, so that the probability is smaller than ε. log 1/ε (µ, ε) µ For large δ: + (µ, ε) < log (1/ε) µ 1 4

5 1.3 A More Convenient Form Proof: (of simplified form of Theorem 1.5) Equation () is just Exercise 1.9. As for Equation (1), we prove this only for δ 1/. For details about the case 1/ δ e 1, see MR95. By Theorem 1.5, we have ( ) e δ µ PrX > (1 + δ)µ < exp(µδ µ(1 + δ) ln(1 + δ)). (1 + δ) 1+δ The Taylor expansion of ln(1 + δ) is for δ 1. Thus, δ δ + δ3 3 δ4 4 + δ δ, PrX > (1 + δ)µ < exp ( µ ( δ (1 + δ) ( δ δ / ))) exp ( µ ( δ δ + δ / δ + δ 3 / )) for δ 1/. exp ( µ ( δ / + δ 3 / )) exp ( µδ /4 ), Application of the Chernoff Inequality Routing in a Parallel Computer The following is based on Section 4. in MR95. G: A graph of processors. Packets can be sent on edges. 1,..., N: The vertices (i.e., processors) of G. N n, and G is a hypercube. Each processes is a binary string b 1 b... b n. Question: Given a permutation π, how to send the permutation and create minimum delay? Theorem.1 For any deterministic oblivious permutation routing algorithm on a network of N nodes each of out-degree n, there is a permutation for which the routing of the permutation takes Ω( N/n) time. How do we sent a packet? We use bit fixing. Namely, the packet from the i node, always go to the current adjacent node that have the first different bit as we scan the destination string d(i). For example, packet from (0000) going to (1101), would pass through (1000), (1100), (1101). We assume each edge have a FIFO queue. Here is the algorithm: (i) Pick a random intermediate destination σ(i) from 1,..., N. Packet v i travels to σ(i). (ii) Wait till all the packet arrive to their intermediate destination. (iii) Packet v i travels from σ(i) to its destination d(i). We analyze only (i) as (iii) follows from the same analysis. ρ i - the route taken by v i in (i). 5

6 Exercise. Once a packet v j that travel along a path ρ j can not leave a path ρ i, and then join it again later. Namely, ρ i ρ j is (maybe an empty) path. Lemma.3 Let the route of v i follow the sequence of edges ρ i (e 1, e,..., e k ). Let S be the set of packets whose routes pass through at least one of (e 1,..., e k ). Then, the delay incurred by v i is at most S. Let H ij be an indicator variable that is 1 if ρ i, ρ j share an edge, 0 otherwise. Total delay for v i is j H ij. Note, that for a fixed i, the variables H i1,..., H in are independent (not however, that H 11,..., H NN are not independent!). For ρ i (e 1,..., e k ), let T (e) be the number of packets (i.e., paths) that pass through e. N k N k H ij T (e j ) and thus E H ij E T (e j ). Because of symmetry, the variables T (e) have the same distribution for all the edges of G. On the other hand, the expected length of a path is n/, there are N packets, and there are Nn/ edges. We conclude ET (e) 1. Thus N k µ E H ij E T (e j ) E ρ i n. By the Chernoff inequality (Exercise 1.9), we have Pr H ij > 7n Pr H ij > (1 + 13)µ < 13µ 6n. j j Since there are N n packets, we know that with probability 5n all packets arrive to their temporary destination in a delay of most 7n. Theorem.4 Each packet arrives to its destination in 14n stages, in probability at least 1 1/N (note that this is very conservative). References MR95 R. Motwani and P. Raghavan. Randomized Algorithms. Cambridge University Press, New York, NY,