Probability Theory Overview and Analysis of Randomized Algorithms

Transcription

1 Probability Theory Overview and Analysis of Randomized Algorithms Analysis of Algorithms Prepared by John Reif, Ph.D.

2 Probability Theory Topics a) Random Variables: Binomial and Geometric b) Useful Probabilistic Bounds and Inequalities

3 Readings Main Reading Selections: CLR, Chapter 5 and Appendix C

4 Probability Measures A probability measure (Prob) is a mapping from a set of events to the reals such that: 1) For any event A 0 < Prob(A) < 1 2) Prob (all possible events) = 1 3) If A, B are mutually exclusive events, then Prob(A B) = Prob (A) + Prob (B)

5 Conditional Probability Define Prob(A B) Prob(A B) = Prob(B) for Prob(B) > 0

6 Bayes Theorem If A 1,, A n are mutually exclusive and contain all events then Prob(A B) i = n P j=1 where P = Prob(B A ) Prob(A ) i P j j j j

7 Random Variable A (Over Real Numbers) Density Function f (x) = Prob(A=x) A

8 Random Variable A (cont d) Prob Distribution Function F (x) = Prob(A x) = f (x) dx A x - A

9 Random Variable A (cont d) If for Random Variables A,B x F (x) A F (x) Then A upper bounds B and B lower bounds A B F (x) = Prob (A x) A F (x) = Prob (B x) B

10 Expectation of Random Variable A E(A) = A = - x f (x) dx A Ā is also called average of A and mean of A = µ A

11 Variance of Random Variable A σ = (A A) = A (A) A where 2nd momoment A 2 2 = - x f (x) dx A

12 Variance of Random Variable A (cont d)

13 Discrete Random Variable A

14 Discrete Random Variable A (cont d)

15 Discrete Random Variable A Over Nonnegative Numbers Expectation E(A) = A = x=0 x f (x) A

16 Pair-Wise Independent Random Variables A,B independent if Prob(A B) = Prob(A) * Prob(B) Equivalent definition of independence f (x) = f (x) f (x) A B A B M G (s) = M (s) M (s) A B A B (z) = G (z) G (z) A B A B

17 Bounding Numbers of Permutations n! = n * (n-1) * 2 * 1 = number of permutations of n objects Stirling s formula n! = f(n) (1+o(1)), where n -n f(n) = n e 2πn

18 Bounding Numbers of Permutations (cont d) Note Tighter bound f(n) e 1 1 (12n+1) 12n < n! < f(n) e n! (n-k)! = number of permutations of n objects taken k at a time

19 Bounding Numbers of Combinations n n! = k k! (n-k)! = number of (unordered) combinations of n objects taken k at a time Bounds (due to Erdos & Spencer, p. 18) 2 3 k k k 2n 6n 2 n ne ~ (1 o(1)) k k! 3 for k = o n 4

20 Bernoulli Variable A i is 1 with prob P and 0 with prob 1-P Binomial Variable B is sum of n independent Bernoulli variables A i each with some probability p procedure begin end BINOMIAL with parameters n,p B 0 for i=1 to n do with probability P do B B+1 output

21

22 B is Binomial Variable with Parameters n,p mean µ = n p 2 variance σ = np (1-p)

23 B is Binomial Variable with Parameters n,p (cont d) density fn n = Prob(B=x) = x x n-x p (1-p) distribution fn n = k=0 k x k n-k Prob(B x) = p (1-p)

24 Poisson Trial A i is 1 with prob P i and 0 with prob 1-P i Suppose B is the sum of n independent Poisson trials A i with probability P i for i > 1,, n

25 Hoeffding s Theorem B is upper bounded by a Binomial Variable B Parameters n,p where n i=1 p= n P i

26 Geometric Variable V parameter p x 0 Prob(V=x) = p(1-p) x procedure GEOMETRIC parameter p begin V 0 loop with probability 1-p goto exit

27 Probabilistic Inequalities For Random Variable A mean µ = A variance σ = A (A) 2 2 2

28 Markov and Chebychev Probabilistic Inequalities Markov Inequality Prob (A x) µ x (uses only mean) Chebychev Inequality (uses mean and variance) 2 σ Prob ( A µ Δ) Δ 2

29 Example of use of Markov and Chebychev Probabilistic Inequalities If B is a Binomial with parameters n,p np Then Prob (B x) x Prob ( B np Δ) np (1-p) 2 Δ

30 Gaussian Density Function Ψ(x) = 1 2π e 2 x - 2

31 Normal Distribution Φ (X) = Bounds x > 0 (Feller, p. 175) x - Ψ(Y) dy 1 1 Ψ(x) Ψ(x) 1 Φ(x) 3 x ε [0,1] x x x x 1 x = x Ψ(1) Φ(x) - Ψ(0) = 2πe 2 2π

32 Sums of Independently Distributed Variables Let S n be the sum of n independently distributed variables A 1,, A n Each with mean µ n and variance 2 σ n So S n has mean µ and variance σ 2

33 Strong Law of Large Numbers: Limiting to Normal Distribution The probability density function of T = n (Sn - µ ) limits as n σ to normal distribution Φ(x) Hence Prob (S -µ σx) Φ(x) as n n

34 Strong Law of Large Numbers (cont d) So Prob (S -µ σx) 2(1 Φ(x)) n 2 Ψ(x)/x (since 1- Φ(x) < Ψ(x)/x)

35 Advanced Material Moment Generating Functions and Chernoff Bounds

36 Moments of Random Variable A (cont d) n th Moments of Random Variable A A n = - n x f A(x) dx Moment generating function M (s) A = sx e f A(x) dx - sa = E(e )

37 Moments of Random Variable A (cont d) Note S is a formal parameter A n dm n (s) = A n ds s= 0

38 Moments of Discrete Random Variable A n th moment A n = x=0 n x f (x) A

39 Probability Generating Function of Discrete Random Variable A x A G A(z) = z f A(x) = E(z ) x=0 1st derivative G ' A (1) = A " 2 2nd derivative G A(1) = A A 2 = G " + G ' G ' 2 A A A A variance σ (1) (1) ( (1))

40 Moments of AND of Independent Random Variables If A 1,, A n independent with same distribution f (x) = f (x) for i = 1,... n A A 1 i Then if B = A A... A f (x) = f B 1 2 n n ( ) A (x) 1 ( ) ( ) M (s) = M (s), G (z) = G (z) B A B A 1 1 n n

41 Generating Function of Binomial Variable B with Parameters n,p n G(z) = (1-p+pz) = z p (1-p) k=0 k Interesting fact x n k k n-k Prob(B= µ ) = Ω 1 n

42 Generating Function of Geometric Variable with parameter p G(Z) = k=0 k k p Z (p(1-p) ) = 1-(1-p)Z

43 Chernoff Bound of Random Variable A Uses all moments Uses moment generating function -sx Prob (A x) e M A(s) for s 0 γ(s) - sx = e where γ(s) = ln (M A (s)) e γ(s) - s γ'(s) By setting x = ɣ (s) 1 st derivative minimizes bounds

44 Chernoff Bound of Discrete Random Variable A -x Prob (A x) z G A(z) for z 1 Choose z = z 0 to minimize above bound Need Probability Generating function A x 0 x G (z) = z f (x) = E(z ) A A

45 Chernoff Bounds for Binomial B with parameters n,p Above mean x > µ Prob (B x) n-x n-µ µ n-x x x x x x-µ µ 1 1 e since 1 < e x x e for x µe -x - µ 2

46 Chernoff Bounds for Binomial B with parameters n,p (cont d) Below mean x < µ Prob (B x) n-x n-µ µ n-x x x

47 Anguin-Valiant s Bounds for Binomial B with Parameters n,p Just above mean µ = np for 0 < ε < 1 µ Prob (B (1+ε)µ) e -ε2 2 Just below mean µ < np for 0 < ε < 1 µ Prob (B " (1-ε)µ! ) e -ε2 3

48 Anguin-Valiant s Bounds for Binomial B with Parameters n,p (cont d) Tails are bounded by Normal distributions

49 Binomial Variable B with Parameters p,n and Expectation μ= np By Chernoff Bound for p < ½ N Prob (B ) < 2 p 2 Raghavan-Spencer bound for any > 0 n N 2 Prob (B (1+ )µ)! # " e (1+ ) (1+ ) $ & % µ

50 Probability Theory Analysis of Algorithms Prepared by John Reif, Ph.D.