CS145: Probability & Computing

Transcription

1 CS45: Probability & Computing Lecture 5: Concentration Inequalities, Law of Large Numbers, Central Limit Theorem Instructor: Eli Upfal Brown University Computer Science Figure credits: Bertsekas & Tsitsiklis, Introduction to Probability, 2008 Pitman, Probability, 999

2 CS45: Lecture 5 Outline Ø Markov and Chebyshev Inequalities Ø Law of Large Numbers Ø Central Limit Theorem

3 E[X] = Theorem Markov s Inequality [Markov Inequality] For any non-negative random variable, and for all a > 0, Pr(X a) apple E[X ] a. Z 0 xf X (x) dx Z a xf X (x) dx Z a af X (x) dx = ap (X a) a f X (x)

4 Characterizing Random Variables Suppose we care about some summary of random variable X: Y = g(x) Ø Expected Value: Value of a Z typical sample E[Y ]=E[g(X)] = g(x)f X (x) dx Ø Variance: Expected squared distance from mean Var[Y ]=E[(g(X) E[g(X)]) 2 ] x Ø Cumulative distribution: A complete characterization of g(x) F Y (y) =P (Y apple y) =P (g(x) apple y) Often tractable to compute, & leads to bounds on extreme events! f Y (y) = df Y (y) dy If g(x) is complex or high-dimensional, may be hard to compute!

5 Theorem Chebyshev s Inequality For any random variable X,andanya > 0, Proof. Pr( X E[X ] a) apple Var[X ] a 2. By Markov inequality Pr( X E[X ] a) =Pr((X E[X ]) 2 a 2 ) Pr((X E[X ]) 2 a 2 ) apple E[(X E[X ])2 ] a 2 = Var[X ] a 2

6 Theorem Chebyshev s Inequality For any random variable X,andanya > 0, Pr( X E[X ] a) apple Var[X ] a 2. Ø Another way of parameterizing Chebyshev s inequality: µ = E[X], = p Var[X] P ( X µ k ) apple k 2 Ø Chebyshev bound is vacuous (above one) for events less than one standard deviation from the mean. But this could be likely! k 2

7 Chebyshev s Inequality Ø Another way of parameterizing Chebyshev s inequality: E(X) µ = E[X], = p Var[X] P ( X µ k ) apple k 2 Ø Chebyshev bound is vacuous (above one) for events less than one standard deviation from the mean. But this could be likely! k 2

8 . Markov vs. Chebyshev s Inequalities

9 CS45: Lecture 5 Outline Ø Markov and Chebyshev Inequalities Ø Law of Large Numbers Ø Central Limit Theorem

10 Convergence in Probability Convergence in Probability Let Y,Y 2,... be a sequence of random variables (not necessarily independent), and let a be a real number. We say that the sequence Y n converges to a in probability, if for every ϵ > 0, we have lim n P( Y n a ϵ ) =0. (almost all) of the PMF/PDF of Y n eventually gets concentrated (arbitrarily) close to a Convergence of a Deterministic Sequence Let a,a 2,... be a sequence of real numbers, and let a be another real number. We say that the sequence a n converges to a, or lim n a n = a, if for every ϵ > 0 there exists some n 0 such that a n a ϵ, for all n n 0. a n eventually gets and stays (arbitrarily) close to a

11 Convergence in Probability Convergence in Probability Let Y,Y 2,... be a sequence of random variables (not necessarily independent), and let a be a real number. We say that the sequence Y n converges to a in probability, if for every ϵ > 0, we have lim n P( Y n a ϵ ) =0. Example: X n is a sequence of independent uniform variables on [0, ] and Y n =min{x,...,x n } Ø We expect that Y n converges to zero. To verify: P ( Y n 0 ϵ ) = P(X ϵ,...,x n ϵ) = P(X ϵ) P(X n ϵ) = ( ϵ) n.

12 The (Weak) Law of Large Numbers X,X 2,... i.i.d. finite mean µ and variance 2 E[M n ]= Var[M n ]= E[X ]+ + E[X n ] = nµ n n = µ, M = n var(x + + X n ) = var(x )+ +var(x n ) n 2 n 2 X + + X n n = nσ2 n 2 Var( M 2 n) P( M n µ ) = 2 2 n = σ2 n. Ø Chebyshev s inequality bounds distance between the true mean and the empirical or sample mean: sample mean or empirical mean Ø The empirical mean converges to the true mean in probability lim n! P ( M n µ ) =0 Ø True even if variance not finite, but proof more challenging.

13 Why is it a Weak Law of Large Numbers? Convergence in Probability Let Y,Y 2,... be a sequence of random variables (not necessarily independent), and let a be a real number. We say that the sequence Y n converges to a in probability, if for every ϵ > 0, we have lim n P( Y n a ϵ ) =0. Example: - /n pmf of Y n /n 0 n For every > 0, lim n! P ( Y n 0 ) =0. But even though Y n converges in probability, occasionally it takes on very large values: E[Y n ] = for all n.

14 The Strong Law of Large Numbers The Strong Law of Large Numbers (SLLN) Let X,X 2,... be a sequence of independent identically distributed random variables with mean µ. Then, the sequence of sample means M n =(X + + X n )/n converges to µ, with probability, in the sense that ( P lim n X + + X n n ) = µ =. Ø This stronger (but more technically challenging) notion of convergence rules out cases like the previous example Ø For many practical scenarios, both forms of convergence hold, but convergence in probability is easier to show Ø We focus exclusively on the weak law in this course

15 CS45: Lecture 5 Outline Ø Markov and Chebyshev Inequalities Ø Law of Large Numbers Ø Central Limit Theorem

16 What is the shape of the distribution of Z n for large n? Scaling of the Sample Mean Ø Sequence of independent, identically distributed random variables: X,X 2,...,X n E[X i ]=µ Var[X i ]= 2 < Ø The variance of their sum increases with n: S n = P n i= X i E[S n ]=nµ Var[S n ]=n 2 Ø Law of Large Numbers: variance of the empirical mean decreases with n: M n = n S n E[M n ]=µ Var[M n ]= 2 n Ø Standardized sum: transform so mean and variance constant for all n p Var[Sn ] Z n = S n E[S n ] = S n nµ p n E[Z n ]=0 Var[Z n ]=

17 Central Limit Theorem (CLT) Standardized S n = X + + X n : S n E[S n ] S n Z n = = zero mean S n ne[x] n Theorem: For every c: P(Z n c) P(Z c) unit variance Let Z be a standard normal r.v. (zero mean, unit variance) f Z (z) = p 2 e z2 2 P(Z c) is the standard normal CDF, (c), available from the normal tables

18 Central Limit Theorem (CLT) accurate computational shortcut justification of Usefulness normal models What exactly does it say? universal; only Usefulness means, variances matter universal; accurate CDF of Z computational only n converges to means, variances shortcut normal CDF matter accurate justification notwhat a statement computational exactly of normal about does models convergence shortcut it say? of justification CDF PDFs of or Z PMFs n of converges normal models to normal CDF not a statement about convergence of PDFs What or PMFs exactly does it say? CDF ofnormal Z n converges approximation to normal CDF Treat not azstatement n as if normal about convergence of PDFs also treat Normal PMFs S n as approximation if normal Treat Z n as if normal also treat S n as if normal Normal approximation Can Treat wez n use as it if normal when n is moderate? Yes, alsobut treat nos n nice as if theorems normal to this e ect Can Symmetry we use helps it when a lot n is moderate? Yes, but no nice theorems to this e ect Symmetry helps a lot Can we use it when n is moderate? Theorem: For every c: P(Z n c) P(Z c) P(Z c) is the standard normal CDF, (c), available from the normal tables

19 CLT: Uniform Random Variables n=l o n=2... o C "iii c () Cl Joint density of (X + Y, W) o 2 I t t ",, " n=3 o 2 3 n=4 o n=5 S n = P n i= X i Den ity of X + Y f Xi (x i )=if0apple x i apple, 0 otherwise. o

20 .0 CLT: Exponential Random Variables f Xi (x) = e x,x 0. E[X Var[X i ]= i ]= 2 n= n=2 n=3 S n = P n i= X i n=0 0.0 o time in multiples of /,\

21 CLT: Intuitions for Proof Ø Theorem: A sum of independent Gaussians is Gaussian f Xi (x i )= p e 2 S = P n i i= X i 2 2 i f S (s) = p 2 2 e 2( s µ ) 2 µ = nx µ i, 2 = i= nx i= x µi 2 Ø A family of distributions is stable if a linear combination of independent random variables stays in the same family Ø Theorem: The only (non-degenerate) stable family of random variables with finite variance is the Gaussian. Ø One proof: Take Taylor expansion of log-pdf (or its Fourier transform), show that all terms of order three or higher approach zero as n grows, giving a quadratic (Gaussian) limit 2 i

22 CLT: Binomial Distribution Fix p, where 0 <p< X i : Bernoulli(p) S n = X + + X n : Binomial(n, p) mean np, variance np( p) CDF of q S n np standard normal np( p) k l k l De Moivre Laplace Approximation to the Binomial If S n is a binomial random variable with parameters n and p, n is large, and k, l are nonnegative integers, then P(k S n l) Φ ( l + 2 np ) ( k 2 Φ np ). np( p) np( p) P(S n 2) = P(S n < 22), because S n is integer Compromise: consider P(S n 2.5)

23 CLT: Binomial Distribution When the /2 correction is used, CLT can also approximate the binomial p.m.f. (not just the binomial CDF) P(S n = 9) = P(8.5 S n 9.5) n = 36,p= S n S n Z n 0.5 P(S n = 9) P(0.7 Z 0.5) = P(Z 0.5) P(Z 0.7) = = 0.24 k l k l De Moivre Laplace Approximation to the Binomial If S n is a binomial random variable with parameters n and p, n is large, and k, l are nonnegative integers, then P(k S n l) Φ ( l + 2 np ) ( k 2 Φ np ). np( p) np( p) P(S n 2) = P(S n < 22), because S n is integer Compromise: consider P(S n 2.5) Exact answer: =

24 Pollster s Problem: Chebyshev f: fraction of population that... M n =(X + + X n )/n fraction of yes in our sample ith (randomly selected) person polled: Goal: 95% confidence of % error, if yes, X i = 0, if no. P( M n f.0).05 Use Chebyshev s inequality: P( M n M 2 n f.0) (0.0) 2 = 2 x n(0.0) 2 4n(0.0) 2 For any binary variable, Var(X i ) apple 2 2 If n = 50, 000, then P( M n f.0).05 (conservative)

25 M n =(X + + X n )/n < Suppose we want: : f: fraction of population that... ith (randomly selected) person polled:, if yes, X i = 0, if no. Event of interest: M n f.0 X + + X n n Pollster s Problem: CLT P( Mn f.0).05 Event of interest: M n f.0 X + + X n n nf nf. 0 X + + X n nf.0 n n. 0 X + + X n nf.0 n n M n =(X + + X n )/n fraction of yes in our sample Goal: 95% confidence of % error P( M n f.0).05 For any binary variable, Var(X i ) apple 2 2 Std(X i ) apple 2 P( M n f.0) P( Z.0 n/ ) P( Z.02 n) p n = = 00,n= 0, 000