LECTURE 18: Inequalities, convergence, and the Weak Law of Large Numbers. Inequalities

Transcription

1 LECTURE 18: Inequalities, convergence, and the Weak Law of Large Numbers Inequalities bound P(X > a) based on limited information about a distribution Markov inequality (based on the mean) Chebyshev inequality (based on the mean and variance) WLLN: X, X 1,...,X n i.i.d. X X n - E[X] n a pplication to polling Precise defn. of convergence convergence in probability 1

2 The Markov inequality Use a bit of information about a distribution to learn something about probabilities of "extreme events" "If X > 0 and E[X) is small, then X is unlikely to be very large" Markov inequality: If X > 0 and a> 0, then P(X > a) < E[X). a 'Y - 2

3 The Markov inequality Markov inequality: If X > 0 and a > 0, t hen P(X > a) < E[X) a I Example: X is Exponential(A = 1): P (X > a) < 't I o 1 )/. 4-1 I/o. I Example: X is Uniform[- 4, 4) : P (X > 3) < (I)() ~~) ~ ~)(}]= 3 2 L...,... =~.r(ixk3.)'=~ o 3 '1 '- 3 '/; I 3

4 The Chebyshev inequality Random variable X, with finite mean I" and variance a 2 "If the variance is small, then X is unlikely to be too far from the mean" Chebyshev inequality: Markov inequality: If X > 0 and a> 0, then P(X > a) < E[X] a 4

5 The Chebyshev inequality Chebyshev inequality: 1, :: 9 Example: X is Exponential(A = 1): P(X > a) < (Markov) e - Cl N 'felt '/a. ~I~~~---/~~l------~I a 1 (,(>, 0.) ;, 1 (/( - I ~ Q - )) - ) 5

6 The Weak Law of Large Numbers (WLLN) Xl, X2,... - i.i.d.; finite mean I" and variance a _X.=. l --, +_ _ _. +-,----X--", n Sample mean: M n = n 2 var(iit1... ) ~f. Xl+ + Xn WLLN: For E> 0, p ( IM n - 1"1> E) = P ( n - I" as n -+ <Xl 6

7 Interpreting the WLLN Mn = (Xl X n )/ n Xl Xn WLLN: For < > 0, P ( IMn - 1'1> <) = P ( n - I' as n One experiment many measurements X i = I' + Wi W;: measurement noise; E[W;J = 0; independent Wi sample mean Mn is unlikely to be far off from true mean I' Many independent repetitions of the same experiment event A, with p = P(A) X ;: indicator of event A X'.. := :i., 'f Ccu " } O J. ".W. the sample mean Mn is the empirical freljuency of event A r;[7f'-]=r 7

8 The pollster's problem p: fraction of population that will vote "yes" in a referendum {1' ith (randomly selected) person polled : if yes, G ex;1=-p X i = 0, if no. prj -p) CA"'10"""p,),..depe..de... Uy [ Mn = (Xl _L'\ X n)/n: fraction of "yes" in our sample Would like "small error," e.g.: IMn - pi < '1. ['(I-f) C = 'h f:.t 10'1./0-'1 '-I o Try n = 10,000 J J _C> p (IMlO,ooo - pi > 0.01) < r 1.v4. ~? <, 1/", -C <; - ~ 'I'l.> rn. 10-'1 IO~ 10' :: '70, r" w(rp ~uhi~ 8

9 Convergence "in probability" [ WLLN: For any <> 0, P(IMn - 1'1 2: <) --+ 0, as n J Would like to say that "Mn converges to 1''' Need to define the word "converges". f' ~f' Sequence of random variables Y n : not necessarily independent Definition: A sequence Yn converges in probability to a number a if: 9

10 Understanding convergence "in probability" Ordinary convergence Convergence in probability Sequence an; number a Sequence Yn ; number a "an eventually gets and (arbitrarily) close to a" stays Y n --+ a for any E> 0, d-< f., I a..'.j ".." ~-E ~--~--~~~--~ 01 0 For every E > 0, there exists no, "(almost all) of the PMF/PDF of Y n such that for every n > no, eventually gets concentrated we have Ian - al < E (arbitrarily) close to 0.;' 10

11 Some properties Suppose that X n --+ a, Yn --+ b, in probability If 9 is continuous, then g(xn ) --+ g(a) X n + Yn --+ a + b But: E[XnJ need not converge to a 11

12 Convergence in probability examples PY,, ( Y) 1 - (lin) E[Y n ] = f ( I Y h ) ~ t 'h.- I _ "" l in 'I"" y y" convergence in probability does not imply convergence of expectations 12

13 Convergence in probability examples X;: i.i.d., uniform on [0, 1] I. I I 'i y~+\ ~ :: f ( )( I ~ t: J ') X"1 >, 'i) - f (X,>''i.) ~ l(x.,~'i..) 13

14 Related topics - ~ "'-- Better bounds/approximations on tail probabilities... Markov and Chebyshev inequalities Chernoff bound Central limit theorem Different types of convergence Convergence in probability Convergence "with probability 1" f (fw : Yo, (w)-> y (w)~) =1..."" Strong law of large numbers M... w P~ /" "''''co Convergence of a sequence of distributions (CDFS) to a limiting CDF 14

15 MIT OpenCourseWare Resource: Introduction to Probability John Tsitsiklis and Patrick Jaillet The following may not correspond to a particular course on MIT OpenCourseWare, but has been provided by the author as an individual learning resource. For information about citing these materials or our Terms of Use, visit: