An Introduction to Laws of Large Numbers

Transcription

1 An to Laws of John CVGMI Group

2 Contents 1

3 Contents 1 2

4 Contents 1 2 3

5 Contents

6 Intuition We re working with random variables. What could we observe? {X n } n=1

7 Intuition We re working with random variables. What could we observe? {X n } n=1... More specifically Bernoulli sequence, looking at probability of average of the sum of N random events. P(x 1 < N i=1 X i < x 2 )

8 Intuition We re working with random variables. What could we observe? {X n } n=1... More specifically Bernoulli sequence, looking at probability of average of the sum of N random events. P(x 1 < N i=1 X i < x 2 ) Coin Flipping anybody?

9 Intuition We re working with random variables. What could we observe? {X n } n=1... More specifically Bernoulli sequence, looking at probability of average of the sum of N random events. P(x 1 < N i=1 X i < x 2 ) Coin Flipping anybody? as N increases we see that the probability of observing an equal amount of 0 or 1 is equal

10 Intuition We re working with random variables. What could we observe? {X n } n=1... More specifically Bernoulli sequence, looking at probability of average of the sum of N random events. P(x 1 < N i=1 X i < x 2 ) Coin Flipping anybody? as N increases we see that the probability of observing an equal amount of 0 or 1 is equal This is intuitive: As the number of samples increases the average observation should tend toward the theoretical mean

11 Weak Law Let s try to work out this most basic fundamental theorem of random variables arising from repeatedly observing a random event. How do we build such a theorem?

12 Weak Law Let s try to work out this most basic fundamental theorem of random variables arising from repeatedly observing a random event. How do we build such a theorem? In blackbox scenario we want to be able to use independence.

13 Weak Law Let s try to work out this most basic fundamental theorem of random variables arising from repeatedly observing a random event. How do we build such a theorem? In blackbox scenario we want to be able to use independence. We want to be able to use variance and expectation: so let s make X n L 2 for all n, and associate with each X n it s mean, we ll denote by µ n, and variance, by σ n.

14 Weak Law Let s try to work out this most basic fundamental theorem of random variables arising from repeatedly observing a random event. How do we build such a theorem? In blackbox scenario we want to be able to use independence. We want to be able to use variance and expectation: so let s make X n L 2 for all n, and associate with each X n it s mean, we ll denote by µ n, and variance, by σ n. We will create new random variables from the sequence and work with these; aim for results in terms of them.

15 Weak Law Let s try to work out this most basic fundamental theorem of random variables arising from repeatedly observing a random event. How do we build such a theorem? In blackbox scenario we want to be able to use independence. We want to be able to use variance and expectation: so let s make X n L 2 for all n, and associate with each X n it s mean, we ll denote by µ n, and variance, by σ n. We will create new random variables from the sequence and work with these; aim for results in terms of them. This seems like a start, let s try to prove a theorem...

16 Weak Law Theorem Let {X n } be a sequence of independent L 2 random variables with means µ n and variances σ n. Then n i=1 X i µ i 0. So a sequence of functions on Ω, the sample space, are going towards a function = zero...

17 Weak Law Theorem Let {X n } be a sequence of independent L 2 random variables with means µ n and variances σ n. Then n i=1 X i µ i 0. So a sequence of functions on Ω, the sample space, are going towards a function = zero... Can we prove this?

18 Weak Law Theorem Let {X n } be a sequence of independent L 2 random variables with means µ n and variances σ n. Then n i=1 X i µ i 0. So a sequence of functions on Ω, the sample space, are going towards a function = zero... Can we prove this? We haven t used (σ n ): we ll clearly need these since otherwise the X i are wildly unpredictable.

19 Weak Law Theorem Let {X n } be a sequence of independent L 2 random variables with means µ n and variances σ n. Then n i=1 X i µ i 0. So a sequence of functions on Ω, the sample space, are going towards a function = zero... Can we prove this? We haven t used (σ n ): we ll clearly need these since otherwise the X i are wildly unpredictable. IDEA: Constrain lim n σi 2 = 0 i=1

20 Counterexample

21 Weak Law: Pitfalls We haven t specified a mode of convergence: this is a technical point, but if you recall uniform, pointwise, a.e., normed convergence from analysis then the goal of the proof can change radically. IDEA: We want our rule to hold with high probability, that is it should hold for essentially all values of ω Ω. in the normed space is pretty ambitious at this point.

22 Weak Law: Pitfalls We haven t specified a mode of convergence: this is a technical point, but if you recall uniform, pointwise, a.e., normed convergence from analysis then the goal of the proof can change radically. IDEA: We want our rule to hold with high probability, that is it should hold for essentially all values of ω Ω. in the normed space is pretty ambitious at this point. We haven t specified a rate of convergence. The rate of convergence of the σ n should imply something about the rate of convergence of the X n µ n. The X n µ n should converge slower than the σ n.

23 Weak Law: Pitfalls We haven t specified a mode of convergence: this is a technical point, but if you recall uniform, pointwise, a.e., normed convergence from analysis then the goal of the proof can change radically. IDEA: We want our rule to hold with high probability, that is it should hold for essentially all values of ω Ω. in the normed space is pretty ambitious at this point. We haven t specified a rate of convergence. The rate of convergence of the σ n should imply something about the rate of convergence of the X n µ n. The X n µ n should converge slower than the σ n. IDEA: Weaken constraint to lim n 2 n i=1 σ 2 i = 0

24 Weak Law: Pitfalls We haven t specified a mode of convergence: this is a technical point, but if you recall uniform, pointwise, a.e., normed convergence from analysis then the goal of the proof can change radically. IDEA: We want our rule to hold with high probability, that is it should hold for essentially all values of ω Ω. in the normed space is pretty ambitious at this point. We haven t specified a rate of convergence. The rate of convergence of the σ n should imply something about the rate of convergence of the X n µ n. The X n µ n should converge slower than the σ n. IDEA: Weaken constraint to lim n 2 n i=1 The X n µ n should converge on average σ 2 i = 0

25 Law of 2.0 Theorem Let {X n } be a sequence of independent L 2 random variables n with means µ n and variances σn. 2 Then 1 n X i µ i 0 in measure if lim 1 n 2 n σi 2 = 0. i=1 i=1 Let Y n (ω) = 1/n n (X i (ω) µ i ). Then E(Y n ) = 0 and σ 2 (Y n ) = 1/n 2 n i=1 σi 2 i=1 by independence. Now we can use the limit of σ 2 (Y n ) from the hypothesis, and need to prove P({ω : Y n (ω) > ɛ}) σ2 (Y n ) ɛ 2 0 as n

26 Aside: L p bounds What does actually say? P({ω : Y n (ω) > ɛ}) σ2 (Y n ) ɛ 2 0 as n Part of Y n > ɛ is bounded by the integral of Y 2 n divided by epsilon squared... Thinking about this makes it obvious. Better bounds can be obtained: Chernoff bounds

27 Law of 2.1 Theorem Let {X n } be a sequence of independent L 2 random variables n with means µ n and variances σn. 2 Then 1 n X i µ i 0 in measure if lim n 2 n i=1 σ 2 i = 0. weakened; use P(max k k i=1 X i µ i ɛ) ɛ 2 n i=1 σ2 i Theorem Let {X n } be a sequence of independent L 2 random variables n with means µ n and variances σn. 2 Then 1 n X i µ i 0 a.e. if lim n 2 n i=1 σ 2 i <. i=1 i=1

28 Khinchine s Law Theorem Let {X n } be a sequence of independent L 1 random variables, n identically distributed, with means µ. Then 1 n X i µ i 0 a.e. as n. i=1

29 Sample Size in Coding Now we need to use the laws. Start with an example in source coding. C : X (Ω) Σ encoding events Interesting to know how complex C must be to encode (Y i ) N i=1 Entropy: H(x) = E[ lg(p(x))]; representation of how uncertain a r.v. is Problem: p( ) is unknown to the function and the distribution needs to be learned. WLLN can be used to answer: how uncertain is the distribution?

30 Asymptotic Equipartition Property Theorem i.i.d. Asymptotic Equipartition Property If X 1,..., X n p(x) then i.i.d. X i 1 n lg(p(x 1,..., x n )) H(x). p(x) so lgp(x i) are i.i.d. The weak law of large numbers says that P( 1 n (lg( (p(x i ))) E[ lg(p(x)))] > ɛ) 0 i i P( 1 lg((p(x i ))) E[ lg(p(x))] > ɛ) 0 n i so the sample entropy approaches the true entropy in probability as the sample size increases.

31 Want to minimize R(f ) = E(l(x, y, f (x))) eg l = (x, y, f ) 1/2 f (x) y Stuck minimizing over all f, under a distribution we don t know... hopeless... IDEA: Take Law of numbers and apply it to this framework, and hopefully R emp (f ) = 1/m i l(x i, y i, f (x i )) R(f ) Use L p bounds to prove convergence results on testing error.

32 Mini : Measure I won t go into too much detail in this regard. If we ve made it this far then great. (X, A) is a measurable space if A 2 X is closed under countable unions and complements. These correspond to measurable events. (X, A) with µ is a measure space if µ : A [0, ) is countably additive over disjoint sets: µ( i A i ) = i µ(a i) if A i A j = if i j. More great properties fall out of this quickly: measure of the emptyset is zero, measures are monotonic under the containment (partial) ordering, even dominated and monotone convergence (of sets) come out from this. s Inequality - Let f L p (R) and denote E α = {x : f > α} and now f p p E α f p α p µ(e α ) by monotonicity and positivity of measures.

33 L 2 : Definition We say that a function f : X R belongs to the function space L p (X ) if f p = ( f (x) p dx) 1/p < and say f has finite L p norm. Definition The fundamental object being considered in statistics is the random variable. Given a measure space (Ω, B, µ) X : Ω R is an L 2 random variable if µ(ω) = 1 and ( {X < r} B r R X (ω) 2 dµ(ω)) 1/2 <. We may also say that a random variable with finite 2nd 2

34 L 2 : Questions Why B? Physical paradoxes, eg. Banach-Tarski Want to talk about physical events measurable sets B is as descriptive as we d like most of the time What does µ do here? µ weights the events measurable sets Puts values to {ω Ω : X (ω) < r}; Ω is the sample space X maps random events (patterns occuring in data) to the reals which have a good standard notion of measure. So X induces the distribution P(B) = µ(x 1 (B)) which gives P({X (ω) B}) = µ(x 1 (B)) as expected. This is usually written without the sample variable. For more sophisticated questions/answers see MAA 6616 or STA 7466, but discussion is encouraged

35 Statistics: Functions of Measured Events Now we can start building Theorem If f : R R is measurable, then f (x) dp = f (X (ω)) dµ ie. the random variable pushes forward a measure onto R, and the integrals of measurable functions of random variables are therefore ultimately determined by real integration. This may be proved using Dirac measures and measure properties. Following these lines we may develop the basics of the theory of probability from measure theory. (ok, Homework)

36 Statistics: Moments Definition Define the first moment as the linear functional E(X ) = t dp. Then the p th central moment is is a functional given by m p (X ) = (t E(X )) p dp. Note that when X L 2 these are well defined by the theorem above. Pullbacks are omitted. Since we re talking about L 2 let s define the second central moment as σ 2 for convenience. If we need higher moments, sadly, mathematics says no.

37 Statistics: Now that we know what random variables are, let s try to define how a collection of them interact in the most basic way possible. Definition Let X = {X α } be a collection of random variables. Then X is independent if P( α X 1 α (B α )) = P((X 1,..., X n ) 1 (B 1... B n )) = (µ(x 1 1 )... µ(x 1 n ))(B 1... B n ). (Head Explodes) Really this just means that the X α are independent if their joint distributions are given by the product measure over the induced measures.

38 Limits: Repeating Independent Trials The significance of the belabored definition of independence is that when a joint distribution, just a distribution over several random variables, contains zero information about how the variables are related. If we have an infinite collection of random variables, independent of each other - in spite of independence - we would like to be able to infer information about the lower order moments from the higher order moments and vice versa. If we have ideal conditions on the the random variables then we should be able to deduce information about moments from a very large sequence. The type of convergence we get should be sensitive to the hypotheses.

39 We clearly need to specify how the means are converging, right?

40 We clearly need to specify how the means are converging, right? Why?

41 We clearly need to specify how the means are converging, right? Why? So how could N i=1 (X i µ i )/N 0?

42 We clearly need to specify how the means are converging, right? Why? So how could N i=1 (X i µ i )/N 0? In measure : lim N P({ N i=1 (X i µ i )/N ɛ}) 0 In norm: lim N ( ( N i=1 (X i µ i )/N) p dµ(ω)) 1/p 0 A.E : P({lim N N i=1 (X i µ i )/N > 0}) 0 f n f nk AE µ L p L p i f n f nk

43