Lecture Notes 5 Convergence and Limit Theorems. Convergence with Probability 1. Convergence in Mean Square. Convergence in Probability, WLLN

Transcription

1 Lecture Notes 5 Convergence and Limit Theorems Motivation Convergence with Probability Convergence in Mean Square Convergence in Probability, WLLN Convergence in Distribution, CLT EE 278: Convergence and Limit Theorems Page 5

2 Motivation One of the key questions in statistical signal processing is how to estimate the statistics of a r.v., e.g., its mean, variance, distribution, etc. To estimate such a statistic, we collect samples and use an estimator in the form of a sample average How good is the estimator? Does it converge to the true statistic? How many samples do we need to ensure with some confidence that we are within a certain range of the true value of the statistic? Another key question in statistical signal processing is how to estimate a signal from noisy observations, e.g., using MSE or linear MSE Does the estimator converge to the true signal? How many observations do we need to achieve a desired estimation accuracy? The subject of convergence and limit theorems for r.v.s addresses such questions EE 278: Convergence and Limit Theorems Page 5 2

3 Example: Estimating the Mean of a R.V. Let X be a r.v. with finite but unknown mean E(X) To estimate the mean we generate X,X 2,...,X n i.i.d. samples drawn according to the same distribution as X and compute the sample mean S n = n X i n Does S n converge to E(X) as we increase n? If so, how fast? But what does it mean to say that a r.v. sequence S n converges to E(X)? First we give an example: Let X,X 2,...,X n,... be i.i.d. N(,) We use Matlab to generate 6 sets of outcomes of X,...,X n,...,x We then plot s n for the 6 sets of outcomes as a function of n Note that each s n sequence appears to be converging to, the mean of the r.v., as n increases i= EE 278: Convergence and Limit Theorems Page 5 3

4 Plots of Sample Sequences of S n 2 sn sn sn sn n EE 278: Convergence and Limit Theorems Page 5 4

5 Convergence With Probability Recall that a sequence of numbers x,x 2,...,x n,... converges to x if for every ǫ >, there exists an m(ǫ) such that x n x < ǫ for every n m(ǫ) Now consider a sequence of r.v.s X,X 2,...,X n,... all defined on the same probability space Ω. For every ω Ω we obtain a sample sequence (sequence of numbers) X (ω),x 2 (ω),...,x n (ω),... A sequence X,X 2,X 3,... of r.v.s is said to converge to a random variable X with probability (w.p., also called almost surely) if P{ω : lim n X n(ω) = X(ω)} = This means that the set of sample paths that converge to X(ω), in the sense of a sequence converging to a limit, has probability Equivalently, X,X 2,...,X n,... converges w.p. if for every ǫ >, lim P{ X n X < ǫ for every n m} = m EE 278: Convergence and Limit Theorems Page 5 5

6 Example : Let X,X 2,...,X n be i.i.d. Bern(/2), and define Y n = 2 n n i= X i. Show that the sequence Y n converges to w.p. Solution: To show this, let ǫ > (and ǫ < 2 m ), and consider P{ Y n < ǫ for all n m} = P{X n = for some n m} = P{X n = for all n m} = ( 2 )m as m An important example of convergence w.p.: the Strong Law of Large Numbers (SLLN), which says that if X,X 2,...,X n,... are i.i.d. with finite mean E(X), then the sequence of sample means S n E(X) w.p. The previous Matlab example is a good demonstration of the SLLN each of the 6 sample paths appears to be converging to, which is E(X) The proof of the SLLN and other convergence w.p. results are beyond the scope of this course. Take Stats 3 if you want to learn a lot more about this EE 278: Convergence and Limit Theorems Page 5 6

7 Convergence in Mean Square A sequence of r.v.s X,X 2,...,X n,... converges to a random variable X in mean square (m.s.) if Example: Estimating the mean. lim n E[ (X n X) 2] = Let X,X 2,...,X n,... be i.i.d. with finite mean E(X) and variance Var(X). Then S n E(X) in m.s. Proof: Here we need to show that First note that E(S n ) = E [ n lim E[ (S n E(X)) 2] = n n ] X i i= = n n E(X i ) = n i= So, S n is an unbiased estimate of E(X) n E(X) = E(X) i= EE 278: Convergence and Limit Theorems Page 5 7

8 Now to prove convergence in m.s., consider E [ (S n E(X)) 2] = E [ (S n E(S n )) 2] ( = E n X i n n i= ( = n n 2E X i i= ( n ) = n 2Var X i i= ( n ) = Var(X i ) n 2 i= = n 2(nVar(X)) ) 2 n E(X) i= ) 2 n E(X) i= = Var(X) as n n since {X i } are independent EE 278: Convergence and Limit Theorems Page 5 8

9 Note that the proof works even if the r.v.s are only pairwise independent or even only uncorrelated Example: Consider the best linear MSE estimates found in the first estimation example of Lecture Notes 4 as a sequence of r.v.s ˆX, ˆX 2,..., ˆX n,..., where ˆX n is the best linear estimate of X given the first n observations. This sequence converges in m.s. to X since MSE n Convergence in m.s. does not necessarily imply convergence w.p. Example 2: Let X,X 2,...,X n,... be a sequence of independent r.v.s such that with probability n X n = with probability n Clearly this sequence converges to in m.s., but does it converge w.p.? EE 278: Convergence and Limit Theorems Page 5 9

10 It actually does not, since for < ǫ < and any m n ( P{ X n < ǫ for all n m} = lim ) n i = lim n = lim n = lim n i=m n i=m ( i Also convergence w.p. does not imply convergence in m.s. Consider the sequence in Example. Since E [ (Y n ) 2] = ( 2) n2 2n = 2 n, i ) (m ) (m+) (n ) m m n m n the sequence does not converge in m.s. even though it converges w.p. EE 278: Convergence and Limit Theorems Page 5

11 Example: Convergence to a random variable: Flip a coin with random bias P conditionally independently to obtain the sequence X,X 2,...,X n,..., where as usual X i = if the ith coin flip is heads and X i = otherwise As we already know, the r.v.s X,X 2,...,X n are not independent, but given P = p they are i.i.d. Bern(p) It is easy to show using iterated expectation that E(S n ) = E(X ) = E(P) In a homework exercise, you will show that S n P (not to E(P)) in m.s. EE 278: Convergence and Limit Theorems Page 5

12 Convergence in Probability A sequence of r.v.s X,X 2,...,X n,... converges to a r.v. X in probability if for any ǫ >, lim n P{ X n X < ǫ} = Convergence w.p. implies convergence in probability. The converse is not necessarily true, so convergence w.p. is stronger than in probability Example 3: Let X,X 2,...,X n,... be independent such that { with probability n X n = n with probability n Clearly, this sequence converges in probability to, since P{ X n > ǫ} = P{X n > ǫ} = n as n But does it converge w.p.? The answer is no (see Example 2) EE 278: Convergence and Limit Theorems Page 5 2

13 Convergence in m.s. implies convergence in probability. To show this we use the Markov inequality. For any ǫ >, P{ X n X > ǫ} = P{(X n X) 2 > ǫ 2 } E[(X n X) 2 ] ǫ 2 If X n X in m.s., then lim n E[ (X n X) 2] = lim n P{ X n X > ǫ} =, i.e., X n X in probability The converse is not necessarily true. In Example 3, X n converges in probability. Now consider E [ (X n ) 2] ( = ) +n 2 n n = n as n Thus X n does not converge in m.s. So convergence in probability is weaker than both convergence w.p. and in m.s. EE 278: Convergence and Limit Theorems Page 5 3

14 The Weak Law of Large Numbers The WLLN states that if X,X 2,...,X n,... is a sequence of i.i.d. r.v.s with finite mean E(X) and variance Var(X), then S n = n X i E(X) in probability n i= We already proved that S n E(X) in m.s., and since convergence in m.s. implies convergence in probability, S n E(X) in probability So, WLLN requires only uncorrelation of the r.v.s (SLLN requires independence) EE 278: Convergence and Limit Theorems Page 5 4

15 Confidence Intervals Given ǫ,δ >, how large should n, the number of samples, be so that P{ S n E(X) ǫ} δ, i.e., S n is within ±ǫ of E(X) with probability δ? Let s use the Chebyshev inequality: P{ S n E(X) ǫ} = P{ S n E(S n ) ǫ} Var(S n) ǫ 2 = Var(X) nǫ 2 So n should be large enough that: Var(X)/nǫ 2 δ n Var(X)/δǫ 2 Example: Let ǫ =.σ X and δ =.. The number of samples should satisfy n σ 2 X..σ 2 X = 5, i.e., 5 samples ensure that S n is within ±.σ X of E(X) with probability.999, independent of the distribution of X EE 278: Convergence and Limit Theorems Page 5 5

16 Convergence in Distribution A sequence of r.v.s X,X 2,...,X n,... converges in distribution to a r.v. X if lim n F X n (x) = F X (x) for every x at which F X (x) is continuous Convergence in probability implies convergence in distribution so convergence in distribution is the weakest form of convergence we discuss The most important example of convergence in distribution is the Central Limit Theorem (CLT). Let X,X 2,...,X n,... be i.i.d. r.v.s with finite mean E(X) and variance σx 2. Consider the normalized sum Z n = n n i= X i E(X) The sum is called normalized because E(Z n ) = and Var(Z n ) = The Central Limit Theorem states that Z n Z N(,) in distribution, i.e., { Q(z) z lim F Z n n (z) = Φ(z) = Q( z) z < σ X EE 278: Convergence and Limit Theorems Page 5 6

17 Example: Let X,X 2,... be i.i.d. U[,] r.v.s. The normalized sum is Z n = n i= X i/ n/3. The following plots show the pdf of Z n for n =,2,4,6. Note how quickly the pdf of Z n approaches the Gaussian pdf Z4 pdf Z2 pdf Z pdf Z6 pdf z EE 278: Convergence and Limit Theorems Page 5 7

18 Example: Let X,X 2,... be i.i.d. Bern(/2). The normalized sum is Z n = n i= (X i.5)/ n/4. The following plots show the cdf of Z n for n =,2,6. Z n is discrete and thus has no pdf, but its cdf converges to the Gaussian cdf Z cdf Z2 cdf Z6 cdf z EE 278: Convergence and Limit Theorems Page 5 8

19 Application: Confidence Intervals Let X,X 2,...,X n be i.i.d. with finite mean E(X) and variance Var(X) and let S n be the sample mean Given ǫ, δ >, how large should n, the number of samples, be so that P{ S n E(X) ǫ} δ? We can use the CLT to find an estimate of n as follows: { n } P{ S n E(S n ) ǫ} = P (X i E(X)) ǫ n i= { n } = P (X i E(X)) ǫ n σ X n σ X i= ( ) ǫ n 2Q Example: For ǫ =.σ X, δ =., set 2Q(. n) =., so. n = 3.3 or n = 89 much smaller than n 5 obtained by the Chebyshev inequality σ X EE 278: Convergence and Limit Theorems Page 5 9

20 CLT for Random Vectors The CLT applies to i.i.d. sequences of random vectors Let X,X 2,...,X n,... be a sequence of i.i.d. k-dimensional random vectors with finite mean µ and nonsingular covariance matrix Σ. Define the sequence of random vectors Z,Z 2,...,Z n,... by Z n = n (X i µ) n The Central Limit Theorem for random vectors states that as n i= Z n Z N(,Σ) in distribution Example: Let X,X 2,...,X n,... be a sequence of i.i.d. 2-dimensional random vectors with { x +x 2 < x <, < x 2 < f X (x,x 2 ) = otherwise The following plots show the joint pdf of Y n = n i= X i for n =,2,3,4. Note how quickly it looks Gaussian. EE 278: Convergence and Limit Theorems Page 5 2

21 2 Y pdf.5.5 Y2 pdf Y3 pdf Y4 pdf EE 278: Convergence and Limit Theorems Page 5 2

22 Relationships Between Types of Convergence The following figure summarizes the relationships between the different types of convergence we discussed with probability in probability in distribution in mean square EE 278: Convergence and Limit Theorems Page 5 22