Chapter 6. Convergence. Probability Theory. Four different convergence concepts. Four different convergence concepts. Convergence in probability

Size: px

Start display at page:

Download "Chapter 6. Convergence. Probability Theory. Four different convergence concepts. Four different convergence concepts. Convergence in probability"

Eric Gardner
5 years ago
Views:

1 Probability Theory Chapter 6 Convergence Four different convergence concepts Let X 1, X 2, be a sequence of (usually dependent) random variables Definition 1.1. X n converges almost surely (a.s.), or with probability 1 (w.p.1), to the random variable X as n iff Definition iti 1.2. X n converges in probability to the random variable X as n iff for every ε>0 1 2 Four different convergence concepts Definition 1.3. X n converges in r-mean to the random variable X as n iff Convergence in probability Definition 1.2. X n converges in probability to the random variable X as n iff for every ε>0 Definition 1.4. X n converges in distribution to the random variable X as n iff where C(F X ) is the continuity set of F X. p Notation. X n X as n. In situations where the limiting distribution is degenerate, that is, the limiting random variable X is a constant, convergence in probability is (in statistics) also known as consistency. Chebyshev s inequality. Let X be a random variable with mean μ and finite variance σ 2. Then 3 4 1

2 Problem a Let X₁,X₂, be i.i.d. Pa(1,2)-distributed random variables, and set p Y n = min{x₁,x₂,,x n }. Show that Y n 1 as n. Since it follows that The weak law of large numbers The weak law of large numbers. Let X 1, X 2, be a sequence of i.i.d. random variables with mean μ and finite variance σ 2 and set S n =X+X X+X n. Then The distribution function of Y n is therefore given by and so (for any ε>0) Proof. The statement is a simple consequence of Chebyshev s inequality. It thus follows that it follows that 5 6 Example: Consistency of S 2 Consistency of S 2. Let X 1, X 2, be a sequence of i.i.d. random variables with mean μ and finite variance σ 2. Define the sample variance by Convergence in probability: Extension Theorem 6.7 Theorem 6.7.Suppose that X 1, X 2, converges in probability to a constant a and that h is a continuous function. Then Proof. h is continuous, so given ε>0 there exists a δ>0 such that Since (prove this!) The continuity of h thus makes sure that it follows from Chebyshev s s inequality that and since X n converges in probability to X and thus, a sufficient condition for S 2 p σ 2 is that Var(S 2 ) 0 as n

3 Convergence in probability: Extension Exercise Exercise Suppose that X 1, X 2, converges in probability to a random variable X and that h is a continuous function. Then Convergence in probability: Extension Exercise We therefore divide the sample space into two disjoint subsets Proof. To prove this we use the fact that on a closed interval (compact set) any continuous function h is actually uniformly continuous. To this we use the fact that for any given η>0 there exists an A such that and it now follows that Since h for any A is uniformly continuous on [-A,A] we can (on this interval) for any given η>0 and ε>0 find a δ>0 and an m such that for any n>m 9 10 Almost sure convergence Definition 1.1. X n converges almost surely, or with probability one, to the random variable X as n iff Example: Almost sure convergence Let the sample space S be [0,1] with the uniform distribution. Define random variables X n (s) = s+s n and X(s) = s. As n we have that a.s. Notation. X n X as n. When to prove that X n converges (or fails to converge) almost surely we can use the following result. a.s. X n X as n iff, ε>0 and 0<δ<1, n 0 such that, n>n 0, which means that But since Pr([0,1)) = 1 it follows by Definition 1.1 that

4 Relationship: Almost sure convergence and convergence in probability Comparison of Definitions 1.1 and 1.2. We have that Example: Convergence in probability but not almost sure convergence Let the sample space S be [0,1] with the uniform distribution. Define X(s) = s and the sequence X 1,X 2, by Since for m>n, etc. It is clear that for any 0<ε<1 it is proven that p where I n is the interval related to X n. It is thus clear that t X n X as n. However, since X n (s) alternates between s and s+1 infinitely often, that is it is clear that X n does not converge to X almost surely as n Relationship: Convergence in r-mean and convergence in probability Convergence in distribution (and relationships between concepts) Definition 1.3. X n converges in r-mean to the random variable X as n iff Definition 1.4. X n converges in distribution to the random variable X as n iff r Notation. X n X as n. Convergence in r-mean is stronger convergence concept than convergence in probability. By Markov s inequality (for any ε>0) d where C(F X ) is the continuity set of F X. Notation. X n X as n. Convergence in distribution is the weakest concept of the four but also the most useful. The (complete) relationships can be described as which implies that where all implications are strict

5 Important results concerning limits Consider two functions f and g, such that Then Problem b Let X₁,X₂, be i.i.d. Pa(1,2)-distributed random variables, and set Y n = min{x₁,x₂,,x n }. Show that U n = n(y n -1) converges in distribution as n, and determine the limit distribution. Since If h is a continuous function then it follows that The single most important limit (in its most general form) is the following: Let a n a as n. Then d It is thus clear that U n X as n where X Exp(1/2) Convergence via transforms Important results concerning limits (when using Taylor series expansion) Theorem 4.1. Let X, X₁,X₂, be nonnegative, integer-valued random variables. Then The Taylor series expansion of a function f that is infinitely differentiable in a neighborhood of x=a is the power series Theorem 4.2. Let X₁,X₂, be random variables for which the mgf s exist for h<t<h for some h>0, and suppose that X is a random variable whose mgf ψ X (t) exists for h 1 t h 1 where 0<h 1 <h. If then which implies that Some terms in the expansion might be insignificant in the limit. For such terms we can use the o -concept. The function f(x) is said to be little-o of g(x) if f(x)/g(x) 0 as x 0 and we write

6 Problem Let X₁,X₂, be i.i.d. random variables with mean μ<, and let N n Ge(p n ), 0<p n <1, independent of X₁,X₂,. Determine the limit distribution of Problem We now note that for a general probability distribution with mean μ (where the mgf exists) it holds that as n if p n 0 as n. It follows from Theorem that that is and dfrom Theorem we have thatt Since ψ X (p n t) ψ X (0) = 1 as n it therefore follows that 21 d and so it is clear that Y n Exp(μ) as n. 22 The weak law of large numbers Revisited The weak law of large numbers (LLN). Let X 1, X 2, be a sequence of i.i.d. random variables with mean μ and mgf ψ X (t). Then The Central Limit Theorem The Central Limit Theorem (CLT). Let X 1, X 2, be i.i.d. random variables with mean μ, variance σ 2, and mgf ψ X (t) and set S n = X 1 +X 2 + +X n. Then Proof. Proof. Because of linear properties of moment generating functions it is no restriction to let μ=0 and σ 2 =1. This means that as n. It is clear that, or equvalently,

7 and therefore we get that The Central Limit Theorem Convergence of sums of sequences of random variables Theorem Let X₁,X₂, and Y₁,Y₂, be sequences of random variables. Then as n, and we are done since this is the moment generating function of N(0,1). Theorem Let X₁,X₂, X₂ and Y₁,Y₂, Y₂ be sequences of random variables. Suppose further that X n and Y n are independent for all n and that X and Y are independent. Then Slutsky s theorem (or Cramér s theorem) Theorem 6.5. Let X₁,X₂, and Y₁,Y₂, be sequences of random variables. Suppose that Exercise Let X₁,X₂, be i.i.d. Be(p)-distributed random variables where 0<p<1. We would like to construct a confidence interval for the population proportion p. What about the random behavior of the sample proportion? Set S n = X 1 +X 2 + +X n and consider Y₁,Y₂, where Y n =S n /n. Since where a is a constant. Then it follows by the Central Limit Theorem (CLT) that which, for instance, implies that

8 Exercise Hence, in the denominator we have to replace p(1-p) with Y n (1-Y n ), that is Exercise Now it follows by the third and the fourth result of Theorem 6.5 (Slutsky) that is to be replaced by With the aid of Slutsky s theorem we can prove that CLT still works. First it follows by the law of large numbers that Since the square root is a continuous function, it follows by Theorem 6.7 that Finally, it follows by the fourth result of Theorem 6.5 (Slutsky) that and therefore by the second result of Theorem 6.5 (Slutsky) we have that 29 as n. It is thus clear that the approximation is still valid for sufficiently large sample sizes. 30 Problem Let X₁,X₂, be positive i.i.d. random variables with mean μ and variance σ 2 <, and set S n = X 1 +X 2 + +X n. Determine the limit distribution of So therefore we have that Problem In order to use the central limit theorem we rewrite the expression as Hence, it follows from Theorem 6.5 (Slutsky) that The expression in the numerator meets the requirements of the central limit theorem, and the expression in the denominator meets the requirements of the law of large numbers. since a linear function of a normal random variable also is normal

Lecture 21: Convergence of transformations and generating a random variable

Lecture 21: Convergence of transformations and generating a random variable If Z n converges to Z in some sense, we often need to check whether h(z n ) converges to h(z ) in the same sense. Continuous