7 Convergence in R d and in Metric Spaces

Transcription

1 STA 711: Probability & Measure Theory Robert L. Wolpert 7 Convergence in R d and in Metric Spaces A sequence of elements a n of R d converges to a limit a if and only if, for each ǫ > 0, the sequence a n eventually lies within a ball of radius ǫ centered at a. It s okay if the first few (or few million) terms lie outside that ball and the number of terms that do lie outside the ball may depend on how big ǫ is (if ǫ is small enough it typically will take millions of terms before the remaining sequence lies inside the ball). This can be made mathematically precise by introducing a letter (say, N ǫ ) for how many initial terms we have to throw away, so that a n a if and only if there is an N ǫ < so that, for each n N ǫ, a n a < ǫ: only finitely many a n can be farther than ǫ from a. The same notion of convergence really works in any metric space, where we require that some measure of the distance d(a n,a) from a n to a tend to zero in the sense that it exceeds each number ǫ > 0 for at most some finite number N ǫ of terms. Points a n in d-dimensional Euclidean space will converge to a limit a R d if and only if each of their coordinates converges in R; and, since there are only finitely many coordinates, if they all converge then they do so uniformly (i.e., for each ǫ we can take the same N ǫ for all d of the coordinate sequences), so all notions of convergence in R d are equivalent. For example, [ ]1 max x i y i (xi y i ) i d 1 i d x i y i d max 1 i d x i y i so convergence is identical for all three of these metrics. Convergence is much more complex and interesting for random variables. 7.1 Convergence of Random Variables For random variables X n the idea of convergence to a limiting random variable X is more delicate, since each X n is a function of ω Ω and usually there are infinitely many points ω Ω. What should we mean in saying that a sequence X n converges to a limit X? That X n (ω) converges to X(ω) for each fixed ω? Or that X n (ω) converges uniformly in ω Ω? Or that some notion of the distance d(x n,x) between X n and the limit X decreases to zero? Should the probability measure P be involved in some way? Here are a few different choices of what we might mean by the statement that X n converges to X, for a sequence of random variables X n and a random variable X, all defined on the same probability space (Ω,F,P): 1

2 pw: The real numbers X n (ω) X(ω) for every ω Ω (pointwise cgce): ( ǫ > 0)( ω Ω)( N ǫ,ω < )( n N ǫ,ω ) X n (ω) X(ω) < ǫ. uni: The sequences of real numbers X n (ω) X(ω) uniformly for ω Ω: ( ǫ > 0)( N ǫ < )( ω Ω)( n N ǫ ) X n (ω) X(ω) < ǫ. a.s.: Outside some null event N F, each sequence of real numbers X n (ω) X(ω) (Almost-Sure convergence, or convergence almost everywhere (a.e.)): for some N F with P[N] = 0, ( ǫ > 0)( ω / N)( N ǫ,ω < )( n N ǫ,ω ) X n (ω) X(ω) < ǫ, i.e., P { ǫ>0 N< n N X n (ω) X(ω) ǫ } = 0. L : OutsidesomenulleventN F,thesequencesofrealnumbersX n (ω) X(ω)converge uniformly ( almost-uniform or L convergence): for some N F with P[N] = 0, ( ǫ > 0)( N ǫ < )( ω / N)( n N ǫ ) X n (ω) X(ω) < ǫ. pr.: For each ǫ > 0, the probabilities P[ X n X > ǫ] 0 (convergence in probability, or in measure ): ( ǫ > 0)( η > 0)( N ǫ,η < )( n N ǫ,η ) P[ X n X > ǫ] < η. L 1 : The expectation E[ X n X ] converges to zero (convergence in L 1 ): ( ǫ > 0)( N ǫ < )( n N ǫ ) E[ X n X ] < ǫ. L p : For some fixed number p > 0, the expectation of the p th absolute power E[ X n X p ] converges to zero (convergence in L p, sometimes called in the p th mean ): ( ǫ > 0)( N ǫ < )( n N ǫ ) E[ X n X p ] < ǫ. dist.: The distributions of X n converge to the distribution of X, i.e., the measures P X 1 n converge in some way to P X 1 ( vague or weak convergence, or convergence in distribution, sometimes written X n X): ( ǫ > 0) ( φ C b (R) ) ( N ǫ,φ < )( n N ǫ,φ ) E [ φ(x n ) φ(x) ] < ǫ. Which of these eight notions of convergence is right for random variables? The answer is that all of them are useful in probability theory for one purpose or another. You will want to know which ones imply which other ones, under what conditions. All but the first two (pointwise, uniform) notions depend upon the measure P; it is possible for a sequence X n to converge to X in any of these senses for one probability measure P, but to fail to converge for another P. Most of them can be phrased as metric convergence for some notion of distance between random variables: Page 2

3 pr.: X n X in probability if and only if d 0 (X,X n ) 0 as real numbers, where: ( ) X Y d 0 (X,Y) := E 1+ X Y L 1 : X n X in L 1 if and only if d 1 (X,X n ) := X X n 1 0 as real numbers, where: Z 1 := E Z L p : X n X in L p if and only if d p (X,X n ) 0 as real numbers, where: { (E X Y p ) 1/p p 1 d p (X,Y) := E X Y p 0 < p < 1. L : X n X almost uniformly if and only if d (X,X n ) := X X n 0 as real numbers, where: Z := sup{r 0 : P[ Z > r] > 0} As the notation suggests, convergence in probability and in L are in some sense limits of convergence in L p as p 0 and p, respectively. Almost-sure convergence is an exception: there is no metric notion of distance d(x,y) for which X n X almost surely if and only if d(x,x n ) 0 (unless Ω is countable or P is atomic) Almost-Sure Convergence Let {X n } and X be a collection of RVs on some (Ω,F,P). The set of points ω for which X n (ω) does converge to X(ω) is just ǫ>0 m=1 n=m [ω : X n (ω) X(ω) ǫ], the points which, for all ǫ > 0, have X n (ω) X(ω) less than ǫ for all but finitely-many n. The sequence X n is said to converge almost everywhere (a.e.) to X, or to converge to X almost surely (a.s.), if this set of ω has probability one, or (conversely) if its complement is a null set: [ P ǫ>0m=1 n=m ] [ω : X n (ω) X(ω) > ǫ] = 0. Despite their appearance these intersections and unions over ǫ > 0 are only countable, since we need include only rational ǫ (or, for that matter, any sequence ǫ k tending to zero, such as ǫ k = 1/k). Thus X n X a.e. if and only if, for each ǫ > 0, Page 3

4 [ P m=1 n=m ] [ω : X n (ω) X(ω) > ǫ] = 0. (a.e.) This combination of intersection and union occurs frequently in probability, and has a name; for any sequence E n of events, [ m=1 n=m E n] is called the lim sup of the {E n }, and is sometimes described more colorfully as [E n i.o.], the set of points in E n infinitely often. Its complement is the lim inf of the sets F n = E c n, [ m=1 n=m F n]: the set of points in all but finitely many of the F n. Since P is countably additive, and since the intersection in the definition of lim sup is decreasing and the union in the definition of lim inf is increasing, always we have P[ n=m E n] ց P[ m=1 n=m E n] and P[ n=m F n] ր P[ m=1 n=m F n] as m. Thus, Theorem 1 X n X P-a.s. if and only if for every ǫ > 0, lim P[ X n X > ǫ for some n m] = 0. m In particular, X n X P-a.s. if n< P[ X n X > ǫ] < for each ǫ > 0 (why?) Convergence In Probability The sequence X n is said to converge to X in probability (pr.) if, for each ǫ > 0, P[ω : X n (ω) X(ω) > ǫ] 0. If we denote by E n the event [ω : X n (ω) X(ω) > ǫ] we see that convergence almost surely requires that P[ n m E n] 0 as m, while convergence in probability requires only that P[E n ] 0. Thus: Theorem 2 If X n X a.s. then X n X pr. Here is a partial converse: Theorem 3 If X n X pr., then there is a subsequence n k such that X nk X a.s. Proof. Set n 0 := 0 and, for each integer k 1, set { n k := inf n > n k 1 : P [ ω : X n (ω) X(ω) > 1 k ] 2 k }. (pr.) Page 4

5 For any ǫ > 0 we have 1 ǫ eventually (namely, for k k k 0 := 1 ǫ ) and for each m k0, [ { P ω : Xnk (ω) X(ω) > ǫ }] { P[ ω : X nk (ω) X(ω) > 1 }] k k=m k=m { P ω : X nk (ω) X(ω) > 1 } k k=m k=m 2 k = 2 1 m 0 as m. In fact, this characterizes convergence in probability: Theorem 4 Let {X n }, X be random variables on (Ω,F,P). Then X n X pr. if and only if every sequence N n k ր has a subsequence n ki such that X nki X a.s as i. Proof. The only if ( ) direction is just Theorem 3. Suppose (for contradiction) that X n X pr.; thenforsomeǫ > 0andδ > 0thereareinfinitely-manynforwhichP[ X n X > ǫ] > δ. Let n k be an increasing sequence satisfying this bound. By hypothesis, there is a subsequence along which X nki X a.s.; but by Theorem2, also X nki X pr, so P[ X n X > ǫ] 0, a contradiction. Theorem 5 Let {X n }, X be real-valued random variables on (Ω,F,P) with X n X pr. and let φ : R R be continuous. Then Y n := φ(x n ) Y := φ(x) pr. Proof. Foraneasybutindirectproof,simplyapplyTheorem4. Foramoredirectapproach, beginbyselecting anyǫ > 0andδ > 0. FindacompactsetK ǫ RwithP[X K ǫ ] 1 ǫ/2; since φ is uniformly continuous on K ǫ, find η > 0 such that ( x K ǫ )( y R) x y η φ(x) φ(y) < δ. Now, since X n X pr., find N N such that Then for n N, ( n N) P[ X n X > η] ǫ/2. P[ Y n Y > δ] P[X / K ǫ ]+P[X K ǫ, φ(x n ) φ(x) > δ] P[X / K ǫ ]+P[X K ǫ, X n X > η] ǫ/2+ǫ/2 = ǫ. The same result (with the same proof) is true for random variables X n, X taking values in any σ-compact complete separable metric space X and, in particular, for X = R d. Page 5

6 7.1.3 A Counter-Example If X n X a.s. implies X n X pr., and if the converse holds at least along subsequences, are the two notions really identical? Or is it possible for RVs X n to converge to X pr., but not a.s.? The answer is that the two notions are different, and that a.s. convergence is strictly stronger than convergence pr. Here s an example: First notice that every integer n N can be written uniquely in the form n = i + 2 j for integers j 0 and 0 i < 2 j (set j := log 2 n and i := n 2 j ). Let (Ω,F,P) be the unit interval with Borel sets and Lebesgue measure. Define a sequence of random variables X n : Ω R by X n (ω) = { i 1 if < ω i+1 2 j 2 j 0 otherwise SO, X 1 is one on 1 interval of length 1, X 2,X 3 are one on 2 intervals each of length 1/2, X 4,...,X 7 are one on 4 intervals each of length 1/4, X 8,...,X 15 are one on 8 intervals each of length 1/8, where n = i+2 j, 0 i < 2 j. and, in general, each X n is one on an interval of length 2 j. Since 1 n 1 2 j < 2 n, P[ X n > ǫ] = 2 j < 2 n 0 for each 0 < ǫ < 1 and X n 0 pr. On the other hand, for every j > 0 we have each ω Ω in one of the 2 j intervals of length 2 j where some X n is one, ( j N) Ω = 2 j 1 i=0 ( i i+1 ] 2j, = 2 j 2 j+1 1 n=2 j [ ω : Xn (ω) = 1 ] For every ω Ω and j N there is some n 2 j with X n (ω) = 1 (and so there are infinitelymany such X n ); thus [ω : X n (ω) 0] is empty, not a set of probability one! Obviously X n does not converge a.s, although it does converge pr. This example is a building-block for several examples to come, so getting to know it well is worthwhile. Try to verify that X n 0 in probability (how large must n be to ensure P[ X n > δ] < ǫ?) and in L p (how large must n be to ensure X n p < ǫ?) but not almost surely (what is {ω : X n 0}? Why?). Find X n p explicitly. Why doesn t X n 0 a.s.? What would happen if we multiplied X n by n? By n 2? By j := log 2 n? What about the subsequence Y n := X 2 n? Does X n converge in L? Page 6

7 In summary, for 1 < p < q < the convergence implications are: L a.s. L q pr. L p L 1 with partial converses (pr. a.s. along subsequences, pr. L 1,L p,l q under UI see the next section). All of these imply convergence in distribution, which we ll consider later. 7.2 Uniform Integrability Let Y 0 be integrable on some probability space (Ω,F,P), E[Y] = Y dp <. By Lebesgue s DCT it follows that Ω lim E[Y 1 {Y>t}] = lim Y dp = 0 t t [ω:y(ω)>t] since { Y 1 {Y>t} } is dominated by Y L1 and converges to zero a.s. as t. Y 1(Y>t) t Y Consequently, for any sequence of random variables X n also dominated by Y in the sense that X n Y a.s., E[ X n 1 { Xn >t}] E[Y 1 {Y>t} ] 0, uniformly in n. Page 7

8 Call the sequence X n uniformly integrable (or simply UI) if E[ X n 1 { Xn >t}] 0 uniformly in n, even if it is not dominated by a single integrable random variable Y. The big result is: Theorem 6 If X n X pr. and if X n is UI then X n X in L 1. Proof. Without loss of generality take X 0. Fix any ǫ > 0; find (by UI) t ǫ > 0 such that E[ X n 1 {Xn>t ǫ}] ǫ for all n. Now (by X n X pr.) find N ǫ N such that, for n N ǫ, P[ X n > ǫ] < ǫ/t ǫ. Then: E [ X n ] = X n dp+ X n dp+ X n dp [ X n ǫ] [ǫ< X n t ǫ] [t ǫ< X n ] ǫdp + t ǫ dp + X n dp [ X n ǫ] [ǫ< X n t ǫ] ǫ +t ǫ P [ X n > ǫ ] +ǫ 3ǫ. [t ǫ< X n ] Theorem6hasapartialconverse if{x n } L 1 andx n X pr. andife X n E X <, then {X n } is UI (see Theorem9). For another proof, first note E[ X n 1 { Xn >t}] = E[ X n ] E[ X n 1 { Xn t}]. For ǫ > 0 first pick t < s.t. E[ X 1 { X >t} ] < ǫ (possible, since X L 1 ); then (by DCT) find N s.t. for n N, X n 1 { Xn t} X1 { X t} 1 < ǫ and (by assumption) E Xn E X < ǫ. Then E[ Xn 1 { Xn >t}] 3ǫ for n N; since X j L 1 for j < N, the result follows (with t = max(t,t 1,...,t N 1 )). Similarly, for any p > 0, X n X (pr.) and X n p UI (for example, X n Y L p, or X n q B < for some q > p) gives X n X (L p ). In the special case of X n Y L p this is just Lebesgue s Dominated Convergence Theorem (DCT). We have seen that {X n } is UI whenever X n Y L 1, but UI is more general than that. Here are two more criteria. The first is called the crystal ball condition : Theorem 7 If {X n } is uniformly bounded in L p for some p > 1 then {X n } is UI. Proof. Let B R + be an upper bound for X n p and set q := p. By Hölder s inequality p 1 E [ X n 1 { Xn >t}] Xn p 1 { Xn >t} q B {P[ X n > t]} 1/q B {E[ X n p /t p ]} 1/q = B { X n p p/t p} 1/q = B X n p 1 p /t p 1 B p t 1 p 0 by Markov s inequality since p/q = p 1 uniformly in {n}. Page 8

9 Theorem 8 The random variables {X α } are UI if and only if they are uniformly bounded in L 1 and ( ǫ > 0)( δ > 0)( A F) P(A) < δ E[ X α 1 A ] < ǫ. If (Ω,F,P) is non-atomic, the condition {X α } is uniformly bounded in L 1 is unnecessary. Proof. Straightforward. If (Ω,F,P) is non-atomic, for each ǫ > 0 find δ > 0 by hypothesis and then cover Ω with N := 1 + 1/δ sets A i F with P(A i ) < δ to see E X α < Nǫ uniformly. Try to offer an example to illustrate what can go wrong if P has an atom ω Ω with c = P[{ω }] > 0. Theorem 9 Let {X n },X L 1 (Ω,F,P) with X n X pr. Then the followingare equivalent: 1. {X n } is UI 2. X n X in L 1 3. E X n E X Proof. Since 1 2 by Theorem6 and 2 3 by the triangle inequality, it remains only to show that 3 1. Let X n X pr. and E X n E X. Fix ǫ > 0 and, for each t > 0, set x 0 x t f t (x) := t(t+1 x ) t < x t+1 0 t+1 < x < and note that 0 x 1 { x t} f t (x) x 1 { x t+1} and 0 f t (x) t. Since X L 1, E X 1 { X >t} 0 as t by DCT so we can find t sufficiently large that E X 1 { X >t} < ǫ. (1) Since f t is continuous and bounded, f t (X n ) f t (X) as n in probability by Theorem5 and also in L 1 by DCT. For large enough n (say, n N ), f t (X n ) f t (X) 1 ǫ and so E X n 1 { Xn t+1 } Ef t (X n ) Ef t (X) ǫ E X 1 { X t} ǫ. (2) Since E X n E X, for sufficiently large n (say, n N N ) Upon subtracting (2) from (3) we have Applying (1) we get completing the proof that {X n } is UI. E X n E X +ǫ. (3) E X n 1 { Xn >t+1} E X 1 { X >t} +2ǫ. E X n 1 { Xn >t+1} < 3ǫ Page 9

10 7.3 Cauchy Convergence Every form of convergence we have considered for random variables except a.s. can be represented as convergence in a metric space. A sequence X n converges to a limit X in a metric space if every ball centered at X contains all but finitely-many of {X n }. Sometimes we wish to consider a sequence X n that converges to some limit, without knowing the limit in advance. The concept of Cauchy Convergence is ideal for this we insist that for each ǫ > 0, all but finitely-many of the ǫ-balls centered at points X m of the sequence contain all but finitely-many of the points. For any of distance measures d p of Section(7.1), with 0 p, say X n is a Cauchy sequence in L p if ( ǫ > 0)( N ǫ < )( m,n N ǫ ) d p (X m,x n ) < ǫ. The spaces L p for 0 p are all complete in the sense that if X n is Cauchy for d p then there exists X L p for which d p (X n,x) 0. To see this, take an increasing subsequence N k along which d p (X m,x n ) < 2 k for n m N k, and set X 0 := 0 and N 0 = 0; for k N set Y k := X Nk X Nk 1. Check to confirm that X := k=1 Y k is well-defined and X L p a.s., and that d p (X n,x) Convergence in Distribution Some of the most famous results in probability theory concern the limiting distribution of sequences of random variables. For example, Central Limit Theorem: For iid {X n } L 2 and large n, (S n nµ)/ nσ 2 has approximately the Normal No(0, 1) distribution; Law of Small Numbers: For independent N 0 -valued random variables X i with small means µ i, S n has approximately the Poisson Po(λ n ) distribution with mean λ n = µ i ; Extreme Value or Three Types Theorem: For iid {X n } and suitable sequences {a n }, {b n } of constants, max{(x j a n )/b n : j n} has approximately the Gumbel, Fréchet, or reversed Weibull distribution. But what does it mean to have a distribution approximately??? If X n has the discrete uniform distribution on the points {i/n} for 1 i n, and if n is huge (say, ) then most of us would regard X n as approximately a standard uniform Un(0,1) random variable in fact, this is the best one can hope for in a 64-bit double-precision floating-point representation. Yet each X n has a discrete distribution while the limit is a continuous distribution, so whatever we mean by converging in distribution it can t involve either density functions or probability mass functions, and it is too much to ask for the distributions µ n (B) = P[X n B] to converge for every Borel set B R. In this example the distribution functions F n (x) = P[X n x] do in fact converge at every x R, but that too is a bit too much to ask consider constant random variables Y n := Page 10

11 1/n, for example, which converge almost-surely to Y := 0, but the distribution functions do not converge at the point y = 0. This is closer to the right idea, however: Definition 1 Let {µ n (dx)} and µ(dx) be distributions on a measurable space (X,E). Say µ n converges in distribution to µ or write µ n µ if φ(x)µ n (dx) φ(x) µ(dx) as n for every continuous bounded function φ : X R. X If {X n } are random variables with distributions {µ n }, we also say that X n converges in distribution to µ and write X n µ. This definition works not only for real-valued random variables where X = R and E = B is the Borel sets, but also in R d or any other complete separable metric space X with Borel sets E. The connection with distribution functions is given by: Proposition 1 Let {µ n (dx)} and µ(dx) be distributions on the real line (R,B), and let F n (x) := µ n ( (,x] ) and F(x) := µ ( (,x] ) be the associated DFs. Then the following are equivalent: µ n µ; F n (x) F(x) on any dense set of {x} in R; F n (x) F(x) at every x R where F(x ) = F(x) is continuous. A similar but more general result for complete separable metric spaces X (like R n ) with their Borel sets is that µ n µ if and only if µ n (B) µ(b) for every Borel set B whose boundary B := B B c has zero measure for the limiting distribution, µ( B) = 0. Convergence in distribution is weaker than any of the other forms of convergence we have considered: Proposition 2 Let {X n }, X be random variables on some probability space (Ω,F,P) with distributions {µ n }, µ, and suppose X n X pr. Then X n µ. The proof is a simple application of the UI Convergence Theorem to the (uniformly bounded and hence UI) random variables φ(x n ). Convergence in distribution is strictly weaker than any other notion, because it even applies if the {X n } are defined on different probability spaces (Ω n,f n,p n ), where none of the other forms of convergence even makes sense. As usual there is a partial converse, however: Proposition 3 Let µ n µ on the real line (R,B). Then there exists a probability space (Ω,F,P) and random variables {X n }, X with distributions {µ n }, µ for which X n X a.s. It is enough to take Ω = (0,1], F = B, P = λ, and for ω Ω set X n (ω) := inf { x R + : F n (x) ω } X(ω) := inf { x R + : F(x) ω } and verify that X n (ω) X(ω) for all but countably-many ω Ω. X Page 11

12 7.5 Summary: UI and Convergence Concepts I. Uniform Integrability (UI) A. X n Y L r, r > 0, implies [ X n >t] X n r dp [Y>t] Y r dp 0 as t, uniformly in n, so { X n r } is UI. This is the definition of UI for r = 1. B. If (Ω,F,P) nonatomic, X n UI iff ( ǫ)( δ) P[Λ] δ Λ X n dp < ǫ (take δ = ǫ 2t ). If (Ω,F,P) has atoms, also need uniform bound E X n B. C. E X n q c q < implies X n p UI for each p < q. 1. Remark: not for p = q (counter-eg: X n := n1 (0,1/n] ) D. Main result: If X n X pr., then X n p is UI X n X in L p E X n p E X p. II. Convergence in Distribution (aka Vague Convergence) A. X n X pr. iff ( n k )( n ki ) X nki X a.s. (by contradiction) B. X n X a.s. and g(x) continuous implies g(x n ) g(x) a.s. C. X n X pr. and g(x) continuous g(x n ) g(x) pr. (use A, B) D. Definition: X n X if ( φ C b (R) ) Eφ(X n ) Eφ(X) 1. Prop: X n X pr. implies X n X (use II.C) 2. Prop: X n X implies F n (r) F(r) for all r s.t. F(r) = F(r ). a. Remark: Even if X n X, F n (r) may not converge where F(r) jumps; b. Remark: Even if X n X, f n (r) := F n (r) may not converge to f(r) := F (r); in fact, either may fail to exist. III. Implications among cgce notions: a.s., pr., L p, L q, L, dist. (0<p<q< ): A. a.s. = pr. (by Easy Borel-Cantelli) 1. pr. = a.s. along subsequences 2. pr. a.s. (counter-eg: X n (ω) = 1 (i/2 j,(i+1)/2 j ](ω), n = i+2 j ) B. L p = pr. (by Markov s inequality) 1. pr. = L p under Uniform Integrability 2. pr. L p (counter-eg: X n = n 1/p 1 (0,1/n] ) C. L q = L p for p < q (by Jensen s inequality) 1. L p L q (counter-eg: X n = n 1/q 1 (0,1/n] ) D. L = L p (simple estimate, or X p ր as p ր by Jensen) 1. L p L (counter-eg: X n = 1 (0,1/n] ) E. L = a.s. (almost-uniform cgce implies almost-pointwise cgce) F. pr. = dist. (II.D.1 above) 1. dist. pr. (counter-eg: X n, X on different spaces or iid {X n }) 2. dist. = a.s. ( (Ω,F,P), X n,x X n X a.s.) Page 12 Last edited: October 14, 2015