4 Expectation & the Lebesgue Theorems

Transcription

1 STA 205: Probability & Measure Theory Robert L. Wolpert 4 Expectation & the Lebesgue Theorems Let X and {X n : n N} be random variables on a probability space (Ω,F,P). If X n (ω) X(ω) for each ω Ω, does it follow that E[X n ] E[X]? That is, may we exchange expectation and limits in the equation [ ] lim E[X n] =? E lim X n? () n n In general the answer is no; for a simple example take Ω = (0,], the unit interval, with Borel sets F = B(Ω) and Lebesgue measure P = λ, and for n N set X n (ω) = 2 n (0,2 n ](ω). (2) For each ω Ω, X n (ω) = 0 for all n > log 2 (/ω), so X n (ω) 0 as n for every ω, but E[X n ] = for all n. We will want to find conditions that allow us to compute expectations by taking limits, i.e., to force an equality in Equation (). The two most famous of these conditions are both attributed to Henri Lebesgue: the Monotone Convergence Theorem (MCT) and the Dominated Convergence Theorem (DCT). We will see stronger results later in the course but let s look at these now. 4. Expectation Let E be the linear space of finite-valued F-measureable random variables, and let E + be the positive members of E each X E may be represented in the form k X(ω) = a j Aj (ω) j= for some k N, {a j } R and {A j } F. The representation is only unique if we insist that the {a j } be distinct and that the {A j } be disjoint, in which case X E + if and only if each a j 0. In general we will not need uniqueness of the representation, so don t demand that the {a j } be distinct nor that the {A j } be disjoint.

2 We define the expectation for simple functions in the obvious way: EX = k a j P(A j ). j= For this to be a definition we must verify that it is well-defined in the sense that it doesn t depend on the (non-unique) representation; that s easy. Now we extend the definition of expectation to all non-negative F-measurable random variables as follows: Definition The expectation of any nonnegative random variable X 0 on (Ω,F,P) is EX = lim EX n for any simple sequence X n E + such that X n (ω) ր X(ω) for each ω Ω. For this definition to make sense we have three things to check:. For any X 0 there exists at least one sequence {X n } E + for which X n (ω) ր X(ω) for all ω Ω; 2. For any sequence {X n } E + for which m < n X m (ω) X n (ω) for all ω Ω, the limit lim E[X n ] exists as an extended-real-valued number (i.e., the limit takes values in R + = [0, ] and in particular might be ); 3. If {X n } E + and {Y m } E + are two sequences that both satisfy X n ր X and Y m ր X, then the limits coincide. lim EX n = lim EY m n m In a homework exercise you re asked to prove that X n = min ( 2 n, 2 n 2 n X ) satisfies the conditions above, verifying the first point. By monotonicity the expectations E[X n ] are increasing, so lim E[X n ] = supe[x n ] and the second point holds. For the third it s useful first to prove Lemma Let Z n E + for n N and suppose Z n ց 0. Then EZ n 0. Proof. Since each element in E is bounded, find K with Z K < ; then for any ǫ > 0, 0 E[Z n ] ǫp[z n ǫ] + KP[Z n > ǫ] 0. Item (3) above now follows by symmetry from: Page 2 November, 200

3 Lemma 2 If X n,y m E + are two increasing sequences and if lim n X n (ω) lim m Y m (ω) for each ω Ω, then lim n E[X n ] lim m E[Y m ]. Proof. Fix any n N; then (X n Y m ) ր X n as m, since (by hypothesis) lim Y m lim X m X n. By monotonicity on E +, using Lemma() with E + Z m [X n (X n Y m )] ց 0, Now take n to find E(X n ) = lim m E(X n Y m ) lim m E(Y m). lim E(X n) lim E(Y m) n m as required, completing the verification of Item (3) above (why?). Now that we have EX well-defined for random variabls X 0 we may extend the definition to not-necessarily-non-negative RVs by EX = EX + EX to all random variables for which either of the nonnegative random variables X + = (X 0), X = ( X 0) has finite expectation. If both EX + and EX are infinite, we must leave EX undefined. If both are finite, we call X integrable and note that EX EX+ + EX = E X 4.. Lebesgue Summability Example Does the alternating sum = k N ( ) k+ k (3) converge? Let s look closely. First, for any p R define n { n n p S(n) = k p I(n) = x p p p dx = log n p =. For p < 0 the function x p is increasing on R +, so I(n)+ S(n) < I(n+) and so p < 0 n p + p n k p < (n + ) p, p p Page 3 November, 200

4 and S(n) n p as n. For p > 0 the function x p is decreasing on R +, so I(n+) < S(n) I(n)+ and so p > 0 (n + ) p n < k p n p p p p for p. For 0 < p < we again have S(n) n p as n, but for p > the series converges to some limit S( ) (,p)/(p ). For example, with p = 2 we have S( ) = π 2 / (,2). For any p > the limit is called the Riemann-zeta function S( ) = ζ(p). For p = we again have divergence, with bounds log(n + ) < S(n) log(n) +, so the harmonic series S(n) = n log n. In fact [S(n) log n] γ e converges as n, to Euler s constant γ e = Thus in the Lebesgue sense, the alternating series of Equation (3) does not converge, since its negative and positive parts S (n) = n/2 j= 2j = 2 S(n/2) = 2 [log(n/2) + γ e] + o() S + (n) = n/2 j= 2j = S(n) 2 S(n/2) = [log n + γ e ] 2 [log(n/2) + γ e] + o() = 2 [log(2n) + γ e] + o() each approach as n. Notice that the even partial sums are 2n ( ) k+ k = ( ) ( ) ( ) + = 6 n j= j(j + ), Page 4 November, 200

5 bounded above by π 2 /6 for all n, making the example interesting. More precisely, the difference n ( ) k+ = S + (n) S (n) = [log(2n) log(n/2)] + o() k 2 converges to log 2 as n. What do you think happens with n ξ k/n, for independent random variables ξ k = ± with probability /2 each? Theorem (MCT) Let X and X n 0 be random variables (not necessarily simple) for which X n ր X. Then [ ] lim E[X n] = EX = E lim X n, n n i.e., Equation() is satisfied. For the proof we must find for each n an approximating sequence Y n (m) E + such that Y n (m) ր X n as m and, from it, construct a single sequence Z m = max n m Y (m) n E + that satisfies Z m X m for each m (this is true because, for each n m, Y n (m) X n X m ) and Z m ր X as m (to see this, take ω Ω and ǫ > 0; first find n such that X n (ω) X(ω) ǫ, then find m n such that Y n (m) (ω) X n (ω) ǫ, and verify that Z m (ω) X(ω) 2ǫ), and verify that lim E[X n] lim E[Z m] = EX lim E[X n]. n m n Theorem 2 (Fatou) Let X n 0 be random variables. Then [ ] E lim inf X n lim inf E[X n]. n n To prove this, just set Y n inf m n X n and apply MCT to Y n X n : [ ] [ ] E lim inf X n E lim Y n = lim inf E[Y n] lim inf E[X n] n n n n Notice that equality may fail, as in the example of Equation (2). The condition X n 0 isn t entirely superfluous, but it can be weakened to X n Z for any integrable random variable Z (i.e., one with E Z < ). For indicator random variables X n An of events {A n }, since EX n = P(A n ), Fatou s lemma asserts that ( ) ( ) P lim inf A n lim inf P(A n) lim sup P(A n ) P lim sup A n n n n n Page 5 November, 200

6 Finally we have the most important result of this section: Theorem 3 (DCT) Let X and X n be random variables (not necessarily simple or positive) for which X n X, and suppose that X n Y for some integrable random variable Y with EY <. Then [ ] lim E[X n] = EX = E lim X n, n n i.e., Equation() is satisfied if {X n } is dominated by Y L. Moreover, we have E X n X 0. To show this just apply Fatou s lemma to both (X n X) and to (X X n ); each is bounded below by 2Y. For the moreover part, apply DCT separately to the positive and negative parts (X n X) + 0 (X n X) and (X n X) 0 (X X n ); each is dominated by 2Y and converges to zero. Then use E X n X = E(X n X) + + E(X n X) 0. We will see later that the condition ( ω Ω) X n (ω) X(ω) can be weakened to ( ǫ > 0) P[ X n X > ǫ] L p Spaces and some Expectation Inequalities Fix a probability space (Ω,F,P) and, for any number p > 0, let L p (or L p (Ω,F,P) ) denote the vector space of real-valued (or sometimes complexvalued) random variables X for which E X p <. Note that this is a vector space, since For any X L p and a R, For any X,Y L p, E X + Y p E { X + Y } p E ax p = a p E X p <. E {2max( X, Y )} p = 2 p E {max( X p, Y p )} 2 p E { X p + Y p } = 2 p {E X p + E Y p } <. and hence ax L p and X + Y L p. By Minkowski s Inequality (see item (7) below), the function X p = {E X p } /p Page 6 November, 200

7 is a norm on the space L p for p, inducing a metric d(x,y ) = X Y p that obeys the triangle inequality. We can show that L p is a complete separable metric space in this metric (what does complete mean? Why seperable? What do we need to show to prove each of these?) For 0 < p < the space L p is still a complete separable metric space, but (because ϕ(x) = x p isn t convex for p < ) X Y p doesn t satisfy the triangle inequality and so isn t a metric but X Y p = E X Y p is a metric for 0 < p <. By Jensen s Inequality (see item (5) below) for the convex function ϕ(x) = x q/p, 0 < p < q < X p = {E X p } /p {E X q } /q = X q and hence L p L q for all 0 < p < q <. It is common to treat any two random variables X,Y for which P[X = Y ] as equivalent, and regard L p not as a space of functions, but rather as a space of equivalence classes of functions where X Y if and only if P[X = Y ] =. Distances and norms in L p depend only on the equivalence class. The distinction is only important when we assert the uniqueness of random variables with some specific property. For example, by Hölder s Inequality (item (6) below), for each Y L q the linear functional l Y defined on L p by l Y [X] = E[XY ] is continuous if < p < and p + q =. It happens that these are the only continuous linear functionals on L p, so L p and L q are mutually dual Banach spaces and, in particular, L 2 is a (self-dual) Hilbert space with inner-product X,Y = E[XY ]. Since X p is non-decreasing for each random variable X, we can define X = sup X p L = {X : X < } p< One can show (it s a good exercise, you should do it) that X finite if and only if X is essentially bounded, i.e., if P[ X B] = for some constant B <, and that X is the l.u.b. of the constants B with this property. The space L is also a complete metric space, but except in some trivial cases it isn t separable. Can you prove L (Ω,F,P) isn t separable for Ω = (0,], F = B, and P = λ? What if instead P has finite or countable support {ω j }, with P[{ω j }] = p j > 0, p j =? Page 7 November, 200

8 . For any p > 0, E X p = 0 p x p P[ X > x]dx and E X p < n= np P[ X n] <. The case p = is easiest and most important: if S n= P[ X n] <, then S E X < S+. If X takes on only nonnegative integer values then EX = S. 2. If µ X is the distribution of X, and if f is a measurable real-valued function on R, then Ef(X) = Ω f(x(ω))dp = R f(x)µ X(dx) if either side exists. In particular, µ = EX = xµ X (dx) and σ 2 = E(X µ) 2 = (x µ) 2 µ X (dx). 3. Markov s & Chebychev s Inequalities: If ϕ is positive and increasing, then P[ X u] E[ϕ( X )]/ϕ(u). In particular P[ X µ > u] σ2, P[ X > u] σ2 +µ 2, and for any t > 0, P[X > u] M(t)e tu. u 2 u 2 4. One-Sided Version: P[X > u] σ 2 σ 2 +(u µ) 2 (pf: P[(X µ + t) > (u µ + t)]?; optimize over t µ u) 5. Jensen s Inequality: Let ϕ(x) be a convex function on R, X an integrable RV. Then ϕ(e[x]) E[ϕ(X)]. Examples: ϕ(x) = x p, p ; ϕ(x) = e x ; ϕ(x) = [0 x]. (Introduce L p L q ). The equality is strict if ϕ( ) is strictly convex and X has a non-degenerate distribution. 6. Hölder s Inequality: Let p > and q = p p (e.g., p = q = 2 or p =.0, q = 0). Then E XY [ E X p] p [ E Y q] q. (Pf: XY = exp{ p p log X + q q log Y } { p X p + q Y q }). In particular, for p = q = 2, Cauchy-Schwartz Ineq: EXY E XY EX 2 EY Minkowski s Inequality: Let p and let X,Y L p (Ω,F,P). Then the norm X p (E X p ) p obeys the triangle ineq on L p (Ω,F,P): (E X + Y p ) p (E X p ) p + (E Y p ) p (what if p <?). Pf: E X+Y p E ( X + Y ) X+Y p/q, then Hölder. 8. Hoeffding s Inequality: If {X j } are independent and ( {a j,b j }) s.t. P[a j X j b j ] =, then ( c > 0), S n := n j= X j satisfies P[S n ES n c] exp ( 2c 2 / n b j a j 2). If X j are iid and bounded by X j, e.g., then P[( X n µ) ǫ] e nǫ2 /2. Wassily Hoeffding Page 8 November, 200

9 proved this improvement on Chebychev s inequality in 963 at UNC. Follows from Hoeffding s Lemma: E[e λ(x j µ j ) ] exp ( λ 2 (b j a j ) 2 /8 ), proved in turn from Jensen s ineq and Taylor s thm. Page 9 November, 200