Lecture #17: Expectation of a Simple Random Variable

Transcription

1 Statistics 851 (Fall 2013) October 16, 2013 Prof. Michael Kozdro Lecture #17: Expectatio of a Simple Radom Variable Recall that a simple radom variable is oe that takes o fiitely may values. Defiitio. Let (Ω, F, P) beaprobabilityspace. AradomvariableX :Ω R is called simple if it ca be writte as X = a i 1 Ai where a i R, A i F for i =1, 2,...,.WedefietheexpectatioofX to be E(X) = a i P {A i }. Example Cosider the probability space (Ω, B 1, P) whereω=[0, 1], B 1 deotes the Borel sets of [0, 1], ad P is the uiform probability o Ω. Suppose that the radom variable X :Ω R is defied by 4 X(ω) = a i 1 Ai (ω) where a 1 =4,a 2 =2,a 3 =1,a 4 = 1, ad A 1 =[0, 1 2 ), A 2 =[ 1 4, 3 4 ), A 3 =( 1 2, 7 8 ], A 4 =[ 7 8, 1]. Show that there exist fiitely may real costats c 1,...,c ad disjoit sets C 1,...,C B 1 such that X = c i 1 Ci. Solutio. We fid so that 4, if 0 ω<1/4, 6, if 1/4 ω<1/2, 2, if ω =1/2, X(ω) = 3, if 1/2 <ω<3/4, 1, if 3/4 ω<7/8, 0, if ω =7/8, 1, if 7/8 <ω 1, X = 7 c i 1 Ci where c 1 =4,c 2 =6,c 3 =2,c 4 =3,c 5 =1,c 6 =0,c 7 = 1 ad C 1 =[0, 1 4 ), C 2 =[ 1 4, 1 2 ), C 3 = { 1 2 }, C 4 =( 1 2, 3 4 ), C 5 =[ 3 4, 7 8 ), C 6 = { 7 8 }, C 7 =( 7 8, 1]. 17 1

2 Propositio If X ad Y are simple radom variables, the E(αX + βy )= αe(x)+ βe(y ) for every α, β R. Proof. Suppose that X ad Y are simple radom variables with X = a i 1 Ai ad Y = m b j 1 Bj where A 1,...,A F ad B 1,...,B m F each partitio Ω. Sice αx = α a i 1 Ai = (αa i )1 Ai we coclude by defiitio that E(αX) = (αa i )P {A i } = α a i P {A i } = αe(x). The proof of the theorem will be completed by showig E(X + Y )=E(X)+E(Y ). Notice that {A i B j :1 i, 1 j m} cosists of pairwise disjoit evets whose uio is Ω ad Therefore, by defiitio, X + Y = m (a i + b j )1 Ai B j. E(X + Y )= = = m (a i + b j )P {A i B j } m a i P {A i B j } + a i P {A i } + m b j P {A i B j } m b j P {B j } ad the proof is complete. Fact. If X ad Y are simple radom variables with X Y,the E(X) E(Y ). Exercise Prove the previous fact. 17 2

3 Havig already defied E(X) for simple radom variables,our goal ow is to costruct E(X) i geeral. To that ed, suppose that X is a positive radom variable. That is, X(ω) 0 for all ω Ω. (We will eed to allow X(ω) [0, +] forsomecosistecy.) Defiitio. If X is a positive radom variable, defie the expectatio of X to be E(X) = sup{e(y ):Y is simple ad 0 Y X}. That is, we approximate positive radom variables by simple radom variables. Of course, this leads to the questio of whether or ot this is possible. Fact. For every radom variable X 0, there exists a sequece (X )ofpositive,simple radom variables with X X (that is, X icreases to X). A example of such a sequece is give by k k, if X(ω) < k+1 ad 0 k 2 1, 2 X (ω) = 2 2, if X(ω). (Draw a picture.) Fact. If X 0ad(X )isasequeceofsimpleradomvariableswithx E(X ) E(X). X, the We will prove these facts ext lecture. Now suppose that X is ay radom variable. Write X + =max{x, 0} ad X = mi{x, 0} for the positive part ad the egative part of X, respectively. Note that X + 0adX 0sothatthepositivepartadegativepartofX are both positive radom variables ad X = X + X ad X = X + + X. Defiitio. AradomvariableX is called itegrable (or has fiite expectatio) ifboth E(X + )ade(x )arefiite.ithiscasewedefiee(x) tobe E(X) =E(X + ) E(X ). 17 3

4 Statistics 851 (Fall 2013) October 18, 2013 Prof. Michael Kozdro Lecture #18: Costructio of Expectatio Recall that our goal is to defie E(X) for all radom variables X :Ω R. Weoutliedthe costructio last lecture. Here is the summary of our strategy. Summary of Strategy for Costructig E(X) for Geeral Radom Variables We will (1) defie E(X) for simple radom variables, (2) defie E(X) for positive radom variables, (3) defie E(X) for geeral radom variables. This strategy is sometimes called the stadard machie ad is the outlie that we will follow to prove most results about expectatio of radom variables. Step 1: Simple Radom Variables Let (Ω, F, P) beaprobabilityspace. SupposethatX :Ω R is a simple radom variable so that m X(ω) = a j 1 Aj (ω) where a 1,...,a m R ad A 1,...,A m F.WedefietheexpectatioofX to be E(X) = m a j P {A j }. Step 2: Positive Radom Variables Suppose that X is a positive radom variable. That is, X(ω) 0forallω Ω. (We will eed to allow X(ω) [0, +] forsomecosistecy.)wearealsoassumigatthisstepthat X is ot a simple radom variable. Defiitio. If X is a positive radom variable, defie the expectatio of X to be E(X) = sup{e(y ):Y is simple ad 0 Y X}. (18.1) Propositio For every radom variable X 0, there exists a sequece (X ) of positive, simple radom variables with X X (that is, X icreases to X). 18 1

5 Proof. Let X 0begiveaddefiethesequece(X )by k k, if X(ω) < k+1 ad 0 k 2 1, 2 X (ω) = 2 2, if X(ω). The it follows that X X +1 for every =1, 2, 3,... ad X X which completes the proof. Propositio If X 0 ad (X ) is a sequece of simple radom variables with X X, the E(X ) E(X). That is, if X X +1 ad lim X = X, the E(X +1 ) E(X ) ad lim E(X )=E(X). Proof. Suppose that X 0isaradomvariableadlet(X )beasequeceofsimple radom variables with X 0adX X. Observe that sice the X are icreasig, we have E(X ) E(X +1 ). Therefore, E(X )icreasestosomelimita [0, ]; that is, E(X ) a for some 0 a.(ife(x )isauboudedsequece,thea =. However, if E(X ) is a bouded sequece, the a< follows from the fact that icreasig, bouded sequeces have uique limits.) Therefore, it follows from (18.1), the defiitio of E(X), that a E(X). We will ow show a E(X). As a result of (18.1), we oly eed to show that if Y is a simple radom variable with 0 Y X, thee(y ) a. That is, by defiitio, E(X) = sup{e(y ):Y is simple with 0 Y X} ad so if Y is a arbitrary simple radom variable satisfyig 0 Y X ad E(Y ) a, thethedefiitioofsupremumimpliese(x) a. To this ed, let Y be simple ad write Y = m a k 1{Y = a k }. That is, take A k = {ω : Y (ω) =a k }.Let0< 1addefie Y, =(1 )Y1 {(1 )Y X}. Note that Y, =(1 )a k o the set {(1 )Y X } A k = A k,, ad that Y, =0otheset{(1 )Y >X }.ClearlyY, X ad so m E(Y, )=(1 ) a k P {A k,, } E(X ). 18 2

6 We will ow show that A k,, icreases to A k. Sice X X +1 ad X X we coclude that {(1 )Y X } {(1 )Y X +1 } {(1 )Y X} ad therefore A k,, A k,+1, {(1 )Y X} A k. (18.2) Sice Y X by assumptio, we kow that the evet {(1 )Y X} is equal to Ω. Thus, (18.2) implies A k,, A k for all ad so A k,, A k. (18.3) =1 Coversely, let ω A k = {(1 )Y X} A k = {ω :(1 )Y (ω) X(ω)} A k so that (1 )a k X(ω). Sice Y X, whichistosaythat Y (ω) X(ω) = lim X (ω) for all ω, wekowthatifω A k,thethereissomen such that Y (ω) =a k X (ω) wheever >N.Wethereforecocludethatifω A k ad >N,the(1 )a k <X (ω). (This requires that X is ot idetically 0.) That is, ω A k,, which proves that A k,, A k. (18.4) =1 I other words, it follows from (18.3) ad (18.4) that A k,, icreases to A k, so by the cotiuity of probability theorem (Theorem 10.2), we coclude that Hece, That is, ad so that E(Y, )=(1 ) lim P {A k,,} = P {A k }. m a k P {A k,, } (1 ) m a k P {A k } =(1 )E(Y ) a. E(Y, ) E(X ) E(Y, ) (1 )E(Y ) ad E(X ) a (1 )E(Y ) a (which follows sice everythig is icreasig). Sice 0 < 1isarbitrary, E(Y ) a which, as oted earlier i the proof, is sufficiet to coclude that E(X) a. Combied with our earlier result that a E(X) wecocludee(x) =a ad the proof is complete. 18 3

7 Step 3: Geeral Radom Variables Now suppose that X is ay radom variable. Write X + =max{x, 0} ad X = mi{x, 0} for the positive part ad the egative part of X, respectively. Note that X + 0ad X 0sothatthepositivepartadegativepartofX are both positive radom variables ad X = X + X. Defiitio. AradomvariableX is called itegrable (or has fiite expectatio) ifboth E(X + )ade(x )arefiite.ithiscasewedefiee(x) tobe E(X) =E(X + ) E(X ). Defiitio. If oe of E(X + )ore(x )isifiite,thewecastilldefiee(x) by settig +, if E(X + )=+ ad E(X ) <, E(X) =, if E(X + ) < ad E(X )=+. However, X is ot itegrable i this case. Defiitio. If both E(X + )=+ ad E(X )=+, thee(x) doesotexist. Remark. We see that the stadard machie is really ot that hard to implemet. I fact, it is usually eough to prove a result for simple radom variables ad the exted that result to positive radom variables usig Propositios 18.1 ad The result for geeral radom variables usually the follows by defiitio. 18 4

8 Statistics 851 (Fall 2013) October 21, 2013 Prof. Michael Kozdro Lecture #19: Expectatio ad Itegratio Defiitio. Let (Ω, F, P) beaprobabilityspace,adletx :Ω R be a radom variable. If X is a simple radom variable, say X(ω) = a i 1 Ai (ω) for a 1,...,a R ad A 1,...,A F,thewedefie E(X) = a i P {A i }. If X is a positive radom variable, the we defie E(X) = sup{e(y ):Y is simple ad 0 Y X}. If X is ay radom variable, the we ca write X = X + X where both X + ad X are positive radom variables. Provided that both E(X + )ade(x )arefiite,wedefiee(x) to be E(X) =E(X + ) E(X ) ad we say that X is itegrable (or has fiite expectatio). Remark. If X :(Ω, F, P) (R, B) is a radom variable, the we sometimes write E(X) = X dp = X(ω)dP {ω} = X(ω)P {dω}. Ω Ω That is, the expectatio of a radom variable is the Lebesgue itegral of X. We will say more about this later. Defiitio. Let L 1 (Ω, F, P) bethesetofreal-valuedradomvariableso(ω, F, P) with fiite expectatio. That is, L 1 (Ω, F, P) ={X :(Ω, F, P) (R, B) : X is a radom variable with E(X) < }. Ω We will ofte write L 1 probability space. for L 1 (Ω, F, P) adsuppressthedepedeceotheuderlyig Theorem Suppose that (Ω, F, P) is a probability space, ad let X 1,X 2,..., X, ad Y all be real-valued radom variables o (Ω, F, P). (a) L 1 is a vector space ad expectatio is a liear map o L 1. Furthermore, expectatio is positive. That is, if X, Y L 1 with 0 X Y, the 0 E(X) E(Y ). 19 1

9 (b) X L 1 if ad oly if X L 1, i which case we have E(X) E( X ). (c) If X = Y almost surely (i.e., if P {ω : X(ω) =Y (ω)} =1), the E(X) =E(Y ). (d) (Mootoe Covergece Theorem) If the radom variables X 0 for all ad X X (i.e., X X ad X X +1 ), the lim E(X )=E lim X = E(X). (We allow E(X) =+ if ecessary.) (e) (Fatou s Lemma) If the radom variables X all satisfy X Y almost surely for some Y L 1 ad for all, the E lim if X lim if E(X ). (19.1) I particular, (19.1) holds if X 0 for all. (f) (Lebesgue s Domiated Covergece Theorem) If the radom variables X X, ad if for some Y L 1 we have X Y almost surely for all, the X L 1, X L 1, ad lim E(X )=E(X). Remark. This theorem cotais ALL of the cetral results of Lebesgue itegratio theory. Theorem Let X be a sequece of radom variables. (a) If X 0 for all, the (b) If E X = =1 E(X ) (19.2) =1 with both sides simultaeously beig either fiite or ifiite. the E( X ) <, =1 coverges almost surely to some radom variable Y L 1. I other words, =1 = X X

10 is itegrable with E X = =1 Thus, (19.2) holds with both sides beig fiite. E(X ). Notatio. For 1 p<, letl p = {radom variables X :Ω R such that X p L 1 }. Theorem 19.3 (Cauchy-Schwartz Iequality). If X, Y L 2, the XY L 1 ad =1 E(XY ) E(X 2 )E(Y 2 ). Theorem Let X :(Ω, F, P) (R, B) be a radom variable. (a) (Markov s Iequality) If X L 1, the for every a>0. (b) (Chebychev s Iequality) If X L 2, the for every a>0. P { X a} E( X ) a P { X a} E(X2 ) a

11 Statistics 851 (Fall 2013) October 23, 2013 Prof. Michael Kozdro Lecture #20: Proofs of the Mai Expectatio Theorems Our goal for today is to start provig all of the importat results for expectatio that were stated last lecture. Note that the proofs geerally follow the so-called stadard machie; that is, we first prove the result for simple radom variables, the exted it to o-egative radom variables, ad fially exted it to geeral radom variables. The key results for implemetig this strategy are Propositio 18.1 ad Propositio Theorem Let (Ω, F, P) be a probability space. If L 1 deotes the space of itegrable radom variables, amely L 1 = {radom variables X such that E(X) < }, the L 1 is a vector space ad expectatio is a liear operator o L 1. Moreover, expectatio is mootoe i the sese that if X ad Y are radom variables with 0 X Y ad Y L 1, the X L 1 ad 0 E(X) E(Y ). Proof. Suppose that X 0adY 0areo-egativeradomvariables,adletα [0, ). We kow that there exist sequeces X ad Y of o-egative simple radom variables such that (i) X X ad E(X ) E(X), ad (ii) Y Y ad E(Y ) E(Y ). Therefore, αx is a sequece of o-egative simple radom variables with (αx ) (αx) ad E(αX ) E(αX ). Moreover, X + Y is a sequece of o-egative simple radom variables with (X + Y ) (X + Y )ade(x + Y ) E(X + Y ). However, we kow that expectatio is liear o simple radom variables so that Thus, we fid E(αX )=αe(x ) ad E(X + Y )=E(X )+E(Y ). E(αX ) = αe(x ) E(αX) αe(x) ad E(X + Y ) = E(X )+E(Y ) E(X + Y ) E(X)+E(Y ) so by uiqueess of limits, we coclude E(αX) =αe(x) ade(x + Y )=E(X + Y ). Also ote that if 0 X Y,thethedefiitioofexpectatioofo-egativeradomvariables immediately implies that 0 E(X) E(Y )sothaty L 1 implies X L 1. Suppose ow that X ad Y are geeral radom variables ad let α R. Sice ad (αx) =(αx) + (αx) (αx) + +(αx) α (X + + X ) (X + Y )=(X + Y ) + (X + Y ) (X + Y ) + +(X + Y ) X + + X + Y + + Y. Hece, sice X +, X, Y +, Y L 1,wecocludethatαX L 1 ad X + Y L 1. Fially, sice E(X) =E(X + ) E(X )bydefiitio,adsiceweshowedabovethatexpectatiois liear o o-egative radom variables, we coclude that expectatio is liear o geeral radom variables. 20 1

12 Theorem If X :(Ω, F, P) (R, B) is a radom variable, the X L 1 if ad oly if X L 1. Proof. Suppose that X L 1 so that E(X + ) < ad E(X ) <. Sice X = X + + X ad sice expectatio is liear, we coclude that so that X L 1. E( X ) =E(X + + X )=E(X + )+E(X ) < O the other had, suppose that X L 1 so that E(X + )+E(X ) <. However, sice X + 0adX 0, we kow that E(X + ) 0adE(X ) 0. We ow use the fact that if the sum of two o-egative umbers is fiite, the each umber must be fiite to coclude that E(X + ) < ad E(X ) <. Thus, by defiitio, E(X) =E(X + ) E(X ) < so that X L 1. Corollary. If X L 1, the E(X) E( X ). I particular, if E( X ) =0, the E(X) =0. Proof. Sice E(X + ) 0adE(X ) 0, we have from the triagle iequality E(X) = E(X + X ) = E(X + ) E(X ) E(X + ) + E(X ) = E(X + )+E(X )=E( X ). Sice E(X) 0, if E( X ) =0,the0 E(X) E( X ) =0implyigE(X) =0. Theorem If X, Y :(Ω, F, P) (R, B) are itegrable radom variables with X = Y almost surely, the E(X) =E(Y ). Proof. Suppose that X = Y almost surely so that P {ω Ω:X(ω) =Y (ω)} =1. Tobegi, assume that X 0adY 0adletA = {ω : X(ω) = Y (ω)} so that P {A} =0. Write E(Y )=E(Y 1 A + Y 1 A c)=e(y 1 A )+E(Y 1 A c)=e(y 1 A )+E(X1 A c). ( ) We kow that there exist sequeces X ad Y of o-egative simple radom variables such that (i) X X ad E(X ) E(X), ad (ii) Y Y ad E(Y ) E(Y ). Thus, X 1 A X1 A ad E(X 1 A ) E(X1 A )adsimilarlyy 1 A Y 1 A ad E(Y 1 A ) E(Y 1 A ). For each, the radom variable X takes o fiitely may values ad is therefore bouded by K, say,where K may deped o. Thus,siceX K, we obtai X 1 A K ad so 0 E(X 1 A ) E(K1 A )=KP {A} =0. This implies that E(X 1 A )=0adsobyuiqueessoflimits,E(X1 A )=0. E(Y 1 A )=0. Therefore,by( ), we obtai Similarly, E(Y )=E(Y 1 A )+E(X1 A c)=0+e(x1 A c)=e(x1 A )+E(X1 A c)=e(x). I geeral, ote that X = Y almost surely implies that X + = Y + almost surely ad X = Y almost surely. 20 2

13 Statistics 851 (Fall 2013) October 25, 2013 Prof. Michael Kozdro Lecture #21: Proofs of the Mai Expectatio Theorems (cotiued) We will cotiue provig the importat results for expectatio that were stated i Lecture #19. Recall that a radom variable is said to have fiite mea or have fiite expectatio or be itegrable if E(X) <. Thevectorspaceofallitegrableradomvariableoagive probability space (Ω, F,P)isdeotedbyL 1 ad if 1 p<, the L p = {radom variables X :Ω R such that X p L 1 }. Theorem 21.1 (Cauchy-Schwartz Iequality). If X, Y L 2, the XY L 1 ad E(XY ) E(X 2 )E(Y 2 ). Proof. Sice 0 (X + Y ) 2 = X 2 + Y 2 +2XY ad 0 (X Y ) 2 = X 2 + Y 2 2XY,we coclude that 2 XY X 2 + Y 2 implyig 2E( XY ) E(X 2 )+E(Y 2 ). Thus, if X, Y L 2, we coclude that XY L 1.Foreveryx R, otethat 0 E((xX + Y ) 2 )=x 2 E(X 2 )+2xE(XY )+E(Y 2 ). Sice x 2 E(X 2 )+2xE(XY )+E(Y 2 )isao-egativequadraticix, itsdiscrimiatis ecessarily o-positive; that is, 4[E(XY )] 2 4E(X 2 )E(Y 2 ) 0, or, equivaletly, as required. E(XY ) E(X 2 )E(Y 2 ) Theorem Let X :(Ω, F, P) (R, B) be a radom variable. (a) (Markov s Iequality) If X L 1, the for every a>0. P { X a} E( X ) a (b) (Chebychev s Iequality) If X L 2, the for every a>0. P { X a} E(X2 ) a

14 Proof. Recall that X L 1 if ad oly if X L 1.Sice X 0, we ca write X a1 { X a}. Takig expectatios of the previous expressio implies E( X ) ae(1 { X a} )=P { X a} ad Markov s iequality follows. Similarly, X 2 a 2 1 {X 2 a 2 } so that if X L 2,takig expectatios implies E(X 2 ) a 2 P X 2 a 2. Hece, P { X a} P X 2 a E(X2 ) a 2 yieldig Chebychev s iequality. Defiitio. If X L 2 we say that X has fiite variace ad defie the variace of X to be Var(X) =E[(X E(X)] 2 )=E(X 2 ) [E(X)] 2. Thus, Chebychev s iequality sometimes takes the form Computig Expectatios P { X E(X) a} Var(X) a 2. Havig proved a umber of the mai expectatio theorems, we will ow take a small detour to discuss computig expectatios. We will also show over the course of the ext several lectures how the formulas for the expectatios of discrete ad cotiuous radom variables ecoutered i itroductory probability follow from the geeral theory developed. The first examples we will discuss, however, are the calculatios of expectatios directly from the defiitio ad theory. Example Cosider ([0, 1], B 1, P) whereb 1 are the Borel sets of [0, 1] ad P is the uiform probability. Let X :Ω R be give by X(ω) =ω so that X is a uiform radom variable. We will ow compute E(X) directly by defiitio. Note that X is ot simple sice the rage of X, amely[0, 1], is ucoutable. However, X is positive. This meas that as acosequeceofpropositios18.1ad18.2,ifx is a sequece of positive, simple radom variables with X X, thee(x ) E(X). Thus, let 0, 0 ω<1/2, X 1 (ω) = 1/2, 1/2 ω 1 0, 0 ω 1/4, 1/4, 1/4 ω<1/2, X 2 (ω) = 1/2, 1/2 ω<3/4, 3/4, 3/4 ω 1, 21 2

15 ad i geeral, X (ω) = 2 2 j 1 j ω< j ω Thus, implyig E(X )= = j 1 j 1 2 P 2, j P, j 1 2 = (j 1) 2 2 j 2 j 1 2 = 1 (2 2)(2 1) = E(X) = lim E(X )= lim = Example Cosider ([0, 1], B 1, P) whereb 1 are the Borel sets of [0, 1] ad P is the uiform probability. Let Q 1 =[0, 1] Q ad cosider the radom variable Y :Ω R give by ω, if ω [0, 1] \ Q 1, Y (ω) = 0, if ω Q 1. Note that Y is ot a simple radom variable, although it is o-egative. I order to compute E(Y )itiseasiesttousetheorem20.3.thatis,letx(ω) =ω for ω [0, 1] ad ote that {ω : X(ω) = Y (ω)} = Q 1 \{0}. (The techical poit here is that X(0) = Y (0) = 0. However, if ω Q 1 with ω = 0,the X(ω) = Y (ω).) Sice P {Q 1 \{0}} =0,weseethatX ad Y differ o a set of probability 0 so that X = Y almost surely. Sice X is a uiform radom variable o [0, 1], we kow from the previous example that E(X) = 1/2. Therefore, usig Theorem 20.3 we coclude E(Y )=E(X) =1/

16 Statistics 851 (Fall 2013) October 28, 2013 Prof. Michael Kozdro Lecture #22: Computig Expectatios of Discrete Radom Variables Recall from itroductory probability classes that a radom variable X is called discrete if the rage of X is at most coutable. The formula for the expectatio of a discrete radom variable X give i these classes is E(X) = jp {X = j}. We will ow derive this formula as a cosequece of the geeral theory developed durig the past several lectures. Suppose that the rage of X is at most coutable. Without loss of geerality, we ca assume that Ω = {1, 2,...,}, F =2 Ω,adX :Ω R is give by X(ω) =ω. Assume further that the law of X is give by P {X = j} = p j where p j [0, 1] ad p j =1. Let A j = {X = j} = {ω : X(ω) =j} so that X ca be writte as X(ω) = j1 Aj. Observe that if p j =0forifiitelymayj, thex is simple ad ca be writte as X(ω) = j 1 Aj for some < which implies that E(X) = j P {A j } = jp {X = j}. O the other had, suppose that p j =0forifiitelymayj. Ithiscase,X is ot simple. We ca approximate X by simple fuctios as follows. Let A j, = {ω : X(ω) =j, j } so that A j, A j,+1 ad A j, = A j. =1 22 1

17 That is, A j, A j ad so by cotiuity of probability we coclude If we ow set so that X (ω) = lim P {A j,} = P {A j }. X(ω)1 Aj, = E(X )= jp {A j, }, j1 Aj, the (X )isasequeceofpositivesimpleradomvariablewithx X. Propositio 18.2 the implies as required. E(X) = lim E(X )= lim jp {A j, } = P {A j } = jp {X = j} Computig Expectatios of Cotiuous Radom Variables We will ow tur to that other formula for computig expectatio ecoutered i elemetary probability courses, amely if X is a cotiuous radom variable with desity f X,the E(X) = xf X (x)dx. It turs out that verifyig this formula is somewhat more ivolved tha the discrete formula. As such, we eed to take a brief detour ito some geeral fuctio theory. Some Geeral Fuctio Theory Suppose that f : X Y is a fuctio. We are implicitly assumig that f is defied for all x X. WecallX the domai of f ad call Y the codomai of f. The rage of f is the set f(x) ={y Y : f(x) =y for some x X}. Note that f(x) Y. Iff(X) =Y, thewesaythatf is oto Y. Let B Y. Wedefief 1 (B) by f 1 (B) ={x X : f(x) =y for some y B} = {f B} = {x : f(x) B}. We call X a topological space if there is a otio of ope subsets of X. TheBorelσ-algebra o X, writteb(x), is the σ-algebra geerated by the ope sets of X. 22 2

18 Let X ad Y be topological spaces. A fuctio f : X Y is called cotiuous if for every ope set V Y, thesetu = f 1 (V ) X is ope. Afuctiof : X Y is called measurable if f 1 (B) B(X) foreveryb B(Y). Sice the ope sets geerate the Borel sets, this theorem follows easily. Theorem Suppose that (X, B(X)) ad (Y, B(Y)) are topological measure spaces. The fuctio f : X Y is measurable if ad oly if f 1 (O) B(X) for every ope set O B(Y). The ext theorem tells us that cotiuous fuctios are ecessarily measurable fuctios. Theorem Suppose that (X, B(X)) ad (Y, B(Y)) are topological measure spaces. If f : X Y is cotiuous, the f is measurable. Proof. By defiitio, f : X Y is cotiuous if ad oly if f 1 (O) X is a ope set for every ope set O Y. Sice a ope set is ecessarily a Borel set, we coclude that f 1 (O) B(X) foreveryopeseto B(Y). However, it ow follows immediately from the previous theorem that f : X Y is measurable. The followig theorem shows that the compositio of measurable fuctios is measurable. Theorem Suppose that (W, F), (X, G), ad (Y, H) are measurable spaces, ad let f :(W, F) (X, G) ad g :(X, G) (Y, H) be measurable. The the fuctio h = g f is a measurable fuctio from (W, F) to (Y, H). Proof. Suppose that H H. Sice g is measurable, we have g 1 (H) G. Sice f is measurable, we have f 1 (g 1 (H)) F.Sice the proof is complete. h 1 (H) =(g f) 1 (H) =f 1 (g 1 (H)) F 22 3

19 Statistics 851 (Fall 2013) October 30, 2013 Prof. Michael Kozdro Lecture #23: Proofs of the Mai Expectatio Theorems (cotiued) Theorem 23.1 (Mootoe Covergece Theorem). Suppose that (Ω, F, P) is a probability space, ad let X 1,X 2,..., ad X be real-valued radom variables o (Ω, F, P). If X 0 for all ad X X (i.e., X X ad X X +1 ), the lim E(X )=E (allowig E(X) =+ if ecessary). lim X = E(X) Proof. For every, lety,k, k =1, 2,..., be o-egative ad simple with Y,k X ad E(Y,k ) E(X )ask which is possible by Propositios 18.1 ad Set Z k =max k Y,k, ad observe that 0 Z k Z k+1 so that (Z k )isaicreasigsequeceofo-egative simple radom variables which ecessarily has a limit Z =lim k Z k. By Propositio 18.2, E(Z k ) E(Z). We ow observe that if k, the Y,k Z k X X k X. (23.1) We ow deduce from (23.1) that X Z X almost surely, ad so lettig implies X = Z almost surely. We also deduce from (23.1) that for k. Fix ad let k to obtai E(Y,k ) E(Z k ) E(X k ) Now let to obtai E(X ) E(Z) lim k E(X k ). Thus, lim E(X ) E(Z) lim E(X k ). k E(Z) = lim E(X ). But X = Z almost surely so that E(X) =E(Z) adwecoclude as required. E(X) = lim E(X ) 23 1

20 Theorem 23.2 (Fatou s Lemma). Suppose that (Ω, F, P) is a probability space, ad let X 1,X 2,... be real-valued radom variables o (Ω, F, P). IfX Y almost surely for some Y L 1 ad for all, the E lim if X lim if E(X ). (23.2) I particular, (23.2) holds if X 0 for all. Proof. Without loss of geerality, we ca assume X 0. Here is the reaso. If X = X Y, the X L 1 with X 0ad E lim if X lim if E( X ) if ad oly if E lim if X lim if E(X ) because Hece, if X 0, set lim if X = lim if X Y. Y =if k X k so that Y 0isaradomvariableadY Y +1.ThisimpliesthatY coverges ad lim Y =limif X. Sice X Y,mootoicityofexpectatioimpliesE(X ) E(Y ). This yields lim if E(X ) lim E(Y )=E lim Y =limif E(X ) by the Mootoe Covergece Theorem ad the proof is complete. Theorem 23.3 (Lebesgue s Domiated Covergece Theorem). Suppose that (Ω, F, P) is a probability space, ad let X 1,X 2,... be real-valued radom variables o (Ω, F, P). If the radom variables X X, ad if for some Y L 1 we have X Y almost surely for all, the X L 1, X L 1, ad lim E(X )=E(X). Proof. Suppose that we defie the radom variables U ad V by U =limif X ad V =limsupx. The assumptio that X X implies that U = V = X almost surely so that E(U) = E(V )=E(X). Ad if we also assume that X Y almost surely for all so that X L 1, the we coclude X Y. Sice Y L 1,wehaveX L 1. Moreover, X Y almost surely ad Y L 1 so by Fatou s Lemma, E(U) lim if E(X ). 23 2

21 However, we also kow X Y almost surely ad so Fatou s lemma also implies V =limif ( X ) E(V )=E( V ) lim if Combiig these two iequalities gives E(X) =E(U) lim if so that E(X ) E(X) asrequired. E( X )= lim sup E(X ). E(X ) lim sup E(X ) E(V )=E(X) Theorem Let X be a sequece of radom variables. (a) If X 0 for all, the (b) If E X = =1 E(X ) (23.3) =1 with both sides simultaeously beig either fiite or ifiite. the E( X ) <, =1 coverges almost surely to some radom variable Y L 1. I other words, is itegrable with =1 =1 X X E X = =1 Thus, (23.3) holds with both sides beig fiite. Proof. Suppose that S = E(X ). =1 X k ad T = X k. Sice S cotais fiitely may terms, we kow by liearity of expectatio that E(S )=E X k = E( X k ). 23 3

22 Moreover, 0 S S +1 so that S icreases to some limit S = X k. Note that S(ω) [0, +]. By the Mootoe Covergece Theorem, E(S) = lim E(S )= lim E( X k )= E( X k ). If X 0forall, thes = T ad (a) follows. If X are geeral radom variables, with E( X k ) <, the E(S) <. Now, for every > 0, sice 1 S= S, wecocludep {S = } = E(1 S= ) E(S). Sice >0is arbitrary ad E(S) <, wecocludethatp {S = } =0. Therefore, =1 is absolutely coverget almost surely ad its sum is the limit of the sequece T.Moreover, T S S ad S L 1,sobytheDomiatedCovergeceTheorem, E X k = E lim X k = lim E(X k )= E(X k ) provig (b). X 23 4

23 Statistics 851 (Fall 2013) November 1, 2013 Prof. Michael Kozdro Lecture #24: The Expectatio Rule Let (Ω, F, P) beaprobabilityspace,adletx :Ω R be a radom variable so that X 1 (B) Ffor every B B. Suppose that h : R R is a measurable fuctio so that h 1 (B) Bfor every B B. Propositio The fuctio h X :Ω R give by h X(ω) =h(x(ω)) is a radom variable. Proof. To prove that h X is a radom variable, we must show that (h X) 1 (B) = X 1 (h 1 (B)) Ffor every B B. Thus, suppose that B B. Sice h is measurable, we kow h 1 (B) B. Let A = h 1 (B). Sice A is a Borel set, i.e., A B,wekowthat X 1 (A) F sice X is a radom variable. That is, (h X) 1 (B) =X 1 (h 1 (B)) = X 1 (A) F so that h X :Ω R is a radom variable as required. We kow that X iduces a probability o (R, B) calledthelaw of X ad defied by P X {B} = P X 1 (B) = P {X B} = P {ω Ω:X(ω) B}. I other words, (R, B, P X )isaprobabilityspace. Thismeasthatifh : R R is a measurable fuctio o (R, B, P X ), the we ca also call h a radom variable. As such, it is possible to discuss the expectatio of h, amely E(h) = h(x) P X {dx}. R As we will ow show, the expectatio of h ad the expectatio of h X are itimately related. Theorem 24.2 (Expectatio Rule). Let X :(Ω, F, P) (R, B) be a radom variable with law P X, ad let h :(R, B) (R, B) be measurable. (a) The radom variable h(x) L 1 (Ω, F, P) if ad oly if h L 1 (R, B, P X ). (b) If h 0, or if either coditio i (a) holds, the E(h(X)) = h(x(ω)) P {dω} = Ω R h(x) P X {dx}. Remark. The secod itegral i the theorem is actually a Lebesgue itegral sice it is the itegral of a radom variable, amely h, with respect to a probability, amely P X. As we will see shortly, we ca always relate this to a Riema-Stieltjes itegral ad i some cases to a Riema itegral. 24 1

24 Proof. Sice h X :Ω R is a radom variable by Propositio 24.1, we kow E(h(X)) = h(x(ω)) P {dω} by defiitio. The cotet of the theorem is that this itegral is equal to h(x) P X {dx}. R Ω I order to complete the proof, we will follow the stadard machie. That is, let h(x) =1 B (x) for B Bso that E(h(X)) = E(1 B (X)) = P {X B} = P X 1 (B) = P X {B} = P X {dx} B = 1 B (x) P X {dx} R = h(x) P X {dx}. Hece, if h is simple, say h(x) = b i 1 Bi (x) for some b 1,...,b R ad B 1,...,B B,the E(h(X)) = E b i 1 Bi (x) = b i P X {B i } = = = R R b i b i 1 Bi (x) P X {dx} h(x) P X {dx} R R 1 Bi (x) P X {dx} usig liearity of expectatio. If h 0, let h 0besimplewithh h so that E(h(X)) = E lim h (X) = lim E(h (X)) = lim h (x) P R X {dx} = lim h (x) P X {dx} R = h(x) P X {dx} usig the Mootoe Covergece Theorem twice. I particular, (b) follows for h 0. If we apply the above procedure to h, the(a)follows. Ifh is ot positive, the writig h = h + h implies the result by subtractio. R 24 2