Random Variables. STA 205 Week 3

Transcription

1 Random Varables Let Ω be any set, F any Sgma Feld on Ω, and P any probablty measure defned for each element of F; such a trple (Ω, F, P) s called a probablty space. Let R denote the real numbers (, ) and B the Borel sets on R generated by (for example) the half-open sets (a, b]. Defnton. A real-valued Random Varable s a functon X : Ω R that s F\B-measurable,.e., that satsfes X 1 (B) = {ω : X(ω) B} F for each Borel set B B (or, equvalently, smply for each set B of the form (, b] for some ratonal < b < ). Ths s sometmes denoted smply X 1 (B) F. Snce the probablty measure P s only defned on sets F F, a random varable must satsfy ths condton f we are to be able to fnd the probablty Pr[X B] for each Borel set B, or even f we want to fnd the dstrbuton functon (DF) F X (b) Pr[X b] for each ratonal number b. Note that set-nverses are rather wellbehaved functons from one class of sets to another; specfcally, for any collecton {A α } B, [X 1 (A α )] c = X 1( (A α ) c) X 1 (A α ) = X 1( ) A α α α X 1 (A α ) = X 1( ) A α α α and thus, measurable or not, X 1 (B) s a Sgma Feld f B s; t s denoted F X (or σ(x)), called the sgma feld generated by X, and s the smallest sgma feld G such that X s (G\B)- measurable. In partcular, X s (F\B)- measurable f and only f σ(x) F. In probablty and statstcs, sgma feld s represent nformaton: a random varable Y s measurable over F X f and only f the value of Y can be found from that of X,.e., f there exsts some functon ϕ such that Y = ϕ(x). Note the dfference n perspectve between real analyss, on the one hand, and probablty/statstcs, on the other; n analyss t s only Lebesgue measurablty that mathematcans worry about, and only to avod paradoxes and pathologes. In probablty and statstcs we study measurablty for a varety of sgma feld s, and the (techncal) concept of measurablty corresponds to the (emprcal) noton of observablty. DISTRIBUTIONS. A random varable X on a probablty space (Ω, F, P) nduces a measure µ X on (R, B), called the dstrbuton measure (or smply the dstrbuton), va the relaton µ(b) = P[X B], sometmes wrtten more succnctly as µ X = P X 1 or even PX 1. Functons of Random Varables Let (Ω, F, P) be a probablty space, X a (real-valued) random varable, and f : R R a (realvalued B\B) measurable functon. Then Y = f(x) s a random varable,.e., Y 1 (B) = X 1 (f 1 (B)) F for any B B. Also every contnuous or pecewse-contnuous real-valued functon on R s B\Bmeasurable.

2 Random Vectors Denote by R 2 the set of ponts (x, y) n the plane, and by B 2 the sgma feld generated by rectangles of the form {(x, y) : a < x b, c < y d} = (a, b] (c, d]. Note that fnte unons of those rectangles form a feld F0 2, so the mnmal ) sgma feld and mnmal λ system contanng F 0 2 concde, and the assgnment λ0( 2 (a, b] (c, d] = (b a) (d c) has a unque extenson to a measure on all of B 2, called two-dmensonal Lebesgue measure (and denoted λ 2 ). Of course, t s just the area of sets n the plane. A F\R 2 -measurable mappng X : Ω R 2 s called a (two-dmensonal) random vector, or smply an R 2 -valued random varable, or (a bt ambguously) an R 2 -RV. It s easy to show that the components X 1, X 2 of a R 2 -RV X are each RV s, and conversely that for any two random varables X 1 and X 2 the two-dmensonal RV (X, Y ) : Ω R 2 s F\R 2 -measurable,.e., s a R 2 -RV. Also, any measurable (and n partcular, any pecewse-contnuous) functon f : R 2 R nduces a random varable f(x, Y ): ths shows that such combnatons as X + Y, X/Y, X Y, X Y, etc. are all random varables f X and Y are. The same deas work n any fnte number of dmensons, so wthout any specal notce we wll regard n-tuples (X 1,..., X n ) as R n -valued RV s, or F\B n -measurable functons, and wll use Lebesgue n-dmensonal measure λ n on B n. Agan X, X, mn X, and max X are all random varables. Even f we have nfntely many random varables we can verfy the measurablty of X, nf X, and sup X, and of lm nf X, and lm sup X as well: for example, [ω : sup [ω : lm sup The event X converges s the same as X (ω) r] = X (ω) r] = [ω : X (ω) r] j= [ω : X (ω) r]. [ω : lm sup X (ω) lm nf X (ω) = 0], and so s F- measurable and has a well defned probablty P[lm sup X = lm nf X ]. Ths s one pont where countble addtvty (and not just fnte addtvty) of P s crucal, and where F needs to be a sgma feld (and not just a feld). Example: Dscrete RV s If an RV X can take on only a fnte or countable set of values, say b, then each set Λ = [ω : X(ω) = b ] must be n F, the Λ are dsjont, and X can be represented n the form X(ω) = b 1 Λ (ω), where ( ) { 1 f ω Λ 1 Λ (ω) = 0 f ω / Λ s the so-called ndcator functon of Λ. By ncludng a term wth b = 0, f necessary, we can assume that Ω = Λ so the {Λ } form a countable partton of Ω. Any RV can be approxmated as well as we lke by a smple RV of the form ( ). Page 2

3 EXPLICIT CONSTRUCTION OF SIGMA FIELDS (OMIT ON FIRST READING) Ordnals and Transfnte Inducton Every fnte set S (say, wth n < elements) can be totally ordered a 1 a 2 a 3... n n! ways, but n some sense every one of these s the same f 1 and 2 are two orderngs, there exsts a 1 1 order-preservng somorphsm ϕ : (S, 1 ) (S, 2 ). Thus up to somorphsm there s only one orderng for any fnte set. For countably nfnte sets there are many dfferent orderngs. The obvous one s a 1 a 2 a 3..., ordered just lke the postve ntegers N; ths orderng s called ω, the frst lmt ordnal. But we could pck any element (say, b 1 S) and order the remander of S n the usual way, but declare a n b 1 for every n N; one element s bgger (n the orderng) than all the others. Ths s not somorphc to ω, and t s called ω+1, the successor to ω. If we set asde two elements (say, b 1 b 2 ) to follow all the others we have ω + 2, and smlarly we have ω + n for each n N. The lmt of all these s ω + ω, or 2ω... t s the orderng we would get f we lexcographcally ordered the set {(, j) : = 1, 2 j N} of the frst two rows of ntegers n the frst quadrant, declarng (1, ) (2, j) for every, j and otherwse (, j) (, k) f j < k. We would get the successor to ths, 2ω + 1, by extendng the lexcographcal orderng as we add (3, 1) to S; n an obvous way we get 2ω + n and eventually the lmt ordnals 3ω, 4ω, etc., and the successor ordnals mω+n. The lmt of all these s ωω or ω 2, the lexcographcal orderng of the entre frst quadrant of ntegers (, j). It too has successors ω 2 + n (graphcally you can thnk about nteger trplets (, j, k)), and lmts lke ω 2 + ω and ω 3 and ω ω (whch turns out to be the same as 2 ω ). In general an ordnal s a successor ordnal f t has a maxmal element, and otherwse s a lmt ordnal. Every ordnal α has a successor α + 1, and every set of ordnals {α n } has a lmt (least upper bound) λ. Let Ω be the frst uncountable ordnal. Proofs and constructons by transfnte nducton usually have one step at each successor ordnal, and another at each lmt ordnal. The Borel sets can be defned by transfnte constructon as follows. Let F 1 be any class of subsets of some probablty space X (perhaps F 1 s the open sets n X = R, for example). Succ: For any ordnal α, let F α+1 be the class of countable unons of sets E n F α and ther complements E m : Em c F α. Lm: For any lmt ordnal λ, let F λ = α λ F α. Together these defne F α for all ordnals, lmt and successor; the sgma feld generated by F 1 s just F Ω. It remans to prove that: 1. F 1 F Ω,.e., F Ω contans the open sets; 2. E F Ω = E c F Ω,.e., F Ω s closed under complements; 3. E n F Ω = n=1 E n F Ω,.e., F Ω s closed under countable unons; 4. F Ω G for any sgma feld G contanng F 1. Item 1. s trval snce F Ω = α Ω F α, and n partcular contans F 1. Item 2. follows by transfnte nducton upon notng that E F α = E c F α+1. Item 3 follows by notng that E n F Ω = E n F αn for some α n Ω, and β = sup n< α n s an ordnal satsfyng α n β Ω and hence E n F β for all n and n=1e n F β+1. Verfyng the mnmalty condton Item 4 s left as an exercse. It sn t mmedately obvous from the constructon that we couldn t have stopped earler for example, that F 2 or F ω sn t already the Borel sets, unchangng as we allow successvely more ntersectons and unons. In fact that happens f the orgnal space X s countable or fnte; n the case of R, however, one can show that F α F α+1 for every α Ω. Do you thnk ths explct constructon s clearer or more complcated than the completon argument used n Bllngsly s book? Page 3

4 INFINITE COIN TOSS For each ω Ω = (0, 1] and nteger n N let δ n (ω) be the n th bt n the nontermnatng bnary expanson of ω. There s some ambguty n the dyadc expanson of ratonals... for example, one-half can be wrtten ether as 0.10b or as the nfntely repeatng b. If we had used the conventon that the dyadc ratonals have only fntely many 1 s n ther expanson (so 1/2 = 0.10b) then δ n (ω) = 2 n ω (mod 2); wth our conventon that all expansons must have nfntely many ones, we have δ n (ω) = ( 2 n ω + 1) (mod 2). We can thnk of {δ n } as an nfnte sequence of random varables, all defned on the same measurable space (Ω, B 1 ), wth the random varable δ 1 equal to zero on (0, 1 / 2 ] and one on ( 1 / 2, 1]; δ 2 equal to zero on (0, 1 / 4 ] ( 1 / 2, 3 / 4 ] and one on ( 1 / 4, 1 / 2 ] ( 3 / 4, 1]; and, n general, δ n equal to one on a unon of 2 n 1 ntervals, each of length 2 n (for a total length of 1 / 2 ), and equal to zero on the complementary set, also of length 1 / 2. For the Lebesgue probablty measure P on Ω that just assgns to each event E B 1 ts length P(E), we have P[X n = 0] = P[X n = 1] = 1 / 2. Queston 1: If we had used the other conventon that every bnary expanson must have nfntely many zero s (nstead of one s), so e.g. 1/2 = 0.10b, then what would the event E 1 {ω : δ 1 (ω) = 1} have been? The sgma feld generated by any famly of random varables {X α } (whether countable or not) s defned to be the smallest sgma feld for whch each X α s measurable,.e., the smallest one contanng each X 1 α (B) for every Borel set B R. For each fxed n the σ algebra F n generated by δ 1,..., δ n s just the feld F n = { (a /2 n, b /2 n ]} consstng of all (fnte) unons of leftopen ntervals wth both endponts an nteger over 2 n. Each set n F n can be specfed by lstng whch of the 2 n ntervals (, +1 ] (0 < 2 n ) t contans, so there are 2 2n sets n F 2 n 2 n n altogether. The unon F n conssts of all fnte unons of left-open ntervals wth dyadc ratonal endponts. It s closed under takng complements but t stll sn t a sgma feld, snce t sn t closed under takng countable unons and ntersectons; for example, t contans the set E n = {ω : δ n =1} for each n N and fnte ntersectons lke E 1... E n = (1 2 n, 1], but not ther countable ntersecton n=1e n = {1}. By defnton the jon F = n F n σ( n F n ) s just the smallest sgma feld that contans each F n (and so contans ther unon); ths s just the famlar Borel sets n (0, 1]. Lebesgue measure P, whch assgns to any nterval (a, b] ts length, s determned on each F n by the rule P [ (a /2 n, b /2 n ] ] = (b a )2 n or, equvalently, by the jont dstrbuton of the random varables δ 1,..., δ n : ndependent Bernoull s, each wth P[δ = 1] = 1 / 2. For any number 0 < p < 1 we can make a smlar measure P p on (Ω, F n ) by requrng P p [δ n = 1] = p and, more generally, P[δ = d, 1 n] = p Σd (1 p) n Σd ; the four ntervals n F 2 would have probabltes [(1 p) 2, p(1 p), p(1 p), and p 2 ], for example, nstead of [ 1 / 4, 1 / 4, 1 / 4, 1 / 4 ]. Ths determnes a measure on each F n, whch extends unquely to a measure P p on F = n F n. For p = 1/2 ths s Lebesgue Measure, characterzed by the property that P [ (a, b] ] = b a for each 0 a b 1, but the other P p s are new. Ths example (the famly δ n of random varables on the spaces (Ω, F, P p )) s an mportant one, and lets us buld other mportant examples. Under each of these probablty dstrbutons all the δ n are both dentcally dstrbuted and ndependent,.e., n P[δ 1 A 1,..., δ n A n ] = P[δ 1 A ]. Page 4

5 Any probablty assgnment to ntervals (a, b] Ω determnes some jont probablty dstrbuton for all the {δ n }, but typcally the δ n wll be nether ndependent nor dentcally dstrbuted. For any DF (.e., non-decreasng rght-contnuous functon F (x) satsfyng F (0) = 0 and F (1) = 1), the prescrpton P F ( (a, b] ) F (b) F (a) determnes a probablty dstrbuton on every Fn that extends unquely to F, determnng the jont dstrbuton of all the {δ n }. Queston 2: For F (x) = x 2, are δ 1 and δ 2 dentcally dstrbuted? Independent? Fnd the margnal probablty dstrbuton for each δ n under P F. MEASURABILITY AND OBSERVABILITY Fx any measure P p on (Ω, F) (say, Lebesgue measure P = P.5 ), and defne a new sequence of random varables Y n on (Ω, F, P) by Y n (ω) = n ( 1) δn(ω) = n ( 2δn (ω) 1 ), the sum of n ndependent terms, each ±1 wth probablty 1/2 each. Ths s the symmetrc random walk (t would be assymetrc wth P p for p.5), startng at the orgn and movng left or rght wth equal probablty at each step; each Y n s 2S n n for the bnomal B(n,.5) random varable S n = n δ, the partal sums of the δ n s. The sgma feld generated by the frst n Y s, that generated by the frst n S s, and that generated by the frst n δ s are all the same, the fnte feld F n of all unons of half-open ntervals wth endponts of the form j2 n, and a random varable Z on (Ω, F, P) s F n -measurable f and only f Z can be wrtten as a functon Z = ϕ n (δ 1,..., δ n ) of the frst n δ s. Thus measurablty means somethng for us Z s measureable over F n f and only f you can tell ts value by observng the frst n values of δ (or, equvalently, of Y or S ). We ll see that a functon Z on Ω s F-measurable (.e., s a random varable) f and only f you can approxmate t arbtrarly well by a functon of the frst n δ s, as n. UNIFORMS, NORMALS, AND MORE From the nfnte sequence of ndependent random bts {δ n } we can construct as many random varables as we lke of any dstrbuton, all on the same space (Ω, F, P), the unt nterval wth Lebesgue measure (length). For example, set: U 1 (ω) = U 2 (ω) = 2 δ 2 (ω) U 3 (ω) = 2 δ 3 (ω) U 4 (ω) = 2 δ 5 (ω) 2 δ 7 (ω), each the sum of dfferent (and therefore ndependent) random bts; t s easy to see that {U n } wll be ndependent, unformly dstrbuted random varables for n = 1, 2, 3, 4, and that we could construct as many of them as we lke usng successve prmes {2, 3, 5, 7, 11, 13,...}. Queston 3: Why dd I use δ 2, δ 3, δ 5, δ 7? Gve another choce that would have worked. Let F (x) be any DF (rght-contnuous, non-decreasng functon on R wth lmts 0 and 1 x and x +, respectvely) and defne: X 1 (ω) = nf[x R : F (x) U 1 (ω)] X 2 (ω) = nf[x R : F (x) U 2 (ω)] X 3 (ω) = nf[x R : F (x) U 3 (ω)] X 4 (ω) = nf[x R : F (x) U 4 (ω)]; Page 5

6 t s not hard to see or show (we ll do t n a week or so) that the {X n } are ndependent, each wth DF F (x) = P [X n x]. For example, we could take X n = Φ 1 (U n ) to get ndependent random varables wth the standard normal dstrbuton or X n = log(1 U n ) for the exponental dstrbuton. Independent normal random varables can be constructed even more effcently va: Z 1 (ω) = cos(2πu 1 ) 2 ln U 2 Z 3 (ω) = cos(2πu 3 ) 2 ln U 4 Z 2 (ω) = sn(2πu 1 ) 2 ln U 2 Z 4 (ω) = sn(2πu 3 ) 2 ln U 4 ; We ve seen that from ordnary length measure on the unt nterval (or, equvalently, from a sngle unformly-dstrbuted random varable ω) we can construct frst an nfnte sequence of ndependent 0 1 bts δ n ; then an nfnte sequence of ndependent unform random varables U n ; then an nfnte sequence of ndependent normal random varables Z n or, more generally, random varables X n wth any dstrbuton(s) we choose. The Cantor Dstrbuton Set Y n=1 2δ n3 n ; then the ternery expanson of y = Y (ω) ncludes only zero s (where δ n = 0) and two s (where δ n = 1), and so les n the Cantor set. Snce Y takes on uncountably many dfferent values, t cannot have a dscrete random varable. Its CDF can be gven analytcally by the expresson F (y) = {2 n : t n > 0, t m 1, 1 m < n}, n=1 n terms of the ternary expanson t n 3 n y (mod 3) of y = n=1 t n3 n or graphcally as x Evdently F (x) has dervatve F = 0 wherever t s dfferentable; ths dstrbuton s an example of a sngular dstrbuton, one that s nether absolutely contnuous nor dscrete. Theorem. Let F (x) be any dstrbuton functon. Then there exst unque numbers p d 0, p c 0, p s 0 wth p d + p c + p s = 1 and dstrbuton functons F d (x), F c (x), F s (x) wth the propertes that F d s dscrete wth some probablty mass functon f d (x), F c s absolutely Page 6

7 contnuous wth some probablty densty functon f c (x), and F s s sngular, satsfyng F (x) = p d F d (x) + p c F c (x) + p s F s (x) and F d (x) = f d (t), F c (x) = f c (t) dt, F s(x) = 0. t x t x EXPECTATION AND INTEGRAL INEQUALITIES Dscrete RV s If a random varable Y can take on only a fnte or countably nfnte set of values, say b, then each set Λ = [ω : Y (ω) = b ] must be n F; the Λ are dsjont, and Y can be represented n the form Y (ω) = { 1 f ω Λ b 1 Λ (ω), where 1 Λ (ω) = ( ) 0 f ω / Λ s the so-called ndcator functon of Λ. By addng a term wth b = 0, f necessary, we can assume that Ω = Λ so the {Λ } form a countable partton of Ω. Any RV X can be approxmated as well as we lke by a smple RV of the form ( ) by choosng ɛ > 0, settng b ɛ, and Λ {ω : b X(ω) < b + ɛ} X ɛ (ω) b 1 Λ (ω) = ɛ X(ω)/ɛ It s easy to defne the expectaton of such a smple RV, or (equvalently) the ntegral of X ɛ over (Ω, F, P), f X s bounded below or above (to avod ndetermnate sums): EX ɛ = X ɛ (ω) P(dω) = X ɛ (ω) dp(ω) = X ɛ dp = b P(Λ ) Ω Ω Ω Snce X ɛ (ω) X(ω) < X ɛ (ω) + ɛ, we have EX ɛ EX < EX ɛ + ɛ,.e., ɛp[ɛ X < (+1)ɛ] EX < ɛp[ɛ X < (+1)ɛ] + ɛ. ( ) Ths determnes the value of E X = X dp for each random varable X. If we take ɛ = 2 n Ω above, and smplfy the notaton by wrtng X n for X 2 n = 2 n 2 n X, the sequence X n ncreases monotoncally to X and we can defne EX = lm n EX n. Note that even for Ω = (0, 1], P = λ(dx) (Lebesgue measure), and X contnuous, the passage to the lmt suggested n ( ) s not the same as the lmt of Remann sums that s used to ntroduce ntegraton n undergraduate calculus courses; for the Remann sum t s the x-axs that s broken up nto ntegral multples of some ɛ, determnng the ntegral of contnuous functons, whle here t s the y axs that s broken up, determnng the ntegral of all measurable functons. The two defntons of ntegral agree for contnuous functons where they are both defned, of course, but the present one s much more general. If X s not bounded below or above, we can set X + 0 X and X 0 X, so that X = X + X wth both X + and X bounded below (by zero), so ther expectatons are welldefned; f ether EX + < or EX <, we can unambguously defne EX EX + EX, whle f EX + = EX = we regard EX as undefned. For any measurable set Λ F we wrte Λ X dp for EX1 Λ. For Ω R, f P gves postve probablty to ether {a} or {b} then the ntegrals over the sets (a, b), (a, b], [a, b), and [a, b] may all be dfferent; the notaton b X dp sn t expressve enough to dstngush them. a Page 7

8 Frequently n Probablty and Statstcs we need to calculate or estmate ntegrals and expectatons; usually ths s done through lmtng arguments n whch a sequence of ntegrals s shown to converge to the one whose value we need. Here are some mportant propertes of ntegrals for any measurable set Λ F and random varables {X n }, X, Y, useful for boundng or estmatng the ntegral of a random varable X (they re only lsted here for reference and so we can talk about them don t worry, you won t have to remember them all or know how to prove them!): 1. Λ X dp s well-defned and fnte f and only f Λ X dp <, and Λ X dp X dp. Λ We can also defne X dp for any X bounded below by some b >. Λ 2. Lebesgue s Monotone Convergence Thm: If 0 X n X, then Λ X n dp Λ X dp. In partcular, the sequence of ntegrals converges (possbly to + ). 3. Lebesgue s Domnated Convergence Thm: If X n X, and f X n Y for some RV Y 0 wth EY <, then Λ X n dp Λ X dp and Λ X dp Y dp <. In partcular, Λ the sequence of ntegrals converges to a fnte lmt. 4. Fatou s Lemma: If X n 0 on Λ, then Λ (lm nf X n) dp lm nf ( Λ X n dp ). The two sdes may be unequal (example?), and the result s false for lm sup. 5. Fubn s Thm: If ether each X n 0, or n Λ X n dp <, then the order of ntegraton and summaton can be exchanged: n Λ X n dp = Λ n X n dp. If both these condtons fal, the orders may not be exchangeable (example?) 6. For any p > 0, E X p = p x p 1 P[ X > x] dx and E X p < 0 n=1 np 1 P[ X n] <. The case p = 1 s easest and most mportant: f S n=1 P[ X n] <, then S E X < S+1. If X takes on only nonnegatve nteger values, EX = S. 7. If µ X s the dstrbuton of X, and f f s a measurable real-valued functon on R, then Ef(X) = Ω f(x(ω)) dp = R f(x) µ X(dx) f ether sde exsts. In partcular, µ = EX = x µx (dx) and σ 2 = E(X µ) 2 = (x µ) 2 µ X (dx). 8. Hölder s Inequalty: Let p > 1 and q = p p 1 E XY E XY [ E X p] 1 p [ E Y q] 1 q. In partcular, for p = q = 2, Cauchy-Schwartz Inequalty: E XY E XY EX 2 EY 2. (e.g., p = q = 2 or p = 1.01, q = 101). Then 9. Mnkowsk s Inequalty: Let 1 p and let X, Y L p (Ω, F, P). Then Thus the norm X p (E X p ) 1 p (E X + Y p ) 1 p (E X p ) 1 p + (E Y p ) 1 p obeys the trangle nequalty on Lp (Ω, F, P). 10. Jensen s Inequalty: Let ϕ(x) be a convex functon on R, X an ntegrable RV. Then ϕ(e[x]) E[ϕ(X)]. Examples: ϕ(x) = x p, p 1; ϕ(x) = e x ; ϕ(x) = [0 x]. 11. Markov s & Chebychev s Inequaltes: If ϕ s postve and ncreasng, then P[ X u] E[ϕ( X )]/ϕ(u). In partcular P[ X µ > u] σ2 and P[ X > u] σ2 +µ 2. u 2 u 2 σ One-Sded Verson: P[X > u] 2. σ 2 +(u µ) Hoeffdng s Inequalty: If {X j } are ndependent and ( {a j, b j }) s.t. P[a j X j b j ] = 1, then ( c > 0), S n := n j=1 X j satsfes P[S n ES n c] exp ( 2c 2 / n 1 b j a j 2). Hoeffdng proved ths mprovement on Chebychev s nequalty (at UNC) n See also related Azuma s nequalty (1967), Bernsten s nequalty (1937), and Chernoff bounds (1952). Page 8