Math 6070 A Primer on Probability Theory

Transcription

1 Math 67 A Primer o Probability Theory Davar Khoshevisa Uiversity of Utah Sprig 214 Cotets 1 Probabilities 1 2 Distributio Fuctios Discrete Raom Variables Cotiuous Raom Variables Expectatios Momets A (Very) Partial List of Discrete Distributios A (Very) Partial List of Cotiuous Distributios Raom Vectors Distributio Fuctios Expectatios Multivariate Normals Iepeece 9 6 Covergece Criteria Covergece i Distributio Covergece i Probability Momet Geeratig Fuctios Some Examples Properties Characteristic Fuctios Some Examples

2 9 Classical Limit Theorems The Cetral Limit Theorem (Weak) Law of Large Numbers Variace Stabilizatio Refiemets to the CLT Coitioal Expectatios Coitioal Probabilities a Desities Coitioal Expectatios A Ituitive Iterpretatio Probabilities Let F be a collectio of sets. A probability P is a fuctio, o F, that has the followig properties: 1. P( ) = a P(Ω) = 1; 2. If A B the P(A) P(B); 3. (Fiite aitivity). If A a B are isjoit the P(A B) = P(A)+P (B); 4. For all A, B F, P(A B) = P(A) + P(B) P(A B); 5. (Coutable Aitivity). If A 1, A 2,... F are isjoit, the P( i=1 A i) = i=1 P(A i). 2 Distributio Fuctios Let X eote a raom variable. It istributio fuctio is the fuctio F (x) = P{X x}, (1) efie for all real umbers x. It has the followig properties: 1. lim x F (x) = ; 2. lim x F (x) = 1; 3. F is right-cotiuous; i.e., lim x y F (x) = F (y), for all real y; 4. F has left-limits; i.e., F (y ) := lim x y F (x) exists for all real y. I fact, F (y ) = P{X < y}; 5. F is o-ecreasig; i.e., F (x) F (y) wheever x y. It is possible to prove that (1) (5) are always vali for all what raom variables X. There is also a coverse. If F is a fuctio that satisfies (1) (5), the there exists a raom variable X whose istributio fuctio is F. 2

3 2.1 Discrete Raom Variables We will mostly stuy two classes of raom variables: iscrete, a cotiuous. We say that X is a iscrete raom variable if its possible values form a coutable or fiite set. I other wors, X is iscrete if a oly if there exist x 1, x 2,... such that: P{X = x i for some i 1} = 1. I this case, we are itereste i the mass fuctio of X, efie as the fuctio p such that p(x i ) = P{X = x i } (i 1). (2) Implicitly, this meas that p(x) = if x x i for some i. By coutable aitivity, i=1 p(x i) = x p(x) = 1. By coutable aitivity, the istributio fuctio of F ca be compute via the followig: For all x, F (x) = y x p(y). (3) Occasioally, there are several raom variables arou a we ietify the mass fuctio of X by p X to make the structure clear. 2.2 Cotiuous Raom Variables A raom variable is sai to be (absolutely) cotiuous if there exists a oegative fuctio f such that P{X A} = f(x) x for all A. The fuctio A f is sai to be the esity fuctio of X, a has the properties that: 1. f(x) for all x; 2. f(x) x = 1. The istributio fuctio of F ca be compute via the followig: For all x, F (x) = x By the fuametal theorem of calculus, F x f(y) y. (4) = f. (5) Occasioally, there are several raom variables arou a we ietify the esity fuctio of X by f X to make the structure clear. Cotiuous raom variables have the peculiar property that P{X = x} = for all x. Equivaletly, F (x) = F (x ), so that F is cotiuous (ot just rightcotiuous with left-limits). 3

4 3 Expectatios The (mathematical) expectatio of a iscrete raom variable X is efie as EX = x xp(x), (6) where p is the mass fuctio. Of course, this is well efie oly if x x p(x) <. I this case, we say that X is itegrable. Occasioally, EX is also calle the momet, first momet, or the mea of X. Propositio 1. For all fuctios g, Eg(X) = x g(x)p(x), (7) provie that g(x) is itegrable, a/or x g(x) p(x) <. This is ot a trivial result if you rea thigs carefully, which you shoul. Iee, the efiitio of expectatio implies that Eg(X) = y yp{g(x) = y} = y yp g(x) (y). (8) as The (mathematical) expectatio of a cotiuous raom variable X is efie EX = xf(x) x, (9) where f is the esity fuctio. This is well efie whe x f(x) x is fiite. I this case, we say that X is itegrable. Some times, we write E[X] a/or E{X} a/or E(X) i place of EX. Propositio 2. For all fuctios g, Eg(X) = g(x)f(x) x, (1) provie that g(x) is itegrable, a/or g(x) f(x) x <. As was the case i the iscrete settig, this is ot a trivial result if you rea thigs carefully. Iee, the efiitio of expectatio implies that Eg(X) = yf g(x) (y) y. (11) Here is a result that is sometimes useful, a ot so well-kow to stuets of probability: Propositio 3. Let X be a o-egative itegrable raom variable with istributio fuctio F. The, EX = (1 F (x)) x. (12) 4

5 Proof. Let us prove it for cotiuous raom variables. The iscrete case is prove similarly. We have ( ) (1 F (x)) x = P{X > x} x = f(y) y x. (13) Chage the orer of itegratio to fi that ( y ) (1 F (x)) x = x f(y) y = x yf(y) y. (14) Because f(y) = for all y <, this proves the result. It is possible to prove that for all itegrable raom variables X a Y, a for all reals a a b, E[aX + by ] = aex + bey. (15) This justifies the buzz-phrase, expectatio is a liear operatio. 3.1 Momets Note that ay raom variable X is itegrable if a oly if E X <. For all r >, the rth momet of X is E{X r }, provie that the rth absolute momet E{ X r } is fiite. I the iscrete case, E[X r ] = x r p(x), (16) x a i the cotiuous case, E[X r ] = x r f(x) x. (17) Whe it makes sese, we ca cosier egative momets as well. For istace, if X, the E[X r ] makes sese for r < as well, but it may be ifiite. Propositio 4. If r > a X is a o-egative raom variable with E[X r ] <, the E[X r ] = r x r 1 (1 F (x)) x. (18) Proof. Whe r = 1 this is Propositio 3. The proof works similarly. For istace, whe X is cotiuous, ( x ) E[X r ] = x r f(x) x = r y r 1 y f(x) x ( ) (19) = r y r 1 f(x) x y = r y r 1 P{X > y} y. y 5

6 This verifies the propositio i the cotiuous case. A quatity of iterest to us is the variace of X. If is efie as [ VarX = E (X EX) 2], (2) a is equal to VarX = E[X 2 ] (EX) 2. (21) Variace is fiite if a oly if X has two fiite momets. 3.2 A (Very) Partial List of Discrete Distributios You are expecte to be familar with the followig iscrete istributios: 1. Biomial (, p). Here, < p < 1 a = 1, 2,... are fixe, a the mass fuctio is ( ) p(x) = p x (1 p) x if x =,...,. (22) x EX = p a VarX = p(1 p). The biomial (1, p) istributio is also kow as Beroulli (p). 2. Poisso (λ). Here, λ > is fixe, a the mass fuctio is: p(x) = e λ λ x x! EX = λ a VarX = λ. x =, 1, 2,.... (23) 3. Negative biomial (, p). Here, < p < 1 a = 1, 2,... are fixe, a the mass fuctio is: ( ) x 1 p(x) = p (1 p) x x =, + 1,.... (24) 1 EX = /p a VarX = (1 p)/p A (Very) Partial List of Cotiuous Distributios You are expecte to be familar with the followig cotiuous istributios: 1. Uiform (a, b). Here, < a < b < are fixe, a the esity fuctio is f(x) = 1 if a x b. (25) b a EX = (a + b)/2 a VarX = (b a) 2 /12. 6

7 2. Gamma (α, β). Here, α, β > are fixe, a the esity fuctio is f(x) = βα Γ(α) xα 1 e βx < x <. (26) Here, Γ(α) = t α 1 e t t is the (Euler) gamma fuctio. It is efie for all α >, a has the property that Γ(1+α) = αγ(α). Also, Γ(1+) =! for all itegers, whereas Γ(1/2) = π. EX = α/β a VarX = α/β 2. Gamma (1, β) is also kow as Exp (β). [The Expoetial istributio.] Whe 1 is a iteger, Gamma (/2, 1/2) is also kow as χ 2 (). [The chi-square istributio with egrees of freeom.] 3. N(µ, σ 2 ). [The ormal istributio] Here, < µ < a σ > are fixe, a the esity fuctio is: f(x) = 1 σ 2 2π e (x µ) /(2σ 2 ) < x <. (27) EX = µ a VarX = σ 2. N(, 1) is calle the staar ormal istributio. We have the istributioal ietity, µ+σn(, 1) = N(µ, σ 2 ). Equivaletly, N(µ, σ 2 ) µ = N(, 1). (28) σ The istributio fuctio of a N(, 1) is a importat object, a is always eote by Φ. That is, for all < a <, 4 Raom Vectors Φ(a) := 1 2π a e x2 /2 x. (29) Let X 1,..., X be raom variables. The, X := (X 1,..., X ) is a raom vector. 4.1 Distributio Fuctios Let X = (X 1,..., X ) be a N-imesioal raom vector. Its istributio fuctio is efie by F (x 1,..., x ) = P {X 1 x 1,..., X x }, (3) vali for all real umbers x 1,..., x. 7

8 If X 1,..., X are all iscrete, the we say that X is iscrete. O the other ha, we say that X is (absolutely) cotiuous whe there exists a o-egative fuctio f, of variables, such that for all -imesioal sets A, P{X A} = f(x 1,..., x ) x 1... x. (31) A The fuctio f is calle the esity fuctio of X. It is also calle the joit esity fuctio of X 1,..., X. Note, i particular, that x1 x F (x 1,..., x ) = f(u 1,..., u ) u u 1. (32) By the fuametal theorem of calculus, 4.2 Expectatios If g is a real-value fuctio of variables, the Eg(X 1,..., X ) = F x 1 x 2... x = f. (33) g(x 1,..., x )f(x 1,..., x ) x 1... x. (34) A importat special case is whe = 2 a g(x 1, x 2 ) = x 1 x 2. I this case, we obtai E[X 1 X 2 ] = The covariace betwee X 1 a X 2 is efie as It turs out that u 1 u 2 f(u 1, u 2 ) u 1 u 2. (35) Cov(X 1, X 2 ) := E [(X 1 EX 1 ) (X 2 EX 2 )]. (36) Cov(X 1, X 2 ) = E[X 1 X 2 ] E[X 1 ]E[X 2 ]. (37) This is well efie if both X 1 a X 2 have two fiite momets. I this case, the correlatio betwee X 1 a X 2 is ρ(x 1, X 2 ) := Cov(X 1, X 2 ) VarX1 VarX 2, (38) provie that < VarX 1, VarX 2 <. The expectatio of X = (X 1,..., X ) is efie as the vector EX whose jth cooriate is EX j. Give a raom vector X = (X 1,..., X ), its covariace matrix is efie as C = (C ij ) 1 i,j, where C ij := Cov(X i X j ). This makes sese provie that the X i s have two fiite momets. Lemma 5. Every covariace matrix C is positive semi-efiite. That is, x Cx for all x R. Coversely, every positive semi-efiite ( ) matrix is the covariace matrix of some raom vector. 8

9 4.3 Multivariate Normals Let µ = (µ 1,..., µ ) be a -imesioal vector, a C a ( )-imesioal matrix that is positive efiite. The latter meas that x Cx > for all o-zero vectors x = (x 1,..., x ). This implies, for istace, that C is ivertible, a the iverse is also positive efiite. We say that X = (X 1,..., X ) has the multivariate ormal istributio N (µ, C) if the esity fuctio of X is f(x 1,..., x ) = for all x = (x 1,..., x ) R. EX = µ a Cov(X) = C. 1 (2π) /2 et C e 1 2 (x µ) C 1 (x µ), (39) X N (µ, C) if a oly if there exists a positive efiite matrix A, a i.i.. staar ormals Z 1,..., Z such that X = µ + AZ. I aitio, AA = C. Whe = 2, a multivariate ormal is calle a bivariate ormal. Warig. Suppose X a Y are each ormally istribute. The it is ot true i geeral that (X, Y ) is bivariate ormal. A similar caveat hols for the -imesioal case. 5 Iepeece Raom variables X 1,..., X are (statistically) iepeet if P {X 1 A 1,..., X A } = P {X 1 A 1 } P {X A }, (4) for all oe-imesioal sets A 1,..., A. It ca be show that X 1,..., X are iepeet if a oly if for all real umbers x 1,..., x, P {X 1 x 1,..., X x } = P {X 1 x 1 } P {X x }. (41) That is, the cooriates of X = (X 1,..., X ) are iepeet if a oly if F X (x 1,..., x ) = F X1 (x 1 ) F X (x ). Aother equivalet formulatio of iepeece is this: For all fuctios g 1,..., g such that g i (X i ) is itegrable, E [g(x 1 )... g(x )] = E[g 1 (X 1 )] E[g (X )]. (42) A reay cosequece is this: If X 1 a X 2 are iepeet, the they are ucorrelate provie that their correlatio exists. Ucorrelate meas that ρ(x 1, X 2 ) =. This is equivalet to Cov(X 1, X 2 ) =. If X 1,..., X are (pairwise) ucorrelate with two fiite momets, the Var(X X ) = VarX VarX. (43) 9

10 Sigificatly, this is true whe the X i s are iepeet. I geeral, the formula is messier: ( ) Var X i = VarX i + 2 Cov(X i, X j ). (44) i=1 i=1 1 i<j I geeral, ucorrelate raom variables are ot iepeet. A exceptio is mae for multivariate ormals. Theorem 6. Suppose (X, Y ) N +k (µ, C), where X a Y are respectively -imesioal a k-imesioal raom vectors. The: 1. X is multivariate ormal. 2. Y is multivariate ormal. 3. If EX i Y j = for all i, j, the X a Y are iepeet. For example, suppose (X, Y ) is bivariate ormal. The, X a Y are ormally istribute. If, i aitio, Cov(X, Y ) = the X a Y are iepeet. 6 Covergece Criteria Let X 1, X 2,... be a coutably-ifiite sequece of raom variables. There are several ways to make sese of the statemet that X X for a raom variable X. We ee a few of these criteria. 6.1 Covergece i Distributio We say that X coverges to X i istributio if F X (x) F X (x), (45) for all x R at which F X is cotiuous. We write this as X X. Very ofte, F X is cotiuous. I such cases, X X if a oly if FX (x) F X (x) for all x. Note that if X X a X has a cotiuous istributio the also P{a X b} P{a X b}, (46) for all a < b. Similarly, we say that the raom vectors X 1, X 2,... coverge i istributio to the raom vector X whe F X (a) F X (a) for all a at which F X is cotiuous. This covergece is also eote by X X. 1

11 6.2 Covergece i Probability We say that X coverges to X i probability if for all ɛ >, P { X X > ɛ}. (47) We eote this by X P X. It is the case that if X P X the X X, but the coverse is patetly false. There is oe exceptio to this rule. Lemma 7. Suppose X c where c is a o-raom costat. The, X P c. Proof. Fix ɛ >. The, P{ X c ɛ} P{c ɛ < X c + ɛ} = F X (c + ɛ) F X (c ɛ). (48) But F c (x) = if x < c, a F c (x) = 1 if x c. Therefore, F c is cotiuous at c ± ɛ, whece we have F X (c + ɛ) F X (c ɛ) F c (c + ɛ) F c (c ɛ) = 1. This proves that P{ X c ɛ} 1, which is aother way to write the lemma. Similar cosieratios lea us to the followig. Theorem 8 (Slutsky s theorem). Suppose X X a Y c for a costat c. If g is a cotiuous fuctio of two variables, the g(x, Y ) g(x, c). [For istace, try g(x, y) = ax + by, g(x, y) = xye x, etc.] Whe c is a raom variable this is o loger vali i geeral. 7 Momet Geeratig Fuctios We say that X has a momet geeratig fuctio if there exists t > such that M(t) := M X (t) = E[e tx ] is fiite for all t [ t, t ]. (49) If this coitio is met, the M is the momet geeratig fuctio of X. If a whe it exists, the momet geeratig fuctio of X etermies its etire istributio. Here is a more precise statemet. Theorem 9 (Uiqueess). Suppose X a Y have momet geeratig fuctios, a M X (t) = M Y (t) for all t sufficietly close to. The, X a Y have the same istributio. 7.1 Some Examples 1. Biomial (, p). The, M(t) exists for all < t <, a M(t) = ( 1 p + pe t). (5) 11

12 2. Poisso (λ). The, M(t) exists for all < t <, a M(t) = e λ(et 1). (51) 3. Negative Biomial (, p). The, M(t) exists if a oly if < t < log(1 p). I that case, we have also that ( pe t ) M(t) = 1 (1 p)e t. (52) 4. Uiform (a, b). The, M(t) exists for all < t <, a M(t) = etb e ta t(b a). (53) 5. Gamma (α, β). The, M(t) exists if a oly if < t < β. I that case, we have also that ( ) α β M(t) =. (54) β t Set α = 1 to fi the momet geeratig fuctio of a expoetial (β). Set α = /2 a β = 1/2 for a positive iteger to obtai the momet geeratig fuctio of a chi-square (). 6. N(µ, σ 2 ). The momet geeratig fuctio exists for all < t <. Moreover, M(t) = exp (µt + σ2 t 2 ). (55) Properties Besie the uiqueess theorem, momet geeratig fuctios have two more properties that are of iterest i mathematical statistics. Theorem 1 (Covergece Theorem). Suppose X 1, X 2,... is a sequece of raom variables whose momet geeratig fuctios all exists i a iterval [ t, t ] arou the origi. Suppose also that for all t [ t, t ], M X (t) M X (t) as, where M is the momet geeratig fuctio of a raom variable X. The, X X. Example 11 (Law of Rare Evets). Let X have the Bi(, λ/) istributio, where λ > is iepeet of. The, for all < t <, ( M X (t) = 1 λ + λ ) et. (56) 12

13 We claim that for all real umbers c, ( 1 + c ) e c as. (57) Let us take this for grate for the time beig. The, it follows at oce that That is, M X (t) exp ( λ + λe t) = e λ(et 1). (58) Bi (, λ/) Poisso (λ). (59) This is Poisso s law of rare evets (also kow as the law of small umbers ). Now we wrap up this example by verifyig (57). Let f(x) = (1 + x), a Taylor-expa it to fi that f(x) = 1 + x ( 1)x2 +. Replace x by c/, a compute to fi that ( 1 + c ) ( 1)c 2 = 1 + c j= c j j!, (6) a this is the Taylor-series expasio of e c. [There is a little bit more oe has to o to justify the limitig proceure.] The seco property of momet geeratig fuctios is that if a whe it exists for a raom variable X, the all momets of X exist, a ca be compute from M X. Theorem 12 (Momet-Geeratig Property). Suppose X has a fiite momet geeratig fuctio i a eighborhoo of the origi. The, E( X ) exists for all, a M () () = E[X ], where f () (x) eotes the th erivative of fuctio f at x. Example 13. Let X be a N(µ, 1) raom variable. The we kow that M(t) = exp(µt t2 ). Cosequetly, M (t) = (µ + t)e µt+(t2 /2), a M (t) = [ 1 + (µ + t) 2] e µt+(t2 /2) (61) Set t = to fi that EX = M () = µ a E[X 2 ] = M () = 1 + µ 2, so that VarX = E[X 2 ] (EX) 2 = 1. 8 Characteristic Fuctios The characteristic fuctio of a raom variable X is the fuctio φ(t) := E [ e itx] < t <. (62) Here, the i refers to the complex uit, i = 1. We may write φ as φ X, for example, whe there are several raom variables arou. 13

14 I practice, you ofte treat e itx as if it were a real expoetial. However, the correct way to thik of this efiitio is via the Euler formula, e iθ = cos θ+i si θ for all real umbers θ. Thus, φ(t) = E[cos(tX)] + ie[si(tx)]. (63) If X has a momet geeratig fuctio M, the it ca be show that M(it) = φ(t). [This uses the techique of aalytic cotiuatio from complex aalysis.] I other wors, the aive replacemet of t by it oes what oe may guess it woul. However, oe avatage of workig with φ is that it is always wellefie. The reaso is that cos(tx) 1 a si(tx) 1, so that the expectatios i (63) exist. I aitio to havig this avatage, φ shares most of the properties of M as well! For example, Theorem 14. The followig hol: 1. (Uiqueess Theorem) Suppose there exists t > such that for all t ( t, t ), φ X (t) = φ Y (t). The X a Y have the same istributio. 2. (Covergece Theorem) If φ X (t) φ X (t) for all t ( t, t ), the X X. Coversely, if X X, the φx (t) φ X (t) for all t. 8.1 Some Examples 1. Biomial (, p). The, φ(t) = M(it) = ( 1 p + pe it). (64) 2. Poisso (λ). The, φ(t) = M(it) = e λ(eit 1). (65) 3. Negative Biomial (, p). The, ( pe it ) φ(t) = M(it) = 1 (1 p)e it. (66) 4. Uiform (a, b). The, 5. Gamma (α, β). The, φ(t) = M(it) = eitb e ita t(b a). (67) φ(t) = M(it) = ( ) α β. (68) β it 6. N(µ, σ 2 ). The, because (it) 2 = t 2, φ(t) = M(it) = exp (iµt σ2 t 2 ). (69) 2 14

15 9 Classical Limit Theorems 9.1 The Cetral Limit Theorem Theorem 15 (The CLT). Let X 1, X 2,... be i.i.. raom variables with two fiite momets. Let µ := EX 1 a σ 2 = VarX 1. The, j=1 X j µ σ N(, 1). (7) Elemetary probability texts prove this by appealig to the covergece theorem for momet geeratig fuctios. This approach oes ot work if we kow oly that X 1 has two fiite momets, however. However, by usig characteristic fuctios, we ca relax the assumptios to the fiite mea a variace case, as state. Proof of the CLT. Defie j=1 T := X j µ σ. (71) The, φ T (t) = E = j=1 E j=1 ( exp it [ exp ( it ( )) Xj µ σ ( ))] Xj µ σ, (72) thaks to iepeece; see (42) o page 8. Let Y j := (X j µ)/σ eote the staarizatio of X j. The, it follows that φ T (t) = ( ) [ ( )] φ Yj t/ = φy1 t/, (73) j=1 because the Y j s are i.i.. Recall the Taylor expasio, e ix = 1 + ix 1 2 x2 +, a write φ Y1 (s) as E[e ity1 ] = 1 + itey t2 E[Y 2 1 ] + = t2 +. Thus, φ T (t) = ] [1 t2 2 + e t2 /2. (74) See (57) o page 12. Because e t2 /2 is the characteristic fuctio of N(, 1), this a the covergece theorem (Theorem 15 o page 14) together prove the CLT. The CLT has a multiimesioal couterpart as well. Here is the statemet. 15

16 Theorem 16. Let X 1, X 2,... be i.i.. k-imesioal raom vectors with mea vector µ := EX 1 a covariace matrix Q := CovX. If Q is o-sigular, the j=1 X j µ Nk (, Q). (75) 9.2 (Weak) Law of Large Numbers Theorem 17 (Law of Large Numbers). Suppose X 1, X 2,... are i.i.. a have a fiite first momet. Let µ := EX 1. The, j=1 X j P µ. (76) Proof. We will prove this i case there is also a fiite variace. The geeral case is beyo the scope of these otes. Thaks to the CLT (Theorem 15, page 14), (X X )/ coverges i istributio to µ. Slutsky s theorem (Theorem 8, page 1) proves that covergece hols also i probability. 9.3 Variace Stabilizatio Let X 1, X 2,... be i.i.. with µ = EX 1 a σ 2 = VarX 1 both efie a fiite. Defie the partial sums, S := X X. (77) We kow that: (i) S µ i probability; a (ii) (S µ) N(, σ 2 ). Now use Taylor expasios: For ay smooth fuctio h, ( ) S h(s /) h(µ) + µ h (µ), (78) i probability. By the CLT, (S /) µ N(, σ 2 /). Therefore, Slutsky s theorem (Theorem 8, page 1) proves that [ ( ) ] S h h(µ) N (, σ 2 h (µ) 2). (79) [Techical coitios: h shoul be cotiuously-ifferetiable i a eighborhoo of µ.] 9.4 Refiemets to the CLT There are may refiemets to the CLT. Here are 2 particularly well-kow oes. The first gives a escriptio of the farthest the istributio fuctio of ormalize sums is from the ormal. 16

17 Theorem 18 (Berry Essee). If ρ := E{ X 1 3 } <, the { max <a< P i=1 X } i µ σ a Φ(a) 3ρ σ 3. (8) The seco is a oe-term example of a family is results that are calle Egeworth expasios. Theorem 19 (Egeworth). Suppose E exp(itx 1) t < a E( X 1 ρ ) < for some ρ > 3, the we ca write where: { i=1 P X } i µ σ a = Φ(a) + κ 1(1 a 2 ) 6 φ(a) + R (a), 1. φ(a) := (2π) 1/2 exp( a 2 /2) eotes the staar ormal esity; 2. κ 1 := σ 3 E[(X 1 µ) 3 ] eotes the skewess of the istributio of X 1 ; 3. max <a< R (a) cost 1. Remark 2. The coitio E exp(itx 1) t < hols roughly whe X 1 has a ice pf. Remark 21. Uer further restrictios, oe ca i fact write a asymptotic expasio of the form { i=1 P X } i µ σ a = Φ(a) + r j=1 κ j H j (a) φ(a) + R j/2,r (a), for every [fixe] positive iteger r, where κ j s are fiite costats, each H j is a certai polyomial of egree j [Hermite polyomials], a the remaier is very small i the sese that max <a< R,r (a) cost (r+1)/2. 1 Coitioal Expectatios Let us begi by recallig some basic otios of coitioig from elemetary probability. Throughout this sectio, X eotes a raom variable a Y := (Y 1,..., Y ) a -imesioal raom vector. 1.1 Coitioal Probabilities a Desities If X, Y 1,..., Y are all iscrete raom variables, the the coitioal mass fuctio of X, give that Y = y, is p X Y (x y) := P{X = x, Y 1 = y 1,..., Y = y }, (81) P{Y 1 = y 1,..., Y = y } 17

18 provie that P{Y = y} >. This is a boa fie mass fuctio [as a fuctio of the variable x] for every fixe choice of y. [It oes t make sese to worry about its behavior i the variables y 1,..., y.] Similarly, if the istributio of (X, Y 1,..., Y ) is absolutely cotiuous, the the coitioal esity fuctio of X, give that Y = y, is f X Y (x y) := f X,Y (x, y 1,..., y ), (82) f Y (y 1,..., y ) provie that the observe value y is such that the joit esity f X,Y raom vector (X, Y ) satisfies of the f Y (y 1,..., y ) >. (83) Note that (83) is etirely possible, though P{Y = y} = simply because Y has a absolutely cotiuous istributio. Coitio (83) is quite atural i the followig sese: Let B eote the collectio of all -imesioal vectors y such that f Y (y 1,..., y ) =. The, P{Y B} = f Y (y 1,..., y ) y 1 y =. (84) B I other wors, we o ot have to worry about efiig f X Y (x y) whe y is ot i B. 1.2 Coitioal Expectatios If we have observe that Y = y, for a kow vector y = (y 1,..., y ), the the best liear preictor of X is the [classical] coitioal expectatio E(X Y = y) := { x xp{x = x Y = y} if (X, Y ) is iscrete, xf X Y (x y) x if (X, Y ) has a joit pf. (85) The preceig assumes tacitly that the sum/itegral coverges absolutely. More geerally, we have for ay ice fuctio ϕ, E(ϕ(X) Y = y) := { x ϕ(x)p{x = x Y = y} if iscrete, ϕ(x)f X Y (x y) x if joit pf exists, (86) provie that the sum/itegral coverges absolutely. The preceig is i fact a theorem, but a careful statemet requires writig too may techical etails from itegratio theory. 1.3 A Ituitive Iterpretatio The basic use of coitioal expectatios is this: If we observe that Y = y, the we preict X, base oly o our observatio that Y = y, as E(X Y = y). 18

19 Example 22. We perform 1 iepeet Beroulli trials [p := probability of success per trial]. Let X eote the total umber of successes. We kow that X has a Bi(1, p) istributio. If Y := the total umber of successes i the first 5 trials, the you shoul check that E(X Y = ) = 5p. More geerally, E(X Y = y) = y + 5p for all y {,..., 5}. The previous example shows you that it is frequetly more coveiet to use a slightly ifferet form of coitioal expectatios: We write E(X Y ) for the raom variable whose value is E(X Y = y) whe we observe that Y = y. I the previous example, this efiitio traslates to the followig computatio: E(X Y ) = Y + 5p. This ought to make very goo sese to you, before you rea o! The classical Bayes formula for coitioal probabilities has a aalogue for coitioal expectatios. Suppose (X, Y ) has a joit esity fuctio f X,Y. The, E(X) = xf X (x) x ( ) = x f X,Y (x, y 1,..., y ) y x R ( ) = x f X Y (x y)f Y (y) y x R ( ) = xf X Y (x y) x f Y (y) y R = E(X Y = y)f Y (y) y R = E {E(X Y )}. (87) This is always true. That is, we always have provie that E X <. E(X) = E {E(X Y )}, (88) 19