4 Momet Geeratig Fuctios* This sectio is optioal. The momet geeratig fuctio g : R R of a radom variable X is defied as g(t) = E[e tx ]. Propositio 1. We have g () (0) = E[X ] for = 1, 2,... Proof. Therefore g (1) (0) = E[X]. g (1) (t) = de[etx ] dt g (2) (t) = d2 E[e tx ] dt 2 Therefore g (2) (0) = E[X 2 ]. Iductively, we have [ ] de tx = E = E[Xe tx ]. dt [ d 2 e tx ] = E = E[X 2 e tx ]. dt 2 g () (0) = E[X ] for = 1, 2,...,. 1
Example 1. I this example, we will fid the momet geeratig fuctio of X which follows the geometric distributio with the followig p.m.f.: p(1 p) x 1, x = 1, 2,..., ad 0 < p < 1. By defiitio, for (1 p)e t < 1, we have g(t) = E[e tx p ] = 1 p ((1 p)et ) x = x=1 pe t 1 (1 p)e t. Example 2. I this example, we will fid the momet geeratig fuctio of X which follows the expoetial distributio with the followig p.d.f.: By defiitio, for t < λ, we have λe λx, x 0 ad λ > 0. g(t) = E[e tx ] = 0 e tx λe λx dx = λ λ t. Remark 1. The momet geeratig fuctio uiquely determies the distributio. I other words, if two radom variables X ad Y have the same momet geeratig fuctio, the X ad Y have the same distributio. 2
5 Some Fudametal Results The etire sectio, except the cetral limit theorem, is optioal. The followig result is kow as the Markov iequality. Propositio 2. If a radom variable X takes o oly o-egative values, the for ay a > 0, we have P (X a) E[X] a. Proof. We give a proof for the case whe X is a cotiuous radom variable. The case for discrete radom variable is similar ad therefore omitted. Let f(x) be the probability desity fuctio of X. E[X] = = a 0 a a xf(x)dx = xf(x)dx a 0 a xf(x)dx + f(x)dx = ap (X a). a xf(x)dx af(x)dx (because xf(x) af(x), for x a) 3
Usig the Markov iequality, oe ca also prove Chebyshev s iequality. Propositio 3. Let X be a radom variable (which may take egative values). The, for ay a > 0, we have P ( X E[X] a) V ar(x) a 2. Proof. Note that (X E[X]) 2 is a o-egative radom variable with mea E [ (X E[X]) 2] = V ar(x). Now, applyig the Markov iequality, we deduce that, for ay a > 0, P ( X E[X] a) = P ( (X E[X]) 2 a 2) V ar(x) a 2. 4
The followig result is the famous weak Law of Large Numbers. Propositio 4. Let X 1, X 2,..., X,... be a sequece of idepedet ad idetically distributed (i.i.d.) radom variables havig mea µ ad fiite variace σ 2. The for ay ε > 0 we have ( ) lim P X 1 + X 2 +... + X µ > ε = 0. 5
Proof. Let X = 1 (X 1 + X 2 +... + X ). We the have ad E[ X] = E[X 1] + E[X 2 ] +... + E[X ] = µ + µ +... + µ = µ Var( X) = Var(X 1) + Var(X 2 ) +... + Var(X ) = σ2 + σ 2 +... + σ 2 2 2 By Chebyshev s iequality, we have P ( X µ ε) Therefore for ay positive ε we have V ar( X) ε 2 lim P ( X µ > ε) = 0. = σ2 ε 2. = σ2. 6
Here let us also state without proof the strog law of large umbers. Propositio 5. Let X 1, X 2,..., X,... be a sequece of idepedet ad idetically distributed radom variables havig mea µ ad fiite variace σ 2. The we have P ( lim X 1 + X 2 +... + X ) = µ = 1. The strog law of large umbers is a stroger versio of the weak law of large umbers, which meas that the log-ru average of a sequece of idepedet ad idetically distributed radom variables will coverge to its mea. 7
The followig result is the famous cetral limit theorem (C.L.T.), which is of cetral importace i probability theory ad statistics. The statemet of cetral limit theorem is required (but its proof is optioal) i this course ad will be reviewed i greater details later i the lecture otes. Propositio 6. Let X 1, X 2,..., X,... be a sequece of idepedet ad idetically distributed radom variables with mea µ ad variace σ 2. The the cumulative distributio fuctio of the followig radom variable teds to that of the ormal radom variable with mea 0 ad variace 1 as : X 1 + X 2 +... + X µ σ. E Proof. Suppose first that each X i has mea 0 ad variace 1. The the momet geeratig fuctio of (X 1 + X 2 + + X )/ is [ { ( )}] X1 + X 2 + + X [ exp t = E e tx 1/ e tx 2/... e tx / ] = (E[e tx 1/ ]). 8
Now, for large eough, we obtai from the Taylor series expasio of the fuctio e x that e tx 1/ 1 + tx 1 + t2 X1 2 2. Takig expectatio shows that whe is large, E[e tx 1/ ] 1 + te[x] + t2 E[X 2 ] 2 Therefore, we obtai that whe is large, [ { E exp t ( X1 + X 2 + + X )}] = 1 + t2 2. ) (1 + t2 e t2 /2. 2 I other words, the momet geeratig fuctio of (X 1 + +X )/ coverges to e t2 /2, the momet geeratig of fuctio of a ormal radom variable with mea 0 ad variace 1 (stadard ormal radom variable). Cosequetly, its distributio fuctio also coverges to that of a stadard ormal radom variable. Whe each X i has mea µ ad variace σ 2, the radom variable (X i µ)/σ has mea 0 ad 1. Applyig the previous argumet to (X i µ)/σ, we deduce that the distributio of (X 1 µ + X 2 µ + + X µ)/(σ ) coverges to that of a stadard ormal radom variable, which completes the proof. 9
Summary 1. Combiatio 2. Permutatio 3. Sample space 4. Radom variable 5. Coditioal probability ad idepedet evets 6. Probability distributio 7. Uiform distributio 8. Expoetial distributio 9. Normal distributio 10. Expected value E[X] 11. Variace V ar(x) 12. V ar(x) = E[X 2 ] E 2 [X] 10