Moment Generating Functions

Transcription

1 MATH 382 Moment Generating Functions Dr. Neal, WKU Definition. Let X be a random variable. The moment generating function (mgf) of X is the function M X : R R given by M X (t ) = E[e X t ], defined for all t for which this expected value is finite. When X is a discrete random variable, then M X (t ) is obtained by M X (t ) = e k t P( X = k ), and when X is a continuous random variable, then M X (t ) is obtained by M X (t ) = e x t f X ( x) dx = e x t f X (x ) dx. We shall derive the mgf s of our standard random variables, derive properties of the mgf function, and then show how to use mgf s to derive the distribution of various sums of independent random variables. The Binomial MGF n n Let X ~ b(n, p). By the binomial expansion theorem, k a k b n k = (a + b) n ; thus, k = 0 M X (t ) = e k t n P( X = k ) = e k t n p k k q n k k = 0 n n = ( pe k t ) k q n k = ( pe t + q) n, for all t. k = 0 A special case is the Bernoulli random variable X ~ b(, p) for which M X (t ) = pe t + q. The Geometric MGF Let X ~ geo( p) with p > 0. Then 0 q < and because x k = x / ( x) for < x <, k = we have M X (t ) = e k t P( X = k ) = e k t q k p = p (qe t ) k k = q k = = p q q et qe t = p et qe t, for qet < ; i. e., for t < ln( / q).

2 The Poisson MGF Let X ~ Poi(λ ). Because xk k = 0 k! = e x for all x, we have M X (t ) = e k t P( X = k ) = e k t λ k e λ k = 0 k! = e λ (λ et ) k k = 0 k! = e λ e λ et = e λ (et ), for all t. The Uniform MGF Let X ~ U[a, b]. Then M X (t ) = e x t b f X (t ) = e x t b a dx = a = ebt e at t( b a), for t 0. b a t ex t b a The Exponential MGF Let X ~ exp(θ ) with θ > 0. We shall assume that θ t < 0 so that the improper integral below will converge. We then have M X (t ) = e x t f X (t ) = e x t θ e x /θ dx 0 = θ 0 e x (t /θ) dx = θ 0 = θ θ θ t ex (θ t )/θ 0 Let X ~ N(µ, σ ). Then f X (x ) = e x (θ t )/θ dx = 0 The Normal MGF θ t = θ t, for t < / θ. σ 2π e (x µ )2 / (2σ 2 ). Now let Y ~ N(µ + σ, σ ) with f Y (x ) = σ 2π e ( x (µ+σ )) 2 / (2σ 2 ) = σ 2π e ( x2 2x(µ+σ )+(µ +σ ) 2 ) /(2σ 2 ) = σ 2π e ( x2 2x(µ+σ ) + µ 2 +2µσ +σ 4 t 2 ) / (2σ 2 ). = σ 2π e ( x2 2x(µ+σ ) + µ 2 ) /(2σ 2 ) e µ t e σ 2 /2.

3 Then because f Y ( x) dx =, we have M X (t ) = e x t f X (t ) = e x t e (x µ )2 / (2σ 2 ) dx σ 2π = e x t ( x2 2 x µ + µ 2 ) /(2σ 2 ) dx = e (x 2 2 x µ 2σ 2 x t + µ 2 ) /(2σ 2 ) dx = e (x 2 2 x (µ +σ )+ µ 2 )/ (2σ 2 ) dx = e µ t e σ 2 /2 e ( x2 2 x (µ +σ ) + µ 2 ) /(2σ 2 ) e µ t e σ 2 /2 dx = e µ t e σ 2 /2 f Y ( x) dx = e µ t e σ 2 /2. Dr. Neal, WKU Thus, M X (t ) = e µ t e σ 2 /2 for all t. A special case is the standard normal distribution Z ~ N(0, ). M Z (t) = e t 2 /2 for all t. We then have Generating Moments Let X be a random variable. The first moment of X is simply the expected value E[ X ]. In general for j, the j th moment of X is E[ X j ], which can be found by E[ X j ] = x j P( X = k) E[ X j ] = x j f X (x )dx for X discrete for X continuous. But the mgf of X also can be used to obtain these moments. When X is discrete, then M X (t ) = e k t P( X = k ), and then the j th derivative of M X (t ) with respect to t is ( j) M X (t) = k j e k t P( X = k ). ( j) Then M X (0) = k j P( X = k) = E[ X j ].

4 Similarly, for X continuous, M X (t ) = M X (t ) with respect to t is M X ( j) (t) = M X ( j) (0) = e x t f X ( x) dx. Then the j th derivative of x j e x t f X ( x) dx, and thus x j f X ( x) dx = E[X j ]. Thus, the j th moment of X can be found by computing the j th derivative of M X (t ) and then evaluating it at t = 0. In particular, E[ X ] = M ʹ X (0) E[ X 2 ] = M X ʹ (0) Var( X) = M ʹ X ʹ (0) ( M ʹ X (0)) 2 Example. Let X ~ b(n, p) with M X (t ) = ( pe t + q) n, for all t. Then the first and second derivatives of M X (t ) are M X ʹ (t ) = n( pe t + q) n pe t and M X ʹ (t ) = n(n )( pe t + q) n 2 ( pe t ) 2 + n( pe t + q) n pe t. Thus, E[ X ] = and E[ X 2 ] = M ʹ X (0) = n( p + q) n p = n p M X ʹ (0) = n(n )p 2 + np Thus, Var( X) = n(n ) p 2 + np (np) 2 = np np 2 = np( p) = n pq. Properties of MGF s (i) M X +Y (t ) = M X (t ) M Y (t ) for independent random variables X and Y. (ii) M ax (t) = M X (a t ) for all constants a (iii) M ( X+c ) (t) = e c t M X ( t) for all constants c Proof. (i) Let X and Y be independent random variables. Then M X +Y (t ) = E[e (X+Y)t ] = E[e Xt+Yt ] = E[e Xt e Yt ] = By Indep. E[eXt ] E[e Yt ] = M X (t) M Y (t). (ii) For all constants a, we have M ax (t) = E[e (ax)t ] = E[e X(at ) ] = M X (a t ). (iii) For all constants c, we have M ( X+c ) (t) = E[e (X +c)t ] = E[e Xt e ct ] = e c t E[e Xt ] = e c t M X ( t ).

5 The following result, stated without proof, is perhaps the most important property of mgf s, and can used to determine the distribution of independent sums of known distributions. Theorem. The moment generating function completely determines the distribution of a random variable. That is, if two random variables have the same mgf, then they have the same distribution. Example 2. Let X ~ N(µ, σ ) and let Y = X µ. We have seen previously that Y is now σ a N(0, ) distribution. We can prove this fact by using the properties of mgf's and the preceding theorem. The mgf of X is M X (t ) = e µ t e σ 2 /2. Thus the mgf of Y = σ X µ σ is given by M Y (t ) = M σ X µ (t ) = e ( µ /σ)t M σ σ X (t) = e ( µ /σ)t M X (t / σ ) = e ( µ /σ)t e µ t/σ e σ2 (t 2 /σ 2 )/2 = e t 2 /2. which is the mgf of the N(0, ) distribution. Thus, Y itself must be N(0, ) because it has the same mgf as N(0, ). Example 3. (Sum of Independent Binomial Distributions) Let X ~ b(n, p) and Y ~ b(m, p), with X and Y being independent. Derive the distribution of X + Y. Solution. We know M X (t ) = ( pe t + q) n and M Y (t ) = ( pe t + q) m for all t. Then by independence, the mgf of X + Y is M X +Y (t ) = M X (t ) M Y (t ) = ( pe t + q) n ( pe t + q) m = ( pe t + q) n+m, which is the mgf of a b(n + m, p) distribution. Thus, X + Y ~ b(n + m, p) because it has the same mgf as b(n + m, p). Note: For both X and Y, the probability of success on any attempt is p. X counts the number of successes in n independent attempts, and Y counts the number of successes in m further independent attempts. So X + Y counts the total number of successes in n + m independent attempts; hence, X + Y ~ b(n + m, p).

6 Example 4. (Sum of Independent Normal Distributions) Let X ~ N(µ,σ ) and Y ~ N(µ 2,σ 2 ), with X and Y being independent. Derive the distribution of X + Y. Solution. We know M X (t ) = e µ t e σ 2 /2 and MY (t ) = e µ e σ 2 2 /2 for all t. Then by independence, the mgf of X + Y is M X +Y (t ) = M X (t ) M Y (t ) = e µ t e σ 2 /2 e µ e σ 2 2 /2 = e (µ +µ 2 )t e (σ 2 +σ 2 2 )t 2 /2, which is the mgf of a Normal distribution having mean µ = µ + µ 2 and variance σ 2 = σ 2 + σ2 2. Because the mgf completely determines the distribution, we must have X + Y ~ N(µ, σ ) = N µ + µ 2, σ 2 + σ2 2 ( ). Distribution of the Sample Mean of Independent Sample Measurements from a N(µ, σ ) Distribution Let x = x x n be the sample mean where x n i ~ N (µ,σ ) is the i th independent sample of a Normally distributed measurement. Then each x i has the same mgf given by M i (t) = e µ t e σ 2 /2. By independence, the mgf of the sum S = x x n is the product of the mgf's: M S (t ) = e µ t e σ 2 /2 n = e nµ t e nσ 2 /2. Then the mgf of x is M x (t) = M n S(t ) = M S (t / n) = enµ t/n e nσ 2 (t/n) 2 /2 = e µ t e (σ 2 /n) t 2 /2, which is the mgf of a Normal distribution having mean µ and variance σ 2 / n. Because the mgf completely determines the distribution, we must have x ~ N µ, σ. n Note: When sampling with a normally distributed measurement, the sample mean x is still normally distributed regardless of the sample size. And we have the following: µ x = µ The average of all possible sample means from samples of size n equals the original average µ. σ x = σ n The standard deviation of all possible sample means from samples of size n is a fraction of the original standard deviation σ. Important Statistical Consequence: As the sample size n increases, then σ x decreases to 0. Thus with large sample sizes, the values of x have very small deviation and therefore the possible values of x are consistently close to their average µ. Thus, with a large sample size, virtually any sample mean x will be a decent approximation of µ.

7 Exercises. Use the moment generating function to derive the mean and variance of X ~ exp(θ ). 2. Let X ~ Poi(λ ) and Y ~ Poi (β ), with X and Y being independent. Derive the distribution of X + Y. 3. The negative binomial distribution X ~ nb(r, p) is the sum of r independent geometric distributions: X = X + X X r, where each X i ~ geo( p). (a) Use the mgf's of the X i to derive the mgf of X ~ nb(r, p). (b) Use the mgf to derive the expected value of X ~ nb(r, p).