Expectation, variance and moments John Appleby Contents Expectation and variance Examples 3 Moments and the moment generating function 4 4 Examples of moment generating functions 5 5 Concluding remarks 7 In this lecture, we revisit the notion of the expectation and variance of a random variable, with particular reference to continuous random variables. We also describe how higher order moments of both continuous and discrete random variables can be computed via their moment generating functions. Expectation and variance We remember from our study of discrete random variables that the expectation of a discrete random variable makes precise the notion of the average value of the random variable, while the variance measures the expected spread around the average. However, we cannot compute the expectation of a continuous random variable X using the formula E[X] = X(ω)P[{ω}] = xp[x = x], ω Ω x R X because the summation involved is not defined: the random variable X(ω) takes on more than countably many values. Moreover, as P[X = x] = for all x R for a continuous random variable X, this definition, if taken seriously, always gives a value of zero! So how should one seek to define the expectation of a continuous random variable?
Suppose that the continuous random variable X is supported on (a, b), and has a probability density function f. Let s try to estimate E[X] by cutting [a, b] into n equal subintervals, each of width h, so h = (b a)/n. Define x j = a + jh, j =,,..., n to be the junction points between the subintervals. Then, the probability of X assuming a value on (x j, x j+ ) is xj+ ( ) xj+ + x j p j = P[x j < X < x j+ ] = f(x) dx (x j+ x j )f. x j Since the average value of the random variable on the subinterval (x j, x j+ ) is (x j + x j+ )/, an estimate of the desired expectation is approximately n j= x j + x j+ f ( xj + x j+ ) (x j+ x j ). As n gets ever larger, we see that this converges to the integral a b x=a xf(x) dx. This yields the required definition. Similarly, as the variance is defined as E[X ] E[X], we have ( b b Var[X] = x f(x) dx xf(x) dx). Definition. The expectation of the continuous random variable X with probability density f is E[X] = a xf(x) dx, provided that the integral converges absolutely (i.e., x f(x) dx < ; the variance of X is ( Var[X] = E[X ] E[X] = x f(x) dx xf(x) dx). Examples Example. Find the expectation and variance of the Gaussian random variable X with distribution N (µ, σ ). Solution: By definition, X has density function The expectation therefore is given by E[X] = f(x) = e ( x µ σ ), x R. xf(x) dx = xe ( x µ σ ) dx. Probability I(MS7) c John Appleby,
3 Making the change of variables y = (x µ)/σ, we get E[X] = = π (σy + µ)e y σ dy ye y / dy }{{} =e y / y== +µ The variance is similarly computed: firstly, note that π e y / dy } {{ } = = µ. Var[X] = E[X ] E[X] = E[(X E[X]) ] = E[(X µ) ], so, by writing this as an integral and making the change of variable y = (x µ)/σ again, we get Var[X] = = (x µ) e ( x µ σ Since d dy (e y / ) = ye y /, we have ) dx (σy) e y σ dy = σ y e y dy. π Var[X] = σ π Integrating the last integral by parts gives Var[X] = σ π = σ π y ye y d y dy (e ) dy. y= } {{ } = e y } {{ } = dy = σ. e y dy Exercise.. If X is a random variable with the E(λ) distribution (i.e., an exponential distribution with parameter λ), show that E[X] = λ. Hint: Compute xλe λx dx using integration by parts. In Tutorials, we introduce another continuous distribution: the Cauchy distribution. A Cauchy distribution function is one which has associated probability density f(x) = π + x, x R. Probability I(MS7) c John Appleby,
4 Here we mention an important property of the Cauchy distribution, which also highlights an important facet of the definition of the expectation of a continuous random variable: namely, that the expectation can fail to exist. This happens when x f(x) dx =. To see that this is the case for the Cauchy distribution, as the density is an even function, we have x f(x) dx = = ( x) π π + x dx + x + x dx. Integration by substitution (u = + x ) yields x f(x) dx = x π + x dx π u du = π log u =. u= 3 Moments and the moment generating function We have already seen how to calculate the mean (E[X]), and (in calculating the variance), the mean squared, (E[X ]) of a continuous random variable X. These two statistics give an indication of the location and spread of a distribution. It transpires that one can also determine whether a distribution is skewed to the left or right of its mean µ by calculating its skewness E[(X µ) 3 ], and that one can determine whether the distribution is strongly centred around its mean by calculating its kurtosis, E[(X µ) 4 ]. In each case, one is required to compute the expected value of the random variable raised to an integer power. Such a list of values for a random variable are called its moments. Definition. The moment of order n of a random variable X is E[X n ]. Observe that the definition holds equally well for discrete random variables. We saw above that it was relatively straightforward (but required quite a lot of hard work!) to compute the mean and variance of a Gaussian random variable. Now suppose that it was requested that we calculate all the moments of a Gaussian random variable, or for a random variable that was Poisson distributed! Such a request seems quite a difficult one to respond to. Fortunately, there is a neat gimmick called the moment generating function which streamlines, and makes systematic, such calculations. The following demonstration is not rigorous, and is for motivation only. Let s consider a random variable X for which all moments E[X n ] exist. Recall for any real values t and x, that we can write e tx as a power series, according to e tx = n! (tx)n. Probability I(MS7) c John Appleby,
5 Now, replace x by the random variable X to get e tx = n! tn X n. Now take expectations both sides, and use the linearity of the expectation, to get [ E[e tx ] = E n! tn X n] = E n! E[Xn ]t n. (3.) Notice that we have created or generated all the moments E[X n ] on the right-hand side of (3.) from the single function M X (t) = E[e tx ]. To obtain the moments E[X n ], we just need to remember the following result from the theory of power/taylor series: if M is a function which has arbitrarily many derivatives at zero, then M(t) = n! M (n) ()t n, (3.) where M (n) (t) is the n-th derivative evaluated at t. Comparing coefficients of t n in (3.) and (3.), we see that E[X n ] = dn dt n M X(t). t= This explains why the function M X (t) = E[e tx ] is called the moment generating function of the random variable X. Definition 3. The moment generating function of a random variable X (the m.g.f. of X) is the function M X (t) = E[e tx ] defined for those values of t for which E[e tx ] <. From the above discussion, the following result does not seem implausible. Proposition. If the moment generating function M X (t) exists in some interval containing, then X has finite moments of all orders, and E[X n ] = dn dt n M X(t). t= 4 Examples of moment generating functions We compute some of the moment generating functions of the random variables studied in this course, and then use them to evaluate the desired moments. Example. Let X be an exponentially distributed random variable with parameter λ. Compute its moment generating function, and use it to find the mean and variance of X. Probability I(MS7) c John Appleby,
6 Solution: Recalling that the density function of an E(λ) distribution is f(x) = λe λx for x, and f(x) = otherwise, we have M X (t) = E[e tx ] = e tx f(x) dx = e tx λe λx dx = λ The last integral is well-defined for t < λ: hence ) M X (t) = λ e (λ t)x ( (λ t) = λ x= (λ t) Now, we obtain so Similarly, so M X(t) = λ. (λ t) = λ (λ t), E[X] = M X() = λ. M X(t) = λ (λ t) 3, E[X ] = M X() = λ. = λ λ t. e (λ t)x dx. The variance of X is therefore Var[X] = E[X ] E[X] = λ λ = λ. Example 3. Consider the Poisson random variable X with probability mass function given by P[X = n] = e λ λ n, n =,,.... n! Compute the moment generating function of X, and thereby evaluate E[X], Var[X]. Solution: M X (t) = E[e tx ] = e tn λn e λ n! = e λ (λe t ) n. n! By noticing that the last summation is just exp(λe t ), we obtain M X (t) = e λ(et ). To compute the mean and variance, we use Proposition as before: by the chain rule, we get M X(t) = e λ(et ).λe t = λe t M X (t). Notice that M X () =. Then E[X] = M X () = λe M X () = λ. To calculate the second derivative of M X, we use the product rule to obtain Putting t =, we get M X(t) = M X(t)λe t + M X (t)λe t. E[X ] = M X() = M X()λe + M X ()λe ) = λ + λ, which gives Var[X] = E[X ] E[X] = λ + λ λ = λ. Probability I(MS7) c John Appleby,
7 Example 4. Obtain the moment generating function of the random variable X which has a N (µ, σ ) distribution. Solution: Recalling that the density function of a N (µ, σ ) distribution is f(x) = e ( x µ σ ), x R, we have M X (t) = E[e tx ] = = e tx e ( x µ σ ) dx e σ [(x µ) σ tx ] dx. (4.) We now attempt to complete the square in x of the term inside square brackets in the exponent in (4.): (x µ) σ tx = x µx + µ σ tx = x x(µ + σ t) + µ = ( x (µ + σ t) ) (µ + σ t) + µ = (x µ σ t) µσ t σ 4 t. Replacing this into the expression in (4.) yields M X (t) = e σ [ (x µ σ t) µσ t σ 4 t ] dx = e σ ( µσ t σ 4 t ) e σ (x µ σ t) ) dx = e µt+ σ t σ π e σ (x µ σ t) ) dx = e µt+ σ t. }{{} = Exercise 4.. Use the above to show that E[X] = µ, E[X ] = µ + σ. 5 Concluding remarks We close with some remarks concerning moment generating functions. The moment generating function of a random variable characterises its distribution: that is, if two random variables have the same m.g.f, they must have the same distribution function. For example, if a random variable X has moment generating function M X (t) = e µt+ σ t, it must be a Gaussian distributed random variable with distribution N (µ, σ ). Moment generating functions are most useful for random variables which are either bounded below (i.e., taking values on (a, )), or bounded above (i.e., Probability I(MS7) c John Appleby,
8 taking values on (, b)). For instance, if X is positive M X (t) = E[e tx ] = e tx f(x) dx exists for all t, and M X ( t) is the Laplace transform of the probability density f. If the random variable X is unbounded (that is, it can assume values on (, )), it can happen that its m.g.f does not exist at t. Instead one can make use of the characteristic function C X (t) = E[e itx ]. The characteristic function always exists, and has similar properties to m.g.fs: i n E[X n ] = dn dt n C X(t). t= If X has a density f, C X (t) is closely related to the Fourier transform of f: C X (t) = e itx f(x) dx. For example, a Cauchy-distributed random variable has no moment generating function, but it has characteristic function C(t) = π e itx + x dx = e t. Very often, the characteristic function of continuous random variables can be computed by contour integral and residue theory techniques from Complex Analysis. Probability I(MS7) c John Appleby,