Econ 508B: Lecture 5

Econ 508B: Lecture 5 Expectation, MGF and CGF Hongyi Liu Washington University in St. Louis July 31, 2017 Hongyi Liu (Washington University in St. Louis) Math Camp 2017 Stats July 31, 2017 1 / 23

Outline 1 Expected Values 2 Moment Generating Functions 3 Cumulative Generating Functions Hongyi Liu (Washington University in St. Louis) Math Camp 2017 Stats July 31, 2017 2 / 23

Outline 1 Expected Values 2 Moment Generating Functions 3 Cumulative Generating Functions Hongyi Liu (Washington University in St. Louis) Math Camp 2017 Stats July 31, 2017 3 / 23

Motivation:Probability v.s. Expectation To start with, people probably have a better understanding for an expected value than for probability. Like optimization and approximation problems, they are phrased in terms of expectations. Expectations are indeed seen as special cases and are treated with uniformity and economy. Hongyi Liu (Washington University in St. Louis) Math Camp 2017 Stats July 31, 2017 4 / 23

Definition 1.1 Let X be a random variable on (Ω, F, P ).The expected value of X, EX, is defined as EX = XdP, given the integral is well-defined, i.e., at least one of the two quantities X + dp and X dp is finite. Ω Proposition 1.1 (Change of variable formula) If X is a random variable on (Ω, F, P ) and g : R R is Borel measurable and Y = g(x) is also a random variable on (Ω, F, P ). Ω Y dp = R g(x) P X(dx) = R y P Y (dy). If Ω Y dp <, then Y dp = h(x)p X (dx) = yp Y (dy). Ω R R Hongyi Liu (Washington University in St. Louis) Math Camp 2017 Stats July 31, 2017 5 / 23

Moment Definition 1.2 For any positive integer n, the n th moment µ n and the n th central moment µ n of a random variable X is defined by µ n EX n, µ n E(X EX) n provided the expectation is well-defined. In particular, the variance of a random variable X is the 2 th central moment, namely V ar(x) = E(X EX) 2, provided EX 2 <. Hongyi Liu (Washington University in St. Louis) Math Camp 2017 Stats July 31, 2017 6 / 23

Outline 1 Expected Values 2 Moment Generating Functions 3 Cumulative Generating Functions Hongyi Liu (Washington University in St. Louis) Math Camp 2017 Stats July 31, 2017 7 / 23

MGF Definition 2.1 The moment generating function (MGF) of a random variable X is M X (t) E(e tx ), for all t R e tx is always non-negative, therefore, E(e tx ) is well-defined but could be infinity (Why?). Hongyi Liu (Washington University in St. Louis) Math Camp 2017 Stats July 31, 2017 8 / 23

MGF Definition 2.1 The moment generating function (MGF) of a random variable X is M X (t) E(e tx ), for all t R e tx is always non-negative, therefore, E(e tx ) is well-defined but could be infinity (Why?). The payoff of MGF is that it gives the direct connection between MGF and the moments of a random variable X as follows. Hongyi Liu (Washington University in St. Louis) Math Camp 2017 Stats July 31, 2017 8 / 23

non-negative case Proposition 2.1 Let X be a non-negative random variable t > 0. Then M X (t) E(e tx ) = Proof: By Taylor expansion, e tx = this comes from M.C.T. n=0 n=0 tn X n t n µ n n! n! and X is non-negative, Hongyi Liu (Washington University in St. Louis) Math Camp 2017 Stats July 31, 2017 9 / 23

bounded case Proposition 2.2 Let X be a random variable and let M X (t) be finite for all t < ɛ, for some ɛ > 0, then (1) E X n < for all n 1, (2) M X (t) = µn n=0 tn n! for all t < ɛ, (3) M X ( )is infinitely differentiable on ( ɛ, +ɛ) and for r N, the r th derivative of M X ( ) is M (r) X (t) = n=0 µ n+r t n n! = E(etX X r )for t < ɛ. In particular, M (r) X (0) = µ r = EX r Hongyi Liu (Washington University in St. Louis) Math Camp 2017 Stats July 31, 2017 10 / 23

Proof (1) : According to M X (t) is finite and the fact that t n X n n! e tx for all n N, then E(e tx ) E(e tx ) + E(e tx ) < for t < ɛ Therefore, choosing a t ( ɛ, +ɛ) leads to the outcome of (1). Hongyi Liu (Washington University in St. Louis) Math Camp 2017 Stats July 31, 2017 11 / 23

Proof (1) : According to M X (t) is finite and the fact that t n X n n! e tx for all n N, then E(e tx ) E(e tx ) + E(e tx ) < for t < ɛ Therefore, choosing a t ( ɛ, +ɛ) leads to the outcome of (1). (2) : Notice that n (tx) j j=0 j! e tx for all x R and n N, then D.C.T. implies (2) holds. Hongyi Liu (Washington University in St. Louis) Math Camp 2017 Stats July 31, 2017 11 / 23

Proof (1) : According to M X (t) is finite and the fact that t n X n n! e tx for all n N, then E(e tx ) E(e tx ) + E(e tx ) < for t < ɛ Therefore, choosing a t ( ɛ, +ɛ) leads to the outcome of (1). (2) : Notice that n (tx) j j=0 j! e tx for all x R and n N, then D.C.T. implies (2) holds. (3) : The derivative of M X ( ) can be found by term-by-term differentiation of the power series. Hence, M (r) dr X (t) = = dt r ( n=0 n=0 t n µ n n! ) = n=0 n=0 d r (t n ) µ n dt r n! t n r µ n (n r)! = t n µ n+r n! ongyi Liu (Washington University in St. Louis) Math Camp 2017 Stats July 31, 2017 11 / 23

Example 2.1 Let X N(0, 1), then for all t R, Thus µ n = M X (t) = + e tx 1 2π e x2 /2 dx = e t2 /2 = { 0 if n is odd (2k)! k!2 k if n = 2k, k = 1, 2,... (t 2 ) k 1 k! 2 k. k=0 Hongyi Liu (Washington University in St. Louis) Math Camp 2017 Stats July 31, 2017 12 / 23

Example 2.1 Let X N(0, 1), then for all t R, Thus µ n = M X (t) = Remark 2.1 + e tx 1 2π e x2 /2 dx = e t2 /2 = { 0 if n is odd (2k)! k!2 k if n = 2k, k = 1, 2,... (t 2 ) k 1 k! 2 k. k=0 If M X (t) finite within a finite circle is fulfilled, then all the moments {µ n } n 1 of X are determined and its probability distribution as well. However, in general, probability distributions are not completely determined by their moments. Hongyi Liu (Washington University in St. Louis) Math Camp 2017 Stats July 31, 2017 12 / 23

Intuitively speaking, if the sequence of moments does not grow so quickly, then the distribution is determined by its moments. Example 2.2 A standard example of two distinct distributions with the same moment is based on the density of lognormal distribution (Billingsley, Probability and Measure, chapter 30.) f(x) = 1 2π 1/x exp( (log x) 2 /2) And its perturbed density: f a (x) = f(x)(1 + a sin(2π log x)) They have the same moments and the n th moment of each of them is exp(n 2 /2). Proof: Homework! Hongyi Liu (Washington University in St. Louis) Math Camp 2017 Stats July 31, 2017 13 / 23

Joint moment generating function Definition 2.2 The joint moment generating function of a random vector X = (X 1,..., X k ) is defined by M X1,...,X k (t 1,..., t k ) E(e t 1X 1 + t k X k ), for all t 1,..., t k R. And the definition applied here for M X1,...,X k ( ) is similar to M X (t), namely the MGF of X exists if M X1,...,X k ( ) is finite in a neighborhood of the origin of R d, t < t 0, t 0 > 0. M X (t) = 1 + k κ i t i + 1 2 i=1 r κ ij t i t j + i,j=1 where κ i 1 i r = E(Y i1 Y ir ) for i 1,..., i r = 1,..., k, which is referred to as the moment about the origin of order r of X, moments of order r form an array, symmetrical w.r.t permutations of indices. Hongyi Liu (Washington University in St. Louis) Math Camp 2017 Stats July 31, 2017 14 / 23

Moreover, The relationship κ i 1 i r = r M X (t) t i1 t ir t=0 M X (t) = M X1 M Xk holds if and only if the components of X are independent. Hongyi Liu (Washington University in St. Louis) Math Camp 2017 Stats July 31, 2017 15 / 23

Example Suppose M X (t) = 1 8 e 5t + 1 4 et + 5 8 e7t. E(X n )? Answer: M (n) X (t) = 1 8 ( 5)n e 5t + 1 4 et + 5 8 7n e 7t E[X n ] = M (n) X (0) = 1 8 ( 5)n + 1 4 + 5 8 7n Hongyi Liu (Washington University in St. Louis) Math Camp 2017 Stats July 31, 2017 16 / 23

Example Suppose M X (t) = 1 8 e 5t + 1 4 et + 5 8 e7t. E(X n )? Answer: M (n) X (t) = 1 8 ( 5)n e 5t + 1 4 et + 5 8 7n e 7t E[X n ] = M (n) X (0) = 1 8 ( 5)n + 1 4 + 5 8 7n Alternatively, by the definition of expectation and MGF, random variable X, occurs 5 with probability 1/8, occurs 1 with probability 1/4, and occurs 7 with probability 5/8. Thus its E(X n ) is trivially E[X n ] = M (n) X (0) = 1 8 ( 5)n + 1 4 + 5 8 7n. Hongyi Liu (Washington University in St. Louis) Math Camp 2017 Stats July 31, 2017 16 / 23

Outline 1 Expected Values 2 Moment Generating Functions 3 Cumulative Generating Functions Hongyi Liu (Washington University in St. Louis) Math Camp 2017 Stats July 31, 2017 17 / 23

Cumulant Generating Function Definition 3.1 Let M X (t) be finite for t < t 0. The cumulant generating function of X is defined as K X (t) = log M X (t) The CGF also completely determines the distribution of X and it can be expanded in a power series with same radius of convergence R t 0 as follows t 2 K X (t) = κ 1 t + κ 2 2! + κ t 3 3 3! +. The coefficient κ r of t r /r! is referred to as the cumulant of order r of X, κ r = κ r (X) = dr dt r K X(t) t=0 Hongyi Liu (Washington University in St. Louis) Math Camp 2017 Stats July 31, 2017 18 / 23

Multivariate Cumulative generating function When X = (X 1,..., X k ) is a vector, the CGF is defined as K X (t) = logm X (t) If M X (t) exists, then the CGF admits a multivariate Taylor series expansion in a neighborhood of the origin, with the coefficients corresponding to cumulants of X. Definition 3.2 The joint cumulant of order r is κ i 1,i 2,,i r = r K X (t) t i1 t ir t=0. Hongyi Liu (Washington University in St. Louis) Math Camp 2017 Stats July 31, 2017 19 / 23

Sums of I.I.D. random variables Let S n = n i=1 X i and M Xi exists, then Also, M Sn (t) = (M X (t)) n, K Sn (t) = nk X (t), κ r (S n ) = nκ r (X) = nκ r. In a word, when working with sums of i.i.d random variables, its cumulants are simply times n by each random variable s cumulants. Hongyi Liu (Washington University in St. Louis) Math Camp 2017 Stats July 31, 2017 20 / 23

Example 3.1 Let X N(µ, σ 2 ) and then M X (t) = e µt+σ2 t2 2, KX (t) = µt + σ 2 t2 2 Therefore, κ 1 = µ, κ 2 = σ 2, κ r = 0 for r = 3, 4,... Cumulants of order larger than 2 are all zero if and only if X has a normal distribution. Hongyi Liu (Washington University in St. Louis) Math Camp 2017 Stats July 31, 2017 21 / 23

Location Shifts Shifting from X to X + a induce the corresponding transformation of M X ( ) and K X ( ), respectively and M X+a (t) = E(e t(x+a) ) = e at M X (t), K X+a (t) = at + K X (t). Only the first cumulant is affected, i.e., κ 1 (X + a) = a + κ 1. Hongyi Liu (Washington University in St. Louis) Math Camp 2017 Stats July 31, 2017 22 / 23

Scale Changes Scaling change of X by b, b > 0 obtains that X/b. It follows that M X/b (t) = E(e tx/b ) = M X (t/b), K X/b (t) = K X (t/b), κ r (X/b) = κ r (X)/b r = κ r /b r. All cumulants are affected by a scale change unless b = 1. Hongyi Liu (Washington University in St. Louis) Math Camp 2017 Stats July 31, 2017 23 / 23