Moment Generating Function STAT/MTHE 353: 5 Moment Generating Functions and Multivariate Normal Distribution T. Linder Queen s University Winter 07 Definition Let X (X,...,X n ) T be a random vector and t (t,...,t n ) T R n.themoment generating function (MGF) is defined by M X (t) E e tt X for all t for which the exectation exists (i.e., finite). Remarks: M X (t) E e P n i tixi For 0 (0,...,0) T,wehaveM X (0). If X is a discrete random variable with finitely many values, then M X (t) E e tt X is always finite for all t R n. We will always assume that the distribution of X is such that M X (t) is finite for all t ( t 0,t 0 ) n for some t 0 > 0. STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution / 34 STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution / 34 The single most imortant roerty of the MGF is that is uniquely determines the distribution of a random vector: Theorem Assume M X (t) and M Y (t) are the MGFs of the random vectors X and Y and such that M X (t) M Y (t) for all t ( t 0,t 0 ) n.then F X (z) F Y (z) for all z R n where F X and F Y are the joint cdfs of X and Y. Remarks: F X (z) F Y (z) for all z R n clearly imlies M X (t) M Y (t). Thus M X (t) M Y (t) () F X (z) F Y (z) Most often we will use the theorem for random variables instead of random vectors. In this case, M X (t) M Y (t) for all t ( t 0,t 0 ) imlies F X (z) F Y (z) for all z R. STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution 3/ 34 Connection with moments Let k,...,k n be nonnegative integers and k k + + k n. Then @ k M X (t) E e tx+ +tnxn @t k @tkn n @t k @tkn n @ k E e tx+ +tnxn @t k @tkn n E X k Xkn n e tx+ +tnxn Setting t 0 (0,...,0) T, we get @ k @t k @tkn n @ k M X (t) t0 E X k Xkn n For a (scalar) random variable X we obtain the kth moment of X: d k dt k M X(t) t0 E X k STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution 4/ 34
Theorem Assume X,...,X m are indeendent random vectors in R n and let X X + + X m.then Proof: my M X (t) M Xi (t) i M X (t) E e tt X E e tt (X + +X m) E e tt X e tt X m E e tt X E e tt X m M X (t) M Xm (t) Examle: MGF for X Gamma(r, ) and X + + X m where the X i are indeendent and X i Gamma(r i, ). Examle: MGF for X Poisson( ) and X + + X m where the X i are indeendent and X i Gamma( i ). Also, use the MGF to find E(X), E(X ),andvar(x). Note: This theorem gives us a owerful tool for determining the distribution of the sum of indeendent random variables. STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution 5/ 34 STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution 6/ 34 Theorem 3 Assume X is a random vector in R n, A is an m n real matrix and b R m. Then the MGF of Y AX + b is given at t R m by Proof: M Y (t) e tt b M X (A T t) M Y (t) E e tt Y E e tt (AX+b) e tt b E e tt AX e tt b E e (AT t) T X e tt b M X (A T t) Alications to Normal Distribution Let X N(0, ). Then M X (t) E(e tx ) Z e t / e t / Z Z e tx e x / dx e (x tx) dx e (x t) {z } N(t,) df dx Z e (x t) t dx Note: In the scalar case Y ax + b we obtain M Y (t) e tb M X (at) We obtain that for all t R M X (t) e t / STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution 7/ 34 STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution 8/ 34
Moments of a standard normal random variable Recall the ower series exansion e z P k0 zk valid for all z R. Aly this to z tx X M X (t) E(e tx (tx) k )E k0 X E (tx) k X E X k t k and to z t / Matching the coe k0 M X (t) e t / X i0 k0 t / i cient of t k,fork,,... we obtain 8 >< E(X k ) k/ (k/)!, k even, >: 0, k odd. STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution 9/ 34 i! Sum of indeendent normal random variables Recall that if X N(0, ) and Y X + µ, where > 0, then Y N(µ, ). Thus the MGF of a N(µ, ) random variable is M Y (t) e tµ M X ( t) e tµ e ( t) / e tµ+t / Let X,...,X m be indeendent r.v. s with X i N(µ i, i ) and set X X + + X m. Then my my M X (t) M Xi (t) e tµi+t i / e t P m i µi + P m i This imlies i i X m X N µ i, i mx i i i.e., X + + X m is normal with mean µ X P m i µ i and variance X P m i i. i t / STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution 0 / 34 Multivariate Normal Distributions Linear Algebra Review Recall that an n n real matrix C is called nonnegative definite if it is symmetric and x T Cx 0 for all x R n and ositive definite if it is symmetric and x T Cx > 0 for all x R n such that x 6 0 Let A be an arbitrary n n real matrix. Then C A T A is nonnegative definite. If A is nonsingular (invertible), then C is ositive definite. Proof: A T A is symmetric since (A T A) T (A T )(A T ) T A T A. Thus it is nonnegative definite since x T (A T A)x x T A T Ax (Ax) T (Ax) kaxk 0 STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution / 34 For any nonnegative definite n n matrix C the following hold: () C has n nonnegative eigenvalues,..., n (counting multilicities) and corresonding n orthogonal unit-length eigenvectors b,...,b n : Cb i i b i, i,...,n where b T i b i, i,...,n and b T i b j 0if i 6 j. () (Sectral decomosition) C can be written as C BDB T where D diag(,..., n) is the diagonal matrix of the eigenvalues of C, andb is the orthogonal matrix whose ith column is b i,i.e.,b [b...b n ]. (3) C is ositive definite () C is nonsingular () all the eigenvalues i are ositive STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution / 34
(4) C has a unique nonnegative definite square root C /,i.e.,there exists a unique nonnegative definite A such that C AA Proof: We only rove the existence of A. Let D / diag( / /,..., n ) and note that D / D / D. Let A BD / B T. Then A is nonnegative definite and Remarks: A AA (BD / B T )(BD / B T ) BD / B T BD / B T BD / D / B T C If C is ositive definite, then so is A. If we don t require that A be nonnegative definite, then in general there are infinitely many solutions A for AA T C. STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution 3 / 34 Lemma 4 If is the covariance matrix of some random vector X (X,...,X n ) T, then it is nonnegative definite. Proof: We know that Cov(X) is symmetric. Let b R n be arbitrary. Then b T b b T Cov(X)b Cov(b T X) Var(b T X) 0 so is nonnegative definite Remark: It can be shown that an n n matrix is nonnegative definite if and only if there exists a random vector X (X,...,X n ) T such that Cov(X). STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution 4 / 34 Defining the Multivariate Normal Distribution Let Z,...,Z n be indeendent r.v. s with Z i N(0, ). The multivariate MGF of Z (Z,...,Z n ) T is M Z (t) E e ttz E e P n i tizi ny i ny E e tizi i e t i / e P n i t i / e tt t Now let µ R n and A an n n real matrix. Then the MGF of X AZ + µ is M X (t) e tt µ M Z (A T t) e tt µ e (AT t) T (A T t) e tt µ e tt AA T t e tt µ+ tt t where AA T. Note that is nonnegative definite. Definition Let µ R n and let be an n n nonnegative definite matrix. A random vector X (X,...,X n ) is said to have a multivariate normal distribution with arameters µ and if its multivariate MGF is Notation: X N(µ, ). Remarks: M X (t) e tt µ+ tt t If Z (Z,...,Z n ) T with Z i N(0, ), i,...,n,then Z N(0, I), wherei is the n n identity matrix. We saw that if Z N(0, I), thenx AZ + µ N(µ, ), where AA T. One can show the following: X N(µ, ) if and only if X AZ + µ for a random n-vector Z N(0, I) and some n n matrix A with AA T. STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution 5 / 34 STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution 6 / 34
Mean and covariance for multivariate normal distribution Consider first Z N(0, I), i.e.,z (Z,...,Z n ) T,wheretheZ i are indeendent N(0, ) random variables. Then If X N(µ, ), thenx AZ + µ for a random n-vector Z N(0, I) and some n n matrix A with AA T. and Thus E(Z) E(Z ),...,E(Z n ) T (0,, 0) T 8 < if i j, E (Z i E(Z i ))(Z j E(Z j )) E(Z i Z j ) : 0 if i 6 j We have E(AZ + µ) AE(Z)+µ µ Also, Cov(AZ + µ) Cov(AZ) A Cov(Z)A T AA T Thus E(X) µ, Cov(X) E(Z) 0, Cov(Z) I STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution 7 / 34 STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution 8 / 34 Joint df for multivariate normal distribution Lemma 5 If a random vector X (X,...,X n ) T has covariance matrix that is not of full rank (i.e., singular), then X does not have a joint df. Theorem 6 If X (X,...,X n ) T N(µ, ), where is nonsingular, then it has a joint df given by f X (x) ( )n det e (x µ)t (x µ), x R n Proof sketch: If is singular, then there exists b R n such that b 6 0 and b 0. Consider the random variable b T X P n i b ix i : Var(b T X)Cov(b T X) b T Cov(X)b b T b 0 Therefore P (b T X c) for some constant c. If X had a joint df f(x), thenforb {x : b T x c} we should have Z Z P (b T X c) P (X B) f(x,...,x n ) dx dx n But this is imossible since B is an (n )-dimensional hyerlane whose n-dimensional volume is zero, so the integral must be zero. B Proof: We know that X AZ + µ where Z (Z,...,Z n ) T N(0, I) and A is an n n matrix such that AA T. Since is nonsingular, A must be nonsingular with inverse A. Thus the maing h(z) Az + µ is invertible with inverse g(x) A (x J g (x) deta By the multivariate transformation theorem µ) whose Jacobian is f X (x) f Z (g(x)) J g (x) f Z A (x µ) det A STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution 9 / 34 STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution 0 / 34
Proof cont d: Since Z (Z,...,Z n ) T,wheretheZ i are indeendent N(0, ) random variables, we have f Z (z) ny e z i / e P n i z i e zt z ( ) n ( ) n i Secial case: bivariate normal For n we have µ µ µ and so we get where µ i E(X i ), i Var(X i ), i,, and f X (x) f Z A (x µ) det A e (A (x µ)) T (A (x µ)) det A ( ) n since det A ( ) n e (x µ)t (A ) T A (x µ) det A ( )n det e (x µ)t (x µ) det and (A ) T A (exercise!) (X,X ) Cov(X,X ) Thus the bivariate normal distribution is determined by five scalar arameters µ, µ,,,and. is ositive definite () is invertible () det > 0: det ( ) > 0 () < and > 0 so a bivariate normal random variable (X,X ) has a df if and only if the comonents X and X have ositive variances and <. STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution / 34 STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution / 34 We have det Thus the joint df of (X,X ) T N(µ, ) is f(x,x ) e ( ) (x µ ) + (x µ ) (x µ )(x µ ) and (x µ) T (x µ) i x µ hx µ, x µ ( ) x µ h i ( ) x µ, x µ (x µ ) (x µ ) (x µ ) (x µ ) ( ) (x µ ) ( ) (x µ ) (x µ )(x µ )+ (x µ ) + (x µ ) (x µ )(x µ ) STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution 3 / 34 Remark: If 0,then f(x,x ) e e (x µ ) (x µ ) f X (x )f X (x ) + (x µ ) e (x µ ) Therefore X and X are indeendent. It is also easy to see that f(x,x )f X (x )f X (x ) for all x and x imlies 0. Thus we obtain Two jointly normal random variables X and X are indeendent if and only if they are uncorrelated. STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution 4 / 34
In general, the following imortant facts can be roved using the multivariate MGF: (i) If X (X,...,X n ) T N(µ, ), thenx,x,...x n are indeendent if and only if they are uncorrelated, i.e., Cov(X i,x j )0if i 6 j, i.e., is a diagonal matrix. (ii) Assume X (X,...,X n ) T N(µ, ) and let X (X,...,X k ) T, X (X k+,...,x n ) T Then X and X are indeendent if and only if Cov(X, X )0 k (n k),thek (n k) matrix of zeros, i.e., can be artitioned as 0 k (n k) 0 (n k) k where Cov(X ) and Cov(X ). Marginals of multivariate normal distributions Let X (X,...,X n ) T N(µ, ). If A is an m n matrix and b R m,then Y AX + b is a random m-vector. Its MGF at t R m is M Y (t) e tt b M X (A T t) Since M X ( )e T µ+ T for all R n, we obtain M Y (t) e tt b e (AT t) T µ+ (AT t) T (A T t) e tt (b+aµ)+ tt A A T t This means that Y N(b + Aµ, A A T ),i.e.,y is multivariate normal with mean b + Aµ and covariance A A T. Examle: Let a,...,a n R and determine the distribution of Y a X + + a n X n. STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution 5 / 34 STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution 6 / 34 For some ale m<nlet {i,...,i m } {,...,n} such that i <i < <i m. Let e j (0,...,0,, 0,...,0) t be the jth unit vector in R n and define the m n matrix A by 3 Then e T i A 6 7 4. 5 e T i e T i m 3 e T i m X X n 3 AX 6 7 6 7 4. 5 4. 5 6 4 Thus (X i,...,x im ) T N(Aµ, A A T ). X i. X im 3 7 5 Note the following: Aµ 6 4 and the (j, k)th entry of A A T is µ i. µ im 3 7 5 (A A T ) jk A (i k th column of ) j ( ) iji k Cov(X ij,x ik ) Thus if X (X,...,X n ) T N(µ, ), then(x i,...,x im ) T is multivariate normal whose mean and covariance are obtained by icking out the corresonding elements of µ and. Secial case: For m we obtain that X i N(µ i, µ i E(X i ) and i Var(X i), foralli,...,n. i ),where STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution 7 / 34 STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution 8 / 34
Conditional distributions Let X (X,...,X n ) T N(µ, ) and for ale m<ndefine X (X,...,X m ) T, X (X m+,...,x n ) T We know that X N(µ, ) and X N(µ, ) where µ i E(X i ), ii Cov(X i ), i,. Then µ and can be artitioned as µ µ µ, where ij Cov(X i, X j ), i, j,. Note that is m m, is (n m) (n m), is m (n m), and is (n m) m. Also, T. We assume that is nonsingular and we want to determine the conditional distribution of X given X x. Recall that X AZ + µ for some Z (Z,...,Z n ) T where the Z i are indeendent N(0, ) random variables and A is such that AA T. Let Z (Z,...,Z m ) T and Z (Z m+,...,z n ) T. We want to determine such A in a artitioned form with dimensions corresonding to the artitioning of : B 0 m (n m) A C D We can write AA T as B 0 m (n m) B T C T C D 0 (n m) m D T BB T BC T CB T CC T + DD T STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution 9 / 34 STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution 30 / 34 We want to solve for B, C and D. First consider BB T. We choose B to be the unique ositive definite square root of : B / Recall that B is symmetric and it is invertible since is. Then CB T imlies Then CC T + DD T gives C (B T ) B DD T CC T B B ( ) T (BB) Now note that X AZ + µ gives X BZ + µ, X CZ + DZ + µ STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution 3 / 34 Since B is invertible, given X x,wehavez B (x µ ). So given X x, we have that the conditional distribution of X and the conditional distribution of are the same. CB (x µ )+DZ + µ But Z is indeendent of X, so given X x, the conditional distribution of CB (x µ )+DZ + µ is the same as its unconditional distribution. We conclude that the conditional distribution of X given X x is multivariate normal with mean E(X X x ) µ + CB (x µ ) µ + B B (x µ ) µ + (x µ ) and covariance matrix DD T STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution 3 / 34
Secial case: bivariate normal Suose X (X,X ) T N(µ, ) with µ µ and µ We have and µ + (x µ )µ + (x µ ) ( ) Thus the conditional distribution of X given X x is normal with (conditional) mean and variance E(X X x )µ + (x µ ) Var(X X x ) ( ) Equivalently, the conditional distribution of X given X x is N µ + (x µ ), ( ) If <, then the conditional df exists and is given by f X X (x x ) x µ ( ) e Remark: Note that E(X X x )µ + (x (a ne) function of x. (x µ ) ( ) µ ) is a linear Examle: Recall the MMSE estimate roblem for X N(0, X ) from the observation Y X + Z, wherez N(0, Z ) and X and Z are indeendent. Use the above the find g (y) E[X Y y] and comute the minimum mean square error E (X g (Y )). STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution 33 / 34 STAT/MTHE 353: 5 MGF & Multivariate Normal Distribution 34 / 34