The Multivariate Normal Distribution. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

Transcription

1 The Multivariate Normal Distribution Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

2 The Moment Generating Function (MGF) of a random vector X is given by M X (t) = E(e t X ) provided h > 0 E(e t X ) exists t = (t 1,..., t n ) t i ( h, h) i = 1,..., n. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

3 Result 5.1: If the MGFs of two random vectors X 1 and X 2 exist in an open rectangle R that includes the origin, then the cumulative distribution functions (CDFs) of X 1 and X 2 are identical iff M X1 (t) = M X2 (t) t R. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

4 A random variable Z with MGF M Z (t) = E(e tz ) = e t2 /2 is said to have a standard normal distribution. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

5 Show that a random variable Z with density f Z (z) = 1 2π e z2 /2 has a standard normal distribution. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

6 E(e tz ) = = = = e t2 /2 e tz 1 2π e z2 /2 dz 1 2π e 1/2(z2 2tz) dz 1 2π e 1/2(z2 2tz+t 2 t 2) dz = e t2 /2. 1 2π e 1/2(z t)2 dz Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

7 Suppose Z has a standard normal distribution. Then E(Z) = M Z(t) t t=0 = et2 /2 t t=0 = e t2 /2 (t) t=0 = 0. E(Z 2 ) = 2 M Z (t) t 2 t=0 = e t2 /2 + t 2 e t2 /2 t=0 = 1. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

8 Thus, E(Z) = 0 and Var(Z) = 1. If Z is standard normal, then Y = µ + σz has mean E(Y) = E(µ + σz) = µ + σe(z) = µ and variance Var(Y) = Var(µ + σz) = Var(σZ) = σ 2 Var(Z) = σ 2. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

9 Furthermore, the MGF of Y is M Y (t) = E(e ty ) = E(e t(µ+σz) ) = e tµ E(e tσz ) = e tµ M Z (tσ) = e tµ e t2 σ 2 /2 = e tµ+t2 σ 2 /2. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

10 A random variable Y with MGF M Y (t) = e tµ+t2 σ 2 /2 is said to have a normal distribution with mean µ and variance σ 2. We denote the distribution of Y by N(µ, σ 2 ). Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

11 If Y N(µ, σ 2 ), then Thus, the density of Y is P(Y y) y P(Y y) = P(µ + σz y) = P(Z y µ σ ). P(Z y µ σ ) = y ( y µ = f Z σ ) 1 σ = 1 2πσ 2 e 1 2( y µ σ ) 2. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

12 That is, f Y (y) = = 1 2πσ 2 e 1 2( y µ σ ) 2 1 2πσ 2 e 1 2σ 2 (y µ)2. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

13 Suppose Z 1,..., Z p i.i.d. N(0, 1). Then Z = Z 1. Z p is said to have a standard multivariate normal distribution. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

14 E(Z) = 0 Var(Z) = I. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

15 Find the Moment Generating Function of a standard multivariate normal random vector Z. p 1 Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

16 E(e t Z ) = E(e p i=1 t iz i ) p = E( e t iz i ) = = = i=1 p E(e t iz i ) i=1 p M Zi (t i ) i=1 p i=1 e t2 i /2 = e p i=1 t2 i /2 = e t t/2. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

17 A p-dimensional random vector Y has the Multivariate Normal Distribution with mean µ and variance Σ (Y N(µ, Σ)) iff the MGF of Y is M Y (t) = e t µ+t Σt/2. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

18 Suppose Z N(0, I). Show that Y = µ + AZ has a multivariate normal distribution. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

19 If Z standard multivariate normal, then Y = µ + AZ has mean E(Y) = E(µ + AZ) = µ + AE(Z) = µ and variance Var(Y) = Var(µ + AZ) = Var(AZ) = AVar(Z)A = AA. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

20 Furthermore, M Y (t) = E(e t Y ) = E(e t (µ+az) ) = e t µ E(e t AZ ) = e t µ M Z (A t) = e t µ+t AA t/2. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

21 Note that if rank( q p A ) < q, then will be singular. Var(Y) = Var(µ + AZ) = AA In this case, the support of the q 1 random vector Y will lie within a rank(a)(< q)- dimensional vector space. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

22 Give a specific example of a singular multivariate normal distribution (Y N(µ, Σ), Σ singular). Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

23 Suppose Z = Z N(0, 1). [ ] 1 Suppose A =. Then 1 rank( A) = 1 < Let Y = AZ. Then ( [ ]) 1 1 Y N 0, AA =. 1 1 Y lies in the 1-dimensional vector space C = {Y = (Y 1, Y 2 ) R 2 : Y 1 = Y 2 } with probability 1. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

24 Result 5.3: If X N(µ, Σ) and Y = a + BX, then Y N(a + Bµ, BΣB ). Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

25 Proof of Result 5.3: E(e t Y ) = E(e t a+t BX ) = e t a E(e t BX ) = e t a M X (B t) = e t a e t Bµ+t BΣB t/2 = e t (a+bµ)+t BΣB t/2. Thus, Y N(a + Bµ, BΣB ). Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

26 Corollary 5.1: If X is multivariate normal (MVN), then the joint distribution of any p 1 subvector of X is MVN. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

27 Proof of Corollary 5.1: Suppose {i 1,..., i q } {1,..., p}. Then X i1. = e i 1. X, X iq e i q where e i denotes the ith row of the p p identity matrix. e i 1. X MVN by Result 5.3. e i q Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

28 Corollary 5.2: If X N(µ, Σ) and Σ is nonsingular, then p 1 (a) a nonsingular matrix A Σ = AA, (b) A 1 (X µ) N(0, I), and (c) The probability density function of X is f X (t) = (2π) p/2 Σ 1/2 e 1/2(t µ) Σ 1 (t µ). Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

29 Proof of Corollary 5.2: (a) We can take A = Σ 1/2 because Σ is symmetric and positive definite. Because Σ positive definite, (Σ 1/2 ) 1 = Σ 1/2 exists. (b) By Result 5.3, A 1 (X µ) N(A 1 µ A 1 µ, A 1 Σ(A 1 ) ), with A 1 µ A 1 µ = 0 and A 1 Σ(A 1 ) = Σ 1/2 ΣΣ 1/2 = I. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

30 (c) Homework problem. You may wish to use the multivariate change of variables result on page 185 of Casella and Berger. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

31 Result 5.2: Suppose the MGF of X i is M Xi (t i ) i = 1,..., p. Let X = [X 1, X 2,..., X p] and t = [t 1, t 2,..., t p]. Suppose X has MGF M X (t). Then X 1,..., X p are mutually independent iff p M X (t) = M Xi (t i ) i=1 t in an open rectangle that includes 0. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

32 Result 5.4: If X N(µ, Σ) and we partition X = X 1. X p, µ = µ 1. µ p, Σ = Then X 1,..., X p are mutually independent iff Σ 11 Σ 1p..... Σ p1 Σ pp. Σ ij = 0 i j. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

33 Proof of Result 5.4: (= ) X 1,..., X p mutually independent, then Cov(X i, X j ) = E[(X i µ i )(X j µ j ) ] = [E(X i µ i )][E(X j µ j ) ] = [µ i µ i ][(µ j µ j ) ] = 0 i j. Thus, Σ ij = 0 i j. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

34 ( =) Partition t as [t 1,..., t p]. Then t Σt = p p t iσ ij t j. i=1 j=1 If Σ ij = 0 i j, then Thus, t Σt = p t iσ ii t i. i=1 M X (t) = e t µ+t Σt/2 = e p i=1 (t i µ i +t i Σ iit i /2) p p = e t i µ i +t i Σ iit i /2 = M Xi (t i ). i=1 i=1 By Result 5.2, X 1,..., X p are mutually independent. opyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

35 Corollary 5.3: Suppose X N(µ, Σ) Y 1 = a 1 + B 1 X, Y 2 = a 2 + B 2 X. and Then Y 1 and Y 2 are independent iff B 1 ΣB 2 = 0. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36

36 Proof of Corollary 5.3: Let Then Y MVN with Y = [ Y1 Y 2 Var(Y) = = ] = [ B1 B 2 [ a1 ] a 2 ] + [ B1 B 2 ] [ ] Σ B 1 B 2 [ B1 ΣB 1 B 1 ΣB 2 B 2 ΣB 1 B 2 ΣB 2 X. ]. By Result 5.4, Y 1 and Y 2 independent B 1 ΣB 2 = 0. Copyright c 2012 Dan Nettleton (Iowa State University) Statistics / 36