The Multivariate Normal Distribution 1

Transcription

1 The Multivariate Normal Distribution 1 STA 302 Fall See last slide for copyright information. 1 / 37

2 Overview 1 Moment-generating Functions 2 Definition 3 Properties 4 χ 2 and t distributions 2 / 37

3 Joint moment-generating function Of a p-dimensional random vector X ( M X (t = E e t X For example, M (X1,X 2,X 3 (t 1, t 2, t 3 = E ( e X 1t 1 +X 2 t 2 +X 3 t 3 Section 4.3 of Linear models in statistics has some material on moment-generating functions (optional. 3 / 37

4 Two big theorems Proof omitted 1 Joint moment-generating functions correspond uniquely to joint probability distributions. 2 Two random vectors X 1 and X 2 are independent if and only if the moment-generating function of their joint distribution is the product of their moment-generating functions. These results assume only that the moment-generating functions exist in a neighborhood of t = 0. Nothing else is required. 4 / 37

5 A helpful distinction If X 1 and X 2 are independent, M X1 +X 2 (t = M X1 (tm X2 (t X 1 and X 2 are independent if and only if M X1,X 2 (t 1, t 2 = M X1 (t 1 M X2 (t 2 5 / 37

6 Theorem: Functions of independent random vectors are independent Show X 1 and X 2 independent implies that Y 1 = g 1 (X 1 and Y 2 = g 2 (X 2 are independent. Let Y = ( Y1 Y 2 = ( g1 (X 1 g 2 (X 2 and t = ( t1 t 2. Then M Y (t = E (e t Y ( ( = E e t 1 Y 1+t 2 Y 2 = E e t 1 Y 1 e t 2 Y 2 = E (e t 1 g 1(X 1 e t 2 g 2(X 2 = e t 1 g 1(x 1 e t 2 g 2(x 2 f X1 (x 1 f X2 (x 2 d(x 1 d(x 2 = M g1 (X 1 (t 1 M g2 (X 2 (t 2 6 / 37

7 M AX (t = M X (A t Analogue of M ax(t = M X(at ( M AX (t = E e t AX = E (e ( A t X = M X (A t Note that t is the same length as Y = AX: The number of rows in A. 7 / 37

8 M X+c (t = e t c M X (t Analogue of M X+c(t = e ct M X(t ( M X+c (t = E ( = E = e t c E e t (X+c e t X+t c ( e t X = e t c M X (t 8 / 37

9 Distributions may be defined in terms of moment-generating functions Build up the multivariate normal from univariate normals. If Y N(µ, σ 2, then M Y (t = e µt+ 1 2 σ2 t 2 Moment-generating functions correspond uniquely to probability distributions. So define a normal random variable with expected value µ and variance σ 2 as a random variable with moment-generating function e µt+ 1 2 σ2 t 2. This has one surprising consequence... 9 / 37

10 Degenerate random variables A degenerate random variable has all the probability concentrated at a single value, say P r{y = y 0 } = 1. Then M Y (t = E(e Y t = e yt p(y y = e y 0t p(y 0 = e y 0t 1 = e y 0t 10 / 37

11 If P r{y = y 0 } = 1, then M Y (t = e y 0t This is of the form e µt+ 1 2 σ2 t 2 with µ = y 0 and σ 2 = 0. So Y N(y 0, 0. That is, degenerate random variables are normal with variance zero. Call them singular normals. This will be surprisingly handy later. 11 / 37

12 Independent standard normals Let Z 1,..., Z p i.i.d. N(0, 1. Z = Z 1. Z p E(Z = 0 cov(z = I p 12 / 37

13 Moment-generating function of Z Using e µt+ 1 2 σ2 t 2 M Z (t = = p M Zj (t j j=1 p j=1 e 1 2 t2 j = e 1 2 p j=1 t2 j = e 1 2 t t 13 / 37

14 Transform Z to get a general multivariate normal Remember: A non-negative definite means v Av 0 Let Σ be a p p symmetric non-negative definite matrix and µ R p. Let Y = Σ 1/2 Z + µ. The elements of Y are linear combinations of independent standard normals. Linear combinations of normals should be normal. Y has a multivariate distribution. We d like to call Y a multivariate normal. 14 / 37

15 Moment-generating function of Y = Σ 1/2 Z + µ Remember: M AX (t = M X (A t and M X+c (t = e t c M X (t and M Z (t = e 1 2 t t M Y (t = M (t Y=Σ 1/2 Z+µ = e t µ M (t Σ 1/2 Z = e t µ M Z (Σ 1/2 t = e t µ M Z (Σ 1/2 t = e t µ e 1 2 (Σ1/2 t (Σ 1/2 t = e t µ e 1 2 t Σ 1/2 Σ 1/2 t = e t µ e 1 2 t Σt So define a multivariate normal random variable Y as one with moment-generating function M Y (t = e t µ e 1 2 t Σt. 15 / 37

16 Compare univariate and multivariate normal moment-generating functions Univariate M Y (t = e µt+ 1 2 σ2 t 2 Multivariate M Y (t = e t µ e 1 2 t Σt So the univariate normal is a special case of the multivariate normal with p = / 37

17 Mean and covariance matrix For a univariate normal, E(Y = µ and V ar(y = σ 2 Recall Y = Σ 1/2 Z + µ. E(Y = µ cov(y = Σ 1/2 cov(zσ 1/2 = Σ 1/2 I Σ 1/2 = Σ We will say Y is multivariate normal with expected value µ and variance-covariance matrix Σ, and write Y N p (µ, Σ. 17 / 37

18 Probability density function of Y N p (µ, Σ Remember, Σ is only positive semi-definite. It is easy to write down the density of Z N p (0, I as a product of standard normals. If Σ is strictly positive definite (and not otherwise, the density of Y = Σ 1/2 Z + µ can be obtained using the Jacobian Theorem as { 1 f(y = exp 1 } Σ 1 2 (2π p 2 2 (y µ Σ 1 (y µ This is usually how the multivariate normal is defined. 18 / 37

19 Σ positive definite? Positive definite means that for any non-zero p 1 vector a, we have a Σa > 0. Since the one-dimensional random variable W = p i=1 a iy i may be written as W = a Y and V ar(w = cov(a Y = a Σa, it is natural to require that Σ be positive definite. All it means is that every non-zero linear combination of Y values has a positive variance. Often, this is what you want. 19 / 37

20 Singular normal: Σ is positive semi-definite. Suppose there is a 0 with a Σa = 0. Let W = a Y. Then V ar(w = V ar(a Y = a Σa = 0. That is W has a degenerate distribution (but it s still still normal. In this case we describe the distribution of Y as a singular multivariate normal. Excluding the singular case creates a lot of extra work in later proofs. We will insist that a singular multivariate normal is still multivariate normal, even though it has no density. 20 / 37

21 Distribution of AY Recall M Y (t = e t µ+ 1 2 t Σt Let A be an r p matrix, and W = AY. M W (t = M AY (t = M Y (A t = e (A t µ e 1 2 (A t Σ(A t = e t (Aµ e 1 2 t (AΣA t = e t (Aµ+ 1 2 t (AΣA t Recognize moment-generating function and conclude W N r (Aµ, AΣA 21 / 37

22 Exercise Use moment-generating functions, of course. Let Y N p (µ, Σ. Show Y + c N p (µ + c, Σ. 22 / 37

23 Zero covariance implies independence for the multivariate normal. Independence always implies zero covariance. For the multivariate normal, zero covariance also implies independence. The multivariate normal is the only continuous distribution with this property. 23 / 37

24 Show zero covariance implies independence By showing M Y (t = M Y1 (t 1M Y2 (t 2 Let Y N(µ, Σ, with Y = ( Y1 Y 2 µ = ( µ1 µ 2 ( Σ1 0 Σ = 0 Σ 2 t = ( t1 t 2 M Y (t = E (e t Y = E (e (t 1 t 2 Y = M Y ( (t 1 t 2 = / 37

25 Continuing the calculation: M Y (t = e t µ+ 1 2 t Σt Y = ( Y1 Y 2 µ = ( µ1 µ 2 ( Σ1 0 Σ = 0 Σ 2 t = ( t1 t 2 ( M Y (t = M Y (t 1 t 2 { ( } { ( = exp (t 1 t µ1 1 2 exp µ 2 2 (t 1 t Σ Σ 2 { = e t 1 µ 1 +t 2 µ 2 1 ( exp t 2 1 Σ 1 t ( } t 1 2Σ 2 t 2 { = e t 1 µ 1 +t 2 µ 2 1 ( exp t 2 1 Σ 1 t 1 + t } 2Σ 2 t 2 = e t 1 µ 1 e t 2 µ 2 e 1 2 (t 1 Σ 1t 1 e 1 2 (t 2 Σ 2t 2 = M Y1 (t 1 M Y2 (t 2 ( t1 t 2 } So Y 1 and Y 2 are independent. 25 / 37

26 An easy example If you do it the easy way Let Y 1 N(1, 2, Y 2 N(2, 4 and Y 3 N(6, 3 be independent, with W 1 = Y 1 + Y 2 and W 2 = Y 2 + Y 3. Find the joint distribution of W 1 and W 2. ( W1 W 2 = ( Y 1 Y 2 Y 3 W = AY N(Aµ, AΣA 26 / 37

27 W = AY N(Aµ, AΣA Y 1 N(1, 2, Y 2 N(2, 4 and Y 3 N(6, 3 are independent Aµ = AΣA = = ( ( ( = ( / 37

28 Marginal distributions are multivariate normal Y N p(µ, Σ, so W = AY N(Aµ, AΣA Find the distribution of ( Y 1 Y 2 Y 3 Y 4 = ( Y2 Y 4 Bivariate normal. The expected value is easy. 28 / 37

29 Covariance matrix cov(ay = AΣA = = = ( ( σ1,2 σ 2 2 σ 2,3 σ 2,4 σ 1,4 σ 2,4 σ 3,4 σ 2 4 ( σ 2 2 σ 2,4 σ 2,4 σ 2 4 σ1 2 σ 1,2 σ 1,3 σ 1,4 σ 1,2 σ2 2 σ 2,3 σ 2,4 σ 1,3 σ 2,3 σ3 2 σ 3,4 σ 1,4 σ 2,4 σ 3,4 σ Marginal distributions of a multivariate normal are multivariate normal, with the original means, variances and covariances / 37

30 Summary If c is a vector of constants, X + c N(c + µ, Σ If A is a matrix of constants, AX N(Aµ, AΣA Linear combinations of multivariate normals are multivariate normal. All the marginals (dimension less than p of X are (multivariate normal, but it is possible in theory to have a collection of univariate normals whose joint distribution is not multivariate normal. For the multivariate normal, zero covariance implies independence. The multivariate normal is the only continuous distribution with this property. 30 / 37

31 Multivariate normal likelihood For reference L(µ, Σ = n i=1 1 Σ 1 2 (2π p 2 = Σ n/2 (2π np/2 exp { exp 1 } 2 (xi µ Σ 1 (x i µ { 1 2 } n (x i µ Σ 1 (x i µ i=1 = Σ n/2 (2π np/2 exp n { tr( 2 ΣΣ } 1 + (x µ Σ 1 (x µ, where Σ = 1 n n i=1 (x i x(x i x is the sample variance-covariance matrix. 31 / 37

32 Showing (X µ Σ 1 (X µ χ 2 (p Σ has to be positive definite this time X N (µ, Σ Y = X µ N (0, Σ Z = Σ 1 2 Y N (0, Σ 1 2 ΣΣ 1 2 = N (0, Σ Σ 2 Σ 2 Σ 1 2 = N (0, I So Z is a vector of p independent standard normals, and p Y Σ 1 Y = (Σ 1 2 Y (Σ 1 2 Y = Z Z = Zi 2 χ 2 (p j=1 32 / 37

33 X and S 2 independent X 1,..., X n i.i.d N(µ, σ 2 X = X 1. X n N ( µ1, σ 2 I Y = Note A is (n + 1 1, so cov(ay = σ 2 AA is (n + 1 (n + 1, singular. X 1 X. X n X = AX X 33 / 37

34 The argument X 1 X. Y = AX = X n X = X Y is multivariate normal. Cov ( X, (X j X = 0 (Exercise So X and Y 2 are independent. So X and S 2 = g(y 2 are independent. Y 2 X 34 / 37

35 Leads to the t distribution If Z N(0, 1 and Y χ 2 (ν and Z and Y are independent, then T = Z t(ν Y/ν 35 / 37

36 Random sample from a normal distribution i.i.d. Let X 1,..., X n N(µ, σ 2. Then n(x µ N(0, 1 and σ (n 1S 2 σ 2 χ 2 (n 1 and These quantities are independent, so T = = n(x µ/σ (n 1S 2 /(n 1 σ 2 n(x µ t(n 1 S 36 / 37

37 Copyright Information This slide show was prepared by Jerry Brunner, Department of Statistical Sciences, University of Toronto. It is licensed under a Creative Commons Attribution - ShareAlike 3.0 Unported License. Use any part of it as you like and share the result freely. The L A TEX source code is available from the course website: brunner/oldclass/302f14 37 / 37