The Multivariate Normal Distribution 1

Transcription

1 The Multivariate Normal Distribution 1 STA 302 Fall See last slide for copyright information. 1 / 40

2 Overview 1 Moment-generating Functions 2 Definition 3 Properties 4 χ 2 and t distributions 2 / 40

3 Joint moment-generating function Of a p-dimensional random vector x ( M x (t = E e t x For example, M (x1,x 2,x 3 (t 1, t 2, t 3 = E ( e x 1t 1 +x 2 t 2 +x 3 t 3 Just write M(t if there is no ambiguity. Section 4.3 of Linear models in statistics has some material on moment-generating functions (optional. 3 / 40

4 Uniqueness Proof omitted Joint moment-generating functions correspond uniquely to joint probability distributions. M(t is a function of F (x. Step One: f(x = x 1 x p F (x. For example, x 1 x 2 x2 x1 f(y 1, y 2 dy 1 dy 2 Step Two: M(t = e t x f(x dx Could write M(t = g (F (x. Uniqueness says the function g is one-to-one, so that F (x = g 1 (M(t. 4 / 40

5 g 1 (M(t = F (x A two-variable example g 1 ( g 1 (M(t = F (x ex 1t 1 +x 2 t 2 f(x 1, x 2 dx 1 dx 2 = x 2 x1 f(y 1, y 2 dy 1 dy 2 5 / 40

6 Theorem Two random vectors x 1 and x 2 are independent if and only if the moment-generating function of their joint distribution is the product of their moment-generating functions. 6 / 40

7 Proof Two random vectors are independent if and only if the moment-generating function of their joint distribution is the product of their moment-generating functions. Independence therefore the MGFs factor is an exercise. M x1,x 2 (t 1, t 2 = M x1 (t 1 M x2 (t 2 ( ( = e x 1t 1 f x1 (x 1 dx 1 e x 2t 2 f x2 (x 2 dx 2 = = e x 1t 1 e x 2t 2 f x1 (x 1 f x2 (x 2 dx 1 dx 2 e x 1t 1 +x 2 t 2 f x1 (x 1 f x2 (x 2 dx 1 dx 2 7 / 40

8 Proof continued Have M x1,x 2 (t 1, t 2 = ex 1t 1 +x 2 t 2 f x1 (x 1 f x2 (x 2 dx 1 dx 2. Using F (x = g 1 (M(t, F (x 1, x 2 = g 1 ( = = = x2 x1 x2 x2 = F x1 (x 1 f x2 (y 2 e x 1t 1 +x 2 t 2 f x1 (x 1 f x2 (x 2 dx 1 dx 2 f x1 (y 1 f x2 (y 2 dy 1 dy 2 ( x1 f x2 (y 2 F x1 (x 1 dy 2 x2 = F x1 (x 1 F x2 (x 2 f x1 (y 1 dy 1 dy 2 f x2 (y 2 dy 2 So that x 1 and x 2 are independent. 8 / 40

9 A helpful distinction If x 1 and x 2 are independent, M x1 +x 2 (t = M x1 (tm x2 (t x 1 and x 2 are independent if and only if M x1,x 2 (t 1, t 2 = M x1 (t 1 M x2 (t 2 9 / 40

10 Theorem: Functions of independent random vectors are independent Show x 1 and x 2 independent implies that y 1 = g 1 (x 1 and y 2 = g 2 (x 2 are independent. ( ( ( y1 g1(x 1 t1 Let y = = and t =. Then g 2(x 2 y 2 t 2 M y(t = E (e t y ( ( = E e t 1 y 1+t 2 y 2 = E e t 1 y 1 e t 2 y 2 = E (e t 1 g 1(x 1 e t 2 g 2(x 2 = e t 1 g 1(x 1 e t 2 g 2(x 2 f x1 (x 1 f x2 (x 2 dx 1 dx 2 ( = e t 2 g 2(x 2 f x2 (x 2 e t 1 g 1(x 1 f x1 (x 1 dx 1 dx 2 = e t 2 g 2(x 2 f x2 (x 2 M g1 (x 1 (t 1 dx 2 = M g1 (x 1 (t 1 M g2 (x 2 (t 2 = M y1 (t 1 M y2 (t 2 So y 1 and y 2 are independent. 10 / 40

11 M Ax (t = M x (A t Analogue of M ax(t = M x(at ( M Ax (t = E e t Ax = E (e ( A t x = M x (A t Note that t is the same length as y = Ax: The number of rows in A. 11 / 40

12 M x+c (t = e t c M x (t Analogue of M x+c(t = e ct M x(t ( M x+c (t = E ( = E = e t c E e t (x+c e t x+t c ( e t x = e t c M x (t 12 / 40

13 Distributions may be defined in terms of moment-generating functions Build up the multivariate normal from univariate normals. If y N(µ, σ 2, then M y (t = e µt+ 1 2 σ2 t 2 Moment-generating functions correspond uniquely to probability distributions. So define a normal random variable with expected value µ and variance σ 2 as a random variable with moment-generating function e µt+ 1 2 σ2 t 2. This has one surprising consequence / 40

14 Degenerate random variables A degenerate random variable has all the probability concentrated at a single value, say P r{y = y 0 } = 1. Then M y (t = E(e yt = e yt p(y y = e y 0t p(y 0 = e y 0t 1 = e y 0t 14 / 40

15 If P r{y = y 0 } = 1, then M y (t = e y 0t This is of the form e µt+ 1 2 σ2 t 2 with µ = y 0 and σ 2 = 0. So y N(y 0, 0. That is, degenerate random variables are normal with variance zero. Call them singular normals. This will be surprisingly handy later. 15 / 40

16 Independent standard normals Let z 1,..., z p i.i.d. N(0, 1. z = z 1. z p E(z = 0 cov(z = I p 16 / 40

17 Moment-generating function of z Using e µt+ 1 2 σ2 t 2 M z (t = = p M zj (t j j=1 p j=1 e 1 2 t2 j = e 1 2 p j=1 t2 j = e 1 2 t t 17 / 40

18 Transform z to get a general multivariate normal Remember: A non-negative definite means v Av 0 Let Σ be a p p symmetric non-negative definite matrix and µ R p. Let y = Σ 1/2 z + µ. The elements of y are linear combinations of independent standard normals. Linear combinations of normals should be normal. y has a multivariate distribution. We d like to call y a multivariate normal. 18 / 40

19 Moment-generating function of y = Σ 1/2 z + µ Remember: M Ax(t = M x(a t and M x+c(t = e t c M x(t and M z (t = e 1 2 t t M y (t = M (t Σ 1/2 z+µ = e t µ M (t Σ 1/2 z = e t µ M z (Σ 1/2 t = e t µ M z (Σ 1/2 t = e t µ e 1 2 (Σ1/2 t (Σ 1/2 t = e t µ e 1 2 t Σ 1/2 Σ 1/2 t = e t µ e 1 2 t Σt = e t µ+ 1 2 t Σt So define a multivariate normal random variable y as one with moment-generating function M y (t = e t µ+ 1 2 t Σt. 19 / 40

20 Compare univariate and multivariate normal moment-generating functions Univariate M y (t = e µt+ 1 2 σ2 t 2 Multivariate M y (t = e t µ+ 1 2 t Σt So the univariate normal is a special case of the multivariate normal with p = / 40

21 Mean and covariance matrix For a univariate normal, E(y = µ and V ar(y = σ 2 Recall y = Σ 1/2 z + µ. E(y = µ cov(y = Σ 1/2 cov(zσ 1/2 = Σ 1/2 I Σ 1/2 = Σ We will say y is multivariate normal with expected value µ and variance-covariance matrix Σ, and write y N p (µ, Σ. Note that because M y (t = e t µ+ 1 2 t Σt, µ and Σ completely determine the distribution. 21 / 40

22 Probability density function of y N p (µ, Σ Remember, Σ is only positive semi-definite. It is easy to write down the density of z N p (0, I as a product of standard normals. If Σ is strictly positive definite (and not otherwise, the density of y = Σ 1/2 z + µ can be obtained using the Jacobian Theorem as { 1 f(y = exp 1 } Σ 1 2 (2π p 2 2 (y µ Σ 1 (y µ This is usually how the multivariate normal is defined. 22 / 40

23 Σ positive definite? Positive definite means that for any non-zero p 1 vector a, we have a Σa > 0. Since the one-dimensional random variable w = p i=1 a iy i may be written as w = a y and V ar(w = cov(a y = a Σa, it is natural to require that Σ be positive definite. All it means is that every non-zero linear combination of y values has a positive variance. Often, this is what you want. 23 / 40

24 Singular normal: Σ is positive semi-definite. Suppose there is a 0 with a Σa = 0. Let w = a y. Then V ar(w = cov(a y = a Σa = 0. That is, w has a degenerate distribution (but it s still still normal. In this case we describe the distribution of y as a singular multivariate normal. Including the singular case saves a lot of extra work in later proofs. We will insist that a singular multivariate normal is still multivariate normal, even though it has no density. 24 / 40

25 Distribution of Ay Recall y N p(µ, Σ means M y (t = e t µ+ 1 2 t Σt Let y N p (µ, Σ, and w = Ay, where A is an r p matrix. M w (t = M Ay (t = M y (A t = e (A t µ e 1 2 (A t Σ(A t = e t (Aµ e 1 2 t (AΣA t = e t (Aµ+ 1 2 t (AΣA t Recognize moment-generating function and conclude w N r (Aµ, AΣA 25 / 40

26 Exercise Use moment-generating functions, of course. Let y N p (µ, Σ. Show y + c N p (µ + c, Σ. 26 / 40

27 Zero covariance implies independence for the multivariate normal. Independence always implies zero covariance. For the multivariate normal, zero covariance also implies independence. The multivariate normal is the only continuous distribution with this property. 27 / 40

28 Show zero covariance implies independence By showing M y(t = M y1 (t 1M y2 (t 2 Let y N(µ, Σ, with y = ( y1 y 2 µ = ( µ1 µ 2 ( Σ1 0 Σ = 0 Σ 2 t = ( t1 t 2 M y (t = E (e t y = E e t 1 t 2 y = / 40

29 Continuing the calculation: M y (t = e t µ+ 1 2 t Σt y = ( y1 y 2 µ = ( µ1 µ 2 ( Σ1 0 Σ = 0 Σ 2 t = ( t1 t 2 M y(t = E e t1 t 2 y { ( = exp (t 1 t µ1 2 µ 2 { = e t 1 µ 1 +t 2 µ 2 1 exp 2 } { ( 1 exp 2 (t 1 t Σ Σ 2 } ( t 1 Σ 1 t 2Σ 2 ( t 1 t 2 { = e t 1 µ 1 +t 2 µ 2 1 ( exp t 1 Σ 1t 1 + t } 2Σ 2t 2 2 = e t 1 µ 1 e t 2 µ 2 e 1 2 (t 1 Σ 1t 1 e 1 2 (t 2 Σ 2t 2 = e t 1 µ (t 1 Σ 1t 1 e t 2 µ (t 2 Σ 2t 2 = M y1 (t 1M y2 (t 2 ( t1 t 2 } So y 1 and y 2 are independent. 29 / 40

30 An easy example If you do it the easy way Let y 1 N(1, 2, y 2 N(2, 4 and y 3 N(6, 3 be independent, with w 1 = y 1 + y 2 and w 2 = y 2 + y 3. Find the joint distribution of w 1 and w 2. ( w1 w 2 = ( y 1 y 2 y 3 w = Ay N(Aµ, AΣA 30 / 40

31 w = Ay N(Aµ, AΣA y 1 N(1, 2, y 2 N(2, 4 and y 3 N(6, 3 are independent Aµ = AΣA = = ( ( ( = ( / 40

32 Marginal distributions are multivariate normal y N p(µ, Σ, so w = Ay N(Aµ, AΣA Find the distribution of ( y 1 y 2 y 3 y 4 = ( y2 y 4 Bivariate normal. The expected value is easy. 32 / 40

33 Covariance matrix Of Ay cov(ay = AΣA = = = ( ( σ1,2 σ 2 2 σ 2,3 σ 2,4 σ 1,4 σ 2,4 σ 3,4 σ 2 4 ( σ 2 2 σ 2,4 σ 2,4 σ 2 4 σ1 2 σ 1,2 σ 1,3 σ 1,4 σ 1,2 σ2 2 σ 2,3 σ 2,4 σ 1,3 σ 2,3 σ3 2 σ 3,4 σ 1,4 σ 2,4 σ 3,4 σ Marginal distributions of a multivariate normal are multivariate normal, with the original means, variances and covariances / 40

34 Summary If c is a vector of constants, x + c N(c + µ, Σ. If A is a matrix of constants, Ax N(Aµ, AΣA. Linear combinations of multivariate normals are multivariate normal. All the marginals (dimension less than p of x are (multivariate normal, but it is possible in theory to have a collection of univariate normals whose joint distribution is not multivariate normal. For the multivariate normal, zero covariance implies independence. The multivariate normal is the only continuous distribution with this property. 34 / 40

35 Showing (x µ Σ 1 (x µ χ 2 (p Σ has to be positive definite this time x N (µ, Σ y = x µ N (0, Σ z = Σ 1 2 y N (0, Σ 1 2 ΣΣ 1 2 = N (0, Σ Σ 2 Σ 2 Σ 1 2 = N (0, I So z is a vector of p independent standard normals, and p y Σ 1 y = (Σ 1 2 y (Σ 1 2 y = z z = zi 2 χ 2 (p j=1 35 / 40

36 x and s 2 independent x 1,..., x n i.i.d N(µ, σ 2 x = x 1. x n N ( µ1, σ 2 I y = x 1 x. x n x = Ax Note A is (n + 1 n, so cov(ax = σ 2 AA is (n + 1 (n + 1, singular. x 36 / 40

37 The argument x 1 x. y = Ax = x n x = x y 2 x y is multivariate normal because x is multivariate normal. Cov (x, (x j x = 0 (Exercise So x and y 2 are independent. So x and S 2 = g(y 2 are independent. 37 / 40

38 Leads to the t distribution If z N(0, 1 and y χ 2 (ν and z and y are independent, then we say T = z y/ν t(ν 38 / 40

39 Random sample from a normal distribution i.i.d. Let x 1,..., x n N(µ, σ 2. Then n(x µ N(0, 1 and σ (n 1S 2 σ 2 χ 2 (n 1 and These quantities are independent, so T = = n(x µ/σ (n 1S 2 /(n 1 σ 2 n(x µ t(n 1 S 39 / 40

40 Copyright Information This slide show was prepared by Jerry Brunner, Department of Statistical Sciences, University of Toronto. It is licensed under a Creative Commons Attribution - ShareAlike 3.0 Unported License. Use any part of it as you like and share the result freely. The L A TEX source code is available from the course website: brunner/oldclass/302f17 40 / 40