Notes on Random Vectors and Multivariate Normal

Transcription

1 MATH 590 Spring 06 Notes on Random Vectors and Multivariate Normal Properties of Random Vectors If X,, X n are random variables, then X = X,, X n ) is a random vector, with the cumulative distribution function cdf) and the probability density function pdf) F x) = F x, x,, x n ) = P X x, X x,, X n x n ) fx) = fx, x,, x n ) = n x x n F x) provided it exists), and the pdf must satisfy fx) 0 and fx)dx = fx,, x n )dx dx n = and Note that marginal f i x i ) = fx)dx dx i dx i+ dx n, so EX i ) = x i f i x i )dx i = x i fx)dx dx n If EX i ) = µ i, then X EX) = E = X n µ µ n = µ x CovX) = EX µ x )X µ x ) = EXX ) EX)EX ) = Σ xx If Y = Y,, Y m ) is another random vector, with EY) = µ y then CovX, Y) = EX µ x )Y µ y ) = Σ xy with the i, j)th element of Σ xy being the scalar covariance CovX i, Y j ) = EX i EX i ))Y j EY j )) If Z = AX + c, where A is p n matrix and c is a p-vector both non-random), then ie, E ) is a linear operator) and EZ) = AEX) + c CovZ) = ACovX)A = AΣ xx A More generally, if W = BY + d where B is a q m matrix and d is a q-vector, then CovZ, W) = ACovX, Y)B = AΣ xy B Claim: the covariance matrix Σ xx is symmetric and psd = We can write Σ xx = PΛP and all the eigenvalues are nonnegative

2 Proof Varb X) = Covb X) = b Σ xx b 0 Change of Variables: Suppose we have X = X,, X n ) with pdf f X x) Let Y = TX c) where T is n n nonsingular and non-random, and c is non-random n-vector Then we get X = T Y + c, and the pdf of y is given by where J is the Jacobian f Y y) = f X T y + c) J J = X X Y X X Y X n Y X Y n X Y n Y Y X n Y X n Y n Now, write T = t ij ) Then from the relationship X = T Y + c, X = t Y + t Y + + t n Y n + c X = t Y + t Y + + t n Y n + c we get X n = t n Y + t n Y + + t nny n + c n J = t t t t t n = T = T t n t n n t nn t and so J = / dett) For example, if T = I ie, Y = X c) then J = Basics of Multivariate Normal Distribution Definition: Let Σ = σ ij ) denote an n n real symmetric pd matrix, and let µ = µ,, µ n ) be a real n-vector Then a random vector X = X,, X n ) is said to have a nonsingular) multivariate normal distribution with mean vector µ and covariance matrix Σ if it has a pdf fx) = π) n/ Σ / x µ) Σ x µ) We write X N n µ, Σ) First, we show that this is a valid pdf ie, fx) 0 and fx)dx = ) Result: Let A be n n symmetric pd matrix, and let X = X,, X n ), a random vector but not necessarily normal Then ) x Ax dx = π) n/ A /

3 Consequently, if Σ is an n n symmetric pd matrix and µ is an n-vector, then x µ) Σ x µ) dx = π) n/ Σ / Alternatively, we can show that it is a valid pdf by Moment Generating Function: Define the n-dimensional multivariate) moment generating function mgf) by Mt) = Ee t X ) where t R n For the normal case, Mt) = Ee t X ) = e t x fx)dx = π) n/ Σ / t x x µ) Σ x µ) dx Since Σ is assume to be pd, we have Σ = PΛ P where Λ = diag/λ,, /λ n ) Now, consider the linear transformation Y = P X µ) Then X = PY + µ remember that P = P ) Hence, J = P and J = why?), and then Mt) = π) n/ Σ / = π) n/ Σ / = t µ)π) n/ Σ / = t µ)π) n/ Σ / = t µ)π) n/ Σ / t x x µ) Σ x µ) dx t Py + µ) Py) Σ Py) t Py y P Σ Py dy dy P t) y y Λ y dy n w j y j n yj /λ j dy where w = P t = w,, w n ) Also, we know that Σ = Λ = n j= λ j, so Mt) = t µ)π) n/ Σ / j= j j= n w j y j j= j= n yj /λ j dy j= n = t µ) π) / λ / w j y j /)yj /λ j dyj and since M j w j ) = Ee wjyj ) = π) / λ / j wj y j /)yj /λ j dyj if Y j N0, λ j ), and we know 3

4 that M j w j ) = λ j wj /), we get Mt) = t µ) = t µ) n π) / λ / j= n λ j wj /) j= n = t µ) λ j wj / j= = t µ + w Λw = t µ + t PΛP t = t µ + t Σt j w j y j /)y j /λ j dyj This is the mgf of X N n µ, Σ) We can easily show that fx)dx = by setting t = 0 in the mgf The mgf each X j is obtained from Mt) by setting all t i = 0 except t j, and we get M Xj t j ) = Ee tjxj ) = t j µ j +σ jj t j /) So X j Nµ j, σ jj ) for j =,, n, and the mgf gives us EX) = µ and CovX) = Σ Alternatively, if we consider Y = P X µ), we saw that each Y j N0, λ j ) and that EY) = 0 and CovY) = Λ, and thus Y N0, Λ) Since X = PY + µ, it follows that EX) = PEY) + µ = µ and CovX) = PCovY)P = PΛP = Σ Multivariate Standard Normal: A random vector Z = Z,, Z n ) is said to have multivariate standard normal distribution if and only if fz) = ) z z = and is denoted Z N n 0, I) π) n/ ) π) n/ z Linear Combinations: Suppose that X N n µ x, Σ xx ), and let c = c,, c n ) Then Y = c X = n j= c jx j has a normal distribution with EY ) = c µ x = n j= c jµ j and VarY ) = c Σ xx c = n n j= c ic j σ ij Quick proof: by mgf, M Y t) = Ee ty ) = Ee tc X ) = Ee tc) X ) = tc µ x + t c Σ xx c/ which is the mgf of Nc µ x, c Σ xx c) Can easily show that X N n µ x, Σ xx ) c X Nc µ x, c Σ xx c) for every c 0 Cramér-Wold) More generally, if Y = BX + d where Y = Y,, Y p ), then Y N p Bµ x + d, BΣ xx B ) Properties of Multivariate Normal Distribution Independence: This is a very nice property of the multivariate normal distribution It says that if X = X,, X k ) and X N k µ x, Σ xx ), then the random variables X,, X k are independent Σ xx is diagonal, or equivalently, the X j are uncorrelated This is easily proved by inspecting the mgf of the multivariate normal Also, if X N k µ x, Σ xx ) and U = AX and V = BX, then U and V are independent if and only if CovU, V) = AΣ xx B = 0 See p5 of the text for the proof 4

5 Marginal Distributions: Let X N k µ, Σ), and partition X = x, x ), where x is k -vector and x is k -vector, with k + k = k Then µ = µ, µ ) where µ i = Ex i ) and ) Σ Σ Σ = Σ Σ where Σ ij = Covx i, x j ) It follows that x i N ki µ i, Σ ii ), and this can be verified by letting x = I k 0X and x = 0 I k X and deriving their distributions In addition, x and x are independent iff Σ = Covx, x ) = 0 which is equivalent to Σ = Σ = 0) For the proof, one direction is obvious As for the other direction, if Σ = 0, then Σ = Σ Σ and Σ = diagσ, Σ ) why?) Then it follows that fx) = π) k/ Σ / x µ) Σ x µ) = π) k/ Σ / Σ / = f x )f x ) ) Σ x µ) 0 0 Σ x µ) Conditional Distribution: Under the same partition as above, the conditional distribution of x given x = c is still normal with conditional mean vector Ex x = c ) = µ + Σ Σ c µ ) and conditional covariance matrix Covx x = c ) = Σ Σ Σ Σ where we write Σ = Σ Σ Σ Σ One proof goes as follows Let y = x and y = x + Mx where M is chosen such that y and y are independent Then we must have 0 = Covy, y ) = Covx + Mx, x ) = Σ + MΣ, and then we can choose M = Σ Σ Then x = y and y = x Σ Σ x are independent, with x N k µ, Σ ) and y N k µ Σ Σ µ, Σ ), since Covy ) = Σ Σ Σ Σ = Σ Hence, the distribution of y x = c is the same as the distribution of unconditional y, so that y x = c ) N k µ Σ Σ µ, Σ ) = x Σ Σ c x = c has the same distribution as y x = c = x x = c ) N k µ + Σ Σ c µ ), Σ ) A few more facts: The matrix B = Σ Σ is called the matrix of regression coefficients or regression matrix) of x on x Ex x = c ) = µ + Bc µ ) is called the linear) regression function of x on x = c The elements of Σ are called partial covariance, adjusted for x, and the corresponding correlation associated with Σ are called partial correlation between elements of x adjusted for x ) Samples From Multivariate Normal Distribution Let X,, X N be an iid sample from the N k µ, Σ) distribution The sample mean is X N = N X i 5

6 The distribution of X N is easily derived from results we saw earlier about multivarite normals, which we can find that X N N k µ, Σ/N) The multivariate analog of sample variance is the sample covariance matrix The sample covariance matrix is S N /N ), where S N = X i X N )X i X N ) = X i X i NX N X N What is the distribution of S N? Wishart Distribution Let X j N k 0, Σ) Then S = m j= X jx j is distributed as central) k- dimensional Wishart Distribution with scale parameter Σ and df m, denoted W k Σ, m) When we have W k I, m), then we say that the Wishart distribution is said to have the standard form Here are some elementary properties of a matrix S that has a k-dimensional Wishart distribution with m degrees of freedom and scale matrix Σ: For each constant vector a R k, a Sa/a Σa χ m For each p k matrix B, BSB W p BΣB, m) ES) = mσ If T W k Σ, n) and is independent of S, then S + T W k Σ, m + n) It can be shown that S N W k Σ, N ) Also, we can prove that X N and S N are independent Note: There is one important non-elementary fact about Wishart matrices If S W k I k, m) with m k, then one over the diagonal entries of S have a χ m k+ distribution It is related to the more general fact that, if we partition S so that the diagonal blocks are k k and k k with k = k + k ), then S = S S S S W k Σ, m k ) Also, S is independent of both S and S S Sufficient Statistics A statistic T is sufficient for a parameter θ for a distribution of X) if the distribution of X given T does not depend on θ We know a factorization theorem: a statistic TX) is sufficient for θ if and only if there exist functions gt θ) and hx) such that fx θ) = gtx) θ)hx) We can show that if X,, X N are independent multivariate normal N k µ, Σ), then X N and S N are sufficient for µ and Σ, respectively To prove this, write the joint density of N independent) normal random vectors X,, X N as fx,, x N ; µ, Σ) = = N ) π) k/ Σ / x i µ) Σ x i µ) π) Nk/ Σ N/ ) x i µ) Σ x i µ) = trσ S N ) N x N µ) Σ x N µ) ) π) Nk/ Σ N/ 6

7 note that we can do this by observing that x i µ) Σ x i µ) = = tr x i µ) Σ x i µ) ) tr Σ x i µ)x i µ) ) = tr = tr = tr +tr Σ N N Σ N Σ Σ x i µ)x i µ) ) x i x N + x N µ)x i x N + x N µ) ) x i x N )x i x N ) ) N x N µ)x N µ) ) = trσ S N ) + Nx N µ) Σ x N µ), since N x i x N )x i µ) = 0) By the method analogous to the univariate case, we see that X N and S N are sufficient statistics Note that we can also show the independence of X N and S N by using Basu s Theorem Maximum likelihood estimates The likelihood function based on a sample of N iid N k µ, Σ) vectors which is actually the same as the joint pdf above) is Lµ, Σ) = Again, onent can be rewritten as π) Nk/ Σ N/ ) x i µ) Σ x i µ) N x N µ) Σ x N µ) trσ S N ) The first term is the only place that µ appears in the likelihood, hence we can maximize the likelihood over µ for each fixed Σ by choosing µ = x N, so the MLE of µ is then Next, µ = X N logl µ, Σ) = Nk logπ) N log Σ ) trσ S N ) One could try to maximize this by taking its derivative with respect to each element of Σ and setting them all to 0 This procedure is facilitated by a couple of results about derivatives with respect to matrices, which are, trab) = B A and log A ) A = A ) 7

8 Now, let A = Σ We get logl µ, Σ) A = N A log A ) ) tras N = N A S N The derivative is 0 k k if and only if A = S N /N The MLE of Σ is then Σ = S N /N Since the MLE has the invariance property, we can show that the MLE of µ Σ µ is µ Σ µ Large Sample Properties First, some preliminaries For x R k, we define the Euclidean norm distance) as x = x x = k x i If X, X, X, are random k-vectors, then we say that, as n, X n as X, if P lim n X n = X) = almost sure convergence) X n L p X, if E X n X p 0 convergence in L p ) X n p X, if for any ɛ > 0, P Xn X > ɛ) 0 convergence in probability) X n d X, if FXn is the cdf of X n and F X is the cdf of X, then F Xn x) F X x), at points of x R k where F X x) is continuous convergence in distribution) Consistency: It can be shown that, if X, X, are iid random k-vectors with EX i ) = µ and CovX i ) = Σ, then X n µ and S n n Σ or S n /n Σ) almost surely, as n Actually, we mostly need convergence in probability consistency) Multivariate Central Limit Theorem If X, X, are iid random k-vectors with EX i ) = µ and CovX i ) = Σ, then n X i µ) = nx n µ) d N k 0, Σ) n as n An important consequence of this result is so-called the Multivriate Delta Method Under the same assumptions as above, and if there exists a real-valued function g such that the first partial derivative exists at µ and is not all zero, then ngxn ) gµ)) d N0, gµ) Σ gµ)) where gx) is the gradient of g 8