Multivariate Gaussian Analysis

Transcription

1 BS2 Statistical Inference, Lecture 7, Hilary Term 2009 February 13, 2009

2 Marginal and conditional distributions For a positive definite covariance matrix Σ, the multivariate Gaussian distribution has density on R d f (x ξ, Σ) = (2π) d/2 (det K) 1/2 e (x ξ) K(x ξ)/2, (1) where K = Σ 1 is the concentration matrix of the distribution. If X 1 N d (ξ 1, Σ 1 ) and X 2 N d (ξ 2, Σ 2 ) and X 1 X 2 X 1 + X 2 N d (ξ 1 + ξ 2, Σ 1 + Σ 2 ). If A is an r d matrix, b R r and X N d (ξ, Σ), then Y = AX + b N r (Aξ + b, AΣA ).

3 Marginal and conditional distributions Partition X into X 1 and X 2, where X 1 R r and X 2 R s with r + s = d and partition mean vector, concentration and covariance matrix accordingly. Then, if X N d (ξ, Σ) X 2 N s (ξ 2, Σ 22 ). If Σ 22 is regular, it further holds that where X 1 X 2 = x 2 N r (ξ 1 2, Σ 1 2 ), ξ 1 2 = ξ 1 + Σ 12 Σ 1 22 (x 2 ξ 2 ) and Σ 1 2 = Σ 11 Σ 12 Σ 1 22 Σ 21. In particular, if Σ 12 = 0 if and only if X 1 and X 2 are independent.

4 Marginal and conditional distributions From the matrix identities and K 1 11 = Σ 11 Σ 12 Σ 1 22 Σ 21 = Σ 1 2 (2) K 1 11 K 12 = Σ 12 Σ 1 22, (3) it follows that then the conditional expectation and concentrations also can be calculated as ξ 1 2 = ξ 1 K 1 11 K 12(x 2 ξ 2 ) and K 1 2 = K 11. Note that the marginal covariance is simply expressed in terms of Σ where as the conditional concentration is simply expressed in terms of K.

5 Trace of matrix Sample with known mean Maximizing the likelihood A square matrix A has trace tr(a) = i a ii. The trace has a number of properties: 1. tr(γa + µb) = γ tr(a) + µ tr(b) for γ, µ being scalars; 2. tr(a) = tr(a ); 3. tr(ab) = tr(ba) 4. tr(a) = i λ i where λ i are the eigenvalues of A.

6 Trace of matrix Sample with known mean Maximizing the likelihood For symmetric matrices the last statement follows from taking an orthogonal matrix O so that OAO = diag(λ 1,..., λ d ) and using tr(oao ) = tr(ao O) = tr(a). The trace is thus orthogonally invariant, as is the determinant: det(oao ) = det(o) det(a) det(o ) = 1 det(a)1 = det(a). There is an important trick that we shall use again and again: For λ R d λ Aλ = tr(λ Aλ) = tr(aλλ ) since λ Aλ is a scalar.

7 Trace of matrix Sample with known mean Maximizing the likelihood Consider first the case where ξ = 0 and a sample X 1 = x 1,..., X n = x n from a multivariate Gaussian distribution N d (0, Σ) with Σ regular. Using (1), we get the likelihood function where L(K) = (2π) nd/2 (det K) n/2 e n ν=1 x ν Kxν /2 (det K) n/2 e n ν=1 tr{kxνx ν }/2 = (det K) n/2 e tr{k n ν=1 xνx ν }/2 = (det K) n/2 e tr(kw)/2. (4) W = n X ν Xν = X X, ν=1 is the matrix of sums of squares and products. Here we have let X be the n d matrix with rows equal to X ν.

8 Trace of matrix Sample with known mean Maximizing the likelihood Writing the trace out tr(kw ) = i k ij W ji j emphasizes that it is linear in both K and W and we can recognize this as a linear and canonical exponential family with K as the canonical parameter and W /2 as the canonical sufficient statistic. Thus, the likelihood equation becomes E( W /2) == nσ/2 = W /2 since E(W ) = nσ. Solving, we get ˆK 1 = ˆΣ = W /n in analogy with the univariate case.

9 Trace of matrix Sample with known mean Maximizing the likelihood Rewriting the likelihood function as log L(K) = n log(det K) tr(kw )/2 2 we can of course also differentiate to find the maximum, leading to k ij log(det K) = w ij /n, which in combination with the previous result yields K log(det K) = K 1. The latter can also be derived directly by writing out the determinant, and it holds for any non-singular square matrix, i.e. one which is not necessarily positive definite.

10 Definition Wishart density Partioning the Wishart distribution is the sampling distribution of the matrix of sums of squares and products. More precisely: A random d d matrix W has a d-dimensional Wishart distribution with parameter Σ and n degrees of freedom if W D = n X ν Xν i=1 where X ν N d (0, Σ). We then write W W d (n, Σ). The Wishart is the multivariate analogue to the χ 2 : W 1 (n, σ 2 ) = σ 2 χ 2 (n). If W W d (n, Σ) its mean is E(W ) = nσ.

11 Definition Wishart density Partioning the Wishart distribution If W 1 and W 2 are independent with W i W d (n i, Σ), then W 1 + W 2 W d (n 1 + n 2, Σ). If A is an r d matrix and W W d (n, Σ), then AWA W r (n, AΣA ). For r = 1 we get that when W W d (n, Σ) and λ R d, λ W λ σλ 2 χ2 (n), where σλ 2 = λ Σλ.

12 Definition Wishart density Partioning the Wishart distribution If W W d (n, Σ), where Σ is regular, then W is regular with probability one if and only if n d. When n d the Wishart distribution has density f d (w n, Σ) = c(d, n) 1 (det Σ) n/2 (det w) (n d 1)/2 e tr(σ 1 w)/2 for w positive definite, and 0 otherwise. The Wishart constant c(d, n) is c(d, n) = 2 nd/2 (2π) d(d 1)/4 d Γ{(n + 1 i)/2}. i=1

13 Definition Wishart density Partioning the Wishart distribution Let X 1,..., X n be independent and identically distributed as N d (ξ, Σ). Let X be the n d matrix with rows equal to Xi assume that Π 1,..., Π k are n n matrices for orthogonal projections onto subspaces L 1,..., L k of R n, that is, and Then, if Π i ξ = 0 we have Π u Π v = δ uv Π u and Π u = Π u. W u = X Π u X W d (f i, Σ), where f u = dim L u = rank Π u = tr Π u. Further, W 1,..., W k are independent.

14 Definition Wishart density Partioning the Wishart distribution Let W W d (n, Σ) with Σ regular and n > d. Then W 22 is regular with probability one and (i) W 1 2 is independent of (W 12, W 22 );

15 Definition Wishart density Partioning the Wishart distribution Let W W d (n, Σ) with Σ regular and n > d. Then W 22 is regular with probability one and (i) W 1 2 is independent of (W 12, W 22 ); (ii) W 1 2 W r (n s, Σ 1 2 );

16 Definition Wishart density Partioning the Wishart distribution Let W W d (n, Σ) with Σ regular and n > d. Then W 22 is regular with probability one and (i) W 1 2 is independent of (W 12, W 22 ); (ii) W 1 2 W r (n s, Σ 1 2 ); (iii) W 22 W s (n, Σ 22 );

17 Definition Wishart density Partioning the Wishart distribution Let W W d (n, Σ) with Σ regular and n > d. Then W 22 is regular with probability one and (i) W 1 2 is independent of (W 12, W 22 ); (ii) W 1 2 W r (n s, Σ 1 2 ); (iii) W 22 W s (n, Σ 22 ); (iv) The conditional distribution of W 12 given W 22 = w 22 is multivariate Gaussian N r s (Σ 12 Σ 1 22 w 22, Λ) where Λ ij,kl = Cov(W ij, W kl W 22 = w 22 ) = w jl σ 1 2 ik w jl.

18 Definition Wishart density Partioning the Wishart distribution In the special case with Σ 12 = 0 this can be simplified to W 1 2 W r (n s, Σ 11 ) and with Λ ij,kl = σ ik w jl. W 12 W 22 = w 22 N r s (0, Λ) It follows that in this case, i.e. when Σ 12 = 0, it holds that cf. Problem sheet 4. W 12 W 1 22 W 21 W r (s, Σ 11 ),