Copyright, 2008, R.E. Kass, E.N. Brown, and U. Eden REPRODUCTION OR CIRCULATION REQUIRES PERMISSION OF THE AUTHORS

Transcription

1 Copright, 8, RE Kass, EN Brown, and U Eden REPRODUCTION OR CIRCULATION REQUIRES PERMISSION OF THE AUTHORS Chapter 6 Random Vectors and Multivariate Distributions 6 Random Vectors In Section?? we etended the notion of distributions from random variables to pairs of random variables, considered together via their joint pdf In Section?? we also defined two random variables to be independent when their joint pdf factored as a product of their marginal pdfs, and in Section?? we said that a set of random variables X, X,, X n constituted a random sample when the variables all followed the same distribution and were independent of each other When we introduced this definition of random sample, what did we mean b independent of each other? We were not eplicit We could have meant that for ever pair of random variables the joint distribution factored as a product of the two marginals, but instead we reall meant something stronger: we were considering all n random variables X, X,, X n simultaneousl, ie, jointl We now need to be more careful

2 CHAPTER 6 RANDOM VECTORS AND MULTIVARIATE DISTRIBUTIONS We will sa that X, X,, X n are independent random variables if their joint pdf f (X,X,,X n)(,,, n ) is equal to the product of their marginal pdfs: n f (X,X,,X n)(,,, n ) = f Xi ( i ) Let us now introduce a general notation We will let X = (X, X,, X n ) be a generic random vector, and we will write f X () = f (X,X,,X n)(,,, n ) Sometimes we must distinguish row vectors from column vectors When we do, we will usuall want X to be an n column vector, so we would instead write X = (X, X,, X n ) T, where the superscript T denotes the transpose of a matri In this section we consider properties of random vectors when their component random variables are not necessaril independent As we discuss in Sections 6, a standard wa to measure the dependence of two random variables is b their correlation Using correlation, we define and interpret the bivariate Normal distribution in 6 and we provide some additional interpretation of correlation in terms of conditional epectation in Section 64 after we discuss conditional densities in Section 6 i= 6 The linear dependence of two random variables ma be quantified b their correlation When we consider X and Y simultaneousl, we ma characterize numericall their joint variation, meaning their tendenc to be large or small together, This is most commonl done via the covariance of X and Y which, for continuous random variables, is Cov(X, Y ) = E(X µ X )(Y µ Y ) = ( µ X )( µ Y )f(, )dd and for discrete random variables the integrals are replaced b sums The covariance is analagous to the variance of a single random variable The covariance depends on the variabilit of X and Y individuall as well as their joint variation For instance, as is immediatel verified from the definition, Cov(X, Y ) = Cov(X, Y ) To obtain a measure of joint variation that does not depend on the variance of X and Y, we standardize The correlation of

3 6 RANDOM VECTORS X and Y is Cor(X, Y ) = Cov(X, Y ) σ X σ Y The correlation ranges between and When X and Y are independent their covariance, and therefore also their correlation, is zero (The converse is not true: it is possible to find random variables with zero correlation that are nonetheless not independent) Illustration (Continued) We ma compute the covariance and correlation of X and Y introduced in the previous eample, as follows: which gives µ X = + (8) + () µ Y = + () + () σ X = 9 ( µ X ) + 8 ( µ X ) + ( µ X ) σ Y = 5 ( µ Y ) + ( µ Y ) + ( µ Y ) µ X = 84 µ Y = 7 σ X = 77 σ Y = 78 We then get f(, )( µx )( µ Y ) = 7 and f(, )( µx )( µ Y ) σ X σ Y = 45 Thus, the correlation is ρ 45 6 A bivariate Normal distribution is determined b a pair of means, a pair of standard deviations, and a correlation coefficient As ou might imagine, to sa that two random variables X and Y have a bivariate Normal distribution is to impl that each of them has a (univariate)

4 4CHAPTER 6 RANDOM VECTORS AND MULTIVARIATE DISTRIBUTIONS Normal distribution and, in addition, the have some covariance Actuall, there is a mathematical subtlet here: the requirement of bivariate Normalit is much more than that each has a univariate Normal distribution; it is, instead, that ever nonzero linear combination of X and Y must again have a Normal distribution, ie, for all numbers a and b that are not both zero, ax + by is Normall distributed (For a countereample, see the footnote on page 5) The bivariate Normal distribution is most easil interpreted in terms of its pdf In order for the pdf to make sense, we have to impose a technical constraint: we must assume no nonzero linear combination of X and Y is degenerate In other words, we must assume that for ever nonzero linear combination we have V (ax + by ) > (as opposed to V (ax + by ) = ) Then the bivariate Normal distribution has a pdf, which ma be written πσx σy ρ e ρ µx σx ρ + µy σy σ = σ, ρ = 75 σ = σ, ρ = 75 σ = σ, ρ = 75 σ = σ, ρ = 75 µy σy µx σx σ = σ, ρ = σ = σ, ρ = f (, ) = Figure 6: The Bivariate Normal pdf Perspective plots and contour plots are shown for various values of σx, σy and ρ, with (µx, µ ) = (, ) An important point about the bivariate Normal pdf is that it has the form

5 6 RANDOM VECTORS 5 e Q(,) where Q(, ) is a quadratic centered at the mean vector (We have inserted the minus sign as a reminder that the densit has a maimum rather than a minimum) An implication involves the contours of the pdf In general, a contour of a function f(, ) is the set of (, ) points such that f(, ) = c for some particular number c When the graph z = f(, ) is considered, a particular contour represents a set of points for which the height of f(, ) is the same The various contours of f(, ) are found b varing c The contours of a bivariate Normal pdf satisf Q(, ) = c, for some number c, and the set of points (, ) satisfing a quadratic equation form an ellipse Therefore, the bivariate Normal distribution has elliptical contours See Figure 6 The orientation and narrowness of these elliptical contours are governed b σ X, σ Y, and ρ When σ X = σ Y the aes of the ellipse are on the lines = and = As ρ increases toward (or decreases toward -) the ellipse becomes more tightl concentrated around = (or = ) When ρ = the contours become circles When σ X σ Y the aes rotate to = σ Y σ X and = σ X σy 6 Conditional probabilities involving random variables are obtained from conditional densities We previousl defined the probabilit of one event conditionall on another, which we wrote P (A B), as the ratio P (A B)/P (B), assuming P (B) > When we have a pair of random variables X and Y with f() >, the conditional densit of X given Y = is f X Y ( ) = f(, ) f Y (), (6) Here is an eample in which X and Y are each Normall distributed, but the do not have a bivariate Normal distribution Let U and V be independent N(, ) random variables Let Y = V and for U <, V > or U >, V < take X = U This amounts to taking the probabilit assigned to (U, V ) in the nd and 4th quadrants and moving it, respectivel, to the st and rd quadrants The distribution of (X, Y ) is then concentrated in the st and rd quadrants ((X, Y ) has zero probabilit of being in the nd or 4th quadrants), et X and Y remain distributed as N(, )

6 6CHAPTER 6 RANDOM VECTORS AND MULTIVARIATE DISTRIBUTIONS with f X Y ( )d being interpreted as the probabilit P ( X +d Y + d) When X and Y are independent we have f X Y ( ) = f X () Illustration: Spike Count Pairs (continued) We return to the joint distribution of spike counts for two neurons, given b the following table: 7 Y We ma calculate the conditional distribution of X given Y = We have f X Y ( ) = /5 = 6, f X Y ( ) = 5/5 =, f X Y ( ) = 5/5 = Note that these probabilities are different than the marginal probabilities 9, 8, In fact, if Y = it becomes more likel that X will also be, and less likel that X will be or X 64 The conditional epectation E(Y X = ) is called the regression of Y on X The conditional epectation of Y X is E(Y X) = f Y X ( )d where the integral is taken over the range of Illustration: Spike Count Pairs (continued) For the joint distribution of spike counts let us compute E(X Y = ) We previousl found f X Y ( ) = 6, f X Y ( ) =, f X Y ( ) = Then E(X Y = ) = (6) + () + () = 5

7 6 RANDOM VECTORS 7 In the illustration above, we computed the conditional epectation E(X Y = ) for a single value of We could evaluate it for each possible value of When we consider E(X Y = ) as a function of, this function is called the regression of X on Y Similarl, the function E(Y X = ) is called the regression of Y on X To understand this terminolog, and the interpretation of the conditional epectation, consider the case in which (X, Y ) is bivariate Normal Son s Height (inches) Son s Height (inches) Father s Height (inches) Father s Height (inches) Figure 6: Conditional epectation for bivariate Normal data mimicking Pearson and Lee s data on heights of fathers and sons Left panel shows contours of the bivariate Normal distribution based on the means, standard deviations, and correlation in Pearson and Lee s data The dashed vertical lines indicate the averaging process used in computing the conditional epectation when X = 64 or X = 7 inches: we average using the probabilit f Y X( ), which is the probabilit, roughl, in between the dashed vertical lines, integrating across In the right panel we generated a sample of,78 points (the sample size in Pearson and Lee s data set) from the bivariate Normal distribution pictured in the left panel We then, again, illustrate the averaging process: when we average the values of within the dashed vertical lines we obtain the two values indicated b the red These fall ver close to the least-squares regression line (the solid line) Eample: regression of son s height on father s height A famous data set, from Pearson and Lee (9) (Pearson, K and Lee, A (9) On

8 8CHAPTER 6 RANDOM VECTORS AND MULTIVARIATE DISTRIBUTIONS the laws of inheritance in man, Biometrika, : 57 46), has been used frequentl as an eample of regression (See Freedman, Pisani, and Purves (7)) (Freedman, D, Pisani, R, and Purves, R (7) Statistics, Fourth Edition, WW Norton) Figure 6 displas both a bivariate Normal pdf and a set of data generated from the bivariate Normal pdf the latter are similar to the data obtained b Pearson and Lee (who did not report the data, but onl summaries of them) The left panel of Figure 6 shows the theoretical regression line The right panel shows the regression based on the data, fitted b the method of least-squares, which will be discussed briefl in Chapter and then at length in Chapter 6 In a large sample like this one, the least-squares regression line (right panel) is ver close to the theoretical regression line (left panel) The purpose of showing both is to help clarif the averaging process represented b the conditional epectation E(Y X = ) The terminolog regression is illustrated in Figure 6 b the slope of the regression line being less than that of the dashed line Here, σ Y = σ X, because the variation in sons heights and fathers heights was about the same, while (µ X, µ Y ) = (68, 69), so that the average height of the sons was about an inch more than the average height among their fathers The dashed line has slope σ Y /σ X = and it goes through the point (µ X, µ Y ) Thus, the points falling on the dashed line in the left panel, for eample, would be those for which a theoretical father-son pair had the height of a son inch more than the height of his father Similarl, in the plot on the left, an data points falling on the dashed line would correspond to a real pair for which the son was an inch taller than the father However, if we look at E(Y X = 7) we see that among these taller fathers, their son s height tends, on average, to be less than the inch more than the father s predicted b the dashed line This is the tendenc for the son s height to regress toward the mean The same tendenc, now in the reverse direction, is apparent when the father s height is X = 64 In general, the regression E(Y X = ) could be a nonlinear function, but in Figure 6 it is a straight line This is not an accident: if (X, Y ) is bivariate Normal, the regression of Y on X is linear with slope ρ σ Y /σ X Specificall, E(Y X = ) = µ Y + ρ σ Y σ X ( µ X ) We sa that Y has a regression on X with regression coefficient β = ρ σ Y σ X This means that when X =, the average value of Y is given b the equa-

9 6 RANDOM VECTORS 9 tion above We should emphasize, again, that we are talking about random variables, which are theoretical quantities, as opposed to observed data In data-analtic contets the word regression almost alwas refers to leastsquares regression, illustrated in the right panel of Figure 6 65 Baes Theorem for random variables and vectors is analogous to Baes Theorem for events Now suppose X and Y are random variables with a joint densit f(, ) Substituting f(, ) = f Y X ( )f() into (6), we have f X Y ( ) = f Y X(, ) f Y () = f Y X( )f X () f Y () If we then substitute f Y () = f(, )d in the denominator and rewrite the joint densit in the integrand using f(, ) = f Y X ( )f X () we obtain Baes Theorem for random variables Theorem For continuous random variables X and Y we have f X Y ( ) = f Y X( )f X () fy X ( )f X ()d The resemblance of this result to Baes Theorem for events ma be seen b comparing the formula (??), identifing X with A and Y with B When X and Y are random vectors we have precisel the same result

10 CHAPTER 6 RANDOM VECTORS AND MULTIVARIATE DISTRIBUTIONS 6 Multivariate Normal Distributions 6 A random vector is multivariate Normal if linear combinations of its components are univariate Normal Now suppose we wish to consider the wa m random variables X,, X m var together If we have µ i = E(X i ), σi = V (X i ), and ρ ij = Cor(X i, X j ), for i =,, m and j =,, m, we ma collect the variables in an m- dimensional random vector X = (X,, X m ) T, and can likewise collect the means in a vector µ = µ µ µ m Similarl, we can collect the variances and covariances in a matri Σ = σ ρ σ σ ρ m σ σ m ρ σ σ σ ρ m σ σ m ρ m σ σ m ρ m σ σ m σm Note that ρ ij = ρ ji so that Σ is a smmetric matri (the element in its ith row and jth column is equal to the element in its jth row and ith column, for ever i and j) In dimensions greater than it becomes hard to visualize the multidimensional variation of X One approach is to use projections onto one or twodimensional subspaces We will discuss such techniques for analzing multidimensional data vectors later Projections are also useful for theoretical characterizations The most important of these involves Normalit We sa that the vector X has an m-dimensional multivariate Normal distribution

11 6 MULTIVARIATE NORMAL DISTRIBUTIONS if for ever nonzero linear combination of its components is Normall distributed In terms of smbols, we sa that for ever nonzero m-dimensional vector w, the random variable w T X is Normall distributed Furthermore, b straightforward matri manipulations we obtain the mean and variance of w T X as E(w T X) = w T µ (6) V (w T X) = w T Σw (6) This simplest wa to visualize multivariate data is to plot the data in bivariate pairs Eample: Tetrode spike sorting One relativel reliable method of identifing etracellular action potentials in vivo is to use a tetrode As pictured in panel A of Figure 6, a tetrode is a set of four electrodes that sit near a neuron and record slightl different voltage readings in response to an action potential The use of all four recordings allows more accurate discrimination of a particular neuronal signal from the man others that affect each of the lectrodes Action potentials corresponding to a particular neuron are identified from a comple voltage recording b first thresholding the recording, ie, identifing all events that have voltages above the threshold Each thresholded event is a four-dimensional vector (,,, 4 ), with i being the voltage amplitude (in millivolts) recorded at the ith electrode or channel Panels B-D displa data from a rat hippocampal CA neuron Because there are si pairs of the four tetrodes (channel and channel, channel and channel, etc) si bivariate plots are shown in panel B The univariate distributions are displaed in panel C and Q-Q plots are in panel D The data in Figure 6 are roughl multivariate Normal: the histograms are fit reasonabl well b univariate Normal distributions, and the bivariate plots have an approimatel elliptical shape On the other hand, the Q-Q plots do indicate deviations from Normalit Thus, whether one is willing to assume multivariate Normalit for such data will depend on the purpose of the analsis, and the techniques used

12 CHAPTER 6 RANDOM VECTORS AND MULTIVARIATE DISTRIBUTIONS Figure 6: Spike sorting from a tetrode recording Panel A is a diagram of a tetrode recording device, which is a set of four electrodes; also shown there are signals being recorded from a particular neuron that is sitting near the tetrode Panel B displas the si pairs of plots of event amplitudes For instance, the top left plot in panel B shows the event amplitudes for channel (-ais) and channel (-ais) Also overlaid (in blue) on the data in panel B are 95% probabilit contours found from a suitable bivariate Normal distribution Panel C displas histograms for the event amplitudes on each channel, together with fitted Normal pdfs, and panel D provides the corresponding Normal Q-Q plots 6 A multivariate Normal distribution is specified b its mean vector and covariance matri Just as the univariate Normal distribution is completel characterized b its mean and variance, and the bivariate Normal distribution is characterized b means, variances, and a correlation, the multivariate Normal distribution is completel characterized b its mean vector and covariance matri In man cases, the components of a multivariate Normal random vector are treated

13 6 MULTIVARIATE NORMAL DISTRIBUTIONS separatel, with each diagonal element of the covariance matri furnishing a variance, and the off-diagonal elements being ignored In some situations, however, the joint distribution, and thus all the elements of the covariance matri, are important Eample: Tetrode spike sorting (continued) In Figure 6, the Normal fits displaed in panels B and C were obtained b setting each theoretical mean equal to the corresponding sample mean, and similarl for each variance and correlation In defining the bivariate Normal pdf for (X, Y ) we had to assume no nonzero linear combination of X and Y was degenerate, ie, for no nonzero combination was V (ax +by ) = The corresponding assumption in the multivariate case is most concisel stated b saing that the covariance matri is positive definite (A smmetric matri is positive definite if no nonzero linear combination of its columns is zero or, equivalentl, if its determinant is positive) We will usuall make this assumption, though there will be at least one occasion (in Chapter 6) when we will want to work with the degenerate case 6 The multivariate Normal pdf has elliptical contours, with probabilit densit declining according to a χ pdf If X is m-dimensional multivariate Normal, with positive definite covariance matri, then its pdf is given b where f() = (π) m Σ e Q() Q() = ( µ X ) T Σ ( µ X ) with Σ being the determinant of Σ We have labeled the eponent b Q() to emphasize that it gives a quadratic in the components of This implies that the contours of f() are multidimensional ellipses Importantl, ever pair of components X i and X j has a bivariate Normal distribution with elliptical contours for its joint distribution

14 4CHAPTER 6 RANDOM VECTORS AND MULTIVARIATE DISTRIBUTIONS Using simple matri multiplication arguments, it is not hard to show that Y = Q(X) has a chi-squared distribution with m degrees of freedom See Appendi B Thus, each contour { : Q() = c} of the multivariate Normal pdf encloses a region { : Q() > c} that has probabilit given b the χ m distribution 64 For large n, the multivariate sample mean is approimatel multivariate Normal The multivariate version of the CLT is analagous to the univariate CLT We begin with a set of multidimensional samples of size n: on the first variable we have a sample X, X,, X n, on the second, X, X,, X n, and so on In this notation, X ij is the jth observation on the ith variable Suppose there are m variables in all, and suppose further that E(X ij ) = µ i, V (X ij = σ i, and Cor(X ij, X ik ) = ρ jk ) for all i =,, m, j =,, m, and k =,, m As before, let us collect the means into a vector µ and the variances and covariances into a matri Σ We assume, as usual, that the variables across different samples are independent Here this means X ij and X hk are independent whenever i h The sample means X = n X j n j= X = n X j n j= X m = n X mj n j= ma be collected in a vector X = X X X m

15 6 MULTIVARIATE NORMAL DISTRIBUTIONS 5 Before stating the multivariate CLT we need a preliminar result Lemma If Σ is a smmetric positive definite matri then there is a smmetric positive definite matri Σ such that Σ = Σ Σ and, furthermore, writing its inverse matri as Σ = (Σ ) we have Σ = Σ Σ Proof: This follows from the Spectral Decomposition See Appendi B Writing Σ = P DP T, with D being diagonal we simpl define D to be the diagonal matri having elements ( D,, D mm ) and take Σ = P D P T The stated results are easil checked Multivariate Central Limit Theorem: Suppose X, X,, X m are means from a set of m random samples of size n, as defined above, with the covariance matri Σ being positive definite For an m-dimensional vector w define Z n (w) = nw T Σ ( X µ) Then for ever nonzero m-dimensional vector w, Z n (w) converges in distribution to a Normal random variable having mean and variance More loosel, the multivariate CLT sas that X is approimatel multivariate Normal with mean µ and variance matri Σ As in the univariate case, there n are much more general versions of the multivariate CLT