Covariance. Lecture 20: Covariance / Correlation & General Bivariate Normal. Covariance, cont. Properties of Covariance

Transcription

1 Covariance Lecture 0: Covariance / Correlation & General Bivariate Normal Sta30 / Mth 30 We have previously discussed Covariance in relation to the variance of the sum of two random variables Review Lecture 8. VarX + Y VarX + VarY + CovX, Y Specifically, covariance is defined as Colin Rundel April, 0 CovX, Y EX EX Y EY ] EXY EX EY this is a generalization of variance to two random variables and generally measures the degree to which X and Y tend to be large or small at the same time or the degree to which one tends to be large while the other is small. Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 / 33 Covariance, cont. The magnitude of the covariance is not usually informative since it is affected by the magnitude of both X and X. However, the sign of the covariance tells us something useful about the relationship between X and Y. Consider the following conditions: X > µ X and Y > µ Y then X µ X Y µ Y will be positive. X < µ X and Y < µ Y then X µ X Y µ Y will be positive. X > µ X and Y < µ Y then X µ X Y µ Y will be negative. X < µ X and Y > µ Y then X µ X Y µ Y will be negative. Properties of Covariance CovX, Y EX µ x Y µ y ] EXY µ x µ y CovX, Y CovY, X CovX, Y 0 if X and Y are independent CovX, c 0 CovX, X VarX CovaX, by ab CovX, Y CovX + a, Y + b CovX, Y CovX, Y + Z CovX, Y + CovX, Z Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 / 33 Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 3 / 33

2 Correlation Correlation, cont. Since CovX, Y depends on the magnitude of X and Y we would prefer to have a measure of association that is not affected by changes in the scales of the random variables. The most common measure of linear association is correlation which is defined as ρx, Y CovX, Y σ Y < ρx, Y < Where the magnitude of the correlation measures the strength of the linear association and the sign determines if it is a positive or negative relationship. Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 4 / 33 Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 5 / 33 Correlation and Independence Given random variables X and Y X and Y are independent CovX, Y ρx, Y 0 Example Let X {, 0, } with equal probability and Y X. Clearly X and Y are not independent random variables. CovX, Y ρx, Y 0 X and Y are independent CovX, Y 0 is necessary but not sufficient for independence! Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 6 / 33 Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 7 / 33

3 Example - Linear Dependence Let X Unif0, and Y X ax + b for constants a and b. Find CovX, Y and ρx, Y Multinomial Distribution Let X, X,, X k be the k random variables that reflect the number of outcomes belonging to categories,..., k in n trials with the probability of success for category i being p i, X,, X k Multinomn, p,, p k PX x,, X k x k f x,, x k n, p,, p k where n! x! x k! px px k k k k x i n and p i i i EX i np i VarX i np i p i CovX i, X j np i p j Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 8 / 33 Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 9 / 33 Example - Covariance of Multinomial Distribution Marginal distribution of X i - consider category i a success and all other categories to be a failure, therefore in the n trials there are X i successes and n X i failures with the probability of success is p i and failure is p i which means X i has a binomial distribution X i Binomn, p i Similarly, the distribution of X i + X j will also follow a Binomial distribution with probability of success p i + p j. X i + X j Binomn, p i + p j General Bivariate Normal Let Z, Z N 0,, which we will use to build a general bivariate normal distribution. f z, z π exp ] z + z We want to transform these unit normal distributions to have the follow parameters: µ X, µ Y,, σ Y, ρ X µ X + Z Y µ Y + σ Y ρ Z + ρ Z Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 0 / 33 Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 / 33

4 General Bivariate Normal - Marginals First, lets examine the marginal distributions of X and Y, General Bivariate Normal - Cov/Corr Second, we can find CovX, Y and ρx, Y Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 / 33 Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 3 / 33 General Bivariate Normal - RNG Consequently, if we want to generate a Bivariate Normal random variable with X N µ X, σx and Y N µ Y, σy where CorrX, Y ρ we can generate two independent unit normals Z and Z and use the transformation: X µ X + Z Y µ Y + σ Y ρz + ρ Z We can also use this result to find the joint density of the Bivariate Normal using a d change of variables. Multivariate Change of Variables Let X,..., X n have a continuous joint distribution with pdf f defined of S. We can define n new random variables Y,..., Y n as follows: Y r X,..., X n Y n r n X,..., X n If we assume that the n functions r,..., r n define a one-to-one differentiable transformation from S to T then let the inverse of this transformation be x s y,..., y n x n s n y,..., y n Then the joint pdf g of Y,..., Y n is { f s,..., s n J for y,..., y n T gy,..., y n 0 otherwise Where s y J det s n y s y n s n y n Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 4 / 33 Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 5 / 33

5 General Bivariate Normal - Density General Bivariate Normal - Density The first thing we need to find are the inverses of the transformation. If X r z, z and Y r z, z we need to find functions Z and Z such that Z s X, Y and Z s X, Y. Next we calculate the Jacobian, J det s x s x s y s y ] det 0 ρ ρ σ Y ρ ] σ Y ρ Therefore, Y µ Y σ Y X Z + µ X Z X µ X Y σ Y ρz + ρ Z ] + µ Y ρ X µ X + ρ Z Y µy Z ρ σ Y ρ X µ X s x, y x µ X y µy s x, y ρ σ Y ] ρ x µ X ] The joint density of X and Y is then given by f x, y f z, z J π exp z + z ] J π σ Y ρ exp π σ Y ρ exp ] z + z x µx + y µy ρ σ Y x µx π σ Y ρ exp / ρ σ X + y µ Y σ Y ρ x µ ]] X ρ x µ ] X y µ Y σ Y Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 6 / 33 Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 7 / 33 General Bivariate Normal - Density Matrix Notation Obviously, the density for the Bivariate Normal is ugly, and it only gets worse when we consider higher dimensional joint densities of normals. We can write the density in a more compact form using matrix notation, x µx σ x µ Σ X ρ σ Y y ρ σ Y µ Y f x π det Σ / exp ] x µt Σ x µ We can confirm our results by checking the value of det Σ / and x µ T Σ x µ for the bivariate case. det Σ / σx σ Y ρ σx / σ Y σ Y ρ / σ Y General Bivariate Normal - Density Matrix Notation Recall for a matrix, Then, a b A c d A det A d b c a x µ T Σ x µ T x µx σ σx Y σ Y ρ y µ y ρ σ Y σ X σ Y ρ ad bc ρ σ Y σ X σ Y x µ X ρ σ Y y µ Y ρ σ Y x µ X + σ X y µ Y d b c a x µx y µ y T x µx y µ y σ σx σ Y ρ Y x µ X ρ σ Y x µ X y µ Y + σx y µ Y x µ X ρ σ X ρ x µ X y µ Y + y µ Y σ Y σy Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 8 / 33 Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 9 / 33

6 General Bivariate Normal - Examples General Bivariate Normal - Examples X N 0,, Y N 0, X N 0,, Y N 0, X N 0,, Y N 0, X N 0,, Y N 0, X N 0,, Y N 0, X N 0,, Y N 0, ρ 0 ρ 0 ρ 0 ρ 0.5 ρ 0.5 ρ 0.75 Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 0 / 33 Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 / 33 General Bivariate Normal - Examples General Bivariate Normal - Examples X N 0,, Y N 0, X N 0,, Y N 0, X N 0,, Y N 0, X N 0,, Y N 0, X N 0,, Y N 0, X N 0,, Y N 0, ρ 0.5 ρ 0.5 ρ 0.75 ρ 0.75 ρ 0.75 ρ 0.75 Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 / 33 Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 3 / 33

7 Multivariate Normal Distribution Matrix notation allows us to easily express the density of the multivariate normal distribution for an arbitrary number of dimensions. We express the k-dimensional multivariate normal distribution as follows, X N k µ, Σ where µ is the k column vector of means and Σ is the k k covariance matrix where {Σ} i,j CovX i, X j. The density of the distribution is f x π k/ det Σ / exp ] x µt Σ x µ Multivariate Normal Distribution - Cholesky In the bivariate case, we had a nice transformation such that we can generate two independent unit normal values and transform them into a sample from an arbitrary bivariate normal distribution. There is a similar method for the multivariate normal distribution that takes advantage of the Cholesky decomposition of the covariance matrix. The Cholesky decomposition is defined for a symmetric, positive definite matrix X as L CholX where L is a lower triangular matrix such that LL T X. Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 4 / 33 Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 5 / 33 Multivariate Normal Distribution - RNG Let Z,..., Z k N 0, and Z Z,..., Z k T then µ + CholΣZ N k µ, Σ this is offered without proof in the general k-dimensional case but we can check that this results in the same transformation we started with in the bivariate case and should justify how we knew to use that particular transformation. Cholesky and the Bivariate Transformation We need to find the Cholesky decomposition of Σ for the general bivariate case where σ Σ X ρ σ Y ρ σ Y σy We need to solve the following for a, b, c a 0 b c a b 0 c a ab ab b + c σ X ρ σ Y ρ σ Y σy This gives us three unique equations and three unknowns to solve for, a σ X ab ρ σ Y b + c σ Y a b ρ σ Y /a ρσ Y c σy b σ Y ρ / Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 6 / 33 Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 7 / 33

8 Cholesky and the Bivariate Transformation Let Z, Z N 0, then X Y µ + CholΣZ µx µ Y µx µ Y σx ρσ Y σ Y ρ / Z Z ρσ Y Z + σ Y ρ / Z Z Conditional Expectation of the Bivariate Normal Using X µ X + Z and Y µ Y + σ Y ρz + ρ / Z ] where Z, Z N 0, we can find EY X. X µ X + Z Y µ Y + σ Y ρz + ρ / Z ] Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 8 / 33 Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 9 / 33 Conditional Variance of the Bivariate Normal Using X µ X + Z and Y µ Y + σ Y ρz + ρ / Z ] where Z, Z N 0, we can find VarY X. Example - Husbands and Wives Example 5.0.6, degroot Suppose that the heights of married couples can be explained by a bivariate normal distribution. If the wives have a mean heigh of 66.8 inches and a standard deviation of inches while the heights of the husbands have a mean of 70 inches and a standard deviation of inches. The correlation between the heights is What is the probability that for a randomly selected couple the wife is taller than her husband? Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 30 / 33 Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 3 / 33

9 Example - Conditionals Example - Conditionals, cont. Suppose that X and X have a bivariate normal distribution where EX X X, EX X X, and VarX X Find EX, VarX, EX, VarX, and ρx, X. Based on the given information we know the following, EX X µ ρ σ σ µ + ρ σ σ X X EX X µ ρ σ σ µ + ρ σ σ X X VarX X ρ σ 3.64 From which we get the following set of equations i µ ρ σ σ µ 3.7 ii ρ σ σ 0.5 iii µ ρ σ σ µ 0.4 iv ρ σ σ 0.6 v ρ σ 3.64 Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 3 / 33 From ii and iv From v From ii From i and iii i µ ρ σ σ µ 3.7 ii ρ σ σ 0.5 iii µ ρ σ σ µ 0.4 iv ρ σ σ 0.6 v ρ σ 3.64 ρ σ σ ρ σ σ ρ 0.09 ρ ± σ 3.64/ ρ 4 σ 0.5σ ρ µ + 0.5µ µ + µ 0.4 µ 4 µ Sta30 / Mth 30 Colin Rundel Lecture 0 April, 0 33 / 33