Outline Properties of Covariance Quantifying Dependence Models for Joint Distributions Lab 4. Week 8 Jointly Distributed Random Variables Part II

Transcription

1 Week 8 Jointly Distributed Random Variables Part II

2 Week 8 Objectives 1 The connection between the covariance of two variables and the nature of their dependence is given. 2 Pearson s correlation coefficient is introduced as a quantification of linear dependence. 3 A hierarchical approach to building joint distribution models is presented. The bivariate normal distribution and the class of regression models are introduced in this context. 4 The relationship between covariance/correlation and the slope of the simple linear regression (slr) model is given. The formulas for the variance of sums, and the connection between covariance/correlation and the slope of the slr model are demonstrated in. The R command lm is introduced.

3 1 2 Monotone Dependence and Covariance Correlation as a measure of (linear) dependence 3 Hierarchical Models Regression Models 4 Properties of the Variance via Simulations Fitting Regression Models

4 Proposition 1. Cov(X, Y ) = Cov(Y, X). 2. Cov(X, X) = Var(X). 3. If X, Y are independent, then Cov(X, Y ) = Cov(aX + b, cy + d) = accov(x, Y ), for any real numbers a, b, c and d. ( m 5. Cov i=1 X i, ) n j=1 Y j = m n i=1 j=1 Cov ( ) X i, Y j.

5 Let X denote the number of short fiction books sold at the State College, PA, location of Barnes and Noble in a given week, and let Y denote the corresponding number sold on line from State College residents. It is given that Cov(X, Y ) = If (X i, Y i ), i = 1,..., 10, denote the sale numbers in 10 weeks, find Cov(X, Y ). Solution: Using parts 4 and 5 of the proposition we have Cov( i=1 X i, j=1 Y j ) = Cov( i=1 10 X i, Y j ) j=1 = i=1 j=1 Cov(X i, Y j ) = i=1 Cov(X i, Y i ) = 0.575, assuming sales in different weeks are independent

6 Zero Covariance Independence Example Find Cov(X, Y ), where X, Y have joint pmf given by y /3 0 1/3 x 0 0 1/3 1/3 1 1/3 0 1/3 2/3 1/3 1.0 Are X and Y independent? Is Cov(X, Y ) = 0? Solution: Since E(X) = 0, the computational formula gives Cov(X, Y ) = E(XY ). But E(XY ) = 0. (Why?) Thus, Cov(X, Y ) = 0 but X and Y are not independent (why?).

7 Example Consider the example with the car and home insurance deductibles, but suppose that the deductible amounts are given in cents. Find the covariance of the two deductibles. Solution: Let X, Y denote the deductibles of a randomly chosen home and car owner in cents. If X, Y denote the deductibles in dollars, then X = 100X, Y = 100Y. According to part 4 of the proposition, σ X Y = σ XY = 18, 750, 000.

8 Monotone Dependence and Covariance Correlation as a measure of (linear) dependence 1 2 Monotone Dependence and Covariance Correlation as a measure of (linear) dependence 3 Hierarchical Models Regression Models 4 Properties of the Variance via Simulations Fitting Regression Models

9 Positive and Negative Dependence Monotone Dependence and Covariance Correlation as a measure of (linear) dependence When two variables are not independent, it is of interest to qualify and quantify dependence. X, Y are positively dependent or positively correlated if large values of X are associated with large values of Y, and small values of X are associated with small values of Y. In the opposite case, X, Y are negatively dependent or negatively correlated. If the dependence is either positive or negative it is called monotone.

10 Monotone Dependence and Covariance Correlation as a measure of (linear) dependence Monotone dependence and covariance The monotone dependence is positive or negative, if and only if the covariance is positive or negative. To gain some intuition for that consider a population of N units, let (x 1, y 1 ), (x 2, y 2 ),..., (x N, y N ) denote the values of the bivariate characteristic for each of the N units, and let (X, Y ) denote the bivariate characteristic of a randomly selected unit. Then σ XY = 1 N N (x i µ X )(y i µ Y ), i=1 where µ X = 1 N N i=1 x i and µ Y = 1 N N i=1 y i are the marginal expected values of X and Y. Thus:

11 Intuition for Covariance - Continued Monotone Dependence and Covariance Correlation as a measure of (linear) dependence If X, Y are positively dependent (think of X = height and Y = weight), then (x i µ X )(y i µ Y ) will be mostly positive. If X, Y are negatively dependent (think of X = stress and Y = time to failure), then (x i µ X )(y i µ Y ) will be mostly negative. Therefore, σ XY will be positive or negative according to whether the dependence of X and Y is positive or negative.

12 Monotone Dependence and Covariance Correlation as a measure of (linear) dependence Caveat: This statement: The monotone dependence is positive, if and only if the covariance is positive SHOULD NOT be interpreted as: If the covariance is positive then the dependence is positive Positive covariance implies positive dependence only if we know that the dependence is monotone.

13 Monotone Dependence and Covariance Correlation as a measure of (linear) dependence Monotone dependence and regression function The dependence is monotone (positive or negative), if and only if the regression function, µ Y X (x) = E(Y X = x), is monotone (increasing or decreasing) in x. Example 1 X = height and Y = weight of a randomly selected adult male, are positively dependent. It is also true that µ Y X (1.82) < µ Y X (1.90). 2 X = stress applied and Y = time to failure, are negatively dependent. It is also true that µ Y X (10) > µ Y X (20).

14 Monotone Dependence and Covariance Correlation as a measure of (linear) dependence 1 2 Monotone Dependence and Covariance Correlation as a measure of (linear) dependence 3 Hierarchical Models Regression Models 4 Properties of the Variance via Simulations Fitting Regression Models

15 Monotone Dependence and Covariance Correlation as a measure of (linear) dependence Quantification of dependence should be unit-free! In particular, a quantification of dependence should not depend on the units of measurement. For example, a quantification of the dependence between height and weight should not depend on whether the units are meters and kilograms or feet and pounds The scale-dependence of the covariance, implied by the property Cov(aX, by ) = abcov(x, Y ) makes it unsuitable as a measure of dependence.

16 Monotone Dependence and Covariance Correlation as a measure of (linear) dependence Pearson s Linear Correlation Coefficient Pearson s linear correlation coefficient, or simply correlation coefficient, between X and Y is defined as ρ XY = Corr(X, Y ) = σ XY σ X σ Y. Corr(X, Y ) can also be thought of as the covariance between the scaled versions of X and Y, i.e., ( X Corr(X, Y ) = Cov, Y ) σ X σ Y Since scaled versions of r.v.s are scale-free, so is ρ XY.

17 Monotone Dependence and Covariance Correlation as a measure of (linear) dependence Proposition (Properties of Correlation) 1 If ac > 0, then Corr(aX + b, cy + d) = Corr(X, Y ). 2 1 ρ XY 1. 3 If X, Y are independent, then ρ XY = 0. 4 ρ XY = 1 or 1 if and only if Y = ax + b, for some constants a, b. Assuming there is a linear trend, it possible to associate values of ρ XY with the degree of concentration of the (X, Y )-values around a line, as illustrated by the following figures.

18 Monotone Dependence and Covariance Correlation as a measure of (linear) dependence Correlation = 0.2 Correlation = 0.45 Correlation = 0.65 Correlation = 0.9 Figure: Correlations and degrees of concentration around a line

19 Monotone Dependence and Covariance Correlation as a measure of (linear) dependence Example Find the correlation coefficient of the deductibles in car and home insurance of a randomly chosen car and home owner, when the deductibles are expressed a) in dollars, and b) in cents. Solution: If X, Y denote the deductibles in dollars, we saw that σ XY = Omitting the details, it can be found that σ X = 75 and σ Y = Thus, ρ XY = = Next, the deductibles expressed in cents are (X, Y ) = (100X, 100Y ). According to the proposition, ρ X Y = ρ XY

20 Linear vs Nonlinear Dependence Monotone Dependence and Covariance Correlation as a measure of (linear) dependence Corr(X, Y ) quantifies accurately the dependence between X and Y provided there is a linear trend. If all (X, Y )-values fall on a straight line (perfect dependence) then ρ XY = ±1 depending on whether the line has positive or negative slope. As ρ XY gets smaller, the concentration of the (X, Y )-values around the line decreases. However, the quantification of dependence is not accurate if the trend is nonlinear. It is possible to have perfect dependence (i.e., knowing one amounts to knowing the other) but ρ XY ±1.

21 Example Let X U(0, 1), and Y = X 2. Find ρ XY. Solution: We have σ XY = E(XY ) E(X)E(Y ) Monotone Dependence and Covariance Correlation as a measure of (linear) dependence = E(X 3 ) E(X)E(X 2 ) = = Omitting calculations, σ X = 1 12, σ Y = ρ XY = = = 2 3. Thus, 5 A similar set of calculations reveals that with X as before and Y = X 4, ρ XY =

22 Monotone Dependence and Covariance Correlation as a measure of (linear) dependence Example Let X U( 1, 1), and Y = X 2. Are X and Y independent? Is the dependence of X and Y monotone? Is ρ XY = 0? Solution: X and Y are not independent. In fact, they are dependent quite strongly since knowing X implies that Y is also known. Moreover, their dependence is not monotone (why?). Since f X (x) = 0.5 for 1 < x < 1, and 0 otherwise, E(X) = 0, and E(XY ) = E(X 3 ) = 0. Thus, Cov(X, Y ) = E(XY ) E(X)E(Y ) = 0. Hence, ρ XY = 0.

23 Uncorrelated Variables Monotone Dependence and Covariance Correlation as a measure of (linear) dependence Two variables having zero correlation are called uncorrelated. Independent variables are uncorrelated. Uncorrelated variables are not necessarily independent.

24 Monotone Dependence and Covariance Correlation as a measure of (linear) dependence Sample versions of covariance and correlation If (X 1, Y 1 ),..., (X n, Y n ) is a sample from the bivariate distribution of (X, Y ), the sample covariance, S X,Y, and sample correlation coefficient, r X,Y, are defined as S X,Y = 1 n 1 r X,Y = S X,Y S X S Y n (X i X)(Y i Y ) i=1 Sample versions of covariance and correlation coefficient

25 Monotone Dependence and Covariance Correlation as a measure of (linear) dependence Computational formula: [ n ( n ( n )] S X,Y = 1 X i Y i 1 X i) Y i. n 1 n i=1 i=1 i=1 R commands: R commands for covariance and correlation cov(x,y) gives S X,Y cor(x,y) gives r X,Y

26 Hierarchical Models Regression Models 1 2 Monotone Dependence and Covariance Correlation as a measure of (linear) dependence 3 Hierarchical Models Regression Models 4 Properties of the Variance via Simulations Fitting Regression Models

27 Hierarchical Models Regression Models Hierarchical models are based on the Multiplication Rules p(x, y) = p Y X=x (y)p X (x), for the joint pmf f (x, y) = f Y X=x (y)f X (x), for the joint pdf. They specify the joint distribution of X, Y by first specifying the conditional distribution of Y given X = x, and then specifying the marginal distribution of X. The hierarchical method of modeling yields a very rich and flexible class of joint distributions.

28 Hierarchical Models Regression Models Example Let X be the number of eggs an insect lays and Y the number of eggs that survive. Model the joint pmf of X and Y. Solution: With the principle of hierarchical modeling, we need to specify the conditional pmf of Y given X = x, and the marginal pmf of X. One possibile specification is Y X = x Bin(x, p), X Poisson(λ), which leads to the following joint pmf of X and Y. p(x, y) = p Y X=x (y)p X (x) = ( x )p y (1 p) x y e λ λ x y x!

29 Hierarchical Models Regression Models In hierarchical models that specify p Y X=x (y) and p X (x) the marginal expected value of Y is most easily found through the Law of Total Expectation (LTE). Find E(Y ) if X and Y have the joint distribution of the previous example, i.e., Y X = x Bin(x, p), X Poisson(λ). Solution: Since Y X Bin(X, p), the regression function of Y on X is E(Y X) = Xp. Thus, according to the LTE, E(Y ) = E[E(Y X)] = E[Xp] = E[X]p = λp.

30 Hierarchical Models Regression Models The Bivariate Normal Distribution The bivariate normal distribution arises as a hierarchical model. Definition X and Y are said to have a bivariate normal distribution if ( ) Y X = x N α 1 + β 1 x, σε 2, and X N(µ X, σx 2 ). Commentaries: 1. The regression function of Y on X is linear in x: µ Y X (x) = α 1 + β 1 x.

31 Hierarchical Models Regression Models Commentaries (Continued): 2. The marginal distribution of Y is normal with E(Y ) = E[E(Y X)] = E[α 1 + β 1 X] = α 1 + β 1 µ X. 3. σ 2 ε is the conditional variance of Y given X and is called error variance. 4. A more common form of the bivariate pdf involves µ X and µ Y, σ 2 X, σ2 Y and ρ XY ; see relation (4.6.13), p. 205, in the book. 5. Any linear combination, a 1 X + a 2 Y, of X and Y has a normal distribution. Plots of bivariate normal pdfs are given in the next slides.

32 Hierarchical Models Regression Models Joint PMF of two independent N(0,1) RVs 0.15 z x x

33 Hierarchical Models Regression Models Joint PMF of two N(0,1) RVs with ρ = z x x Week 8 Jointly3 Distributed Random Variables Part II

34 Hierarchical Models Regression Models 1 2 Monotone Dependence and Covariance Correlation as a measure of (linear) dependence 3 Hierarchical Models Regression Models 4 Properties of the Variance via Simulations Fitting Regression Models

35 Hierarchical Models Regression Models Regression models focus primarily on the regression function of Y on X, and are commonly written as Y = µ Y X (x) + ε, where ε is called the (intrinsic) error variable. Y is called the response variable and X is interchangeably referred to as covariate, or independent variable, or predictor, or explanatory variable. The marginal distribution of X, which is of little interest in such studies, is left unspecified. (Thus regression models do not specify the joint distribution of X and Y.)

36 Hierarchical Models Regression Models The Simple Linear Regression Model The simple linear regression model specifies that the regression function is linear in x, i.e., µ Y X (x) = α 1 + β 1 x, and that Var(Y X = x) is the same for all values x. An alternative expression for the regression line is µ Y X (x) = β 0 + β 1 (x µ X ). With this expression, E(Y ) = β 0.

37 Hierarchical Models Regression Models The following picture illustrates the meaning of the slope of the regression line. Figure: Illustration of Regression Parameters

38 Hierarchical Models Regression Models Proposition In the simple linear regression model 1 The intrinsic error variable, ɛ, has zero mean and is uncorrelated from the explanatory variable, X. 2 The slope is related to the correlation by β 1 = σ XY σ 2 X = ρ XY σ Y σ X. 3 An additional relationship is σ 2 Y = σ2 ε + β 2 1 σ2 X.

39 Hierarchical Models Regression Models Example Suppose Y = 5 2x + ε, and let σ ɛ = 4, µ X = 7 and σ X = 3. a) Find σy 2, and ρ XY. b) Find µ Y. Solution: For a) use the formulas in the above proposition: σy 2 = ( 2) = 52 ρ XY = β 1 σ X /σ Y = 2 3/ 52 = For b) use the Law of Total Expectation: E(Y ) = E[E(Y X)] = E[5 2X] = 5 2E(X) = 9

40 Hierarchical Models Regression Models The normal simple linear regression model specifies, in addition, that the intrinsic error variable is normally distributed, i.e., Y = β 0 + β 1 (X µ X ) + ε, ε N(0, σ 2 ε ). An alternative expression of the normal simple linear regression model is Y X = x N(β 0 + β 1 (x µ X ), σ 2 ε ). In the above, β 0 + β 1 (x µ X ) can also be replaced by α 1 + β 1 x.

41 Hierarchical Models Regression Models Figure: Illustration of the Normal Simple Linear Regression Model

42 Hierarchical Models Regression Models Example Suppose Y = 5 2x + ε, and let ε N(0, σ 2 ε ) where σ ɛ = 4. Let Y 1, Y 2 be observations taken at X = 1 and X = 2, respectively. a) Find the 95th percentile of Y 1 and Y 2. b) Find P(Y 1 > Y 2 ). Solution: a) Y 1 N(3, 4 2 ), so its 95th percentile is = Y 2 N(1, 4 2 ), so its 95th percentile is = b) Y 1 Y 2 N(2, 32) (why?). Thus, ( ) 2 P(Y 1 > Y 2 ) = P(Y 1 Y 2 > 0) = 1 Φ =

43 Properties of the Variance via Simulations Fitting Regression Models 1 2 Monotone Dependence and Covariance Correlation as a measure of (linear) dependence 3 Hierarchical Models Regression Models 4 Properties of the Variance via Simulations Fitting Regression Models

44 Proof of σ 2 X+Y = σ2 X Y = σ2 X + σ2 Y Properties of the Variance via Simulations Fitting Regression Models for X, Y indep. Using normal samples, try: x=rnorm(50000); y=rnorm(50000,10,2); var(x+y); var(x y) Can you guess the output from these commands? The X, Y can have very different distributions. Try: z=rpois(50000,1); var(x+z); var(x-z) Can you guess the output from these commands? Guess the output from the commands: cov(x,y); cov(x,z); cov(y,z) Hint: The x, y and z samples are independently generated.

45 Properties of the Variance via Simulations Fitting Regression Models Proof of σ 2 X+Y = σ2 X + σ2 Y + 2σ X,Y We will generate a sample (x i, y i ), i = 1,..., 50000, from a bivariate population with Var(X) = 1, Var(Y ) = 2 and Cov(X, Y ) = 1, and will confirm the formula for the Var(X + Y ) through the sample variance of x i + y i, i = 1,..., 50000). Let X, V be iid N(0,1), and set Y = X + V. Then Var(X) = 1, Var(Y ) = 2, Cov(X, Y ) = 1. (Why?) Generate a sample (x i, y i ), i = 1,..., 50000, from the bivariate population of (X, Y ) as follows: x=rnorm(50000); v=rnorm(50000); y=x+v Try: var(x); var(y); cov(x,y); var(x+y); var(x)+var(y)+2*cov(x,y)

46 Properties of the Variance via Simulations Fitting Regression Models 1 2 Monotone Dependence and Covariance Correlation as a measure of (linear) dependence 3 Hierarchical Models Regression Models 4 Properties of the Variance via Simulations Fitting Regression Models

47 Properties of the Variance via Simulations Fitting Regression Models If y and x contain the response and predictor values, the model Y = α 1 + β 1 X + ɛ is fitted (i.e., the coefficients α 1 and β 1 are estimated) by Can also do out=lm(y x); coef(out) lm(y x ) or out=lm(y x); out Alternatively, since β 1 = Cov(X, Y )/Var(X), β 1 and α 1 can be estimated by cov(x,y)/var(x); mean(y)-(cov(x,y)/var(x))*mean(x)

48 Illustration with Simulated Data Properties of the Variance via Simulations Fitting Regression Models e=rnorm(50,0,5); x=runif(50,0,10); y=25 3.4*x+e beta1= cov(x,y)/var(x); alpha1 = mean(y)-beta1*mean(x) Do a scatterplot and find the estimated regression line: plot(x,y); out=lm(y x); out Try plot(x,y); abline(coef(out),col=2) Can add the true regression line on the plot by lines(x,25-3.4*x,type= l,col=4)

49 Ramifications of lm Properties of the Variance via Simulations Fitting Regression Models To fit the model Y = β 0 + β 1 (X µ X ) + ɛ, do xc=x-mean(x); lm(y xc) [Note: now β 0 = Y!] lm(y x) is equivalent to lm(y 1 + x). lm(y 1) fits the model Y = β 0 + ɛ and returns β 0 = Y. [However, it contains more information than mean(y)!] Polynomial models can also be fitted. For example x2=xˆ2; lm(y x + x2) (or lm(y 1 + x + x2)) fits the model Y = α 1 + β 1 X + β 2 X 2 + ɛ.

50 The Use of Transformations Properties of the Variance via Simulations Fitting Regression Models Do a scatterplot for the data, given in Problem 13, Section 6.3, on the diameter and age of sugar maple trees : da=read.table( TreeAgeDiamSugarMaple.txt, header=t) x=da$diamet; y=da$age; plot(x,y) Are there any apparent violations of the SLR model assumptions? Take logarithm of the data and do a scatterplot for the transformed data: x1=log(x); y1=log(y); plot(x1,y1) Are the apparent violations less apparent?