I L L I N O I S UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN

Transcription

1 Linear Combinations of Variables Edps/Soc 584 and Psych 594 Applied Multivariate Statistics Carolyn J Anderson Department of Educational Psychology I L L I N O I S UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN c Board of Trustees, University of Illinois Linear Combinations of Variables Slide of 24

2 Outline Two sets of Variables (and more linear algebra) Reading: Johnson & Wichern pages 75 75, Linear Combinations of Variables Slide 2 of 24

3 A major tool for data reduction & insight into why reject H o regarding µs Definition: A Linear Combination of p (random) variables X,X 2,,X p is a X +a 2 X 2 + +a p X p Let a (a,a 2,,a p ) and X X X 2 X p A linear combination in terms of vector operations is a X p The linear combination a X is itself a random variable Linear Combinations of Variables Slide 3 of 24

4 Mean of Linear Combination The mean of a X is Mean of Linear Combination Variance of a Linear Combination Results if we add a constant Multiple Linear Combinations Mean and Covariance forq Kind of an Kind of an where E(a X) E(a X +a 2 X 2 + +a p X p ) E(a X )+E(a 2 X 2 )+ +E(a p X p ) a E(X )+a 2 E(X 2 )+ +a p E(X p ) a µ +a 2 µ 2 + +a p µ p a µ µ µ µ 2 µ p Know this: E(a X) a µ Linear Combinations of Variables Slide 4 of 24

5 Variance of a Linear Combination Mean of Linear Combination Variance of a Linear Combination Results if we add a constant Multiple Linear Combinations Mean and Covariance forq Kind of an Kind of an The variance of a X is var(a X) var(a X +a 2 X 2 + +a p X p ) a 2 var(x )+a 2 2var(X 2 )+ +a 2 pvar(x p ) +a a 2 cov(x,x 2 )+ +a p a p cov(x p,x p ) p ia 2 iσ ii + a i a k σ ik a i a k σ ik i i i k More compactly, for p 2, var(a X) a 2 σ +2a a 2 σ 2 +a 2 2σ 22 ( )( σ σ 2 (a,a 2 ) σ 2 σ 22 a Σa k k a a 2 ) A Quadratic Form For any p, var(a X) a Σa Linear Combinations of Variables Slide 5 of 24

6 Results if we add a constant If we add a constant to the linear combination, Mean of Linear Combination Variance of a Linear Combination Results if we add a constant Multiple Linear Combinations Mean and Covariance forq Kind of an Kind of an a X +a 2 X 2 + +a p X p +c a X +c Then Mean: Changes location E(a X +c) a E(X)+c a µ+c Variance: No chance var(a X +c) a Σa Linear Combinations of Variables Slide 6 of 24

7 Multiple Linear Combinations Sometimes one just isn t enough Suppose that there are q linear combinations of the p random variables, Mean of Linear Combination Variance of a Linear Combination Results if we add a constant Multiple Linear Combinations Mean and Covariance forq Kind of an Kind of an y y 2 y q In matrix form, y q y y 2 a X +a 2 X 2 + +a p X p a 2 X +a 22 X 2 + +a 2p X p a q X +a q2 X 2 + +a qp X p a a 2 a 2 a p a 22 a 2p X X 2 A q px p y q a qp a q2 a qp X p No restriction on q, but usually q p Linear Combinations of Variables Slide 7 of 24

8 Mean and Covariance for q y q A q p X p The Mean vector for y, Mean of Linear Combination Variance of a Linear Combination Results if we add a constant Multiple Linear Combinations Mean and Covariance forq Kind of an Kind of an µy E(y) E(AX) AE(X) Aµx The Covariance matrix for y, Σy cov(y) cov(ax) AΣA These are 2 very important results! µy Aµx Σy AΣxA Linear Combinations of Variables Slide 8 of 24

9 Mean of Linear Combination Variance of a Linear Combination Results if we add a constant Multiple Linear Combinations Mean and Covariance forq Kind of an Kind of an Kind of an Let X be a random vector X X X 2 X 3 with µx Suppose we re interested in µ µ 2 µ 3 The average: Y (/3)(X +X 2 +X 3 ) Contrast: Y 2 (X X 2 ) µ y µ y µ y2 /3 /3 /3 0 /3 /3 /3 0 and Σ E µ µ 2 µ 3 X X 2 X 3 σ σ 2 σ 3 σ 2 σ 22 σ 23 σ 3 σ 32 σ 33 3 (µ +µ 2 +µ 3 ) µ µ 2 Linear Combinations of Variables Slide 9 of 24

10 Mean of Linear Combination Variance of a Linear Combination Results if we add a constant Multiple Linear Combinations Mean and Covariance forq Kind of an Kind of an Kind of an and the covariance matrix of Y, Σy AΣxA /3 /3 /3 0 σ σ 2 σ 3 σ 2 σ 22 σ 23 σ 3 σ 32 σ 33 (σ 3 +σ 2 +σ 3 ) (σ 3 2 +σ 22 +σ 32 ) (σ 3 3 +σ 23 +σ 33 ) (σ σ 2 ) (σ 2 σ 22 ) (σ 3 σ 23 ) 9 (σ +σ 22 +σ 33 +2(σ 2 +σ 3 +σ 23 )) /3 /3 /3 0 3 (σ σ 22 +σ 3 σ 23 ) 3 (σ σ 22 +σ 3 σ 23 ) (σ +σ 22 2σ 2 ) /3 /3 /3 0 Linear Combinations of Variables Slide 0 of 24

11 of µ y st just one linear transformation (combination, composite, etc) Let a (a,a 2,,a p ) and x (x,x 2,,x p ) and ofµy of var(y) ofȳ and var(y) Sample Covariance y a x p a i x i i If we have n cases, the sample observation of the linear transformation for the j th case (individual) is y j, a x i a x j +a 2 x j2 + +px jp for j,,n Sample Mean of y : ȳ n (y +y 2 ++y n ) n (a x +a x 2 + +a x n ) a n n j x j a x Linear Combinations of Variables Slide of 24

12 of var(y) Sample variance of y: ofµy of var(y) ofȳ and var(y) Sample Covariance var(y) a n n n n a Sa n (y j ȳ) 2 j n (a x j a x) 2 j n (a x j a x) (a x j a x) }{{} note: a (x j x)(x j x) a n a (x j x)(x j x) a j j n n (x j x)(x j x) a j Linear Combinations of Variables Slide 2 of 24

13 of ȳ and var(y) a x is the sample analog to a µ ofµy of var(y) ofȳ and var(y) Sample Covariance a Sa is the sample analog to a Σa Let s add a 2 nd linear combination of the vector X: p Z b X +b 2 X 2 + +b p X p b X b i X i The j th sample observation on variable Z is i z j b x j b x j +b 2 x j2 + +b p x jp Applying above results gives us Sample mean: z b x Sample variance: s 2 z b Sb What s the covariance between y and z? Linear Combinations of Variables Slide 3 of 24

14 Sample Covariance Between y and z ofµy of var(y) ofȳ and var(y) Sample Covariance cov(y, z) a n n n (y j ȳ)(z j z) j n (a x j a x)(b x j b x) j n a a Sb n n (x j x)(x j x) b j n (x j x)(x j x) b j Linear Combinations of Variables Slide 4 of 24

15 with Numbers Suppose we have n 3 and p 3, and the data matrix is 2 4 X with Numbers Observations on new variables Mean and Covariance forx s Sample Variances & Covariance for x s Sample Covariance Matrices Note regarding var(y) s and cov(y i,y 2 ) And we want to find the two linear combinations b x b x +b 2 x 2 +b 3 x 3 where b (,,) and x is a row of X written as a column vector, and c x c x +c 2 x 2 +c 3 x 3 where c ( 2,,) Linear Combinations of Variables Slide 5 of 24

16 with Numbers Observations on new variables Mean and Covariance forx s Sample Variances & Covariance for x s Sample Covariance Matrices Note regarding var(y) s and cov(y i,y 2 ) Observations on new variables Observations on these two new variables equal b X (,,) (7,7,8) c X ( 2,,) In terms of single matrix formula, AX b c X (4,4,5) Linear Combinations of Variables Slide 6 of 24

17 with Numbers Observations on new variables Mean and Covariance forx s Sample Variances & Covariance for x s Sample Covariance Matrices Note regarding var(y) s and cov(y i,y 2 ) Mean and Covariance for x s The means on the original variables are x, x 2 3, and x Sample means of new variables computed using the scalar formula with the observations on the new variables (on the left) and matrix formula using the observations on the old variables (on the right): ( 3 (7+7+8) 3 (4+4+5) ) ( ) A x ( 2 ) ( Linear Combinations of Variables Slide 7 of 24

18 Sample Variances & Covariance for x s Sample variances of the old variables: s 0 s 22 2 s 33 2 [ (2 3) 2 +(3 3) 2 +(4 3) 2] [ (4 333) 2 +(3 333) 2 +(3 333) 2] 3 with Numbers Observations on new variables Mean and Covariance forx s Sample Variances & Covariance for x s Sample Covariance Matrices Note regarding var(y) s and cov(y i,y 2 ) Sample covariances between the old variables: s 2 2 [0(2 3)+0(3 3)+0(4 3)] 0 s 3 2 [0(4 333)+0(3 333)+0(3 333)] 0 s 23 2 [(2 3)(4 333)+(3 3)(3 333)+(4 3)(3 333)] 2 ( ) 2 Linear Combinations of Variables Slide 8 of 24

19 Sample Covariance Matrices with Numbers Observations on new variables Mean and Covariance forx s Sample Variances & Covariance for x s Sample Covariance Matrices Note regarding var(y) s and cov(y i,y 2 ) The sample variance-covariance matrix of the x s variables is S For the new variables (the y s): ASA ( ( ( ) 3 3 ) ) 2 2 Linear Combinations of Variables Slide 9 of 24

20 Note regarding var(y) s and cov(y i,y 2 ) The results on the previous slide are the same if we had used scalar formulas with the values of the y j and y j2 : s 2 [(7 733)2 +(7 733) 2 +(8 733) 2 ] 3 with Numbers Observations on new variables Mean and Covariance forx s Sample Variances & Covariance for x s Sample Covariance Matrices Note regarding var(y) s and cov(y i,y 2 ) s 22 2 [(4 433)2 +(4 433) 2 +(5 433) 2 ] 3 s 2 2 [(7 733)(4 433)+(7 733)(4 433) +(8 733)(5 433) 3 It was just by coincidence that all the values are /3 bad choice of data on my part Linear Combinations of Variables Slide 20 of 24

21 of Linear Combinations If we have one set of q linear combinations and another set of r linear combinations of random variables X, then Y q A q p X p Z r B r p X P E(Y ) Aµx E(Z) Bµx of Linear Combinations More Linear Algebra: Partitioned Matrices Partitioned mean and covariance matrices Operations on Partitioned Vectors & Matrices Σy AΣxA Σz BΣxB Σyz AΣxB This is just an application of what we already did Linear Combinations of Variables Slide 2 of 24

22 More Linear Algebra: Partitioned Matrices of Linear Combinations More Linear Algebra: Partitioned Matrices Partitioned mean and covariance matrices Operations on Partitioned Vectors & Matrices Let matrix A k be Partitioned into two parts as follows A a a 2 a p a q a q2 a qp a q+, a q+,2 a q+,p a q+r, a q+r,2 a q+r,p Y Y Y q Y q+ Y q+r Y,(q ) Y 2,(r ) A,(q p) A 2,(r p) Y (q ) Z (r ) A q p B r p Linear Combinations of Variables Slide 22 of 24

23 Partitioned mean and covariance matrices µ (q+r) µ µ q µ q+, µ q+r µ,(q ) µ 2,(r ) µ Y,(q ) µ Z,(r ) of Linear Combinations More Linear Algebra: Partitioned Matrices Partitioned mean and covariance matrices Operations on Partitioned Vectors & Matrices Σ (q+r),(q+r) σ σ 2 σ,q+r σ q σ q2 σ q,q+r σ q+, σ q+,2 σ q+,q+r Σ yy,(q q) Σ zy,(r q) Σ yz,(q r) Σ zz,(r r) σ q+r, σ q+r,2 σ q+r,q+r where Σ yz Σ zy Linear Combinations of Variables Slide 23 of 24

24 Operations on Partitioned Vectors & Matrices Same; that is, treat each sub-matrix (vector) as an element of a matrix (vector) For example of Linear Combinations More Linear Algebra: Partitioned Matrices Partitioned mean and covariance matrices Operations on Partitioned Vectors & Matrices A q p B r p A q p B r p A q p B r p Σ p p ( X p µ x(p ) A p q (AX) q (BX) r (Aµ) q (Bµ) r B p q ) Y q Z r µ y,(q ) µ z,(r ) AΣA AΣB BΣA BΣB Σ yy Σ zy Σ yz Σ zz which gives us the results for linear combinations for 2 sets of variables Linear Combinations of Variables Slide 24 of 24