I L L I N O I S UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN
|
|
- Joanna Todd
- 6 years ago
- Views:
Transcription
1 Canonical Edps/Soc 584 and Psych 594 Applied Multivariate Statistics Carolyn J. Anderson Department of Educational Psychology I L L I N O I S UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN Canonical Slide 1 of 52
2 Outline Introduction Canonical and Tests on & Covariance Matrices (i.e., multiple linear combinations) Matrix Describing the relationship between sets (i.e., specific questions asked and answer in canonical analysis) Some ideas on dealing with More than two sets.. Reading: J&W Reference: Morrison (2005) Canonical Slide 2 of 52
3 Introduction We ended MANOVA talking about checking hypotheses and also made assumptions about equality of covariance matrices in discriminant analysis. There are other tests on covariance matrices that are interesting. For example Single sample: H o : Σ = Σ o (where Σ o is some specified matrix) versus H a : Σ Σ o. Tests for special structures, e.g., 1 ρ ρ ρ 1 ρ H o : Σ = σ ρ ρ 1 When might you want to test this one? Canonical Slide 3 of 52
4 More Tests Population Matrix 1 ρ ρ ρ 1 ρ H o : R = ρ ρ 1 H o : Σ 1 = Σ 2 = = Σ K versus not all are equal. Simultaneously test equality of µ and Σ from K samples. Testing the independence of sets of variables, which in what Canonical analysis deals with. Partition the covariance matrix into two sets ( ) Σ 11 Σ 12 Σ = Σ 21 Σ 22 and test H o : Σ 12 = 0. Canonical Slide 4 of 52
5 Sets of Variables In Canonical correlation analysis, we re concerned with whether two sets of variables are related or not. For example: Teacher ratings: X 1 relaxed, X 2 (motivated), X 3 (organized) and Student Achievement: Y 1 (reading), Y 2 (language), Y 3 (math) Psychological health and Performance or Behavioral measures Job performance and Job satisfaction WAIS sub-tests (e.g., digitspan, vocabulary) and Various measures of experience (e.g., age, education, etc) Canonical Slide 5 of 52
6 Goal of Canonical (Due to Hotelling about 1935) Suppose you have (p+q) variables in a vector and partition it into two parts ( ) X (p+q) = X 1,(p 1) X 2,(q 1) with covariance matrix Σ, which has also been partitioned ( ) Σ 11 Σ 12 } p Σ = Σ 21 Σ 22 } q }{{}}{{} p q Note that Σ 12 = Σ 21. Goal: Determine the relationship between the two sets of variables X 1 and X 2. Canonical Slide 6 of 52
7 Goal continued What linear combination of X 1, i.e., ax 1 = a 1 X 11 + a 2 X a p X 1p, is most closely related to a linear combination of X 2, b X 2 = b 1 X 21 + b 2 X b p X 2p. We want to choose a p 1 and b q 1 to maximize the correlation ρ(a X 1, b X 2 ). These linear combinations are called canonical variates. Plan: 1. Determine whether X 1 and X 2 are related. 2. If related, find linear combinations that maximize the canonical correlation. Canonical Slide 7 of 52
8 Assumptions of Wilk s Test Wilk s Test Statistic Wilk s Test continued Example: WAIS and Age Example of Wilk s Test for Relationship Method II: Canonical Covariance matrix for C X The Idea Solution that Maximizes the The Test Example: Method II Example continued Finishing the Test & Finding a 1 and b 1 Finding a 1 and b 1 and Conclusion (So Far) The More Usual Scaling of U 1 and V 1 Our Example Two methods to test whether the two sets of variables are related. We ll start with Wilk s likelihood ratio test for the independence of several sets of variables. This test pertains to K set of variables measures on n individuals. The i th set of variables consists of p i variables. There are ( 2 K) = K!/(2!(K 2)!) inter-variable covariance matrices. Σ ik which is (p i p k ) covariance matrix whose elements are equal to the covariances between variables in the i th set and the k th set. H o : Σ ik = 0 for all i k. This test is more general than what we need for canonical correlation analysis. Canonical Slide 8 of 52
9 Assumptions of Wilk s Test The requirements are Assumptions of Wilk s Test Wilk s Test Statistic Wilk s Test continued Example: WAIS and Age Example of Wilk s Test for Relationship Method II: Canonical Covariance matrix for C X The Idea Solution that Maximizes the The Test Example: Method II Example continued Finishing the Test & Finding a 1 and b 1 Finding a 1 and b 1 and Conclusion (So Far) The More Usual Scaling of U 1 and V 1 Our Example 1. The within set covariance matrices are positive definite; that is Σ 11, Σ 22,..., Σ KK are all positive definite. 2. A sample of n observations has been drawn from a (single) population and measures taken for p = K i=1 p i variables. For each set of variables, compute Sp p, so we have all the within set covariance matrices and between set matrices: S 11 S 12 S 1K S 21 S 22 S 2K Sp p = S K1 S K2 S KK Canonical Slide 9 of 52
10 Wilk s Test Statistic Assumptions of Wilk s Test Wilk s Test Statistic Wilk s Test continued Example: WAIS and Age Example of Wilk s Test for Relationship Method II: Canonical Covariance matrix for C X The Idea Solution that Maximizes the The Test Example: Method II Example continued Finishing the Test & Finding a 1 and b 1 Finding a 1 and b 1 and Conclusion (So Far) The More Usual Scaling of U 1 and V 1 Our Example Wilk s test statistic equals V = det(s) det(s 11 ) det(s 22 ) det(s KK ) = det(r) det(r 11 ) det(r 22 ) det(r KK ) where R is the correlation matrix and the R ii are the within set correlation matrices. The scale of the variables is not important, so we can use either S or R. The distribution of V is very complicates; however, Box (1949) gave a good approximation of V s sampling distribution. When H o is true and n is large, then (n 1) c 1 c = f(n 1) (2τ 3 + 3τ 2 ) f = (1/2)τ 2 ln(v ) χ 2 f τ 2 = ( K i=1 p i) 2 K i=1 (p i) 2 and τ 3 = ( K i=1 p i) 3 K i=1 (p i) 3. Canonical Slide 10 of 52
11 Wilk s Test continued Reject for large values of V. Assumptions of Wilk s Test Wilk s Test Statistic Wilk s Test continued Example: WAIS and Age Example of Wilk s Test for Relationship Method II: Canonical Covariance matrix for C X The Idea Solution that Maximizes the The Test Example: Method II Example continued Finishing the Test & Finding a 1 and b 1 Finding a 1 and b 1 and Conclusion (So Far) The More Usual Scaling of U 1 and V 1 Our Example For canonical correlation analysis where K 2, the test statistic simplifies to V = det(s) det(s 11 ) det(s 22 ) = det(s 11 S 12 S 1 22 S 21) det(s 11 ) = det(s 22 S 21 S 1 11 S 12) det(s 22 ) = det(i S 1 11 S 12S 1 22 S 21) = det(i S 1 22 S 21S 1 11 S 12) Time for an example Canonical Slide 11 of 52
12 Example: WAIS and Age Assumptions of Wilk s Test Wilk s Test Statistic Wilk s Test continued Example: WAIS and Age Example of Wilk s Test for Relationship Method II: Canonical Covariance matrix for C X The Idea Solution that Maximizes the The Test Example: Method II Example continued Finishing the Test & Finding a 1 and b 1 Finding a 1 and b 1 and Conclusion (So Far) The More Usual Scaling of U 1 and V 1 Our Example The data (from Morrison (1990), pp ) are from an investigation of the relationship between the Wechsler Adult Intelligence Scale (WAIS) and age. Participants: n = 933 white men and women aged Two sets of variables: Set 1: p = 2 X 1 = digit span sub-test of WAIS X 2 = vocabulary sub-test of WAIS Set 2: q = 2 X 3 = chronological age X 4 = years of formal education Sample correlation matrx: R = ( R 11 R 12 R 21 R 22 ) = Canonical Slide 12 of 52
13 Example of Wilk s Test for Relationship Testing H o : Σ 12 = 0: Method I Assumptions of Wilk s Test Wilk s Test Statistic Wilk s Test continued Example: WAIS and Age Example of Wilk s Test for Relationship Method II: Canonical Covariance matrix for C X The Idea Solution that Maximizes the The Test Example: Method II Example continued Finishing the Test & Finding a 1 and b 1 Finding a 1 and b 1 and Conclusion (So Far) The More Usual Scaling of U 1 and V 1 Our Example V = R R 11 R 22 =.4015 (.7975)(.9159) =.5497 When n is large and the null hypothesis is true (n 1) c lnv is approximately distributed as χ 2 f random variable where τ 2 = (p + q) 2 (p 2 + q 2 ) = (2 + 2) 2 ( ) = 8 τ 3 = (p + q) 3 (p 3 + q 3 ) = (2 + 2) 3 ( ) = 48 f = 1 2 τ 2 = (.5)(8) = 4 1/c = f(n 1) (2τ 3 + 3τ 2 ) =.9973 Since (n 1) c ln V = (.9973)(933 1) ln(.5497) = is much larger than a χ 2 4(.05), we reject the hypothesis; the data support the conclusion that the two sets of variables are related. (p value<<.00001). Canonical Slide 13 of 52
14 Method II: Canonical Another approach to testing H o : Σ 12 = 0. Assumptions of Wilk s Test Wilk s Test Statistic Wilk s Test continued Example: WAIS and Age Example of Wilk s Test for Relationship Method II: Canonical Covariance matrix for C X The Idea Solution that Maximizes the The Test Example: Method II Example continued Finishing the Test & Finding a 1 and b 1 Finding a 1 and b 1 and Conclusion (So Far) The More Usual Scaling of U 1 and V 1 Our Example Find the vectors a and b that maximize the correlation ρ(a X 1, b X 2 ) Define C which is (p + q) 2 matrix ( ) a 0 } p rows C = 0 b } q rows Consider the linear combination C X, ( ) ( ) ( C a 0 X 1 X = 0 b = X 2 The next piece that we need is cov(c X)... a X 1 b X 2 ) Canonical Slide 14 of 52
15 Covariance matrix for C X cov(c X) = C ΣC = = ( ( ) ( a 0 Σ 11 Σ 12 0 b Σ 21 Σ 22 ) a Σ 11 a a Σ 12 b b Σ 21 a b Σ 22 b ) ( a 0 0 b ) Assumptions of Wilk s Test Wilk s Test Statistic Wilk s Test continued Example: WAIS and Age Example of Wilk s Test for Relationship Method II: Canonical Covariance matrix for C X The Idea Solution that Maximizes the The Test Example: Method II Example continued Finishing the Test & Finding a 1 and b 1 Finding a 1 and b 1 and Conclusion (So Far) The More Usual Scaling of U 1 and V 1 Our Example and the correlation between a X 1 and b X 2 is ρ(a X 1, b X 2 ) = a Σ 12 b a Σ 11 a b Σ 22 b If Σ 12 = 0, then a Σ 12 b = 0 for all possible choices of a and b. The correlation is estimated by a S 12 b a S 11 a b S 22 b Canonical Slide 15 of 52
16 The Idea Assumptions of Wilk s Test Wilk s Test Statistic Wilk s Test continued Example: WAIS and Age Example of Wilk s Test for Relationship Method II: Canonical Covariance matrix for C X The Idea Solution that Maximizes the The Test Example: Method II Example continued Finishing the Test & Finding a 1 and b 1 Finding a 1 and b 1 and Conclusion (So Far) The More Usual Scaling of U 1 and V 1 Our Example The idea underlying this method for testing whether there is a relationship between two sets of variables is that if the correlation (in the population) is 0, then let s find the maximum possible value of the correlation ρ(a X 1, b X 2 ) and use it as a test statistic for the null hypothesis H o : Σ 12 = 0. To simply the problem, we ll constrain a and b such that var(a X 1 ) = a S 11 a = 1 and var(b X i ) = b S 22 b = 1 Our maximization problem is now to the find the a and b max a,b (a S 12 b) This is the canonical correlation (subject to the constraint that the variances of the linear combinations equal 1). Canonical Slide 16 of 52
17 Solution that Maximizes the Assumptions of Wilk s Test Wilk s Test Statistic Wilk s Test continued Example: WAIS and Age Example of Wilk s Test for Relationship Method II: Canonical Covariance matrix for C X The Idea Solution that Maximizes the The Test Example: Method II Example continued Finishing the Test & Finding a 1 and b 1 Finding a 1 and b 1 and Conclusion (So Far) The More Usual Scaling of U 1 and V 1 Our Example The largest sample correlation is the square root of the the largest eigenvalue ( root ) of S 1 11 S 12S 1 22 S 21 or equivalently S 1 22 S 21S 1 11 S 12 The matrix products S 1 11 S 12S 1 22 S 21 and S 1 22 S 21S 1 11 S 12 have the same characteristic roots (eigenvalues), which we ll call c 1, c 2,...,c r. Assume that we ve ordered the roots: c 1 c r. The eigenvector of S 1 11 S 12S 1 22 S 21 associated with c 1 gives use a 1. The eigenvector of S 1 22 S 21S 1 11 S 12 corresponding to c 1 corresponds to b 1. Setting a = a 1 and b = b 1 yields the maximum: c1 = max a,b (a S 12 b). The sample correlation between U 1 = a 1 X 1 and V 1 = b 1 X 2 equals ± c 1 (you have to determine the sign). Canonical Slide 17 of 52
18 The Test Assumptions of Wilk s Test Wilk s Test Statistic Wilk s Test continued Example: WAIS and Age Example of Wilk s Test for Relationship Method II: Canonical Covariance matrix for C X The Idea Solution that Maximizes the The Test Example: Method II Example continued Finishing the Test & Finding a 1 and b 1 Finding a 1 and b 1 and Conclusion (So Far) The More Usual Scaling of U 1 and V 1 Our Example Formal Test of H o : Σ 12 = 0: Consider c 1 = the largest root (eigenvalue). Reject H o : Σ 12 = 0 if c 1 > θ α;s,m.n where θ is the (1 α) 100% percentile point of the greatest root distribution with parameters s = min(p, q), m = 1 2 ( p q 1), n = 1 (n p q 2) 2 There are charts and tables of upper percentile points of the largest root distribution in various multivariate statistics texts (e.g., Morrison). For online via jstor.org: (Assuming connection to uiuc via netid) D. L. Heck (1960) Charts of Some Upper Percentage Points of the Distribution of the Largest Characteristic Root. The Annals of Mathematical Statistics, Vol. 31, No. 3, pp K. C. Sreedharan Pillai, Celia G. Bantegui (1959). On the Distribution of the Largest of Six Roots a Matrix in Multivariate On the Distribution of the Largest of Six Roots a Matrix Canonical Slide 18 of 52 in Multivariate. Biometrika, vol 46, pp )
19 Example: Method II Testing H o : Σ 12 = 0: Method II the largest root distribution Assumptions of Wilk s Test Wilk s Test Statistic Wilk s Test continued Example: WAIS and Age Example of Wilk s Test for Relationship Method II: Canonical Covariance matrix for C X The Idea Solution that Maximizes the The Test Example: Method II Example continued Finishing the Test & Finding a 1 and b 1 Finding a 1 and b 1 and Conclusion (So Far) The More Usual Scaling of U 1 and V 1 Our Example We first find the roots of R 1 11 R 12R 1 the roots of R 1 22 R 21R 1 R 1 11 = ( and multiplying the matrices gives us ( 22 R 21 (which are equal to 11 R 12). So we need ) ( ) R 1 22 = R 1 11 R 12R 1 22 R 21 = The roots of this matrix product are the solution of the equation (.0937 c) (.3730 c) = 0 (.0937 c)(.3730 c) (.2130)(.0873) = 0 ) Canonical Slide 19 of 52
20 Example continued Assumptions of Wilk s Test Wilk s Test Statistic Wilk s Test continued Example: WAIS and Age Example of Wilk s Test for Relationship Method II: Canonical Covariance matrix for C X The Idea Solution that Maximizes the The Test Example: Method II Example continued Finishing the Test & Finding a 1 and b 1 Finding a 1 and b 1 and Conclusion (So Far) The More Usual Scaling of U 1 and V 1 Our Example (.0937 c)(.3730 c) (.2130)(.0873) = 0 c c = 0 (c.4285)(c.0381) = 0 So c 1 =.4285 and c 2 = The largest correlation between a linear combination of variables in set 1 (U 1 = a X 1 ) and a linear combination of variables in set 2 (V 1 = b X 2 ) equals c1 =.4285 = a Σb =.654 To test whether H o : Σ 12 = 0, we have s = min(p, q) = min(2, 2) = 2 m = (1/2)( p q 1) =.5( 2 2 1) =.5 n = (1/2)(n p q 2) =.5( ) = where p = number of variables in set 1, and q = number of variables in set 2. Canonical Slide 20 of 52
21 Finishing the Test & Finding a 1 and b 1 Assumptions of Wilk s Test Wilk s Test Statistic Wilk s Test continued Example: WAIS and Age Example of Wilk s Test for Relationship Method II: Canonical Covariance matrix for C X The Idea Solution that Maximizes the The Test Example: Method II Example continued Finishing the Test & Finding a 1 and b 1 Finding a 1 and b 1 and Conclusion (So Far) The More Usual Scaling of U 1 and V 1 Our Example Using the chart (page 628 of Heck (1960)) of the greatest root distribution, we find θ 2,.5,463.5 (.01) =.02. Since c 1 =.4285 >.02, we reject H o ; there is a dependency between the sets of variables. The linear combination that Maximizes the correlation. To compute a 1, we use R 1 11 R 12R 1 22 R 21 ( R 1 11 R 12R R 21a 1 = c 1 a 1 ) ( ) a 11 a 21 Two equations, two unknowns: = a a 12 = a a 12 = 0 For convenience, we ll set a 12 = 1, and solve for a 11 = (.0547/.2130)a 12 =.26. ( a 11 a 21 ) Canonical Slide 21 of 52
22 Finding a 1 and b 1 Assumptions of Wilk s Test Wilk s Test Statistic Wilk s Test continued Example: WAIS and Age Example of Wilk s Test for Relationship Method II: Canonical Covariance matrix for C X The Idea Solution that Maximizes the The Test Example: Method II Example continued Finishing the Test & Finding a 1 and b 1 Finding a 1 and b 1 and Conclusion (So Far) The More Usual Scaling of U 1 and V 1 Our Example Any vector that is proportional to (i.e., is a multiple of) a = (.26, 1) is a solution and gives us the correct linear combination. To compute b 1, we use R 1 22 R 21R 1 11 R 12 ( R 1 22 R 21R R 12b 1 = c 1 b 1 ) ( ) b 11 b 12 Two Equations, Two Unknowns = b b 12 = b b 12 = 0 ( b 11 b 12 We ll set b 12 = 1 and solve for b 11 = (.0076/.0378)b 12 =.20. Any vector proportional to (a multiple of) b = (.20, 1) is a solution and gives us the correct linear combination. ) Canonical Slide 22 of 52
23 and Conclusion (So Far) Since the correlation matrix, R, was used, to solve for the vectors a 1 and b 1, we use the standardized scores (i.e., z-scores) rather than the original (raw) variables. U 1 =.26z (digit span) z (vocabulary) V 1 =.20z (age) z (years of formal education) Assumptions of Wilk s Test Wilk s Test Statistic Wilk s Test continued Example: WAIS and Age Example of Wilk s Test for Relationship Method II: Canonical Covariance matrix for C X The Idea Solution that Maximizes the The Test Example: Method II Example continued Finishing the Test & Finding a 1 and b 1 Finding a 1 and b 1 and Conclusion (So Far) The More Usual Scaling of U 1 and V 1 Our Example Interpretation/: The correlation between equals U 1 and V 1, which equals c1 =.4285 =.654 = (a 1R 12 b 1 )/( a 1 R 11a 1 b 1 R 22 b 1 ), is the largest possible one for any linear combination of the variables in sets 1 and 2. U 1 : places four times more weight on vocabulary than on digit span... long term versus short term memory. V 1 : places five times more weight on years of formal education than on chronological age. The major link between the two sets of variables is due to education and vocabulary. Canonical Slide 23 of 52
24 The More Usual Scaling of U 1 and V 1 (What I did wasn t the typical way to scale a 1 and b 1 ). Assumptions of Wilk s Test Wilk s Test Statistic Wilk s Test continued Example: WAIS and Age Example of Wilk s Test for Relationship Method II: Canonical Covariance matrix for C X The Idea Solution that Maximizes the The Test Example: Method II Example continued Finishing the Test & Finding a 1 and b 1 Finding a 1 and b 1 and Conclusion (So Far) The More Usual Scaling of U 1 and V 1 Our Example The standard or typical way that a 1 and b 1 are scaled is such that the variances of U 1 = a 1 X 1 and V 1 = b 1 X 2 equal 1. Since any multiple of a 1 and/or b 1 is a solution, we just need to multiply the vectors by an appropriate constant. For example, and now a 1 = a 1 a 1 R 11 a var(u 1 ) = var(a 1X 1 ) = a 1 R 11 a 1 ( ( ) a 1 a = )R a R 11 a a 1 R 11 a = 1 Canonical Slide 24 of 52
25 Our Example a 1R 11 a 1 = (0.26, 1.00) ( ) ( ) = So a 1 = (0.26, 1.00) = (.2279,.8765) Assumptions of Wilk s Test Wilk s Test Statistic Wilk s Test continued Example: WAIS and Age Example of Wilk s Test for Relationship Method II: Canonical Covariance matrix for C X The Idea Solution that Maximizes the The Test Example: Method II Example continued Finishing the Test & Finding a 1 and b 1 Finding a 1 and b 1 and Conclusion (So Far) The More Usual Scaling of U 1 and V 1 Our Example As a check: var(u 1 ) = (.2279,.8765) ( Doing the same thing for b: ( b R 22 b 1 = (0.20, 1.00) So b 1 = ) ( ) ( ( 1 )b 1 = (.2081, ) ) ) = 1.00 = Canonical Slide 25 of 52
26 Assumptions and Solution Back to our Example The Canonical Variates al Structure of Canonical Variates So far we ve only looked at the largest correlation possible between linear combinations of the variables from two sets; however, there are more c i s. There are more linear combinations: U 1 = a 1X 1 V 1 = b 1X 2 U 2 = a 2X 1 V 2 = b 2X 2.. U r = a rx 1 V r = b rx 2 With the property that the sample correlation between U 1 and V 1 is the largest, the sample correlation between U 2 and V 2 is the largest among all linear combinations uncorrelated with U 1 and V 1, etc. That is, for all i k, cov(u i, U k ) = a is 11 a k = 0 cov(v i, V k ) = b is 22 b k = 0 cov(u i, V k ) = a is 12 b k = 0 cov(u k, V i ) = a ks 12 b i = 0 Canonical Slide 26 of 52
27 Assumptions and Solution Assumptions and Solution Back to our Example The Canonical Variates al Structure of Canonical Variates Assume 1. The elements of Σ (p+q) (p+q) are finite. 2. Σ is full rank; that is, rank = p + q. 3. The first r min(p, q) characteristic roots of Σ 1 11 Σ 12Σ 1 22 Σ 21 are distinct. Then a i and b i are estimated from the data by solving (S 12 S 1 22 S 21 c i S 11 )a i = 0 and (S 21 S 1 11 S 12 c i S 22 )b i = 0 where c i is the i th root of the equation. det(s 12 S 1 22 S 21 c i S 11 ) = 0 or equivalently det(s 21 S 1 11 S 12 c i S 22 ) = 0 c i = squared sample correlation between U i and V i Canonical Slide 27 of 52
28 Back to our Example Assumptions and Solution Back to our Example The Canonical Variates al Structure of Canonical Variates The linear combination that gives the next largest correlation and is orthogonal to the first one. Using the second root of the matrix product, c 2 =.0381, which is the second largest squared correlation, repeat the process we did previously to get the vectors a 2 and b 2 : a 2 = ( 1.00, 0.64) and b 2 = (1.00, 0.10) or using the more typically scaling, we get a 2 = ( ,.7001) and b 2 = (1.0249,.1025) Statistical Hypothesis Test: H o : ρ(u 2, V 2 ) = 0 vs H a : ρ(u 2, V 2 ) 0: (n 1 12 ) (p + q + 1) ln(1 c 2 ) = (932.5(5)) ln(1.0381) = If the null hypothesis is true (and n large), then this statistic is approximately distributed at χ 2 pq. Since χ 2 4(.05) = 9.488, we reject the null and conclude that the second canonical correlation is not zero. Canonical Slide 28 of 52
29 The Canonical Variates To find a 2 and b 2, we use the same process that we used to find a 1 and b 1, which gives us and U 2 = 1.00Z ( digit span) Z ( vocabulary) V 2 = 1.00Z ( age) Z ( years of formal education) Assumptions and Solution Back to our Example The Canonical Variates al Structure of Canonical Variates U 2 is a weighted contrast between digit span (performance) and vocabulary (verbal) sub-tests. V 2 is nearly all age. There s a widening in the gap between performance with advancing age. As people get older, there s a larger difference between accumulated knowledge (vocabulary) and performance skills. Canonical Slide 29 of 52
30 al Structure of Canonical Variates Note: The following covariances (correlations) are all zero: Assumptions and Solution Back to our Example The Canonical Variates al Structure of Canonical Variates cov(u 1, U 2 ) = a 1 R 11 a 2 = 0 cov(v 1, V 2 ) = b 1 R 22 b 2 = 0 cov(u 1, V 2 ) = a 1 R 12 b 2 = 0 cov(v 1, U 2 ) = b 1 R 21 a 2 = 0 In other words,the sample correlation matrix for the canonical variates is U 1 U 2 V 1 V 2 U U V V which is much simpler than the sample correlation matrix back on page 12. Canonical Slide 30 of 52
31 Showing Why this Works Or the how to do this in IML, R, and/or MATLAB: We symmetrize the matrix so that our solution are the eigenvalues and vectors of 1/2 (S 11 S 12 S 1 22 S 21S 1/2 11 )e i = c i e i where S 1/2 11 is the inverse of the square root matrix. Then the combination vectors that you want equal a i = S 1/2 11 e i b i = 1 S 1 22 S 21a i ci OR You can find the eigenvalues and eigenvectors of (S 1/2 22 S 21 S 1 11 S 12S 1/2 22 )f i = c i f i Then the combination vectors that you want equal b i = S 1/2 22 f i a i = 1 S 1 11 S 21b i ci Canonical Slide 31 of 52
32 Showing Why this Works Showing Why this Works S 1/2 11 ( ) S 1/2 11 S 12 S 1 22 S 21S 1/2 11 e = c e ( ) S 1/2 11 S 12 S 1 22 S 21S 1/2 11 e = c S 1/2 11 e ( S 1 11 S 12S 1 22 S ) 1/2 21 S 11 e }{{} a = c S 1/2 11 e }{{} a ( S 1 11 S 12S 1 22 S 21) a = c a This is what we did for discriminant analysis to find eigenvalues and vectors of a non-symmetric matrix. Canonical Slide 32 of 52
33 1. To what extent can one set of two (or more) variables be predicted by or explained by another set of two or more variables? 2. What contributions does a single variable make to the explanatory power of the set of variable to which the variable belongs? of What CCA Does Example: The Simplification of R Question 1 Question 2 The Structure Coefficients Question 3 Relationship between Index and Structure 3. To what extent does a single variable contribute to predicting or explaining the composite of the variables in the variable set to which the variable does not belong? We ll talk about how to answer each of these. Canonical Slide 33 of 52
34 of What CCA Does Started with sample R (or S), which in our example is X 1 X 2 X 3 X 4 set 1 X digit span X vocabulary set 2 X age X years of education of What CCA Does Example: The Simplification of R Question 1 Question 2 The Structure Coefficients Question 3 Relationship between Index and Structure We found linear transformation, canonical variates, of the original variables within sets to maximize the between set correlations: where U i = a ix 1 and V i = b ix 2 (R 1 11 R 12R 1 22 R 21)a i = c i a i and (R 1 22 R 21R 1 11 R 12)b i = c i b i And scaled them so that the variance of U i and V i equal 1 a ir 11 a i = a ir 22 b i = 1 Canonical Slide 34 of 52
35 Example: The Simplification of R of What CCA Does Example: The Simplification of R Question 1 Question 2 The Structure Coefficients Question 3 Relationship between Index and Structure X 1 X 2 X 3 X 4 U 1 U 2 V 1 V 2 X X X X U U V V Canonical Slide 35 of 52
36 Question 1 X 1 X 2 X 3 X 4 U 1 U 2 V 1 V 2 of What CCA Does Example: The Simplification of R Question 1 Question 2 The Structure Coefficients Question 3 Relationship between Index and Structure X 1 R 11 R 12 X 2 X 3 R 21 R 22 X 4 U 1 Structure Index c U 2 coefficients coefficients c2 V 1 Index Structure c V 2 coefficients coefficients 0.00 c Question 1: To what extent can one set of two or more variables be explained by another set of two or more variables? Answer: The (first) canonical correlation c 1. Canonical Slide 36 of 52
37 Question 2 Question 2: What is the contribution between a variable and the canonical (composite) variable for it s set? of What CCA Does Example: The Simplification of R Question 1 Question 2 The Structure Coefficients Question 3 Relationship between Index and Structure Answer: The structure coefficients. Variables Within Sets X 1 = X 11 X 12. X 1p X 2 = Structure coefficients equal, X 21 X 22. X 2q Set 1: }{{} corr (X 1, U i ) = corr(x 1, a ix 1 ) vector Canonical Variates U i = a i X 1 V i = b ix 2 Set 2: }{{} corr (X 2, V i ) = corr(x 2, b ix 2 ) vector Canonical Slide 37 of 52
38 The Structure Coefficients Set 1:corr (p) (X 1, U i ) = corr(x 1, a ix 1 ) 1/ s / s 22 0 = /sqrts pp = diag(1/ s ii )S 11 a i S 11a i of What CCA Does Example: The Simplification of R Question 1 Question 2 The Structure Coefficients Question 3 Relationship between Index and Structure When we use R rather than S to find U i and V i, then set 1: corr(z 1, U i ) (p 1) = R 11 a i and set 2: corr(z 2, V i ) q 1) = R 22 b i Canonical Slide 38 of 52
39 Question 3 Question 3 To what extent does a single variable contribute to explaining the canonical (composite) variable in the set of variables to which it does not belong? Answer: The correlation between it and the canonical variate of the other set of variables: index coefficients. of What CCA Does Example: The Simplification of R Question 1 Question 2 The Structure Coefficients Question 3 Relationship between Index and Structure corr(x 1, V i ) (p 1) = corr(x 1, ī X 2 ) = diag(1/ s 11(ii) )Σ 12 b i and corr(x 2, U i ) (q 1) = corr(x 2, i X 1 ) = diag(1/ s 22(ii) )Σ 21 a i Canonical Slide 39 of 52
40 Relationship between Index and Structure a i can be written as a linear combination of b i (and visa versa). of What CCA Does Example: The Simplification of R Question 1 Question 2 The Structure Coefficients Question 3 Relationship between Index and Structure a i = 1 c i Σ 1 11 Σ 12b i and b i = 1 c i Σ a i Re-arrange terms a bit ci }{{} Σ 11a i = Σ 12 b i So the Index Coefficients corr(x 1, V i ) (p 1) = diag(1/ σ 11(ii) )Σ 12 b i = D 1/2 1 ( c i Σ 11 a i ) = ci D 1/2 1 Σ 11 a i }{{}}{{} cov(u i,v i ) corr(x 1,U i ) ci }{{} Σ 22b i = Σ 21 a i Index coefficient = cov(u i, V i ) (Structure Coefficient) }{{} canonical correlation and corr(x 2, U i ) = cov(u i, v i )corr(x 2, V i ) Canonical Slide 40 of 52
41 Edited Output Canonical s, etc Canonical Coefficients Structure Coefficients Index Coefficients If you have a correlation matrix, you can input it as as data set using: data corrmat (type=corr); input TYPE $ NAME $ x1 x2 x3 x4; list; datalines; N CORR x CORR x CORR x CORR x * If you do not input N, default is to assume that N=10,000; proc cancorr data=corrmat simple corr; var x1 x2; with x3 x4; title Canonical of WAIS ; run; Canonical Slide 41 of 52
42 Edited Output Edited Output Canonical s, etc Canonical Coefficients Structure Coefficients Index Coefficients s Among the VAR Variables x1 x2 x <- R_11 x s Among the WITH Variables x3 x4 x <- R_22 x s Between the VAR Variables and the WITH Variables x3 x4 x <- R_12 x Canonical Slide 42 of 52
43 Canonical s, etc Adjusted Approximate Squared Canonical Canonical Standard Canonical Error rho1 = rho2 = Edited Output Canonical s, etc Canonical Coefficients Structure Coefficients Index Coefficients Test of H0: The canonical correlations in the current row and all that follow are zero Likelihood Approximate Ratio F Value Num DF Den DF Pr > F < <.0001 Note: First line above tests H_o: Sigma_12 = rho1 = rho2 = 0 Second line above tests H_o: rho2 = 0 Canonical Slide 43 of 52
44 Canonical Coefficients Edited Output Canonical s, etc Canonical Coefficients Structure Coefficients Index Coefficients Raw Canonical Coefficients for the VAR Variables V1 x <- columns = a_i V2 x Raw Canonical Coefficients for the WITH Variables W1 x <- columns = b_i W2 x Notes: * Since we input a correlation matrix, "raw" are same as standardized coefficients. * "V" is same as lecture notes "U". * "W" is same as lecture notes "V". Canonical Slide 44 of 52
45 Structure Coefficients s Between the VAR Variables and Their Canonical Variables V1 V2 x x Edited Output Canonical s, etc Canonical Coefficients Structure Coefficients Index Coefficients s Between the WITH Variables and Their Canonical Variables W1 W2 x x Canonical Slide 45 of 52
46 Index Coefficients s Between the VAR Variables and the Canonical Variables of the WITH Variables W1 W2 x x Edited Output Canonical s, etc Canonical Coefficients Structure Coefficients Index Coefficients s Between the WITH Variables and the Canonical Variables of the VAR Variables V1 V2 x x Canonical Slide 46 of 52
47 For M > 2, let p= M m=1 p m, X 1 } p 1 X 2 } p 2 X = and Σ =.. } p M X m Σ 11 Σ 12 Σ 1M Σ 21 Σ 22 Σ 2M Σ M1 Σ M2 Σ MM Consider the linear combinations Z 11 = a 11X 1 Set up for Horst s Suggestions Kettenring s Suggestions GENVAR Z 12 = a 12X 2 Z 1M = a 1MX M where a 1m is the (p m 1) vector for the m th canonical variable. The (estimated) covariance matrix of (Z 1 1, Z 1 2,...,Z 1 M) is ˆ Φ(1) (M M), which equals.... Canonical Slide 47 of 52
48 Set up for ˆ Φ(1) (M M) = 1 ˆφ12 (1) ˆφ13 (1) ˆφ1M (1) ˆφ 21 (1) 1 ˆφ23 (1) ˆφ2M (1) ˆφ M1 (1) ˆφM2 (1) ˆφM3 (1) 1 Set up for Horst s Suggestions Kettenring s Suggestions GENVAR where ˆφ ii(1) = a 1i Σ iia 1i = 1 and ˆφ ik(1) = a 1i Σ ika 1k. In the two set case, we only had one off diagonal element that could maximize, i.e., ˆφ 12 (1). There are (at least) five ways to generalized canonical correlation to a multiple set problems. For M = 2 they are all equivalent to what we ve talked about. Canonical Slide 48 of 52
49 Horst s Suggestions SUMCOR: Horst (1965) Factor analysis of data matrices Horst suggested maximizing max a 1,...,a M i<k ˆφ ik (1) That is, sum of all off-diagonal elements of the correlation matrix between the M canonical variates Z 1,...,Z M ), i.e., ˆΦ(1). Set up for Horst s Suggestions Kettenring s Suggestions GENVAR MAXVAR: Horst also suggest maximizing the variance of a linear combination of Z = (Z 11, Z 12,...,Z 1M, which is the first principal component of ˆΦ(1). The variance of this maximal linear combination is the largest eigenvalue of ˆΦ(1); that is, λ 1. Canonical Slide 49 of 52
50 Kettenring s Suggestions Kettenring (1971), Biometrika SSQCOR: Find the linear combinations that maximize max a 1,...,a M ( i<k ˆφ 2 ik(1)) = M λ 2 ik M i=1 Set up for Horst s Suggestions Kettenring s Suggestions GENVAR MINVAR: Kettenring also suggested minimizing the smallest eigenvalue λ 1M ; that is, the variance of the minimal linear combination. Canonical Slide 50 of 52
51 GENVAR Steel (1951) in the Annals of Mathematical Statistics suggested minimizing the generalized sample variance det( ˆΦ(1)) = M m=1 λ ij Once the first canonical variate has been found, all five methods call all be extended to find Z 2, Z 3,...,Z M such that they are orthogonal to ones previously found. Set up for Horst s Suggestions Kettenring s Suggestions GENVAR Canonical Slide 51 of 52
52 /Discusion /Discusion Discriminant analysis is a special case of canonical correlation analysis where one set of variables is a dummy coded variable that defines populations. i.e., For individual j in group i, X 1j (1 p) = (0,..., }{{} 1,...0). i th The other set of variables are q continuous/numerical ones. Discriminant analysis and MANOVA use the same matrix W 1 B or E 1 H. (Binary or Multinomial) Logistic regression is the flip side of discriminant analysis and MANOVA (i.e., interchange response and predictor roles). Conditional Guassian distribution or the location model. See handout on similarities and differences between PCA, MANOVA, DA, and canonical correlation analysis. Canonical Slide 52 of 52
Inferences about a Mean Vector
Inferences about a Mean Vector Edps/Soc 584, Psych 594 Carolyn J. Anderson Department of Educational Psychology I L L I N O I S university of illinois at urbana-champaign c Board of Trustees, University
More informationMS-E2112 Multivariate Statistical Analysis (5cr) Lecture 8: Canonical Correlation Analysis
MS-E2112 Multivariate Statistical (5cr) Lecture 8: Contents Canonical correlation analysis involves partition of variables into two vectors x and y. The aim is to find linear combinations α T x and β
More informationChapter 4: Factor Analysis
Chapter 4: Factor Analysis In many studies, we may not be able to measure directly the variables of interest. We can merely collect data on other variables which may be related to the variables of interest.
More informationCanonical Correlations
Canonical Correlations Like Principal Components Analysis, Canonical Correlation Analysis looks for interesting linear combinations of multivariate observations. In Canonical Correlation Analysis, a multivariate
More informationI L L I N O I S UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN
Introduction Edps/Psych/Stat/ 584 Applied Multivariate Statistics Carolyn J Anderson Department of Educational Psychology I L L I N O I S UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN c Board of Trustees,
More informationMultilevel Models in Matrix Form. Lecture 7 July 27, 2011 Advanced Multivariate Statistical Methods ICPSR Summer Session #2
Multilevel Models in Matrix Form Lecture 7 July 27, 2011 Advanced Multivariate Statistical Methods ICPSR Summer Session #2 Today s Lecture Linear models from a matrix perspective An example of how to do
More informationI L L I N O I S UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN
Comparisons of Several Multivariate Populations Edps/Soc 584 and Psych 594 Applied Multivariate Statistics Carolyn J Anderson Department of Educational Psychology I L L I N O I S UNIVERSITY OF ILLINOIS
More informationI L L I N O I S UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN
Comparisons of Two Means Edps/Soc 584 and Psych 594 Applied Multivariate Statistics Carolyn J. Anderson Department of Educational Psychology I L L I N O I S UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN c
More informationMore Linear Algebra. Edps/Soc 584, Psych 594. Carolyn J. Anderson
More Linear Algebra Edps/Soc 584, Psych 594 Carolyn J. Anderson Department of Educational Psychology I L L I N O I S university of illinois at urbana-champaign c Board of Trustees, University of Illinois
More informationNext is material on matrix rank. Please see the handout
B90.330 / C.005 NOTES for Wednesday 0.APR.7 Suppose that the model is β + ε, but ε does not have the desired variance matrix. Say that ε is normal, but Var(ε) σ W. The form of W is W w 0 0 0 0 0 0 w 0
More informationEigenvalues, Eigenvectors, and an Intro to PCA
Eigenvalues, Eigenvectors, and an Intro to PCA Eigenvalues, Eigenvectors, and an Intro to PCA Changing Basis We ve talked so far about re-writing our data using a new set of variables, or a new basis.
More informationMultivariate Linear Regression Models
Multivariate Linear Regression Models Regression analysis is used to predict the value of one or more responses from a set of predictors. It can also be used to estimate the linear association between
More informationEigenvalues, Eigenvectors, and an Intro to PCA
Eigenvalues, Eigenvectors, and an Intro to PCA Eigenvalues, Eigenvectors, and an Intro to PCA Changing Basis We ve talked so far about re-writing our data using a new set of variables, or a new basis.
More informationApplied Multivariate Analysis
Department of Mathematics and Statistics, University of Vaasa, Finland Spring 2017 Discriminant Analysis Background 1 Discriminant analysis Background General Setup for the Discriminant Analysis Descriptive
More informationCS281 Section 4: Factor Analysis and PCA
CS81 Section 4: Factor Analysis and PCA Scott Linderman At this point we have seen a variety of machine learning models, with a particular emphasis on models for supervised learning. In particular, we
More informationI L L I N O I S UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN
Principal Analysis Edps/Soc 584 and Psych 594 Applied Multivariate Statistics Carolyn J. Anderson Department of Educational Psychology I L L I N O I S UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN c Board
More informationProblem Set #6: OLS. Economics 835: Econometrics. Fall 2012
Problem Set #6: OLS Economics 835: Econometrics Fall 202 A preliminary result Suppose we have a random sample of size n on the scalar random variables (x, y) with finite means, variances, and covariance.
More information1 Principal Components Analysis
Lecture 3 and 4 Sept. 18 and Sept.20-2006 Data Visualization STAT 442 / 890, CM 462 Lecture: Ali Ghodsi 1 Principal Components Analysis Principal components analysis (PCA) is a very popular technique for
More informationComparisons of Several Multivariate Populations
Comparisons of Several Multivariate Populations Edps/Soc 584, Psych 594 Carolyn J Anderson Department of Educational Psychology I L L I N O I S university of illinois at urbana-champaign c Board of Trustees,
More information4.1 Computing section Example: Bivariate measurements on plants Post hoc analysis... 7
Master of Applied Statistics ST116: Chemometrics and Multivariate Statistical data Analysis Per Bruun Brockhoff Module 4: Computing 4.1 Computing section.................................. 1 4.1.1 Example:
More informationMultivariate Statistical Analysis
Multivariate Statistical Analysis Fall 2011 C. L. Williams, Ph.D. Lecture 17 for Applied Multivariate Analysis Outline Multivariate Analysis of Variance 1 Multivariate Analysis of Variance The hypotheses:
More informationPrincipal component analysis
Principal component analysis Motivation i for PCA came from major-axis regression. Strong assumption: single homogeneous sample. Free of assumptions when used for exploration. Classical tests of significance
More informationMultivariate Regression (Chapter 10)
Multivariate Regression (Chapter 10) This week we ll cover multivariate regression and maybe a bit of canonical correlation. Today we ll mostly review univariate multivariate regression. With multivariate
More informationChapter 7, continued: MANOVA
Chapter 7, continued: MANOVA The Multivariate Analysis of Variance (MANOVA) technique extends Hotelling T 2 test that compares two mean vectors to the setting in which there are m 2 groups. We wish to
More informationSample Geometry. Edps/Soc 584, Psych 594. Carolyn J. Anderson
Sample Geometry Edps/Soc 584, Psych 594 Carolyn J. Anderson Department of Educational Psychology I L L I N O I S university of illinois at urbana-champaign c Board of Trustees, University of Illinois Spring
More informationPrinciple Components Analysis (PCA) Relationship Between a Linear Combination of Variables and Axes Rotation for PCA
Principle Components Analysis (PCA) Relationship Between a Linear Combination of Variables and Axes Rotation for PCA Principle Components Analysis: Uses one group of variables (we will call this X) In
More informationMultivariate Statistics Fundamentals Part 1: Rotation-based Techniques
Multivariate Statistics Fundamentals Part 1: Rotation-based Techniques A reminded from a univariate statistics courses Population Class of things (What you want to learn about) Sample group representing
More informationLeast Squares Estimation
Least Squares Estimation Using the least squares estimator for β we can obtain predicted values and compute residuals: Ŷ = Z ˆβ = Z(Z Z) 1 Z Y ˆɛ = Y Ŷ = Y Z(Z Z) 1 Z Y = [I Z(Z Z) 1 Z ]Y. The usual decomposition
More informationAn Introduction to Multivariate Statistical Analysis
An Introduction to Multivariate Statistical Analysis Third Edition T. W. ANDERSON Stanford University Department of Statistics Stanford, CA WILEY- INTERSCIENCE A JOHN WILEY & SONS, INC., PUBLICATION Contents
More informationMS-E2112 Multivariate Statistical Analysis (5cr) Lecture 6: Bivariate Correspondence Analysis - part II
MS-E2112 Multivariate Statistical Analysis (5cr) Lecture 6: Bivariate Correspondence Analysis - part II the Contents the the the Independence The independence between variables x and y can be tested using.
More informationApplication of Ghosh, Grizzle and Sen s Nonparametric Methods in. Longitudinal Studies Using SAS PROC GLM
Application of Ghosh, Grizzle and Sen s Nonparametric Methods in Longitudinal Studies Using SAS PROC GLM Chan Zeng and Gary O. Zerbe Department of Preventive Medicine and Biometrics University of Colorado
More informationSTA 431s17 Assignment Eight 1
STA 43s7 Assignment Eight The first three questions of this assignment are about how instrumental variables can help with measurement error and omitted variables at the same time; see Lecture slide set
More informationIntermediate Social Statistics
Intermediate Social Statistics Lecture 5. Factor Analysis Tom A.B. Snijders University of Oxford January, 2008 c Tom A.B. Snijders (University of Oxford) Intermediate Social Statistics January, 2008 1
More informationMultivariate analysis of variance and covariance
Introduction Multivariate analysis of variance and covariance Univariate ANOVA: have observations from several groups, numerical dependent variable. Ask whether dependent variable has same mean for each
More informationA Introduction to Matrix Algebra and the Multivariate Normal Distribution
A Introduction to Matrix Algebra and the Multivariate Normal Distribution PRE 905: Multivariate Analysis Spring 2014 Lecture 6 PRE 905: Lecture 7 Matrix Algebra and the MVN Distribution Today s Class An
More informationGregory Carey, 1998 Regression & Path Analysis - 1 MULTIPLE REGRESSION AND PATH ANALYSIS
Gregory Carey, 1998 Regression & Path Analysis - 1 MULTIPLE REGRESSION AND PATH ANALYSIS Introduction Path analysis and multiple regression go hand in hand (almost). Also, it is easier to learn about multivariate
More informationModels for Clustered Data
Models for Clustered Data Edps/Psych/Soc 589 Carolyn J Anderson Department of Educational Psychology c Board of Trustees, University of Illinois Spring 2019 Outline Notation NELS88 data Fixed Effects ANOVA
More informationModels for Clustered Data
Models for Clustered Data Edps/Psych/Stat 587 Carolyn J Anderson Department of Educational Psychology c Board of Trustees, University of Illinois Fall 2017 Outline Notation NELS88 data Fixed Effects ANOVA
More informationVectors and Matrices Statistics with Vectors and Matrices
Vectors and Matrices Statistics with Vectors and Matrices Lecture 3 September 7, 005 Analysis Lecture #3-9/7/005 Slide 1 of 55 Today s Lecture Vectors and Matrices (Supplement A - augmented with SAS proc
More informationFactor Analysis Edpsy/Soc 584 & Psych 594
Factor Analysis Edpsy/Soc 584 & Psych 594 Carolyn J. Anderson University of Illinois, Urbana-Champaign April 29, 2009 1 / 52 Rotation Assessing Fit to Data (one common factor model) common factors Assessment
More informationRejection regions for the bivariate case
Rejection regions for the bivariate case The rejection region for the T 2 test (and similarly for Z 2 when Σ is known) is the region outside of an ellipse, for which there is a (1-α)% chance that the test
More informationEigenvalues, Eigenvectors, and an Intro to PCA
Eigenvalues, Eigenvectors, and an Intro to PCA Eigenvalues, Eigenvectors, and an Intro to PCA Changing Basis We ve talked so far about re-writing our data using a new set of variables, or a new basis.
More informationChapter 9. Multivariate and Within-cases Analysis. 9.1 Multivariate Analysis of Variance
Chapter 9 Multivariate and Within-cases Analysis 9.1 Multivariate Analysis of Variance Multivariate means more than one response variable at once. Why do it? Primarily because if you do parallel analyses
More informationOther hypotheses of interest (cont d)
Other hypotheses of interest (cont d) In addition to the simple null hypothesis of no treatment effects, we might wish to test other hypothesis of the general form (examples follow): H 0 : C k g β g p
More informationLecture 5: Hypothesis tests for more than one sample
1/23 Lecture 5: Hypothesis tests for more than one sample Måns Thulin Department of Mathematics, Uppsala University thulin@math.uu.se Multivariate Methods 8/4 2011 2/23 Outline Paired comparisons Repeated
More informationGroup comparison test for independent samples
Group comparison test for independent samples The purpose of the Analysis of Variance (ANOVA) is to test for significant differences between means. Supposing that: samples come from normal populations
More informationSystems Simulation Chapter 7: Random-Number Generation
Systems Simulation Chapter 7: Random-Number Generation Fatih Cavdur fatihcavdur@uludag.edu.tr April 22, 2014 Introduction Introduction Random Numbers (RNs) are a necessary basic ingredient in the simulation
More informationMeasuring relationships among multiple responses
Measuring relationships among multiple responses Linear association (correlation, relatedness, shared information) between pair-wise responses is an important property used in almost all multivariate analyses.
More informationLinear Regression. In this problem sheet, we consider the problem of linear regression with p predictors and one intercept,
Linear Regression In this problem sheet, we consider the problem of linear regression with p predictors and one intercept, y = Xβ + ɛ, where y t = (y 1,..., y n ) is the column vector of target values,
More informationThe Matrix Algebra of Sample Statistics
The Matrix Algebra of Sample Statistics James H. Steiger Department of Psychology and Human Development Vanderbilt University James H. Steiger (Vanderbilt University) The Matrix Algebra of Sample Statistics
More informationMultivariate Statistical Analysis
Multivariate Statistical Analysis Fall 2011 C. L. Williams, Ph.D. Lecture 4 for Applied Multivariate Analysis Outline 1 Eigen values and eigen vectors Characteristic equation Some properties of eigendecompositions
More informationTHE UNIVERSITY OF CHICAGO Booth School of Business Business 41912, Spring Quarter 2016, Mr. Ruey S. Tsay
THE UNIVERSITY OF CHICAGO Booth School of Business Business 41912, Spring Quarter 2016, Mr. Ruey S. Tsay Lecture 5: Multivariate Multiple Linear Regression The model is Y n m = Z n (r+1) β (r+1) m + ɛ
More informationCanonical Correlation Analysis of Longitudinal Data
Biometrics Section JSM 2008 Canonical Correlation Analysis of Longitudinal Data Jayesh Srivastava Dayanand N Naik Abstract Studying the relationship between two sets of variables is an important multivariate
More informationPrincipal Component Analysis Applied to Polytomous Quadratic Logistic
Int. Statistical Inst.: Proc. 58th World Statistical Congress, 2011, Dublin (Session CPS024) p.4410 Principal Component Analysis Applied to Polytomous Quadratic Logistic Regression Andruski-Guimarães,
More informationFixed Effects, Invariance, and Spatial Variation in Intergenerational Mobility
American Economic Review: Papers & Proceedings 2016, 106(5): 400 404 http://dx.doi.org/10.1257/aer.p20161082 Fixed Effects, Invariance, and Spatial Variation in Intergenerational Mobility By Gary Chamberlain*
More informationTAMS39 Lecture 10 Principal Component Analysis Factor Analysis
TAMS39 Lecture 10 Principal Component Analysis Factor Analysis Martin Singull Department of Mathematics Mathematical Statistics Linköping University, Sweden Content - Lecture Principal component analysis
More informationMULTIVARIATE ANALYSIS OF VARIANCE
MULTIVARIATE ANALYSIS OF VARIANCE RAJENDER PARSAD AND L.M. BHAR Indian Agricultural Statistics Research Institute Library Avenue, New Delhi - 0 0 lmb@iasri.res.in. Introduction In many agricultural experiments,
More informationReview of Multiple Regression
Ronald H. Heck 1 Let s begin with a little review of multiple regression this week. Linear models [e.g., correlation, t-tests, analysis of variance (ANOVA), multiple regression, path analysis, multivariate
More informationPsychology 282 Lecture #4 Outline Inferences in SLR
Psychology 282 Lecture #4 Outline Inferences in SLR Assumptions To this point we have not had to make any distributional assumptions. Principle of least squares requires no assumptions. Can use correlations
More informationRandom Matrices and Multivariate Statistical Analysis
Random Matrices and Multivariate Statistical Analysis Iain Johnstone, Statistics, Stanford imj@stanford.edu SEA 06@MIT p.1 Agenda Classical multivariate techniques Principal Component Analysis Canonical
More informationHierarchical Generalized Linear Models. ERSH 8990 REMS Seminar on HLM Last Lecture!
Hierarchical Generalized Linear Models ERSH 8990 REMS Seminar on HLM Last Lecture! Hierarchical Generalized Linear Models Introduction to generalized models Models for binary outcomes Interpreting parameter
More informationAn Introduction to Matrix Algebra
An Introduction to Matrix Algebra EPSY 905: Fundamentals of Multivariate Modeling Online Lecture #8 EPSY 905: Matrix Algebra In This Lecture An introduction to matrix algebra Ø Scalars, vectors, and matrices
More informationUnsupervised Learning: Dimensionality Reduction
Unsupervised Learning: Dimensionality Reduction CMPSCI 689 Fall 2015 Sridhar Mahadevan Lecture 3 Outline In this lecture, we set about to solve the problem posed in the previous lecture Given a dataset,
More informationMultivariate Statistics
Multivariate Statistics Chapter 4: Factor analysis Pedro Galeano Departamento de Estadística Universidad Carlos III de Madrid pedro.galeano@uc3m.es Course 2017/2018 Master in Mathematical Engineering Pedro
More informationDeciphering Math Notation. Billy Skorupski Associate Professor, School of Education
Deciphering Math Notation Billy Skorupski Associate Professor, School of Education Agenda General overview of data, variables Greek and Roman characters in math and statistics Parameters vs. Statistics
More informationYORK UNIVERSITY. Faculty of Science Department of Mathematics and Statistics MATH M Test #1. July 11, 2013 Solutions
YORK UNIVERSITY Faculty of Science Department of Mathematics and Statistics MATH 222 3. M Test # July, 23 Solutions. For each statement indicate whether it is always TRUE or sometimes FALSE. Note: For
More informationShort Answer Questions: Answer on your separate blank paper. Points are given in parentheses.
ISQS 6348 Final exam solutions. Name: Open book and notes, but no electronic devices. Answer short answer questions on separate blank paper. Answer multiple choice on this exam sheet. Put your name on
More informationCarapace Measurements for Female Turtles
Carapace Measurements for Female Turtles Data on three dimensions of female turtle carapaces (shells): X 1 =log(carapace length) X 2 =log(carapace width) X 3 =log(carapace height) ince the measurements
More informationAnalysis of variance, multivariate (MANOVA)
Analysis of variance, multivariate (MANOVA) Abstract: A designed experiment is set up in which the system studied is under the control of an investigator. The individuals, the treatments, the variables
More informationSTAT 501 EXAM I NAME Spring 1999
STAT 501 EXAM I NAME Spring 1999 Instructions: You may use only your calculator and the attached tables and formula sheet. You can detach the tables and formula sheet from the rest of this exam. Show your
More informationSlides 3: Random Numbers
Slides 3: Random Numbers We previously considered a few examples of simulating real processes. In order to mimic real randomness of events such as arrival times we considered the use of random numbers
More informationLECTURE 4 PRINCIPAL COMPONENTS ANALYSIS / EXPLORATORY FACTOR ANALYSIS
LECTURE 4 PRINCIPAL COMPONENTS ANALYSIS / EXPLORATORY FACTOR ANALYSIS NOTES FROM PRE- LECTURE RECORDING ON PCA PCA and EFA have similar goals. They are substantially different in important ways. The goal
More informationECON 4160, Spring term Lecture 12
ECON 4160, Spring term 2013. Lecture 12 Non-stationarity and co-integration 2/2 Ragnar Nymoen Department of Economics 13 Nov 2013 1 / 53 Introduction I So far we have considered: Stationary VAR, with deterministic
More informationTHE UNIVERSITY OF CHICAGO Graduate School of Business Business 41912, Spring Quarter 2008, Mr. Ruey S. Tsay. Solutions to Final Exam
THE UNIVERSITY OF CHICAGO Graduate School of Business Business 41912, Spring Quarter 2008, Mr. Ruey S. Tsay Solutions to Final Exam 1. (13 pts) Consider the monthly log returns, in percentages, of five
More informationFrom Lay, 5.4. If we always treat a matrix as defining a linear transformation, what role does diagonalisation play?
Overview Last week introduced the important Diagonalisation Theorem: An n n matrix A is diagonalisable if and only if there is a basis for R n consisting of eigenvectors of A. This week we ll continue
More informationMachine Learning 2nd Edition
INTRODUCTION TO Lecture Slides for Machine Learning 2nd Edition ETHEM ALPAYDIN, modified by Leonardo Bobadilla and some parts from http://www.cs.tau.ac.il/~apartzin/machinelearning/ The MIT Press, 2010
More informationCluster Validity. Oct. 28, Cluster Validity 10/14/ Erin Wirch & Wenbo Wang. Outline. Hypothesis Testing. Relative Criteria.
1 Testing Oct. 28, 2010 2 Testing Testing Agenda 3 Testing Review of Testing Testing Review of Testing 4 Test a parameter against a specific value Begin with H 0 and H 1 as the null and alternative hypotheses
More informationLecture 4: Principal Component Analysis and Linear Dimension Reduction
Lecture 4: Principal Component Analysis and Linear Dimension Reduction Advanced Applied Multivariate Analysis STAT 2221, Fall 2013 Sungkyu Jung Department of Statistics University of Pittsburgh E-mail:
More informationRepeated Measures ANOVA Multivariate ANOVA and Their Relationship to Linear Mixed Models
Repeated Measures ANOVA Multivariate ANOVA and Their Relationship to Linear Mixed Models EPSY 905: Multivariate Analysis Spring 2016 Lecture #12 April 20, 2016 EPSY 905: RM ANOVA, MANOVA, and Mixed Models
More information1. Introduction to Multivariate Analysis
1. Introduction to Multivariate Analysis Isabel M. Rodrigues 1 / 44 1.1 Overview of multivariate methods and main objectives. WHY MULTIVARIATE ANALYSIS? Multivariate statistical analysis is concerned with
More informationSpatial inference. Spatial inference. Accounting for spatial correlation. Multivariate normal distributions
Spatial inference I will start with a simple model, using species diversity data Strong spatial dependence, Î = 0.79 what is the mean diversity? How precise is our estimate? Sampling discussion: The 64
More informationz = β βσβ Statistical Analysis of MV Data Example : µ=0 (Σ known) consider Y = β X~ N 1 (β µ, β Σβ) test statistic for H 0β is
Example X~N p (µ,σ); H 0 : µ=0 (Σ known) consider Y = β X~ N 1 (β µ, β Σβ) H 0β : β µ = 0 test statistic for H 0β is y z = β βσβ /n And reject H 0β if z β > c [suitable critical value] 301 Reject H 0 if
More informationECON 4160, Lecture 11 and 12
ECON 4160, 2016. Lecture 11 and 12 Co-integration Ragnar Nymoen Department of Economics 9 November 2017 1 / 43 Introduction I So far we have considered: Stationary VAR ( no unit roots ) Standard inference
More informationTopic 9: Canonical Correlation
Topic 9: Canonical Correlation Ying Li Stockholm University October 22, 2012 1/19 Basic Concepts Objectives In canonical correlation analysis, we examine the linear relationship between a set of X variables
More informationMATH5745 Multivariate Methods Lecture 07
MATH5745 Multivariate Methods Lecture 07 Tests of hypothesis on covariance matrix March 16, 2018 MATH5745 Multivariate Methods Lecture 07 March 16, 2018 1 / 39 Test on covariance matrices: Introduction
More informationSAMPLE OF THE STUDY MATERIAL PART OF CHAPTER 1 Introduction to Linear Algebra
1.1. Introduction SAMPLE OF THE STUDY MATERIAL PART OF CHAPTER 1 Introduction to Linear algebra is a specific branch of mathematics dealing with the study of vectors, vector spaces with functions that
More information1 A factor can be considered to be an underlying latent variable: (a) on which people differ. (b) that is explained by unknown variables
1 A factor can be considered to be an underlying latent variable: (a) on which people differ (b) that is explained by unknown variables (c) that cannot be defined (d) that is influenced by observed variables
More informationMTH 2032 SemesterII
MTH 202 SemesterII 2010-11 Linear Algebra Worked Examples Dr. Tony Yee Department of Mathematics and Information Technology The Hong Kong Institute of Education December 28, 2011 ii Contents Table of Contents
More informationEconometrics I KS. Module 2: Multivariate Linear Regression. Alexander Ahammer. This version: April 16, 2018
Econometrics I KS Module 2: Multivariate Linear Regression Alexander Ahammer Department of Economics Johannes Kepler University of Linz This version: April 16, 2018 Alexander Ahammer (JKU) Module 2: Multivariate
More informationStatistical Foundations:
Statistical Foundations: t distributions, t-tests tests Psychology 790 Lecture #12 10/03/2006 Today sclass The t-distribution t ib ti in its full glory. Why we use it for nearly everything. Confidence
More informationMachine learning for pervasive systems Classification in high-dimensional spaces
Machine learning for pervasive systems Classification in high-dimensional spaces Department of Communications and Networking Aalto University, School of Electrical Engineering stephan.sigg@aalto.fi Version
More informationExperimental Design and Data Analysis for Biologists
Experimental Design and Data Analysis for Biologists Gerry P. Quinn Monash University Michael J. Keough University of Melbourne CAMBRIDGE UNIVERSITY PRESS Contents Preface page xv I I Introduction 1 1.1
More informationModel Estimation Example
Ronald H. Heck 1 EDEP 606: Multivariate Methods (S2013) April 7, 2013 Model Estimation Example As we have moved through the course this semester, we have encountered the concept of model estimation. Discussions
More informationIntroduction to Machine Learning
10-701 Introduction to Machine Learning PCA Slides based on 18-661 Fall 2018 PCA Raw data can be Complex, High-dimensional To understand a phenomenon we measure various related quantities If we knew what
More informationSTAT 730 Chapter 5: Hypothesis Testing
STAT 730 Chapter 5: Hypothesis Testing Timothy Hanson Department of Statistics, University of South Carolina Stat 730: Multivariate Analysis 1 / 28 Likelihood ratio test def n: Data X depend on θ. The
More informationRandom Intercept Models
Random Intercept Models Edps/Psych/Soc 589 Carolyn J. Anderson Department of Educational Psychology c Board of Trustees, University of Illinois Spring 2019 Outline A very simple case of a random intercept
More informationMachine Learning Linear Classification. Prof. Matteo Matteucci
Machine Learning Linear Classification Prof. Matteo Matteucci Recall from the first lecture 2 X R p Regression Y R Continuous Output X R p Y {Ω 0, Ω 1,, Ω K } Classification Discrete Output X R p Y (X)
More informationPRINCIPAL COMPONENTS ANALYSIS
121 CHAPTER 11 PRINCIPAL COMPONENTS ANALYSIS We now have the tools necessary to discuss one of the most important concepts in mathematical statistics: Principal Components Analysis (PCA). PCA involves
More informationAn Introduction to Mplus and Path Analysis
An Introduction to Mplus and Path Analysis PSYC 943: Fundamentals of Multivariate Modeling Lecture 10: October 30, 2013 PSYC 943: Lecture 10 Today s Lecture Path analysis starting with multivariate regression
More informationVectors To begin, let us describe an element of the state space as a point with numerical coordinates, that is x 1. x 2. x =
Linear Algebra Review Vectors To begin, let us describe an element of the state space as a point with numerical coordinates, that is x 1 x x = 2. x n Vectors of up to three dimensions are easy to diagram.
More information