I L L I N O I S UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN

Comparisons of Several Multivariate Populations Edps/Soc 584 and Psych 594 Applied Multivariate Statistics Carolyn J Anderson Department of Educational Psychology I L L I N O I S UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN MANOVA Slide 1 of 66

Outline Multivariate SAS IML and 1 way ANOVA Classic Treatment As a general linear model 1 way MANOVA The Model: Generalization of ANOVA to multivariate Example 1: Massed vs distributed practice Multivariate General Linear Model and Example 2: Increased survival Following up to a significant result Multivariate contrasts Simultaneous confidence intervals Discriminant function Summary of PCA, MANOVA, DA SAS IML and PROC Reading: Johnson & Wichern pages 296 323 MANOVA Slide 2 of 66

Generalizing 1 way ANOVA to Multivariate Data Generalizing 1 way ANOVA to Multivariate Data Basic Assumptions Multivariate SAS IML and and Generalizing multivariate T 2 to more than two populations Suppose that we have random samples from g populations and measures on p variables: Population 1: Population 2: Population g: where each x lj is a (p 1) vector x 11, x 12,,x 1n1 x 21, x 22,,x 2n2 x g1, x g2,,x gng Examples: Are 5 standardized tests scores the same for high school students who attend different high school programs (ie, general, vo/tech, academic) Are survival times measured in two ways different between those treated with supplemental vitamin C the over six types of cancer? MANOVA Slide 3 of 66

Basic Assumptions These are needed for Statistical Inference Generalizing 1 way ANOVA to Multivariate Data Basic Assumptions 1 X l1, X l2,,x lnl is a random sample of size n l from a population with means µ l for l = 1,,g (ie, observations within populations are independent and representative of their populations) Multivariate 2 Random samples from different populations are independent 3 All populations have the same covariance matrix, Σ 4 X lj N(µ l,σ); that is, each population is multivariate normal If a population is not multivariate normal, then for large n l central limit theorem may kick-in SAS IML and MANOVA Slide 4 of 66

Review The univariate case where p = 1 Review The Model for an Observation The Sums of Squares Sums of Squared Decomposition & Geometry ANOVA Summary Table Multivariate SAS IML and Assumptions: X lj N(µ l, σ 2 ) iid for j = 1,,n l and l = 1,,g Hypotheses: H o : µ 1 = µ 2 = = µ g versus H a : not H o We usually express µ l as the sum of a grand mean and deviations from the grand mean µ l }{{} l th pop mean = µ }{{} grand mean = µ + τ l + µ l µ }{{} l th pop treatment effect If µ 1 = µ 2 = = µ g, then an equivalent way to write the null hypothesis is H o : τ 1 = τ 2 = = τ g = 0 MANOVA Slide 5 of 66

The Model for an Observation X lj = µ + τ l + ɛ lj Review The Model for an Observation The Sums of Squares Sums of Squared Decomposition & Geometry ANOVA Summary Table where ɛ lj N(0, σ 2 ) and independent ɛ lj is random error We typically impose the condition g l=1 τ l = 0 as an identification constraint The decomposition of an observation is X lj }{{} observation = X }{{} overall sample mean + ( X l X) }{{} estimated treatment effect + (X lj = X l ) }{{} residual error Multivariate SAS IML and X is the estimator of µ ˆτ l = ( X l X) is the estimator of τ l (X lj X l ) is the estimator of ɛ lj MANOVA Slide 6 of 66

The Sums of Squares The sum of squared observations SS obs = SS total = g l=1 n l j=1 X 2 lj Review The Model for an Observation The Sums of Squares Sums of Squared Decomposition & Geometry ANOVA Summary Table Multivariate We also take the three components of X lj and form sums of squares ( g n l g ) SS mean = X 2 = n l X 2 SS treatment = SS residual = l=1 g l=1 g l=1 j=1 n l j=1 n l j=1 ˆτ 2 l = ˆɛ 2 lj = l=1 g n l ( X l X) 2 = l=1 j=1 g n l (X lj X l ) 2 l=1 j=1 g n l ( X l X) 2 l=1 SAS IML and MANOVA Slide 7 of 66

Sums of Squared Decomposition & Geometry SS obs = SS mean + SS tr + SS res Review The Model for an Observation The Sums of Squares Sums of Squared Decomposition & Geometry ANOVA Summary Table Multivariate or SS corrected = SS obs SS mean = SS tr + SS res This work because the components (sums of squares) are orthogonal Geometry: Consider the n = ( g l=1 n l) dimensional observation space where each observation defines a dimension We break this space into three orthogonal sub-spaces corresponding to each component The dimensionality of the sub-space corresponds to the degrees of freedom for the corresponding SS (see text for more details) SAS IML and MANOVA Slide 8 of 66

ANOVA Summary Table Let n + = g l=1 n l, the total sample size Source of Variation Sum of Squares df Treatment SS tr = ( g l=1 n l) X l 2 g 1 Review The Model for an Observation The Sums of Squares Sums of Squared Decomposition & Geometry ANOVA Summary Table Multivariate SAS IML and Residual Total (SS obs SS mean ) (corrected for mean) = g l=1 SS res = g nl l=1 j=1 (X lj X l ) 2 n + g nl j=1 (X lj X) 2 n + 1 Test statistic for H o : µ 1 = = µ g (or H o : τ 1 = = τ g ) and its sampling distribution are F = SS tr/(g 1) SS res /(n + g) F (g 1),(n + g) Reject H o for large values of SS tr /SS res large values of 1 + SS tr /SS res small values of (1 + SS tr /SS res ) 1 = SS res SS tr +SS res MANOVA Slide 9 of 66

One-Way ANOVA as a One-Way ANOVA as a Least Squares Estimates of Testing Using C Multivariate X 11 X 1n1 X 21 X 2n2 X g1 X gng = 1 1 0 0 1 1 0 0 1 0 1 0 1 0 1 0 1 1 1 1 1 1 1 1 µ τ 1 τ 1 τ 2 τ 2 τ g 1 τ g 1 + ɛ 11 ɛ 1n1 ɛ 21 ɛ 2n2 ɛ g1 ɛ gng SAS IML and X n+ 1 }{{} Dependent = A n+ g }{{} Design Matrix β g 1 }{{} Parameters + ɛ n+ 1 }{{} Residuals MANOVA Slide 10 of 66

Least Squares Estimates of One-Way ANOVA as a Least Squares Estimates of Testing Using C Multivariate SAS IML and How we get parameter estimates depends on how the design matrix is set up There are multiple ways of setting up the design matrix We ll use the rank g matrix A on the previous slide ˆβ = (A A) 1 A X ˆx = Aˆβ = { x l } n+ 1 ˆɛ = X ˆX = X A(A A) 1 A X = (I A(A A) 1 A )X Our hypothesis test of equal population means, H o : µ 1 = µ 2 = = µ g τ 1 = τ 2 = = τ g = 0 can be expressed as H o : Cβ = 0 where C is a contrast matrix MANOVA Slide 11 of 66

One-Way ANOVA as a Least Squares Estimates of Testing Using C Multivariate SAS IML and Testing Using C For example, 0 1 1 0 0 0 0 0 1 1 0 0 C (g 1) g = 0 0 0 0 1 1 So τ 1 τ 2 τ 2 τ 3 H o : Cβ = 0 = τ g 2 τ g 1 Our F test (given before) tests H o : Cβ = 0 From framework, you can introduce continuous (numerical) variables ANOVA and multiple regression are essentially the same We can generalize the to the multivariate SAS PROC will make more sense MANOVA Slide 12 of 66

MANOVA model for comparing g population mean vectors parallels univariate ANOVA: Observation Vectors Sums-of-Squares and Cross-Products (SSCP) Sum of Squares A Closer Look at Within Groups SSCP X lj observation vector p 1 Random = µ overall mean vector p 1 + Fixed τ l l th treatment effect vector p 1 + ɛ lj residual for l th group, j th case p 1 Random Multivariate where ɛ lj N p (0,Σ) and all independent for j = 1,,n l cases per group, and l = 1,,g groups For Identification, g l=1 n lτ l = 0 SAS IML and MANOVA Slide 13 of 66

Observation Vectors Each component of X lj satisfies the 1-way ANOVA model, but now the model includes covariances among the components Observation Vectors Sums-of-Squares and Cross-Products (SSCP) Sum of Squares A Closer Look at Within Groups SSCP Multivariate These covariances are assumed to be equal across populations A vector of observations can be decomposed as X lj Observation = X overall sample mean + ( X l X) estimated treatment effect = ˆµ + ˆτ l + ˆɛ lj We also have a decomposition of sums-of-squares and crossproducts, or SSCP for short + (X lj X l ) residual SAS IML and MANOVA Slide 14 of 66

Sums-of-Squares and Cross-Products (SSCP) First we ll find the total corrected squares and cross-products (x lj x)(x lj x) = [(x lj x l ) + ( x l x) ] [(x lj x l ) + ( x l x) ] Observation Vectors Sums-of-Squares and Cross-Products (SSCP) Sum of Squares A Closer Look at Within Groups SSCP Multivariate = (x lj x l )(x lj x l ) + ( x l x)( x l x) }{{} squares & cross-products (x lj x l )( x l x) + ( x l x)(x lj x l ) }{{} cross-products and now sum all of this over cases and groups Since addition is distributive, we ll do this in pieces and look just at cross-product first SAS IML and MANOVA Slide 15 of 66

Observation Vectors Sums-of-Squares and Cross-Products (SSCP) Sum of Squares A Closer Look at Within Groups SSCP Multivariate Sum of Squares n l j=1 (x lj x l )( x l x) = = n l (x lj x l ) ( x l x) j=1 n l j=1 x lj n l x l ( x l x) = n l ( x l x l )( x l x) = 0 }{{} 0 Now summing the rest over j and l we get g l=1 n l j=1 (x lj x)(x lj x) = g l=1 n l ( x l x)( x l x) + Total (corrected) SSCP = Treatment + Residual g l=1 n l j=1 (x lj x l )(x lj x l ) = Between Groups + Within Groups = Hypothesis + Error SAS IML and MANOVA Slide 16 of 66

A Closer Look at Within Groups SSCP Observation Vectors Sums-of-Squares and Cross-Products (SSCP) Sum of Squares A Closer Look at Within Groups SSCP Multivariate W = E = = g n l (x lj x l )(x lj x l ) l=1 n 1 j=1 + j=1 (x 1j x 1 )(x 1j x 1 ) + n g n 2 j=1 (x gj x g )(x gj x g ) j=1 = (n 1 1)S 1 + (n 2 2)S 2 + + (n g )S g (x 2j x 2 )(x 2j x 2 ) where S l is the sample covariance matrix for the l th group (treatment, condition, etc) W (E) is proportional to a pooled estimated of the common Σ SAS IML and MANOVA Slide 17 of 66

Between Groups SSCP & Test Statistic With respect to between groups SSCP, g g B = H = n l ( x l x)( x l x) = n lˆτ lˆτ l l=1 l=1 If H o : τ 1 = τ 2 = = τ g = 0 is true, Then B (H) should be close to 0 Between Groups SSCP & Test Statistic withλ Distribution of Wilk s Lambda Λ Other Test Statistics Pillai s Trace and Roy s Largest Root How They are All Related to Λ Which Test Statistic to Use Other Cases and Summary MANOVA To test H o, we consider the ratio of generalized SSCPs, Λ = W W + B = W T where T = W + B (ie, the total corrected SSCP) Λ is known as Wilk s Lambda It s equivalent to likelihood ratio statistic Multivariate MANOVA Slide 18 of 66

with Λ Λ is a ratio of generalized sampling variances Between Groups SSCP & Test Statistic withλ Distribution of Wilk s Lambda Λ Other Test Statistics Pillai s Trace and Roy s Largest Root How They are All Related to Λ Which Test Statistic to Use Other Cases and Summary MANOVA Multivariate Λ = W T = p i=1 λ i p i=1 λ i where λ i are eigenvalues of W, and λ i are eigenvalues of T If H o : τ 1 = τ 2 = = τ g = 0 is true then B is close to 0 = T W = λ i λ i = Λ close to 1 If H o : τ 1 = τ 2 = = τ g = 0 is false then B is not close 0 = values on diagonals of T, which will be positive, will be large = λ i < λ i = Λ is small The exact distribution of Λ can be derived for special cases of p and g MANOVA Slide 19 of 66

Distribution of Wilk s Lambda Λ Wilk s Λ = SSCP e SSCP e + SSCP h Between Groups SSCP & Test Statistic withλ Distribution of Wilk s Lambda Λ Other Test Statistics Pillai s Trace and Roy s Largest Root How They are All Related to Λ Which Test Statistic to Use Other Cases and Summary MANOVA Number df for variables Hypothesis Sampling distribution for multivariate data p = 1 ν h 1 p = 2 ν h 1 p 1 ν h = 1 p 2 ν h = 2 ( νe ( ν h ) ( 1 Λ Λ ) F νh,ν e ) ( ν e 1 1 ) Λ ν h Λ F 2νh,2(ν e 1) ( ) ( ) νe +ν h p 1 Λ p Λ F p,(νe +ν h p) ( νe +ν h p 1 p )( 1 Λ Λ ) F 2p,2(νe +ν h p 1) Multivariate where ν h = degrees of freedom for hypothesis, and ν e = degrees of freedom for error (residual) MANOVA Slide 20 of 66

Other Test Statistics Between Groups SSCP & Test Statistic withλ Distribution of Wilk s Lambda Λ Other Test Statistics Pillai s Trace and Roy s Largest Root How They are All Related to Λ Which Test Statistic to Use Other Cases and Summary MANOVA Multivariate There is more than one way to combine the information in B and W (or H and E) Wilk s Λ Λ = W W + B = E p E + H = i=1 λ i where λ i are eigenvalues of E, and λ i (E + H) Hotelling-Lawely Trace Criteria = trace(e 1 H) = tr(he 1 ) = where λ i is eigenvalue of HE 1 Reject H o when tr(he 1 ) is large When H o is true, ntr(he 1 ) χ 2 p(g 1) p i=1 λ i are eigenvalues of g λ i i=1 Note: df = rank of design matrix ( approach) MANOVA Slide 21 of 66

Pillai s Trace and Roy s Largest Root Pillai s Trace Criterion = trace(b(b + W) 1 ) = trace(h(h + E) 1 ) = p i=1 λ i 1 + λ i where λ i is the eigenvalue (root) of HE 1 Between Groups SSCP & Test Statistic withλ Distribution of Wilk s Lambda Λ Other Test Statistics Pillai s Trace and Roy s Largest Root How They are All Related to Λ Which Test Statistic to Use Other Cases and Summary MANOVA Roy s Largest Root Criterion θ = largest root of (E + H) 1 H = largest root of H(E + H) 1 ( ) λ1 = 1 + λ 1 where λ 1 is the largest root of E 1 H = HE 1 Multivariate MANOVA Slide 22 of 66

How They are All Related to Λ Let λ i be root of HE 1 ( eigenvalue of H relative to E ) and if all λ i s > 0 (ie, λ 1 λ 2 λ p 0), Between Groups SSCP & Test Statistic withλ Distribution of Wilk s Lambda Λ Other Test Statistics Pillai s Trace and Roy s Largest Root How They are All Related to Λ Which Test Statistic to Use Other Cases and Summary MANOVA Then we can write Λ = = = E E + H 1 I + HE 1 1 p i=1 (1 + λ i ) So Λ is a decreasing function of λ i Note: θ j = because AB = A B λ i 1 + λ i and λ i = θ i 1 θ i Multivariate MANOVA Slide 23 of 66

Which Test Statistic to Use Wilk s Λ = likelihood ratio statistic If all statistics lead to the same conclusion, use Λ Between Groups SSCP & Test Statistic withλ Distribution of Wilk s Lambda Λ Other Test Statistics Pillai s Trace and Roy s Largest Root How They are All Related to Λ Which Test Statistic to Use Other Cases and Summary MANOVA If statistics lead to different conclusion, need to figure out why From Simulation studies (power & robustness): Roy s largest root found to be the least useful, except when the population structure is such that groups differ in one dimension and one group is much more different from the rest Others all do pretty good w/rt power (they use more information in E and H than Roys s) Pillai s trace criterion is Least affected by departures from usual population model (ie, more robust against departures from normality) Better for diffuse alternative hypotheses versus sharper ones When roots are approximately equal, it has best power Wilk s and Hotwelling-Lawley have about the same power for a wider-range (spectrum) of alternative hypotheses Multivariate MANOVA Slide 24 of 66

Other Cases and Summary MANOVA For cases not covered If H o is true and g l=1 n l = n is large, then ( lnλ n 1 1 ) 2 (p + g) = ( W / B+W ) (n 1 12 ) (p + g) χ 2 p(g 1) Between Groups SSCP & Test Statistic withλ Distribution of Wilk s Lambda Λ Other Test Statistics Pillai s Trace and Roy s Largest Root How They are All Related to Λ Which Test Statistic to Use Other Cases and Summary MANOVA Multivariate You should examine the residual vectors for normality and outliers (ie, ˆɛ lj s) maybe use PCA or methods mentioned in the text Source of variation SSCP df Λ Wilk s Treatment B = g l=1 n l( x l x)( x l x) g 1 W / T (Between) Residual W = g l=1 (Within) n l j=1 (x jl x l )(x jl x l ) n g Total (corrected T = W + B n 1 for mean) = g l=1 n l j=1 (x jl x)(x jl x) MANOVA Slide 25 of 66

1 Way MANOVA: Data from Tatsuoka (1988), Multivariate Analysis: Techniques for Educational and Psychological Research, pp 273 279 An experiment was conducted for comparing 2 methods (A & B) of teaching shorthand to 60 female seniors in a vocational high school (a dated example) Also of interest were the effects of distributed versus massed practice Descriptive Statistics Means and Confidence Regions Hypothesis Test Hypothesis Test continued Test Statistic & Distribution Multivariate C 1 : C 2 : C 3 : 2 hours of instruction/day for 6 weeks 3 hours of instruction/day for 4 weeks 4 hours of instruction/day for 3 weeks So each subject received a total of 12 hours of instruction For now, we ll just look the effect of distributed versus massed practice Note: n l = 20 for l = 1, 2, 3 Two variables (dependent measures): X 1 = speed X 2 = accuracy SAS IML and MANOVA Slide 26 of 66

Descriptive Statistics The overall mean vector and mean vectors for each condition: ( ) ( ) ( ) ( 3362 3855 3400 x = x 1 = x 2 = x 3 = 1825 2370 1820 2830 1285 ) Descriptive Statistics Means and Confidence Regions Hypothesis Test Hypothesis Test continued Test Statistic & Distribution Multivariate The treatment effect vectors (ie, ˆτ i = x i x) ( ) ( ) 493 038 ˆτ 1 = ˆτ 2 = ˆτ 3 = 545 005 Sample covariance matrices: ( ) 4952 1317 S 1 = 1317 759 S 3 = ( S 2 = ( 1633 442 442 319 ( 532 540 2747 421 421 448 ) ) ) SAS IML and MANOVA Slide 27 of 66

Means and Accuracy 50 40 95% confidence regions for µ 1, µ 2 and µ 3 where n l = 20 Descriptive Statistics Means and Confidence Regions Hypothesis Test Hypothesis Test continued Test Statistic & Distribution 30 20 10 x1 (2 hours/day for 6 weeks) x x2 (3 hours/day for 4 weeks) x3 (4 hours/day for 3 weeks) 0 10 20 30 40 50 Speed Multivariate SAS IML and MANOVA Slide 28 of 66

Hypothesis Test No difference between massed versus distributed practice on either speed or accuracy: H o : τ 1 = τ 2 = τ 3 = 0 versus H a : τ l 0 for all l = 1, 2, 3 The within groups (residual) sums of squares and cross-products matrix Descriptive Statistics Means and Confidence Regions Hypothesis Test Hypothesis Test continued Test Statistic & Distribution Multivariate W = (n 1 1)S 1 + (n 2 1)S 2 + (n 3 1)S 3 ( ) ( 4952 1317 2747 421 = 19 + 19 1317 759 421 448 ( ) 1633 442 +19 442 319 ( ) 177315 41420 = 41420 28995 ) SAS IML and MANOVA Slide 29 of 66

Hypothesis Test continued Descriptive Statistics Means and Confidence Regions Hypothesis Test Hypothesis Test continued Test Statistic & Distribution Multivariate The between groups SSCP matrix: 3 B = n l ( x l x)( x l x) l=1 ( ) ( = 20 493 545 (493, 545) + 20 ( ) +20 532 540 ( 532, 540) ( ) = 1055033 111155 111155 117730 T = W + B = ( 038 005 282818 152575 152575 146725 ) (038, 05) Or T = (n 1)S where S is the covariance matrix computed over all groups and n is the total sample Then B = T W ) SAS IML and MANOVA Slide 30 of 66

Test Statistic & Distribution W = (177315)(28995) (41420) 2 = 3425632 Descriptive Statistics Means and Confidence Regions Hypothesis Test Hypothesis Test continued Test Statistic & Distribution Multivariate SAS IML and T = (282818)(146725) (152575) 2 = 18217389 Λ = W T = 3425632 18217389 = 0188 For p = 2 and g = 3, we can use the exact sample distribution: ( ) ( n g 1 1 ) ( ) ( Λ n p 2 1 ) Λ = F g 1 Λ p Λ 2(g 1),2(n g 1) which for this example, (60 3 1) (3 1) ( 1 ) 188 188 = 56 2 ( ) 566 434 = 36568 Since F 4,112 (α = 05) F 4,120 (α = 05) = 245, reject H o that treatment vectors are all equal to 0 The data support the conclusion that there is an effect of massed versus distributed practice MANOVA Slide 31 of 66

1 Multivariate contrasts & confidence regions 2 Tests on individual variables (simultaneous confidence intervals for group/treatment differences) 3 Discriminant Analysis Multivariate Contrasts We need the multivariate generalization of the general linear model: X gn p = A gn (g+1) B (g+1) p + E gn p Multivariate SAS IML and where A is the design matrix (it could have g or g + 1 columns depending on the parameterization), and B is a matrix of coefficients (model parameters) some examples MANOVA Slide 32 of 66

A is n + g with dummy codes Multivariate A is n + g with dummy codes SAS PROC design: A is gn (g + 1) What s Interesting Estimable and Testable Example: Cameron & Pauling Data Example: Descriptive statistics s of MANOVA Hypothesis Test MANOVA Estimated MANOVA Slide 33 of 66 parameters X n+ p = 1 1 0 0 1 1 0 0 1 0 1 0 1 0 1 0 1 0 0 0 1 0 0 0 β o1 β o2 β op β 11 β 12 β 1p β 21 β 22 β 2p β g 1,1 β g 1,2 β g 1,p Given the design matrix above, β ok = µ gk, and β lk = µ lk µ gk If p = 1, we would have 1-way ANOVA +E n+ p

Multivariate A is n + g with dummy codes SAS PROC design: A is gn (g + 1) What s Interesting Estimable and Testable Example: Cameron & Pauling Data Example: Descriptive statistics s of MANOVA Hypothesis Test MANOVA Estimated MANOVA Slide 34 of 66 parameters SAS PROC design: A is gn (g + 1) An alternative design matrix and parameter vector: 1 1 0 0 1 1 0 0 1 0 1 0 β o1 β o2 β op β 11 β 12 β 1p X n+ p = β 1 0 1 0 21 β 22 β 2p β 1 0 0 0 g 1,1 β g 1,2 β g 1,p 1 0 0 1 n + (g+1) Normally, ˆB = (A A) 1 A X; however, the rank of A defined above (and hence A A) is only g = There s no unique solution to A X = A AB +E n+ p

What s Interesting We re interested in differences between group means; that is, µ i µ k = (µ + τ i ) p 1 (µ + τ k ) p 1 = τ i τ k Even if we can t get unique estimates of elements of B, we can get unique estimates of differences between parameter estimates, which correspond to differences between group means regardless of what inverse of (A A) is used Moore-Penrose inverses of non-full rank square matrix (A A) is denoted by (A A) Multivariate A is n + g with dummy codes SAS PROC design: A is gn (g + 1) What s Interesting Estimable and Testable Example: Cameron & Pauling Data Example: Descriptive statistics s of MANOVA Hypothesis Test MANOVA Estimated MANOVA Slide 35 of 66 parameters SAS PROC uses the Moore-Penrose inverse of (A A) In SAS/PROC IML, the Moore-Penrose inverse is obtained by the command ginv( ), for example giaa = ginv(a *A);

Estimable and Testable What we can do is test linear combinations of elements of B if the linear combination is a contrast Estimable: A linear function c B is estimable if (A A)(A A) c = c Multivariate A is n + g with dummy codes SAS PROC design: A is gn (g + 1) What s Interesting Estimable and Testable Example: Cameron & Pauling Data Example: Descriptive statistics s of MANOVA Hypothesis Test MANOVA Estimated MANOVA Slide 36 of 66 parameters Testable: A linear function is testable if it only involves the estimable functions of B Contrasts of elements of B are estimable and therefore testable These correspond to differences between means We ll demonstrate multivariate General linear model by example

Example: Cameron & Pauling Data Increase in survival of cancer patients given supplemental treatment with vitamin C Increase in survival = the number of days a patient survives minus the number of days matched control survives x 1 = d 1 = increase in survival measured as days from first hospitalization Multivariate A is n + g with dummy codes SAS PROC design: A is gn (g + 1) What s Interesting Estimable and Testable Example: Cameron & Pauling Data Example: Descriptive statistics s of MANOVA Hypothesis Test MANOVA Estimated MANOVA Slide 37 of 66 parameters x 2 = d 2 = increase in survival measured days from un-treatability type = type of cancer (1 =stomach, 2 =bronchus, 3 =colon, 4 =rectum, 5 =bladder, 6 =kidney)

Example: Descriptive statistics Multivariate A is n + g with dummy codes SAS PROC design: A is gn (g + 1) What s Interesting Estimable and Testable Example: Cameron & Pauling Data Example: Descriptive statistics s of MANOVA Hypothesis Test MANOVA Estimated MANOVA Slide 38 of 66 parameters l Type n l x i = d i S l 1 Stomach 12 7067 12586988 1520670 9492 1520670 1223536 2 Bronchus 16 4950 2391507 1746147 10688 1746147 1661945 3 Colon 16 11719 25202776 14788476 29319 14788476 11827456 4 Rectum 7 29743 50871562 15525005 22657 15525005 5634062 5 Bladder 5 130420 374766320 21407105 12980 21407105 1469770 6 Kidney 7 11886 12901848 334462 10171 334462 1739291 ( ) 5 41966842 7681585 S pool = (n l 1)S l = 7681585 45848122 l=1

Plot of Means and 95% d2 (untreatable) Multivariate A is n + g with dummy codes SAS PROC design: A is gn (g + 1) What s Interesting Estimable and Testable Example: Cameron & Pauling Data Example: Descriptive statistics s of MANOVA Hypothesis Test MANOVA Estimated MANOVA Slide 39 of 66 parameters colon x 3 bronchus x 2 x1 stomach x 6 kidney x 4 rectum x 5 bladder d1 (1 st hospitalization) Using S i to the compute regions n 1 = 12, n 2 = 16, n 3 = 16, n 4 = 7, n 5 = 5, n 6 = 7

Plot of Means and 95% d2 Multivariate A is n + g with dummy codes SAS PROC design: A is gn (g + 1) What s Interesting Estimable and Testable Example: Cameron & Pauling Data Example: Descriptive statistics s of MANOVA Hypothesis Test MANOVA Estimated MANOVA Slide 40 of 66 parameters colon x 3 bronchus x 2 x1 stomach x 6 kidney x 4 rectum Using S pool to the compute regions x 5 bladder d1

s of MANOVA Hypothesis Test H o : µ stomach = µ bronchus = µ colon = µ rectum = µ bladder = µ kidney or equivalently Multivariate A is n + g with dummy codes SAS PROC design: A is gn (g + 1) What s Interesting Estimable and Testable Example: Cameron & Pauling Data Example: Descriptive statistics s of MANOVA Hypothesis Test MANOVA Estimated MANOVA Slide 41 of 66 parameters H o : τ stomach = τ bronchus = τ colon = τ rectum = τ bladder = τ kidney df type of cancer (hypothesis) ν h = g 1 = 6 1 = 5 df within (error) = ν e = l n l g = 63 6 = 57 Wilk s Λ = det(w)/ det(t) = 05817749 Since p = 2 dependent variables, Wilk s Λ has an exact sampling distribution that is F, in particular ( ) ( (νe 1) 1 ) Λ F = F 2νh,2ν Λ e ν h F = 34838 and p-value = 0005 Reject H o The data support the conclusion that not all of the means (or τ s are equal)

Estimated MANOVA parameters ˆµ = ( 20556 16646 ) Multivariate A is n + g with dummy codes SAS PROC design: A is gn (g + 1) What s Interesting Estimable and Testable Example: Cameron & Pauling Data Example: Descriptive statistics s of MANOVA Hypothesis Test MANOVA Estimated MANOVA Slide 42 of 66 parameters Type hospitalization untreatable Stomach ˆτ 1 = ( 134888, 71543) Bronchus ˆτ 2 = (156055, 59585) Colon ˆτ 3 = ( 88368, 126727) Rectum ˆτ 4 = (91873, 60111) Bladder ˆτ 5 = (1098644, 36660) Kidney ˆτ 6 = ( 86698, 64746) Recall that µ l = µ + τ l

MANOVA as a multivariate A main effect and six dummy variables (this is what PROC does) So the design matrix looks like 1 1 0 0 0 0 0 } n stomach 1 0 1 0 0 0 0 } n bronchus A n+ 7 = 1 0 0 1 0 0 0 } n colon 1 0 0 0 1 0 0 } n rectum 1 0 0 0 0 1 0 } n bladder 1 0 0 0 0 0 1 } n kidney Multivariate A is n + g with dummy codes SAS PROC design: A is gn (g + 1) What s Interesting Estimable and Testable Example: Cameron & Pauling Data Example: Descriptive statistics s of MANOVA Hypothesis Test MANOVA Estimated MANOVA Slide 43 of 66 parameters MANOVA (multivariate general linear model): Estimation: y n+ 2 = A n+ 7B 7 2 + ɛ n+ 2 B = (A A) A y Predicted values: ŷ = AB where ŷ jl = ( x 1l, x 2l )

MANOVA as a multivariate (continued) For ŷ jl = ( x 1l, x 2l ), it is the the case that x j1l = b o1 + b l1 and x j2l = b o2 + b l2 So to compare two groups (types of cancer), Multivariate A is n + g with dummy codes SAS PROC design: A is gn (g + 1) What s Interesting Estimable and Testable Example: Cameron & Pauling Data Example: Descriptive statistics s of MANOVA Hypothesis Test MANOVA Estimated MANOVA Slide 44 of 66 parameters x il x il = (b oi + b li ) (b oi + b l i) = b li b l i Consider a contrast between means for two types of cancer, for example, stomach and bronchial, c B = (0, 1, 1, 0, 0, 0, 0) b o1 b o2 b 11 b 12 b 21 b 22 b 31 b 32 b 41 b 42 b 51 b 52 b 61 b 62 = ((b 11 b 21 ), (b 12 b 22 )) = (( x 11 x 21 ), ( x 12 x 22 ))

With the Parameter Estimates B = 32631 15884 25564 6393 27681 5197 20912 13434 2888 6773 97789 2904 Multivariate A is n + g with dummy codes SAS PROC design: A is gn (g + 1) What s Interesting Estimable and Testable Example: Cameron & Pauling Data Example: Descriptive statistics s of MANOVA Hypothesis Test MANOVA Estimated MANOVA Slide 45 of 66 parameters stomach: x 11 = b o1 + b 11 = 32631 + ( 25564) = 7067 x 12 = b o2 + b 12 = 15884 + ( 6393) = 9491 bronchus: x 21 = b o1 + b 21 = 32631 + ( 27681) = 4950 x 22 = b o2 + b 22 = 15884 + ( 5197) = 10687 (0, 1, 1, 0, 0, 0, 0)B = (( 25564 + 27681), ( 6393 5197)) = (2117, 1196) = (( x 11 x 21 ), ( x 12 x 22 )) = 0 ie, Are effects for stomach and bronchus the same?

Testing H o : CB = 0 Our hypothesis tests can be of the form H o : C r g B g p M p s = 0 C defines hypotheses (contrasts) on the elements of columns of B; that is, comparison between means on same variables over groups M defines hypotheses (contrasts) on the elements of rows of B; that is, comparison between means on same group over variables For now M = I and we ll consider hypotheses of the form Multivariate A is n + g with dummy codes SAS PROC design: A is gn (g + 1) What s Interesting Estimable and Testable Example: Cameron & Pauling Data Example: Descriptive statistics s of MANOVA Hypothesis Test MANOVA Estimated MANOVA Slide 46 of 66 parameters H o : CB = 0 Specifically, we want to consider H o : 0b 0k +c 1 b 1k +c 2 b 2k + +c g b gk c 1 τ 1 +c 2 τ 2 + +c g τ g where g l=1 c l = 0

Testing Contrasts: The H matrix For a simple contrast, such as c = (0, 1, 1, 0, 0) we could do this as a multivariate T 2 test for independent groups; however, we ll stay within the MANOVA and multivariate linear model framework (so we can test multiple ones) Suppose that we have a contrast matrix C r (g+1) where the rows are r orthogonal contrasts, the hypothesis matrix equals H = (CB) (C(A A) C ) 1 (CB) Multivariate A is n + g with dummy codes SAS PROC design: A is gn (g + 1) What s Interesting Estimable and Testable Example: Cameron & Pauling Data Example: Descriptive statistics s of MANOVA Hypothesis Test MANOVA Estimated MANOVA Slide 47 of 66 parameters For a balanced design (ie, n 1 = n 2 = = n g = n and a single contrast (ie, r = 1), this reduces to ( g ) ( g ) n H = g l=1 c c l x l c l x l l l=1 l=1 Note: c o = 0

Multivariate A is n + g with dummy codes SAS PROC design: A is gn (g + 1) What s Interesting Estimable and Testable Example: Cameron & Pauling Data Example: Descriptive statistics s of MANOVA Hypothesis Test MANOVA Estimated MANOVA Slide 48 of 66 parameters Testing Contrasts The Error matrix is W ; that is, E = W = g l=1 n (x jk x l )(x jk x l ) j=1 = X X B (A A)B Wilk s Lambda for the test H o : CB = 0 is Λ = det(e) det(h + E) To find the transformation of this to an F distribution: p = anything ν hypothesis = r ν error = l n l p

Example: Five types the same? Three equivalent forms for Hypothesis 1: H o : H o : H o : µ bronchus = µ colon = µ kidney = µ rectum = µ stomach τ bronchus = τ colon = τ kidney = τ rectum = τ stomach β bronchus = β colon = β kidney = β rectum = β stomach Multivariate A is n + g with dummy codes SAS PROC design: A is gn (g + 1) What s Interesting Estimable and Testable Example: Cameron & Pauling Data Example: Descriptive statistics s of MANOVA Hypothesis Test MANOVA Estimated MANOVA Slide 49 of 66 parameters where β l is a p 1 column vector of B (ie, a row of B written as a column) For the contrast matrix we need to know the order of the effects in the I re-order them so that they are in alphabetical order, because PROC puts them in alphabetical order (or numerical if groups are coded this way)

Multivariate A is n + g with dummy codes SAS PROC design: A is gn (g + 1) What s Interesting Estimable and Testable Example: Cameron & Pauling Data Example: Descriptive statistics s of MANOVA Hypothesis Test MANOVA Estimated MANOVA Slide 50 of 66 parameters H o : Four types the same? H o : CB = 0 = So H o : 0 0 1 1 0 0 0 0 0 1 0 1 0 0 0 0 1 0 0 1 0 0 0 1 0 0 0 1 (β 21 β 31 ) (β 22 β 32 ) (β 21 β 41 ) (β 22 β 42 ) (β 21 β 51 ) (β 22 β 52 ) (β 21 β 61 ) (β 22 β 62 ) = β 01 β o2 β 11 β 12 β 21 β 22 β 31 β 32 β 41 β 42 β 51 β 52 β 61 β 61 (τ 21 τ 31 ) (τ 22 τ 32 ) (τ 21 τ 41 ) (τ 22 τ 42 ) (τ 21 τ 51 ) (τ 22 τ 52 ) (τ 21 τ 61 ) (τ 22 τ 62 ) intercept bladder bronchus colon kidney rectum stomach = 0

Hypothesis Matrices H = (CB) (C(A A)(ns + nb + nc + nr + nd + nk 6); C ) 1 (CB) ( ) 32436992785 18071748204 18071748204 42924340815 The E error SSCP is the same as W that we used before, which equals Multivariate A is n + g with dummy codes SAS PROC design: A is gn (g + 1) What s Interesting Estimable and Testable Example: Cameron & Pauling Data Example: Descriptive statistics s of MANOVA Hypothesis Test MANOVA Estimated MANOVA Slide 51 of 66 parameters E = W = X B(A A)B X ( = 24340768476 44553193042 44553193042 2659191047 Wilk s Lambda, Λ = det(e) det(h + E) = 44877E13 54684E13 = 0820661 and ν h = number of rows of C = 4 ν e = number of rows of X p = 57 )

s Referring to the table for transformations of Λ that have sampling distributions that are F, we use the one for p = 2 and ν h 1, which is F = ( νe 1 ν h ) ( 1 Λ Λ ) = ( 56 4 ) ( 1 0820661 0820661 ) = 14541861 Multivariate A is n + g with dummy codes SAS PROC design: A is gn (g + 1) What s Interesting Estimable and Testable Example: Cameron & Pauling Data Example: Descriptive statistics s of MANOVA Hypothesis Test MANOVA Estimated MANOVA Slide 52 of 66 parameters If the null is true, then this should have a F 2νh,2(ν e 1) sampling distribution Comparing F = 145 to the F 4,112, we find that the p-value is 18 Retain the null hypothesis The data suggest not difference in increased survival of patients over different types of cancer (except bladder)

Multivariate A is n + g with dummy codes SAS PROC design: A is gn (g + 1) What s Interesting Estimable and Testable Example: Cameron & Pauling Data Example: Descriptive statistics s of MANOVA Hypothesis Test MANOVA Estimated MANOVA Slide 53 of 66 parameters Five versus the Rest H o : τ bladder = τ bronchus +τ colon +τ kidney +τ rectum +τ stomach )/5 or equivalently H o : CB = 0 = (0, 5, 1, 1, 1, 1, 1) β o1 β o2 β 11 β 12 β 21 β 22 β 31 β 32 β 41 β 42 β 51 β 52 β 61 β 61 E is the same as before, but now ( ) 62660386574 1861059247 H = 1861059247 5527481893 intercept bladder bronchus colon kidney rectum stomach

Multivariate A is n + g with dummy codes SAS PROC design: A is gn (g + 1) What s Interesting Estimable and Testable Example: Cameron & Pauling Data Example: Descriptive statistics s of MANOVA Hypothesis Test MANOVA Estimated MANOVA Slide 54 of 66 parameters The Test and Λ = 44877E13 63332E13 = 07085934 ν e = g l=1 n l g = 57 ν h = 1, the number of rows of C So, for ν h = 1, use ( )( ) νe + ν h p 1 Λ F = p Λ = 11514901, which if the null is true (and assumptions valid), F should have a sampling distribution that is F p,(νe +ν h p) Comparing F to F 2,56, we get a p-value< 01 Reject H o In summary, the mean survival of patients with bladder cancer differs from that of those with other types of cancer; however, no support for differences between the other types Question: Are there differences for survival from first hospitalization and/or from time of unreadability?

We can construct simultaneous confidence intervals for components of differences τ l τ l (which equal µ l µ l ) Multivariate Using CBM = 0 Confidence interval for CBM Our example: CI for CBM Notes about These CIs Bonferroni Roy s Method A Truly Multivariate Follow-Up MANOVA Slide 55 of 66 or other linear combinations such as τ 1 (τ 2 + τ 3 )/2 There are at least three ways of doing this Specify a matrix M in the hypothesis test H o : CBM = 0 that is a (p 1) vector with all M = (0,, 1 }{{} i th, 0) Bonferroni-type: Same as above but split the α into pieces, on part for each of the planned comparisons Roys method, which is based on the union-intersection principle

Using CBM = 0 C picks out which two (or more groups to compare) eg, want to compare bladder with the rest, C = (0, 1, 2, 2, 2, 2, 2) M picks out which variable (or linear combination of variables) eg, Just compare d 1, increase in survival from first hospitalization, M = (1, 0) Multivariate Using CBM = 0 Confidence interval for CBM Our example: CI for CBM Notes about These CIs Bonferroni Roy s Method A Truly Multivariate Follow-Up MANOVA Slide 56 of 66 Putting these together in our example gives us (0, 1, 2, 2, 2, 2, 2) β o1 β o2 β 11 β 12 β 21 β 22 β 31 β 32 β 41 β 42 β 51 β 52 β 61 β 61 ( 1 0 ) = ( β 11 1 5 ) 6 β l1 l=2

Confidence interval for CBM β 11 1 5 We need two things: 6 6 β l1 = τ 11 l=2 l=2 A fudge-factor a value from a probability distribution τ l1 Multivariate Using CBM = 0 Confidence interval for CBM Our example: CI for CBM Notes about These CIs Bonferroni Roy s Method A Truly Multivariate Follow-Up MANOVA Slide 57 of 66 An estimate of the standard error A (1 α)100% confidence statement given vectors C 1 (g+1) and M p 1 is CBM ± F 1,νe (α) (M S pool M)(C(A A) C ) Note: Consider two columns of B, β i and β k, the covariance matrix between them is cov(β i, β k ) = s pool,ik (A A)

Our example: CI for CBM β 11 1 5 6 β l1 = 130420 1 (4950 + 11719 + 11886 + 29743 + 7067) 5 l=2 = 117347 (M S pool M)(C(A A) C ) = s pool,11 (02197619) = (42703103)(02197619) = 93845152 Multivariate Using CBM = 0 Confidence interval for CBM Our example: CI for CBM Notes about These CIs Bonferroni Roy s Method A Truly Multivariate Follow-Up MANOVA Slide 58 of 66 And F 1,57 (05) = 401 So our 95% confidence interval is 130420 ± 401 93845152 130420 ± 401(30634) (56003, 178691) Since 0 is not in the interval, the mean increase in survival from first hospitalization due to bladder cancer is larger than the average of the others means Should we test whether the same is true for increase in survival from time of untreatability?

Plot of Means and 95% d2 Multivariate Using CBM = 0 Confidence interval for CBM Our example: CI for CBM Notes about These CIs Bonferroni Roy s Method A Truly Multivariate Follow-Up MANOVA Slide 59 of 66 x 3 x 4 x 2 x 5 x 6 x1 d1

Notes about These CIs If you re only looking (testing) the difference between two means, eg, τ li τ l i Multivariate Using CBM = 0 Confidence interval for CBM Our example: CI for CBM Notes about These CIs Bonferroni Roy s Method A Truly Multivariate Follow-Up MANOVA Slide 60 of 66 Then the standard error is just s pool,ii 1/nl + 1/n l When looking at a difference for a variable (eg, above), these confidence statements are equivalent to what you would get from 1-way ANOVA using Fisher s least significant differences; that is, they are univariate CIs When considering a linear combination of variables, these CIs are equivalent to univariate CIs where you ve analyzed a new or composite variable defined by the linear combination In our example, we don t have to worry too much about inflated Type I error rate, because we only did one CI after rejecting the overall test and using multivariate contrasts to narrow down where differences exist If you do all pairwise differences, there are g(g 1)/2 pairs times p variables (eg, 2(6)(5)/2 = 30)

Bonferroni If you have planned to look at all pairwise comparisons before looking at the data (ie, m = pg(g 1)/2), then you can use as your fudge factor t νe (α/(2m)) Multivariate Using CBM = 0 Confidence interval for CBM Our example: CI for CBM Notes about These CIs Bonferroni Roy s Method A Truly Multivariate Follow-Up MANOVA Slide 61 of 66 Let n + = g l=1 n l For the model X lj = µ + τ l + ɛ lj with j = 1,,n l and l = 1,,g with confidence at least (1 α), (τ li τ l i) belongs to ( x li x l i) ± t νe (α/(2m)) s pool,ii ( 1 n l + 1 n l for all components (variables) i = 1,,p and all differences l < l = 1,,g No need to do this with our cancer data )

Multivariate Using CBM = 0 Confidence interval for CBM Our example: CI for CBM Notes about These CIs Bonferroni Roy s Method A Truly Multivariate Follow-Up MANOVA Slide 62 of 66 Roy s Method This is based on the union intersection principle This is more like the first method that we considered (ie, CBM); however, we use a different distribution for our fudge factor We use Greatest Root, of H θ(e + H) = 0 where H (between groups or hypothesis SSCP matrix) and E = W (error or within groups SSCP matrix) are independent Wishart matrices To apply this result, we need percentiles of the greatest root distribution of the largest root λ of the equation H λe = 0 Percentile can be found in tables This distribution does not depend on Σ but only on df = n g p 1 Recommendation: I ld suggest using Scheffé s method where you do 1-way ANOVA on a linear combination of variables and then specify the contrast that you want

d2 1 st PC Multivariate Summary: PCA, MANOVA, DA SAS IML and x 3 x x 4 x 2 x 5 x 6 x1 d1 1 st Discriminant MANOVA Slide 63 of 66

SAS IML and SAS IML code using traditional approach & one PROC data=vitc; Multivariate SAS IML and SAS IML and SAS Output class type ; model d1 d2 = type /solution; * Note: The order of the values in the contrast are alphabetical, in this case order is bladder bronchus colon kidney rectum stomach; contrast bronchus=colon=kidney=rectum=stomach type 0 1 0 0 0-1, type 0 0 1-1 0 0, type 0 1-1 -1 0 1, type 0 1 1 1-4 1; contrast bladder vs others type -5 1 1 1 1 1 ; manova h=type /printh printe; estimate b vs o type 1-2 -2-2 -2-2; lsmeans type; title MANOVA of vitamin C and Cancer ; Alternate MANOVA statement where M is entered as M : manova h=type M=(1 0); MANOVA Slide 64 of 66

SAS Output Univariate ANOVAs for each dependent variable If requested, E (printe) and H (printh) SSCP matrices Multivariate p characteristic roots and vectors of E 1 H (ie, discriminant functions) Other requested statistics: contrasts estimates of contrasts cell means etc Test statistics for no overall effect specified in MANOVA statement SAS IML and SAS IML and SAS Output MANOVA Slide 65 of 66