Analysis of Covariance (ANCOVA) Lecture Notes

Transcription

1 1 Analysis of Covariance (ANCOVA) Lecture Notes Overview: In experimental methods, a central tenet of establishing significant relationships has to do with the notion of random assignment. Random assignment solves a couple of problems. Statistically, it ensures that, in the main, the resulting probability will be independent of the starting conditions of an experiment. Secondly, it is a way of establishing parity, which is to say that it is a method of controlling for what you don t know. ANCOVA is a method that can be thought of as a cross between ANOVA and Regression. It is, in fact, an ANOVA where the effects of some other variable have been controlled for statistically. There are three reasons why one would choose to use an ANCOVA: 1. Reduction of within group or error variance to increase the sensitivity of a test of main effects and/or interactions, by reducing the error term. 2. Elimination of systematic bias to adjust the means on the d.v. to what they would be if all subjects scored equally on the c.v. 3. Stepdown Analysis to compare scores on a d.v. after they are adjusted for scores on another DV, which is treated as a c.v. (MANOVA model). In the first case, some nuisance variable(s) might be used as the covariate(s). As an example, consider the following. One is trying to study the effectiveness of several teaching methods. The DV (variate) might be a score on an achievement test. Since some of the within condition variability will be attributable to individual differences in test taking ability, another measure of test-taking ability might be used as a covariate. If students were randomly assigned, this tactic would increase the sensitivity of the experiment by reducing error variance. A common example for the second case involves situations where random assignment is not possible so called intact groups. For instance, consider the example of whether training for detection of deception is more effective for some careers. People from three careers are randomly sampled, Psychiatrists, Judges, and Secret Service Agents. Their ability to detect whether or not someone is trying to deceive them is measured and it is the DV. However, the study as it stands is flawed because there could be some personality variables that resulted in some types of people being more inclined toward certain professions. So, the ability to detect deception may be a function of personality rather than training e.g., perhaps Psychiatrists are, on average, the most trusting and Secret Services are the least trusting. To remedy this, one could use some appropriate measure of personality as a covariate. ANCOVA is applied to the same kinds of research questions as ANOVA, however, it is a special case of ANOVA where one has some other effect one wishes to control for prior to conducting the ANOVA. An ANCOVA yields the following effects Covariate the effect of the covariate is tested for significance and is paramount to testing whether a regression equation is significant. Main Effects for independent variables are tested for significance. Interaction Effects for independent variables are tested. Can you have interactions between IVs and the CV(s)? Yes, but it implies the assumption of homogeneous regression has been violated (see below). As with ANOVA, one can estimate effect sizes, confidence intervals, etc. Much of ANCOVA is redundant with ANOVA. Also, more complicated designs can be used, such as repeated measures, multiple covariates, n-way interactions, etc. Not all software packages may be able to handle the more complicated designs.

2 2 While ANCOVA seems like a procedure tailor made for non-experimental situations and situations where experimental control couldn t be appropriately practiced, it has its problems. It is tempting to think that if the effect of some nuisance variable was partialed out, the resulting observed significant differences among treatment means are somehow more pure, i.e., any remaining significant differences must be causal in nature. This is a false sense of insight, as causation is something of a different matter. Stevens (2002) highlights some problems with the use of ANCOVA in intact groups. Groups can still differ in unknown ways. Question whether groups that are equivalent on the covariate ever exist since ANCOVA adjusts for equivalence on the covariate. Assumptions of linearity and homogeneity of regression slopes need to be satisfied. Differential growth of subjects i.e., is difference due to treatment or differential growth (such as in pre-post designs where a pre measure is used as a covariate)? Measurement error, which exists, can produce spurious results. The use of a CV. results in the loss of one degree of freedom in the denominator. Hence, a good CV, or a few good orthogonal CVs can yield a much more precise error term that is worth the trade-off. Poor choice of a CV will result in a loss of a degree of freedom with no real gain. Care should be taken such that the relationship between the CV and the DV is independent of the relationship between the IV and DV. Otherwise, some of the relationship between the IV and DV will be captured by the CV and not attributable to the IV. The inclusion of more and more CVs will result in overall loss of variation which will result in the adjusted treatment means moving closer together. Hence, one doesn t want to remove all the variation in the DV prior to analyzing the effect of the IV on it. Sample-specific problems, or bias in the CV can result in over or under correction. Assumptions of ANCOVA: Larger sample sizes (because of the regression of the d.v. on the c.v.) Absence of Multicollinearity and Singularity Normality of sampling distributions (of the means) Homogeneity of Variance Linearity of relationship between covariate and dependent variable Homogeneity of regression that the slope of the regression line is the same for all cells. If, in the population, the relationship between the CV and DV varies as a function of level of one or more IVs, then this assumption would be violated. Reliability of covariates an implicit assumption is that the CV is measured without error. This is often not a tenable assumption. Measures that have some reliability exceeding.80 are recommended. Testing assumption of homogeneity of regression can be done with SPSS MANOVA (or another GLM type procedure) by conducting a test of a main effect for the covariate, the IV and the interaction between the covariate and the IV. A nonsignificant F for the interaction implies the assumption has not been violated. This can be accomplished through GLM, using the menus, by specifying the model complete with the covariate, then selecting the Model button and selecting custom. From here you can build a model with the effect of the IV, the CV and the interaction between the two.

3 3 Additionally, the covariates should be evaluated for utility. An F test is associated with each covariate and a significant F value implies a significant adjustment to cell means was made. With multiple CVs, the significance test is based on the assumption that each CV entered the equation last, even though they are entered together. Entering several correlated CVs is not a particularly useful strategy. It can be useful to do some preliminary research & analysis to identify which measures will make the best covariates, preferably finding those that are minimally correlated with each other (assuming you are picking more than one) and that best predict the DV. Alternatives to ANCOVA: In pre-post situations, using difference scores (assuming same metric). Be aware there is a large body of literature on the measurement of change, and simple change scores should not be used without developing some familiarity with contemporary thinking on the measurement of change. o For instance, it is known that as the correlation between pre- and post-test approaches the reliability of the test, the reliability of the difference score approaches zero. Incorporating pre-scores into a RM ANOVA design. Residualize DV and run an ANOVA on the residualized scores. Not extremely popular. Blocking, assigning/matching people based on pre-scores or creating appropriate IV categories of intact groups. Utilizing the CV as a factor in the experiment, if it lends itself well to categorization. This side-steps many issues, such as homogeneity of regression. Comparisons & Trends Specific comparisons on the between-subjects portion of the design can be easily made using the usual post-hoc analyses or contrast procedures. Repeated measures comparisons can become tricky, especially in subjects by trials designs. For fixed factors, post-hoc comparisons can be made using a Bonferoni type correction, as well as using methods such as Tukey s HSD. For some reason, SPSS doesn t allow the use of multitudes of post-hoc tests, it allows for no correction (LSD), Bonferoni (using the options menu) and the Sidak method. A priori comparisons, such as the Dunn-Bonferroni procedure can also be used. Any good treatment of ANOVA will include at least one chapter on ANCOVA, in which these post-hoc procedures will be elaborated upon. Stevens (2002) presents a Bryant-Paulson Simultaneous Test Procedure for conducting post-hoc tests. It is an extension of the Tukey simultaneous interval procedure and specifically makes the assumption that the covariates(s) is (are) random, which is typically true in Psychology. Formulas are presented on page 365 and vary depending upon whether the study was randomized and whether there were one or more covariates, hence four equations are presented. Basic Approach: ANCOVA can be thought of as covarying out an unwanted influence on the DV, then proceeding with an ANOVA. That is useful for understanding how it works, however, it actually proceeds much as ANOVA with the partitioning of variation due to between and within effects. These partitions are based on deviations of group means from the GM and deviations of individual scores from their group mean. However, two additional partitions are sought.

4 4 1. First a partition around Sums of Squares Regression and Error for the covariate. Differences between CV scores and their GM are partitioned into between and within SS. 2. Next, the relationship between the linear relationship between the DV and CV is partitioned into sums of products associated with covariance between groups and products associated with covariance within groups. Information from these two partitions are used to adjust the between and within group SS. The hope is that the adjustment that narrows the within group variation will outstrip the downward adjustment to the between groups variance. Example: Appendix A contains a numerical example, which will be briefly discussed here. All computations are contained in the appendix. In this example there are three instructional methods and achievement is the dependent measure. Intelligence is used as a covariate. Also, since there was random assignment, this ANCOVA was used in order to reduce error variance and make the test more sensitive. First, running the analysis as a simple ANOVA yields a non-significant F. While there appear to be differences, based on examining the subsequent table of means (look at the unadjusted means), the analysis lacked the power to yield a significant effect. ANOVA Sum of Squares df Mean Square F Sig. Between Groups Within Groups Total Group Mean for X Unadjusted Mean for Y Adjusted Mean for Y Group Group Group Total Now, let s look at the same analysis run as an ANCOVA Dependent Variable: achievement Tests of Between-Subjects Effects Source Type III Sum of Squares df Mean Square F Sig. Corrected Model (a) Intercept x gpid Error Total Corrected Total a R Squared =.527 (Adjusted R Squared =.483)

5 5 Here we see there is a significant effect for X, the covariate. Also, we see there is a significant effect for group differences now. Also note that the mean square for between groups decreased (gpid) from 102 to 92.67, but the error term decreased substantially more, from 42.3 to We can also test for an interaction between the covariate and the between factor, a significant result reflecting a violation of assumption of equal regression slopes within cells. To do this, we set up a second run in SPSS as follows: UNIANOVA y BY gpid WITH x /METHOD = SSTYPE(3) /INTERCEPT = INCLUDE /CRITERIA = ALPHA(.05) /DESIGN = x, gpid, x BY gpid. TITLE 'test of assumption using UNIANOVA'. Which yields the following: Dependent Variable: y Tests of Between-Subjects Effects Source Type III Sum of Squares df Mean Square F Sig. Corrected Model (a) Intercept x gpid gpid * x Error Total Corrected Total a R Squared =.570 (Adjusted R Squared =.498) The insignificant interaction means our homogeneity of regression assumption is tenable and p >.05. MANCOVA: In the same way we can generalize from ANOVA to MANOVA, we can generalize from ANCOVA to MANCOVA. The parallels hold fairly well, where instead of carrying out the significance test on adjusted MSB and MSW, we carry out the significance test on adjusted W and T matrices: * * W Λ = * T The logic of extending from ANCOVA to MANCOVA doesn t change from the logic for extending from ANOVA to MANOVA, except that testing the assumptions of MANCOVA becomes somewhat more difficult, especially if there are multiple covariates involved. Basically it involves testing for interactions among all the covariates and the covariates with the factors. It can also be extended to N-Way MANCOVA.

6 6 Example: While I generally do not prefer taking an example from the book being used, the One-Way MANCOVA example from Stevens (2002) is a bit of a disaster, so using it is not without merit to take it up as our example. It comes from Novince (1977). The syntax for the example is listed below TITLE 'NOVINCE DATA MANCOVA'. DATA LIST FREE/GPID AVOID NEGEVAL SOCINT SRINV PREAVOID PRENEG PRESOCI PRESR. BEGIN DATA. END DATA. MANOVA AVOID NEGEVAL SOCINT SRINV PREAVOID PRENEG PRESOCI PRESR BY GPID(1,3) /ANALYSIS=AVOID NEGEVAL SOCINT SRINV WITH PREAVOID PRENEG PRESOCI PRESR /PRINT=PMEANS /DESIGN /ANALYSIS=AVOID NEGEVAL SOCINT SRINV /DESIGN = PREAVOID + PRENEG + PRESOCI + PRESR, GPID, PREAVOID BY GPID + PRENEG BY GPID + PRESOCI BY GPID + PRESR BY GPID /ANALYSIS=PREAVOID,PRENEG,PRESOCI,PRESR. TITLE 'INCLUDING ALL FOUR VARS & COVARIATES'. The term in bold represents the test for the homogeneity of regression slopes and the underlined portion generates results needed for the Bryant Paulson post-hoc procedure. This code generates, among other things, multivariate and univariate tests for the significance of the covariates: EFFECT.. WITHIN CELLS Regression Multivariate Tests of Significance (S = 4, M = -1/2, N = 10 1/2) Test Name Value Approx. F Hypoth. DF Error DF Sig. of F Pillais Hotellings Wilks Roys EFFECT.. WITHIN CELLS Regression (Cont.) Univariate F-tests with (4,26) D. F. Variable Hypoth. SS Error SS Hypoth. MS Error MS F AVOID NEGEVAL SOCINT SRINV We see here that there is a significant relationship among covariates and the DVs. All F s are significant at p <.05.

7 7 We also can evaluate the assumptions of homogeneity of regression slopes via the following information included in the output. EFFECT.. PREAVOID BY GPID + PRENEG BY GPID + PRESOCI BY GPID + PRESR BY GPID Multivariate Tests of Significance (S = 4, M = 1 1/2, N = 6 1/2) Test Name Value Approx. F Hypoth. DF Error DF Sig. of F Pillais Hotellings Wilks Roys EFFECT.. PREAVOID BY GPID + PRENEG BY GPID + PRESOCI BY GPID + PRESR BY GPID (Cont.) Univariate F-tests with (8,18) D. F. Variable Hypoth. SS Error SS Hypoth. MS Error MS F Sig. of F AVOID NEGEVAL SOCINT SRINV This table indicates to us that the assumption of homogeneity of regression slopes is tenable. Finally, of particular interest is whether there are group differences on the DVs after correcting for the covariates. This can be found in the table below: EFFECT.. GPID Multivariate Tests of Significance (S = 2, M = 1/2, N = 10 1/2) Test Name Value Approx. F Hypoth. DF Error DF Sig. of F Pillais Hotellings Wilks Roys Note.. F statistic for WILKS' Lambda is exact EFFECT.. GPID (Cont.) Univariate F-tests with (2,26) D. F. Variable Hypoth. SS Error SS Hypoth. MS Error MS F Sig. of F AVOID NEGEVAL SOCINT SRINV The multivariate test indicates a significant multivariate effect, and the following ANOVAs indicate that there is a significant effect for each DV. Now we need to collect information for the Bryant Paulson procedure to test for significant differences among the avoidance response between groups 1 & 2, after correcting for the covariates. The bold term in the table starting with EFFECT.. WITHIN CELLS Regression (above) is needed, as well as Hotelling s Trace from the following table (in bold)

8 8 EFFECT.. GPID Multivariate Tests of Significance (S = 2, M = 1/2, N = 12 1/2) Test Name Value Approx. F Hypoth. DF Error DF Sig. of F Pillais Hotellings Wilks Roys Note.. F statistic for WILKS' Lambda is exact. BP = Y * i Y * j * 1 1 MSW [1 + TR( BλWλ )]/ n J BP = = = [1 + (.312) / This comparison does not exceed the critical value of 3.76 (using 3 instead of four covariates) and rounding down to df e = 24. The comparison between 2 & 3 would exceed this value. To correct Steven s 2002 example, run the following syntax and repeat the same post-hoc test. MANOVA AVOID NEGEVAL PREAVOID PRENEG BY GPID(1,3) /ANALYSIS AVOID NEGEVAL WITH PREAVOID PRENEG /PRINT=PMEANS /DESIGN /ANALYSIS AVOID NEGEVAL /DESIGN = PREAVOID + PRENEG, GPID, PREAVOID BY GPID + PRENEG BY GPID /ANALYSIS=PREAVOID PRENEG. TITLE 'FOR STEVENS 2002 MANCOVA EXAMPLE 4, P BP = Y * i Y * j * 1 1 MSW [1 + TR( BλWλ )]/ n J BP = = = [1 + (.222)]/ The critical value (Table G from Stevens, 2002) is 3.69 (conf. page 366 on use of table, ignoring the fact that Stevens uses 3 covariates when only 2 were included in the design). Furthermore, there are specific equations for making contrasts. So, for instance, in a three group between subjects design, if one wanted to compare groups one and two, to group three, one could set up this contrast where the adjusted means are being evaluated.

9 9 Effect sizes can be calculated in ANCOVA as in ANOVA. In addition to calculating an effect size for a given effect, one can also determine an effect size for the covariate. When calculating an effect size, one needs to use the adjusted Sums of Squares for the effect.

10 10 Appendix A: ANCOVA Numerical Example Step 1: Set up data table and calculate summary statistics. In this example, X is a measure of intelligence and Y is a measure of achievement. We are interested in whether there are differences among achievement scores in three different conditions (Groups 1-3). We wish to use intelligence (X) as a covariate. Participants were randomly assigned, so in this instance we are using ANCOVA to increase the sensitivity of the experiment. Group 1 Group 2 Group 3 Total Subj. # X Y X*Y X Y X*Y X Y X*Y X Y n ΣX,Y,X*Y Means ΣX 2,Y SS Pearson-r Summary of Statistics: We obtain X and Y scores during the experiment, so, we need to complete the following for this table: A cross-products column: this is X*Y. For the first subject in Group 1, 98 x 60 = The n, sums and means for each column. The sum of the squared scores for each column. The sums of squares (SS) for each column The Correlation between X and Y for each column of X s and Y s. Step 2: Calculate Bracket Terms for X, Y & X*Y. 2 2 ( ΣX ) ( ΣY ) ( ΣX )( ΣY ) [1] =,, nk nk nk Σ Σ Σ Σ Σ Σ Σ [2] =,, n n n 2 2 ( X j ) ( Yj ) ( X j Yj )

11 [3],, = ΣX ΣY Σ XY These will now be calculated using the data from the table above. First, the X s [1] = = 456, [2] = = 456, [3] = = 458,552 Then the Y s [1] = = 192, [2] = = 192, [3] = = 194,175 And finally, the Cross-Products (XY) [1] = = 296, ( ) + ( ) + ( ) [2] = = 296, [3] = 94, , , 690 = 297, 660 Step 3: Calculate Sums of Squares between groups and within groups for X, Y & X*Y. The between groups SS (SS A ) is calculated by subtracting term 1 from 2 (i.e., 2 1) and SS w is calculated by subtracting 2 from 3 (3 2). SS A(x) = = SS W(x) = 458, ,630.5 = SS A(y) = 192, , = SS W(y) = 194, , = SS TOT(y) = [3] [1] = 194, , =

12 12 SP A = 296, , = SP W = 297, ,551.5 = Step 4: Computed Adjusted SS A and SS W The following adjusted Sums of Squares, which are adjusted after taking the effect of the covariate out, are the ones actually used for the significance test. The formulas look a bit daunting, but they are fairly efficient. SS ' A ( SPA + SPW ) SP ( ) W = SS A( y) = = [ ] = SS A( x) + SSW ( x) SSW ( x) SS ' W 2 2 SPW = SSW ( y) = = = SS W ( x) Step 5: Construct ANOVA Summary Table and obtain SS COV Source SS df MS F p(f) Covariate Adjusted Between Adjusted Within Total The Total Sums of Squares is taken from above. Y is the dependent variable, therefore, SS TOT(y) becomes the SS TOT. The df TOT as always is N-1. The SS Covariate is taken as the result of SSTOT SS A SS B and each covariate has one degree of freedom. SS A has 2 degrees of freedom (k-1) and the SS W has N k c degrees of freedom, where c is the number of covariates. Mean Squares and F statistics are calculated in the usual way. A significant Covariate implies that the covariate accounts for a significant proportion of variation in Y. Finally, an effect size can be estimated as SS ' η 2 = A ' ' SS A + SS = W = Step 6: Mean Comparisons with Adjsuted Means In order to make comparisons between groups, we need to adjust the group means so that the new group means have the effect of the covariate removed. These adjusted means are used in post-hoc comparisons. We begin by calculating the pooled Unstandardized Regression Coefficient. Here pooled means it is the regression coefficient arrived at by pooling information across our treatment conditions. The formula is simple and we have already calculated the intermediate terms we need. SPW b = = = SS W ( x)

13 Next, we need to adjust the means. The equation is simple, and it is convenient to construct a table with the means for X, the unadjusted means for Y and the adjusted means for Y. Group Mean for X Unadjusted Mean for Y Adjusted Mean for Y Group Group Group Total The formula for computing adjusted means is as follows: Y ' = Y j b( X j X ) j So, for group 1, the adjusted mean would be ANCOVA 13 Y ' 1 = ( ) = ( 2.194) = = Notice that the Grand Mean (Total row for X), is not adjusted, just the group means. ANCOVA will adjust the group means so that they are closer to the Grand Mean. It is hoped that the reduction in error variance (MS W ) more than off-sets this adjustment. Prior to doing any post-hoc tests, we must make a second adjustment to the error term to be used in the Tukey HSD procedure. This adjustment corrects for group differences on X and can be completed as follows. MS SS A( x) = MS 1 + = = [ ] = ( k 1) SS W ( x) (3 1) " ' W W Looking up Q from the table in your text book (using k=3 and df W = 30 1 ) we find 3.49 for alpha of.05. Substituting MS W for MS W, we calculate HSD as " MS HSD = q W = 3.49 = n 12 Hence, the only post-hoc test that is significant is that Group 3 is significantly higher than Group 2. Another approach is the Bryant-Paulson generalization of Tukey s simultaneous interval approach. For our example, the formula is: 1 Note, the actual df W was 32, because it wasn t in our table, I rounded down to 30, which would make the HSD value slightly more conservative than rounding up from 32.

14 14 BP = Y Y * * i j MS * [1 + MS / SS ]/ n W B W X X Where MS * W is the adjusted MS within, MSBX is the between MS for the covariate, SS WX is the SS within for X, Y * i and Y * j are the adjusted means for the two comparisons, and n is the common group size, or the harmonic mean in the case of unequal cell sizes. MS BX and SS WX can be obtained by running an ANOVA with the covariate as the DV. Note, there is a different formula for nonrandomized studies. So, for our example BP = = = = [ /1921.5]/ (1.0273) / The critical value (requires a special table, can be found in Stevens, 2002), is The Bryant-Paulson test appears to be consistent with Tukey s, finding significant differences only for groups 2 and 3.