Analysis of variance

Analysis of variance Tron Anders Moger 3.0.007 Comparing more than two groups Up to now we have studied situations with One observation per subject One group Two groups Two or more observations per subject We will now study situations with one observation per subject, and three or more groups of subjects The most important question is as usual: Do the numbers in the groups come from the same population, or from different populations? Parametric model (normal distribution): Differences in mean Non-parametric model: Differences in median

ANOVA If you have three groups, could plausibly do pairwise comparisons. But if you have 0 groups? Too many pairwise comparisons: You would get too many false positives! You would really like to compare a null hypothesis of all equal, against some difference ANOVA: ANalysis Of VAriance Testing different types of wheat in a field Interested in finding out if different types of wheat yields different crops Outcome: E.g. wheat in pounds per acre Wheat IIIIIIIIIIIIIIIII IIIIIIIIIIIIIIIII IIIIIIIIIIIIIIIII IIIIIIIIIIIIIIIII Group Wheat IIIIIIIIIIIIIIII IIIIIIIIIIIIIIII IIIIIIIIIIIIIIII IIIIIIIIIIIIIIII Wheat 3 IIIIIIIIIIIIIIIII IIIIIIIIIIIIIIIII IIIIIIIIIIIIIIIII IIIIIIIIIIIIIIIII One-way ANOVA: Testing if mean crop per acre is different for different types of wheat! Find out which wheat type is best!

One-way ANOVA: Example Assume treatment results from 3 patients visiting one of three doctors are given: Doctor A: 4,6,3,7 Doctor B: 9,3,30,36,33 Doctor C: 9,7,34,6 H 0 : The means are equal for all groups (The treatment results are from the same population of results) H : The means are different for at least two groups (They are from different populations) Comparing the groups Averages within groups: Doctor A: 7 Doctor B: 3.8 Doctor C: 9 47 + 53.8 + 49 Total average: = 9.46 4+ 5+ 4 Variance around the mean matters for comparison. We must compare the variance within the groups to the variance between the group means. 3

Variance within and between groups Sum of squares within groups: SSW = ( x x ) i= j= SSW = (4 7) + (6 7) +... + (9 3.8) +... = 94.8 Compare it with sum of squares between K groups: SSG = ni( xi x) i= SSG = + + + + = 4(7 9.46) + 5(3.8 9.46) + 4(9 9.46) = 5.43 (7 9.46) (7 9.46)... (3.8 9.46)... Comparing these, we also need to take into account the number of observations and sizes of groups K n i ij i Adjusting for group sizes Divide by the number of degrees of freedom SSW MSW = n K Both are estimates of population variance of error under H 0 MSG = SSG K n: number of observations K: number of groups Test statistic: MSG MSW Reject H 0 if this is large 4

Test statistic thresholds If populations are normal, with the same variance, then we can show that under the null hypothesis, MSG and MSW are Chisquare distributed with K- and n-k d.f. MSG MSW F ~ K, n K The F distribution, with K- and n-k degrees of freedom MSG FK n K α Reject at confidence level α if >,, MSW Find this value in table p. 87 Continuing example MSW = SSW 94.8 9.48 n = K 3 3 = SSG 5.43 MSG = 6. K = 3 = MSG 6..76 MSW = 9.48 = Page 87: F3,3 3,0.05= 4.0 Thus we can NOT reject the null hypothesis in our case. 5

ANOVA table Source of variation Sum of squares Deg. of freedom Mean squares F ratio Between groups Within groups SSG SSW K- n-k MSG MSW MSG MSW Total SST n- SST = (4 9.46) + (6 9.46) +... + (6 9.46) NOTE: SSG + SSW = SST Formulation of the model: H 0 : µ =µ = =µ K X ij =µ i +ε ij Let G i be the difference between the group means and the population mean. Then: G i =µ i -µ of µ i =µ+g i GivingX ij =µ+g i +ε ij And H 0 : G =G = =G K =0 6

One-way ANOVA in SPSS: Analyze - Compare Means - One-way ANOVA Move dependent variable to Dependent list and group to Factor Choose Bonferroni in the Post Hoc window to get comparisons of all groups Choose Descriptive and Homogeneity of variance test in the Options window One-way ANOVA in SPSS Value Between Groups Within Groups Total ANOVA Sum of Squares df Mean Square F Sig. 5,43 6,5,765, 94,800 0 9,480 47,3 Last column: P-value=0., do not reject H 0 Note that the p-value can also be seen as the smallest value of α at which the null hypothesis is rejected. 7

Energy expenditure example: Let us say we have measurements of energy expenditure in three independent groups: Anorectic, lean and obese Wantto test H 0 : Energy expenditure is the same for anorectic, lean and obese Data for anorctic: 5.40, 6.3, 5.34, 5.76, 5.99, 6.55, 6.33, 6. Energy Lean Obese Anorectic Total SPSS output: Descriptives 95% Confidence Interval for Mean N Mean Std. Deviation Std. Error Lower Bound Upper Bound Minimum Maximum 3 8,066,3808,34338 7,380 8,843 6,3 0,88 9 0,978,39787,46596 9,33,373 8,79,79 8 5,976,4403,5568 5,608 6,3444 5,34 6,55 30 8,783,98936,363 7,4355 8,9 5,34,79 Test for equal Variances H 0 : All variances are equal Test of Homogeneity of Variances Energy Levene Statistic df df Sig.,84 7,078 Energy Between Groups Within Groups Total ANOVA Sum of Squares df Mean Square F Sig. 79,385 39,693 30,88,000 35,384 7,3 4,769 9 Dependent Variable: Energy Bonferroni (I) Group Lean Obese Anorectic (J) Group Obese Anorectic Lean Anorectic Lean Obese Multiple Comparisons Mean Difference 95% Confidence Interval (I-J) Std. Error Sig. Lower Bound Upper Bound -,36*,4964,000-3,4987 -,9646,08990*,544,00,7769 3,409,36*,4964,000,9646 3,4987 4,353*,5566,000,907 5,744 -,08990*,544,00-3,409 -,7769-4,353*,5566,000-5,744 -,907 *. The mean difference is significant at the.05 level. See that there is a difference between groups. See also between which groups the difference is! 8

Conclusion: There is a significant overall difference in energy expenditure between the three groups (p-value<0.00) There are also significant differences for all two-by-two comparisons of groups The Kruskal-Wallis test ANOVA is based on the assumption of normality There is a non-parametric alternative not relying this assumption: Looking at all observations together, rank them Let R, R,,R K be the sums of ranks of each group If some R s are much larger than others, it indicates the numbers in different groups come from different populations 9

The Kruskal-Wallis test The test statistic is K Ri W = 3( n+ ) nn ( + ) n i= i Under the null hypothesis, this has an approximate χk distribution. The approximation is OK when each group contains at least 5 observations. Doctor A 4 (rank ) 6 (rank.5) 3 (rank 9.5) 7 (rank 4.5) R =7.5 Example: previous data Doctor B 9 (rank 6.5) 3 (rank 9.5) 30 (rank 8) 36 (rank 3) 33 (rank ) R =48 Doctor C 9 (rank 6.5) 7 (rank 4.5) 34 (rank ) 6 (rank.5) R 3 =5.5 (This is just an example. We really have too few observations for this test!) 0

Kruskal-Wallis in SPSS Use Analyze=>Nonparametric tests=>k independent samples For our data, we get Value Group Doctor A Doctor B Doctor C Total Ranks N Mean Rank 4 4,38 5 9,60 4 6,38 3 Test Statistics a,b Chi-Square df Asymp. Sig. Value 4,95,3 a. Kruskal Wallis Test b. Grouping Variable: Group For the energy data: Same result as for one-way ANOVA! Energy Group Lean Obese Anorectic Total Ranks N Mean Rank 3 5,6 9 4,67 8 5,00 30 Test Statistics a,b Chi-Square df Asymp. Sig. Energy,46,000 a. Kruskal Wallis Test b. Grouping Variable: Group Reject H 0

When to use what method In situations where we have one observation per subject, and want to compare two or more groups: Use non-parametric tests if you have enough data For two groups: Mann-Whitney U-test (Wilcoxon rank sum) For three or more groups use Kruskal-Wallis If data analysis indicate assumption of normally distributed independent errors is OK For two groups use t-test (equal or unequal variances assumed) For three or more groups use ANOVA What if you have more information on the subjects? When you in addition to the main observation have some observations that can be used to pair or block subjects, and want to compare groups, and assumption of normally distributed independent errors is OK: For two groups, use paired-data t-test For three or more groups, we can use two-way ANOVA

Two-way ANOVA: Want to test different fertilizers also Block: Fertilizer Fertilizer Fertilizer 3 Wheat IIIIIIIII IIIIIIIII IIIIIIIII Group: Wheat IIIIIIIIII IIIIIIIIII IIIIIIIIII Wheat 3 IIIIIIIIII IIIIIIIIII IIIIIIIIII Do different wheat types give different wheat crop per acre? Do different fertilizers give different wheat crop per acre? Two-way ANOVA without interaction! Do e.g fertilizer work better for wheat type than for Wheat types and 3? Is there interaction between wheat and fertilizer? Two-way ANOVA with interaction! Two-way ANOVA (without interaction) In two-way ANOVA, data fall into categories in two different ways: Each observation can be placed in a table. Example: Both doctor and type of treatment should influence outcome. Sometimes we are interested in studying both categories, sometimes the second category is used only to reduce unexplained variance (like an independent variable in regression!). Then it is called a blocking variable Compare means, just as before, but for different groups and blocks 3

Data from exercise 7.46: Three types of aptitude tests (K=3) given to prospective management trainers: Profile fit, Mindbender, Psych Out Each test type is given to members of each of four groups of subjects (H=4) based on scores in preliminary interviews Test type Subject type Profile fit Mindbender Psych Out Poor Fair Good Excellent 65 74 64 83 69 7 68 78 75 70 78 76 Sums of squares for two-way ANOVA Assume K groups, H blocks, and assume one observation x ij for each group i and each block j, so we have n=kh observations (have to be independent!). Mean for category i: Mean for block j: x j x x i Overall mean: Model: X ij =µ+g i +B j +ε ij 4

Sums of squares for two-way ANOVA K SSG = H ( x x) K i= H i= j= i SSE = ( x x x + x) ij i j H SSB = K ( x x) K j= i= j= j SST = ( x x) H ij SSG + SSB + SSE = SST ANOVA table for two-way data Source of variation Sums of squares Deg. of freedom Mean squares F ratio Between groups SSG K- MSG= SSG/(K-) MSG/MSE Between blocks SSB H- MSB= SSB/(H-) MSB/MSE Error SSE (K-)(H-) MSE= SSE/(K-)(H-) Total SST n- Test for between groups effect: compare Test for between blocks effect: compare MSG MSE MSB MSE to to FK,( K )( H ) FH,( K )( H ) 5

Two-way ANOVA (with interaction) The setup above assumes that the blocking variable influences outcomes in the same way in all categories (and vice versa) We can check if there is interaction between the blocking variable and the categories by extending the model with an interaction term Need more observations per block Other advantages: More precise estimates Data from exercise 7.46 cont d: Each type of test was given three times for each type of subject Test type Subject type Profile fit Mindbender Psych Out Poor Fair Good Excellent 65 68 6 74 79 76 64 7 65 83 8 84 69 7 67 7 69 69 68 73 75 78 78 75 75 75 78 70 69 65 78 8 80 76 77 75 6

Sums of squares for two-way ANOVA (with interaction) Assume K groups, H blocks, and assume L observations x ij, x ij,,x ijl for each category i and each block j block, so we have n=khl observations (independent!). Mean for category i: Mean for block j: Mean for cell ij: x ij Overall mean: x x j x i Model: X ijl =µ+g i +B j +I ij +ε ijl Sums of squares for two-way ANOVA (with interaction) SSG = HL ( x x) K i= i H SSB = KL ( x x) j= j K H L SSE = ( x x ) i= j= l= ijl ij K H L SST = ( x x) i= j= l= ijl K H SSI = L ( x x x + x) i= j= ij i j SSG + SSB + SSI + SSE = SST 7

ANOVA table for two-way data (with interaction) Source of variation Between groups Sums of squares SSG Deg. of freedom K- Mean squares MSG= SSG/(K-) F ratio MSG/MSE Between blocks SSB H- MSB= SSB/(H-) MSB/MSE Interaction SSI (K-)(H-) MSI= SSI/(K-)(H-) MSI/MSE Error SSE KH(L-) MSE= SSE/KH(L-) Total SST n- Test for interaction: compare MSI/MSE with Test for block effect: compare MSB/MSE with F( K )( H ), KH( L ) FH, KH( L ) Test for group effect: compare MSG/MSE with FK, KH( L ) Two-way ANOVA in SPSS Analyze->General Linear Model-> Univariate Move dependent variable (Score) to Dependent Variable Move group variable (test type) and block variable (subject type) to Fixed Factor(s) Under Options, may check Descriptive Statistics and Homogeneity Tests, and also get two-by-two comparisons by checking Bonferroni under Post Hoc Gives you a full model (with interaction) 8

Levene's Test of Equality of Error Variances a Dependent Variable: Score F df df Sig.,47 4,06 Tests the null hypothesis that the error variance of the dependent variable is equal across groups. a. Design: Intercept+Subjectty+Testtype+Subjectty * Testtype Dependent Variable: Score Some SPSS output: Tests of Between-Subjects Effects Source Type IV Sum of Squares df Mean Square F Sig. Corrected Model 03,556 a 93,869 5,360,000 Intercept 93306,778 93306,778 363,08,000 Subjectty 389,000 3 9,667,8,000 Testtype 57,556 8,778 4,709,09 Subjectty * Testtype 586,000 6 97,667 5,98,000 Error 46,667 4 6, Total 94486,000 36 Corrected Total 79, 35 a. R Squared =,876 (Adjusted R Squared =,89). Subjectty Dependent Variable: Score Subjectty Poor Fair Good Excellent 95% Confidence Interval Mean Std. Error Lower Bound Upper Bound 70,000,84 68,99 7,70 7,444,84 69,744 73,45 73,000,84 7,99 74,70 78,667,84 76,966 80,367 Equal variances can be assumed See that there is a significant block effect, significant group effect, and a significant interaction effect Means (in plain words) that the test score is different for the four subject types, for the three test types, and that differences between test types depend on what subject type you consider Two-by-two comparisons Dependent Variable: Score Bonferroni Multiple Comparisons (I) Testtype Profile fit Mindbender Psych Out (J) Testtype Mindbender Psych Out Profile fit Psych Out Profile fit Mindbender Mean Difference 95% Confidence Interval (I-J) Std. Error Sig. Lower Bound Upper Bound,83,009,000 -,76 3,43 -,7,009,6-4,76,43 -,83,009,000-3,43,76-3,00*,009,00-5,60 -,40,7,009,6 -,43 4,76 3,00*,009,00,40 5,60 Based on observed means. Multiple Comparisons *. The mean difference is significant at the,05 level. Dependent Variable: Score Bonferroni (I) Subjectty Poor Fair Good Excellent (J) Subjectty Fair Good Excellent Poor Good Excellent Poor Fair Excellent Poor Fair Good Based on observed means. *. The mean difference is significant at the,05 level. Mean Difference 95% Confidence Interval (I-J) Std. Error Sig. Lower Bound Upper Bound -,44,65,000-4,79,9-3,00,65,00-6,35,35-8,67*,65,000 -,0-5,3,44,65,000 -,9 4,79 -,56,65,000-4,9,79-7,*,65,000-0,57-3,87 3,00,65,00 -,35 6,35,56,65,000 -,79 4,9-5,67*,65,000-9,0 -,3 8,67*,65,000 5,3,0 7,*,65,000 3,87 0,57 5,67*,65,000,3 9,0 9

Notes on ANOVA All analysis of variance (ANOVA) methods are based on the assumptions of normally distributed and independent errors The same problems can be described using the regression framework. We get exactly the same tests and results! There are many extensions beyond those mentioned In fact, the book only briefly touches this subject More material is needed in order to do two-way ANOVA on your own Next time: How to design a study? Different sampling methods Research designs Sample size considerations 0