Analysis of Variance. One-way analysis of variance (anova) shows whether j groups have significantly different means. X=10 S=1.14 X=5.8 X=2.4 X=9.

Transcription

1 Analysis of Variance One-way analysis of variance (anova) shows whether j groups have significantly different means. A B C D X=2.4 X=5.8 X=9.8 X=10 S=1.14 S=1.3 S=1.92 S=1.56

2 Procedure Merge the data into a single sample of 20 trials; calculate grand mean x G = 7.0 and grand variance s G 2 = The deviation from x G = 7.0 of the kth datum in the jth group is (x jk x G ) = (x jk x j ) + (x j x G ) For example, the first datum in group A: (3 7) = (3 2.4)+ (2.4 7)

3 Procedure, continued Sums of squared deviations are additive, too: 2 2 (x jk x G ) = (x jk x j ) + n j (x j x G ) j k j k So grand variance can be decomposed into two parts : within group, (x jk x j ) 2, represents "background" noise" j k between group, n j (x j x G ) 2, represents the effect, if any, j of being in one group or another. j 2 Divide these sums of squares by degrees of freedom to get mean square deviations MS within and MS between Under the null hypothesis that the groups are identical, these terms should be equal

4 Procedure, the math For j groups and a total of N data, calculate grand mean, x G =7.0, and sums of squares and mean squares: Sum of Squares Mean Squares Calculate F, the ratio of the mean squares: Total j k (x jk x G ) 2 F = MS between MS within Between Within j n j (x j x G ) j k (x jk x j ) 2 2 j j n j (x j x G ) k J 1 (x jk x j ) N J 2 2 Under the null hypothesis that the j groups are equal, F = 1. Look up the F statistic in a table with appropriate degrees of freedom for a p value

5 Example: Are the mean runtimes in the Rosenberg experiment equal at all levels of num-proc? Sum Squares DF Mean Square F Ratio P NUM-PROC Error Total Group Means Numproc = Average Average

6 Example: Do the systems built by the four SMEs in Team A in the RKF Summer Trials have the same scores? Anova Sum Squares DF Mean Square F P USER Error Total Group Means User= IFLU AMOEBA CELULA VACCINIA Average Average

7 Which SMEs have different scores? How many pairwise comparisons are there? Sum Squares DF Mean Square F P USER Error Total Group Means User= IFLU AMOEBA CELULA VACCINIA Average Average One approach is unplanned pairwise comparisons with the Bonferroni adjustment or the Scheffe test to protect experimentwise error. Scheffe: F = ( ) 2 x 1 x 2 1 MS within + 1 (j 1) n 1 n 2 F s = ( ) (3) 213 = 29.11

8 Which SMEs have different scores? How many pairwise comparisons are there? Sum Squares DF Mean Square F P USER Error Total Group Means User= IFLU AMOEBA CELULA VACCINIA Average Average F = ( ) 2 x 1 x 2 1 MS within + 1 (j 1) n 1 n 2 F s = ( ) (3) 213 = 5.6 F s = ( ) (3) 213 =.433 F s = ( ) (3) 213 = 5.6

9 A less conservative parwise test: Least Significant Differences (LSD) First run an analysis of variance. If F is not significant do not run pairwise tests. This protexts the experimentwise null hypothesis. Then run pairwise tests at the regular α level but use Mswithin to estimate the standard error (why?) t = x a x b 1 MS within + 1 n a n b t = = 1.14 Notice that t2 = F (in general) so the Scheffe test is a factor of j-1 harder: t = x a x b 1 MS within + 1 n a n b, t 2 = (x a x b ) 2 1 MS within + 1 n a n b,f s = (x a x b ) 2 1 MS within + 1 ( j 1) n a n b

10 Two-way analysis of variance The beauty of interaction terms A Knowledge engineers Naïve users B Team Team A Team B Knowledge Engineers Naïve Users

11 Two-way analysis of variance The beauty of interaction terms ANOVA-TABLE DF Sum Squares Mean Square F P Interaction VENDOR USERTYPE Error Total GROUP-MEANS-TABLE CYCORP SRI Average KE SME Average

12 How two-way analysis of variance works One-way ANOVA Two-way ANOVA y1 y2 y3 y4 x 1 x 2 x 3 x 4 x 1 x 2 x 3 x 4 Use column means to estimate effects, cell contents to estimate background variance Use column means and row means to estimate main effects, cell contents to estimate background variance, and both to estimate interaction effects

13 Two way ANOVA example Factor 1: Teams A and B; Factor 2: SMEs and KEs Dependent variable: Problem solving scores Team A Team B KE SME Is there an effect of Expertise? Relative to the grand mean, yes: 0.32 = = Is there an effect of Team? Relative to the grand mean, yes: = = A cell mean can be viewed as the sum of the grand mean, a row effect, and a column effect: Cell mean for KE,Team A: = 2.44 off by 0.2 Cell mean for SME, Team A = 1.8, off by 0.2 (approx.)

14 Model for the two-way ANOVA Cell Mean (row I, column J) = Grand Mean + Effect of being in row I + Effect of being in column J + Interaction effect Interaction effects are what you need to reconstruct the cell means from the grand mean and row and column effects

15 Reconstituting cell means from the grand mean, main effects, and interaction effects Team A Team B KE 2.64 = = SME 1.61 = = Effect of Team = = Effect of Expertise 0.32 = = Just as main effects sum to zero, so interaction effects sum to zero when cell sizes are equal. Here they nearly sum to zero.

16 Effects as components of grand variance x ij = x G + Effect i, + Effect, j + Effect i, j x ij = x G + (x i, x G ) + (x, j x G ) + (x i, j x i, x, j + x G ) SS total = (x ijk x G ) 2 i j SS within = (x ijk x ij ) 2 i j k k SS columns = n i ( x i, x G ) 2, SS rows = n j ( x, j x G ) 2 i j SS interaction = (x i, j x i, x, j + x G ) 2 i j n ij MS columns = SS columns /(C 1), MS rows = SS rows /(R 1) MS interaction = SS interaction /(C 1)(R 1) MS within = SS within /RC(n 1)

17 Null Hypotheses in Two Way ANOVA The column factor has no effect The row factor has no effect There is no interaction effect Team A Team B Knowledge Engineers Naïve Users ANOVA-TABLE for VENDOR x USERTYPE DF Sum Squares Mean Square F P Interaction VENDOR USERTYPE Error Total

18 Thinking about interactions "The effect of Expertise on Score depends on Team" "The effect of Team on Score depends on Expertise" The effect of X 1 on Y depends on X 2 Score Score Team A Knowledge engineers Team B Naïve users Knowledge Engineers Naïve Users A Team B

19 Thinking about a world with no interactions Good students Good students Poor students Poor students Nonadaptive tutoring system Adaptive tutoring system Nonadaptive tutoring system Adaptive tutoring system

20 Number of Processors X Algorithm Dependent variable: Gregory's adjusted run time KOSO KOSO* NUM-PROC

21 You try it Main effects and interaction effects B1 B2 B3 B1 B2 B3 B1 B2 B3 A1 A2 A1 A2 A1 A2 B1 B2 B3 B1 B2 B1 B2 B3 B3 A1 A2 A1 A2 A1 A2

22 How factors can soak up variance One-way ANOVA (i.e., a t test) SS DF MS F P ALGORITHM Error Total Two-way ANOVA DF SS MS F P Interaction ALGORITHM ALPHA Error Total

23 Soaking up variance Look what happens to SS error, MS error, F num-proc One way ANOVA on NUM-PROC df SS MS F P NUM-PROC Error Total Two way ANOVA, NUM-PROC X Alpha df SS MS F P Interaction NUM-PROC ALPHA Error Total

24 A bit more about analysis of variance. The Algorithm x Problem ANOVA, Revisited Source d.f. SS MS F p Algorithm Problem Algorithm x Problem Error No main effect of algorithm, but does algorithm have an effect on Nodes Examined? Nodes Examined Eight Tiles Travelling Salesman + + Tower of Hanoi A* Q*

25 Two-way analysis of area burned at three different wind speeds; quickly-contained and uncontained fires uncontained Area Burned quickly-contained 0 low medium high Wind Speed

26 Three-way analysis: Area burned at three wind speeds, in quickly and uncontained fires, with adequate or inadequate thinking speeds Area Burned RTK = Inadequate uncontained quickly-contained RTK = Adequate uncontained quickly-contained 0 low medium high 0 low medium high Wind Speed Wind Speed How many main effects? How many interaction effects?

27 Effects in three-way analysis of variance Source df SS MS F p RTK Cont'ment WindSpeed RTK x Containment RTKx WindSpeed Containment x Windspeed RTK x Containment x WindSpeed Error

28 Main and interaction effects in two- and three-way designs An interaction between A and B means the effect of A on the dependent variable depends on B, or vice versa Area Burned uncontained quickly-contained low medium high Wind Speed An interaction between A, B and C means the interaction between A and B depends on C, or vice versa Area Burned RTK = Inadequate uncontained quickly-contained low medium high Wind Speed RTK = Adequate uncontained quickly-contained low medium high Wind Speed

29 Designs Random and fixed factors Factorial, repeated measures, and nested designs Error terms (denominator of F) are different

30 You do it... Think of some experiments for which analysis of variance seems the obvious choice Think of some examples of interaction effects Reflect on the idea that explanation begins with interaction effects

31 Assumptions of Analysis of Variance The dependent variable is assumed to be a sum of terms and each term is assumed to be linear in the factors. Thus α=1/x is a problem. X must be transformed. Errors are assumed to be normally distributed with expectation zero for each combination of factors. Not a problem provided cell sizes are not small (i.e., >10) Errors are assumed to have the same variance for each combination of factors (homoscedasticity). This becomes a problem if it is false and the cell sizes are unequal. Errors are independent both within and across treatment combinations. This can be a problem if there is some "carryover" in a repeated-measures design, e.g., if a judge is used in three conditions and his or her errors become smaller with practice.