Multiple t Tests. Introduction to Analysis of Variance. Experiments with More than 2 Conditions

Size: px

Start display at page:

Download "Multiple t Tests. Introduction to Analysis of Variance. Experiments with More than 2 Conditions"

Alaina Hicks
5 years ago
Views:

1 Introduction to Analysis of Variance 1 Experiments with More than 2 Conditions Often the research that psychologists perform has more conditions than just the control and experimental conditions You might want to compare the effectiveness of a new treatment to an old treatment and to a placebo control group Factor the variable (independent or quasi independent) that designates the groups being compared Level the values that the factor can take on 2 Multiple t Tests How would you analyze data from an experiment with more than two levels? Given what we know so far, our only real option would be to perform multiple t-tests The problem with this approach to the data analysis is that you might have to perform many, many t-tests to test all possible combinations of the levels 3 1

2 Multiple t-tests As the number of levels (or conditions) increases, the number of comparisons needed increases more rapidly # comparisons = (n 2 - n) / 2 n = number of levels 4 Why Not Perform Multiple t Tests? With a computer, it is easy, quick, and error-free to perform multiple t-tests, so why shouldn t we do it? The answer lies in α α is the probability of making a Type-I error, or incorrectly rejecting H 0 when H 0 is true α applies to a single comparison It is correctly called α comparison wise 5 Why Not Perform Multiple t Tests? What happens to the probability of making at least one Type-I error as the number of comparisons increases? It increases too The probability of making at least one Type-I error across all of your inferential statistical tests is called α familywise or α fw 6 2

3 α fw If the probability of making a Type-I error on a given comparison is given by α, what is the probability of making at least 1 Type- I error when you make n comparisons? 7 α fw α fw = 1 - (1 - α) n This assumes that the Type I errors are independent As the number of comparisons increases, the probability of making at least one Type-I error increases rapidly 8 Why Not Multiple t-tests? Performing multiple t-tests is bad because it increases the probability that you will make at least one Type-I error You can never determine which statistically significant results, if any, are probably due to chance 9 3

4 ANOVA Part of the solution to this problem rests in a statistical procedure known as the analysis of variance or ANOVA for short ANOVA replaces the multiple comparisons with a single omnibus null hypothesis Omnibus -- covers all aspects In ANOVA, H 0 is of the form: H 0 : µ 1 = µ 2 = µ 3 =. = µ n 10 That is, H 0 is that all the means are equal ANOVA Alternative Hypothesis Given the H 0 and H 1 must be both mutually exclusive and exhaustive, what is H 1 for ANOVA? Why isn t this H 1? H 1 : µ 1 µ 2 µ 3. µ n In ANOVA, the alternative hypothesis is of the form: H 1 : not H 0 or H 1 : at least one population mean is different from another 11 Two Estimates of Variance ANOVA compares two estimates of the variability in the data in order to determine if H 0 can be rejected: Between-treatments variance Also called between-groups variance Within-treatment variance Also called within-groups variance 12 4

5 Within-Treatment Variance Within-treatment variance is the weighted mean variability within each group or condition Which of the two sets of distributions to the right has a larger within-treatment variance? Why? 13 Within-Treatment Variance What causes variability within a treatment? Within-treatment variation is caused by factors that we cannot or did not control in the experiment, such as individual differences between the participants Do you want within-treatment variability to be large or small? 14 Between-Treatments Variance Between-treatments variance is a measure of how different the groups are from each other Which set of distributions has a greater betweentreatments variance? 15 5

6 Sources of Between-Treatments Variance What causes, or is the source of, betweentreatments variance? That is, why are not all the groups identical to each other? Between-treatments variance is partially caused by the effect that the treatment has on the dependent variable The larger the effect is, the more different the groups become and the larger betweentreatments variance will be 16 Sources of Between-Treatments Variance Between-treatments variance also measures sampling error Even if the treatment had no effect on the dependent variable, you still would not expect the distributions to be equal because samples rarely are identical to each other That is, some of the between-treatments variance is due to the fact that the groups are initially different due to random fluctuations (or errors) in our sampling 17 Variance Summary Within-treatments variance measures error Between-treatments variance measures the effect of the treatment on the dependent variable and error 18 6

7 F Ratio Fisher s F ratio is defined as: 19 F Ratio What should F equal if H 0 is true? When H 0 is true, then there is no effect of the treatment on the DV Thus, when H 0 is true, we have random, unsystematic differences / random, unsystematic differences which should equal 1 20 F Ratio What should F be if H 0 is not true? When H 0 is not true, then there is an effect of the treatment on the DV Thus, when H 0 is not true, F should be larger than

8 F Ratio The denominator of the F ratio is called the error term or error variance. It provides a measure of the variance due to random, unsystematic differences 22 How Big Does F Have to Be? Several factors influence how big F has to be in order to reject H 0 To reject H 0, the calculated F must be larger than the critical F which can be found in a table Anything that makes the critical F value large will make it more difficult to reject H 0 23 Factors That Influence the Size of the Critical F α level -- as α decreases, the size of the critical F increases Sample size -- as the sample size increase, the degrees of freedom for the withintreatments variance increases, and the size of the critical F decreases As the sample becomes larger, it becomes more representative of the population 24 8

9 Factors That Influence the Size of the Critical F Number of conditions -- as the number of conditions increase, so does the degrees of freedom for the between-treatments variance term For any reasonable sample size, the size of the critical F will decrease as the number of conditions increases 25 Factors That Influence the Size of the Observed F Experimental Control -- as experimental control increases, within-treatments variance decreases, and the observed F increases 26 ANOVA Example Students used one of three types of mnemonics to learn a list of words: rehearsal, visual imagery, peg word. The number of words recalled in the correct order was measured. There were 10 students in each level of the factor 27 9

10 ANOVA Example Rehearsal Visual Imagery Peg Word n = 10 =6.5 ΣX = 65 ΣX 2 = 627 SS Reh = / 10 = n = 10 =4.9 ΣX = 49 ΣX 2 = 445 SS Vis = / 10 = n = 10 =16.8 ΣX = 168 ΣX 2 = 2936 SS Peg = / 10 = N = 30 =9.4 ΣX= 282 = ΣX 2 =4008= SS Total = / 30 = ANOVA Summary Table SS df MS F p Between treatments Within treatment Total SS = sum of squares -- the numerator of the variance formula = Σ(X- ) 2 = ΣX 2 (ΣX) 2 / n If n 1 = n 2 = n 3 = SS Between = n SS M ΣM = = 28.2 ΣM 2 = = SS Between = 10 ( / 3) = ANOVA Summary Table SS df MS F p Between treatments Within treatment Total SS Within = SS Rehearsal + SS Visual Imagery + SS Peg Word = = SS Total = SS Between + SS Within = =

11 ANOVA Summary Table SS df MS F p Between treatments Within treatment Total df Between = # of conditions 1 = 3 1 = 2 df Within = Σ(n i 1) = (10 1) + (10 1) + (10 1) = 27 df Total = N 1 = 30 1 = 29 df Total = df Between + df Within = = ANOVA Summary Table SS df MS F p Between treatments Within treatment Total MS Between = SS Between / df Between = / 2 = MS Within = SS Within / df Within = / 27 = F = MS Between / MS Within = / = Critical Value of F Reject H 0 when the calculated F ratio is at least as large as the critical F ratio Find the critical F ratio in a table of critical F ratios You need to know df between treatments, df within treatment, and α For the summary table on slide 29, df between treatments = 2, df within treatment = 27, and α =.05 Critical F = 3.35 Since the calculated F = is larger than the critical F = 3.35, we should reject H

12 Effect Sizes with ANOVA r 2 is the proportion of the total variance that is explainable by the treatment For ANOVA, r 2 is reported as η 2 (eta squared) in the literature 34 ANOVA Assumptions ANOVA makes certain assumptions: Population from which the samples are selected are normally distributed Homogeneity of variance -- the variability within each group is approximately equal Independence of observations 35 ANOVA Assumptions ANOVA is fairly robust (it will give good results even if the assumptions are violated) to the normality assumption and the homogeneity of variance assumption as long as: the number of participants in each group is equal the number of participants in each group is fairly large 36 12

13 Post Hoc Tests / Multiple Comparisons ANOVA only tells us if there is an effect That is, are the means of the groups not all equal? ANOVA does not tell us which means are different from which other means Multiple comparisons, or post hoc tests, are used after you reject ANOVA s H 0 to determine which means are probably different from which other means 37 Post Hoc Tests Post hoc tests are not fundamentally different from performing a t-test That is, both post hoc tests and t-tests answer the same question: are two means different from each other The difference is that the post hoc tests protect us from making a Type-I error even though we perform many such comparisons 38 Post Hoc Tests There are many types of post hoc tests E.g. Tukey, Newman-Keuls, Dunnett, Scheffé, Bonferroni There is little agreement about which particular one is best The different tests trade off statistical power for protection from Type-I errors 39 13

14 Tukey Tests We will consider only the Tukey test; other people may feel that other tests are more appropriate The Tukey test offers reasonable protection from Type-I errors while maintaining reasonable statistical power 40 Tukey Tests You should perform Tukey tests only when two criteria have been met: There are more than two levels to the factor / IV The factor / IV is statistically significant 41 Steps in Performing Tukey Tests Write the hypotheses H 0 : µ 1 = µ 2 H 1 : µ 1 µ 2 Specify the α level α =

15 Steps in Performing Tukey Tests Calculate the honestly significant difference (HSD) q α,k,dfwithin-treatment = tabled q value α = α level k = number of levels of the IV df within-treatment = degrees of freedom for MS within-treatment MS within-treatment = within-treatment variance estimate n = number of scores in each treatment 43 Steps in Performing Tukey Tests Take the difference of the means of the conditions you are comparing If the difference of the means is at least as large as the HSD, you can reject H 0 Repeat for whatever other comparisons need to be made 44 Tukey Example Mnemonic Rehearsal 6.5 Visual Imagery 4.9 Peg Word 16.8 SS df MS F p Between treatments Within treatment Total

16 Tukey Example SS df MS F p Between treatments Within treatment Total Tukey Example Mnemonic Rehearsal 6.5 Visual Imagery 4.9 Peg Word 16.8 If a pair of means is at least the HSD, 4.88, apart, then they are reliably different 47 Tukey Example Mnemonic Rehearsal 6.5 Visual Imagery 4.9 Peg Word 16.8 H 0 : µ Rehearsal = µ Visual Imagery H 1 : µ Rehearsal µ Visual Imagery If Rehearsal - Visual Imagery HSD, then reject H < 4.88 fail to reject H

17 Tukey Example Mnemonic Rehearsal 6.5 Visual Imagery 4.9 Peg Word 16.8 H 0 : µ Rehearsal = µ Peg Word H 1 : µ Rehearsal µ Peg Word If Rehearsal - Peg Word HSD, then reject H reject H 0 49 Tukey Example Mnemonic Rehearsal 6.5 Visual Imagery 4.9 Peg Word 16.8 H 0 : µ Visual Imagery = µ Peg Word H 1 : µ Visual Imagery µ Peg Word If Visual Imagery - Peg Word HSD, then reject H reject H 0 50 F = t 2 When there are only two levels to the factor and you are performing a two-tailed test, you can use either the t-test or ANOVA The calculated value of F will equal the square of the calculated value of t The critical value of F will equal the square of the critical value of t Thus, you will always reach the same conclusion either reject H 0 or fail to reject H

Independent Samples ANOVA

Independent Samples ANOVA In this example students were randomly assigned to one of three mnemonics (techniques for improving memory) rehearsal (the control group; simply repeat the words), visual imagery