psyc3010 lecture 2 factorial between-ps ANOVA I: omnibus tests

Transcription

1 psyc3010 lecture 2 factorial between-ps ANOVA I: omnibus tests last lecture: introduction to factorial designs next lecture: factorial between-ps ANOVA II: (effect sizes and follow-up tests) 1 general announcements tutorials start this week attendance strongly recommended! list of private tutors available on Blackboard revisiting upcoming assessment : quiz 1 on ANOVA in late Aug assignment 1 on ANOVA in early Sept 2 1

2 more about quiz 1 multiple-choice and problem-solving questions assesses material on between-subjects ANOVA covered in Lectures 1, 2, 3, and 4 Practice Questions and Answers posted on Blackboard next week Practice Quiz available on Blackboard in Week 4 (to get used to format) ICEBREAKER! 3 last lecture this lecture in the last lecture, we introduced the concept of factorial designs, focusing on key terminology and concepts factors / independent variables, dependent variables crossed designs: A x B cell means, marginal means, grand means main effects, interaction effects, simple effects how one factor qualifies or moderates the effect of another factor ordinal and disordinal interactions in this lecture, we learn more about factorial between-participants ANOVA, focusing on the logic and computation of omnibus tests 4 2

3 today conceptual underpinnings of ANOVA relationship between hypotheses, variance, graphical representations of data, and formulae (one-way and two-way analyses) research questions and hypotheses building up to the F ratio the F test structural model assumptions conducting 2-way factorial ANOVA data table and key terms results in source/summary table graphing results effect sizes what they are and why they are important three estimates: η 2, ω 2, partial η 2 5 anova: conceptual underpinnings like most statistical procedures we use, anova is all about partitioning variance we want to see if variation due to our experimental manipulations or groups of interest is proportionally greater than the rest of the variance (i.e., that is not due to any manipulations etc) do participants scores (on some DV) differ from one another because they are in different groups of our study, more so than they differ randomly and due to unmeasured influences? 6 3

4 'There are three kinds of lies: lies, damned lies, and statistics.' - Disraeli 7 review of one-way ANOVA 8 4

5 remember variance? variance = dispersion or spread of scores around a point of central tendency the mean there is always some variability in a dataset: every score does not sit on the mean error variance = can t be explained (random or due to unmeasured influences) treatment variance = systematic differences due to our IV (e.g., experimental manipulation) x x x x x x x x x x x x x x x xx x x x x x x x x x x x x xx x TEST SCORES x x x x x x x x x x x x x x x xx x x x x x x x x x x x x xx x TEST SCORES 9 sources of variance in 1-way ANOVA total variation Variance between-groups due to treatment variance within-groups Error variance between-groups variance = systematic variance due to membership in different groups / treatments within-groups variance = error variance (due to random chance or unmeasured influences) 10 5

6 sources of variance in 1-way ANOVA between-groups variance = distribution of group means around the grand mean μ 1 μ. μ 2 11 within-groups variance = distribution of individual DV scores around the group mean μ 1 = population mean for group 1 μ 2 = population mean for group 2 μ. = population grand mean 11 hypotheses in 1-way ANOVA differences between 2 means μ j = population mean of group j μ. = grand mean H 0 : μ 1 = μ 2 null hypothesis: no differences between treatment means H 1 : μ 1 μ 2 alternative hypothesis: difference between treatment means Or another way of putting it: H 0 : μ j = μ. H 1 : μ j μ. for at least one group 12 6

7 hypotheses in 1-way ANOVA differences among 3+ means H 0 : μ 1 = μ 2 = μ 3 = = μ j null hypothesis: no differences between any treatment means Could also write, H 0 : μ j = μ. H 1 : null hypothesis is false μ j = population mean of group j μ. = grand mean alternative hypothesis: at least one difference between treatment means Could also write, H 1 : μ j μ. for at least one j 13 sources of variance in 1-way ANOVA μ 1 μ. μ 2 14 What line is hypothesized to be zero in our null hypothesis? 14 7

8 15 what happens in 1-way ANOVA Ok, but it s never really zero, right? we compare the ratio of between-groups variance to within-groups variance given that there is always within-group error variance in the sample, we want to know whether there is more between-groups variation than expected based on this error variance want to be sure that differences between groups are not due simply to error μ 1 μ. μ2 μ 1 μ. μ

9 univariate anova Total Variation Variance Between-groups due to treatment variance Within-groups Error variance n ( X j X.) n (X bar j X bar dot) 2 = people per group x sum of squared differences between group means and grand mean = estimate of between groups variability 2 ( X ij X j 2 ) (X ij X bar j) 2 = sum of squared differences between individual scores and group mean = estimate of within groups variability 17 logic of univariate (one-way) anova the test statistic is the F-ratio F = MS treat /MS error where MS treat = index of variability among treatment means (SS TR /df TR ) or (SS j /df j ) and MS error = index of variability among participants within a cell, i.e. pooled within-cell variance (SS Error /df Error ) = average of s 2 from each sample, a good estimate of σ e2 (population variance) if MS treat is a good estimate of error variance, F = MS treat /MS error = 1 if MS treat is more than just error variance, F = MS treat /MS error > 1 larger values of F indicate that H0 is probably wrong 18 9

10 So what is ANOVA? 1. Estimate of between-groups variability 2. Estimate of within-groups variability 3. Weight each variability estimate by # of observations used to generate the estimate ( degrees of freedom ) Compare ratio [[n (X bar j X bar dot) 2 ] / (j -1) ] [[ (X ij X bar j) 2 ] / [ j (n-1)] ] When the F ratio is > 1, the treatment effect (variability between groups) is bigger than the error variability (variability within groups). Or more specifically: The sum of the squared differences between the group means and the grand mean x the number of people in each group, divided by the number of groups minus 1, is bigger than the sum of the squared differences between the observations and the group means, weighted by the number of observations in each group minus 1 x the # of groups 19 building up to the F ratio for a refresher on DFs: * Field (3 rd ed) pg. 37 * Field (2 nd ed) pg. 319 if you want to know more about the underlying math, take a look online at the extra materials for Lecture

11 21 partitioning variance in 1-way ANOVA error treatment 22 11

12 structural model of 1-way ANOVA X ij = μ. + τ j + e ij for i cases and j treatments X ij, any DV score is a combination of: μ. the grand mean τ j the effect of the j-th treatment (μ j - μ.) e ij error for i person in j-th treatment 23 structural model of 1-way ANOVA X ij = μ.. + τ j + e ij μ 1 μ. μ 2 X i

13 Derivation for one-way ANOVA: expected mean squares an expected value of a statistic is defined as the long-range average of a sampling statistic our expected mean squares are: E(MS error ) σ 2 e i.e., the long term average of the variances within each sample (S 2 ) would be the population variance σ 2 e E(MS treat ) σ 2 e + nσ 2 τ where σ 2 τ is the long term average of the variance between sample means and n is the number of observations in each group i.e., the long term average of the variances within each sample PLUS any variance between each sample Basically - if group means don t vary then nσ 2 τ = 0, and so then E(MS treat ) = σ 2 e + 0 = σ 2 = e E(MS error ) = σ 2 e See e.g., Howell (2007) p two-way way factorial ANOVA 26 13

14 one-way ANOVA: research questions - is there a treatment effect (is there between group variability)? two-way way ANOVA: is there a main effect of Factor A? Is there variability between the levels of A, averaging over the other factor? Do the A group means differ from each other? Do the marginal means of A differ from the grand mean? is there a main effect of Factor B? Is there variability between the levels of B, averaging over the other factor? Do the B group means differ from each other? Do the marginal means of B differ from the grand mean? is there an A x B interaction? Does the simple effect of A change for different B groups? Does the simple effect of B change for different A groups? Does the simple effect change across the levels of the other factor? Do the cell means differ from the grand mean more than would be expected given the effects of A and B? 27 1-way ANOVA total variation Variance between-groups due to treatment variance within-groups Error variance variance due to factor B variance due to factor A 2-way ANOVA variance due to A X B 28 14

15 sources of variance: 2 x 2 ANOVA DV = exam score Factor 1 = note quality (bad or good) Factor 2 = study time (little or much) 4 cells bad notes, little study bad notes, much study μ. good notes, little study good notes, much study 29 sources of variance: main effects of note quality & study time marginal means to assess main effects bad notes, little study bad notes, much study μ. good notes, little study good notes, much study Μ bad notes Μ little Μ much Μ good notes 30 15

16 sources of variance: simple effects of study time simple effect of study time for bad notes (compares cell means for S at N1) bad notes, little study bad notes, much study μ. good notes, little study good notes, much study simple effect of study time for good notes (compares cell means for S at N2) the interaction effect will be significant if the simple effects of one factor differ at the various levels of the other factor 31 sources of variance: simple effects of note quality simple effect of note quality for little study (compares cell means for N at S1) bad notes, little study bad notes, much study μ. good notes, little study good notes, much study simple effect of note quality for much study (compares cell means for N at S2) The interaction effect will be significant if the simple effects of one factor differ at the various levels of the other factor 32 16

17 degrees of freedom summary df total = N 1 df factor = # of levels of the factor 1 df B = b 1 df A = a 1 df interaction = product of df for factors in the interaction df BA = (b 1) x (a 1) df error = total # of observations # of treatments = N ba = df for each cell x # of cells = (n 1) ba 33 1-way ANOVA total df = N -1 between-groups df Variance due to treatment = ab -1 within-groups df Error = N ab = ab (n - 1) 2-way ANOVA df for factor A = a 1 df for factor B = b - 1 df for A X B = (a - 1) x (b - 1) 34 17

18 partitioning variance in 2-way ANOVA main effect of A main effect of B AxB interaction error

19 hypothesis testing in 2-way ANOVA main effects (shown for an IV with 3 levels) H 0 : μ 1 = μ 2 = μ 3 no differences among means across levels of the factor H 1 : null is false interaction (one possibility for a 2 x 3 design) H 0 : μ 11 - μ 21 = μ 12 - μ 22 = μ 13 - μ 23 if there are differences between particular factor means, they are constant at each level of the other factor (hence the parallel lines) the difference of the differences is zero H 1 : null is false μ 11 = mean of the following group: 1st level of Factor A and 1st level of Factor B 37 hypothesis testing in 2-way ANOVA interaction (other possibility for a 2 x 3 design) Or H 0 : μ 1k - μ.. = μ 2k - μ.. differences between the means of factor B and the grand mean are the same for the 2 levels of factor A (because B has 3 means, can t just put up pairs) H 1 : μ 1k - μ.. μ 2k - μ

20 So what is factorial ANOVA? 1. In 2-way design, estimate between- groups variability Due to main effect of first factor Due to main effect of second factor Due to interaction of two factors 2. Estimate within-groups variability 3. Weight each variability estimate by # of observations used to generate the estimate ( degrees of freedom ) Compare ratios of (1) between-groups variability among levels of A to error; (2) B to error; and (3) ABcells (adjusted for main effects) to error 39 the F test in 2-way ANOVA one-way ANOVA F = MS treat / MS error as for 1-way ANOVA, MS error = pooled variance (average s 2 ) but now we have a separate MS treat for each effect: 1) MS treat for effect of factor A = MS A (1st main effect) 2) MS treat for effect of factor B = MS B (2nd main effect) 3) MS treat for effect of factor AB = MS AB (intx effect) a ratio of the systematic variance of EACH EFFECT (i.e. of your experimental manipulations or treatments) to the unsystematic variance 40 20

21 three omnibus tests Main effect of A Main effect of B Interaction 2 way ANOVA 41 structural model of 2-way ANOVA X ijk ijk = μ. + α j + β k + αβ jk + e ijk for i cases, factor A with j treatments, factor B with k treatments, and the AxB interaction with jk treatments: X ijk, any DV score is a combination of: μ. the grand mean α j the effect of the j-th treatment of factor A (μ Aj - μ.) β k the effect of the k-th treatment of factor B (μ Bk - μ.) αβ jk the effect of differences in factor A treatments at different levels of factor B treatments (μ. - μ Aj - μ Bk + μ jk ) e ijk error for i person in j-th and k-th treatments 42 21

22 assumptions of ANOVA population treatment populations are normally distributed (assumption of normality) treatment populations have the same variance (assumption of homogeneity of variance) sample samples are independent no two measures are drawn from the same participant c.f. repeated-measures ANOVA (more on that later in semester) each sample obtained by independent random sampling within any particular sample, no choosing of respondents on any kind of systematic basis each sample has at least 2 observations and equal n data (DV scores) measured using a continuous scale (interval or ratio) mathematical operations (calculations for means, variance, etc) do not make sense for other kinds of scales

23 conducting a 2-way between-subjects factorial ANOVA 45 application of between-subjects factorial ANOVA a psychological study of creativity in complex socio- chemical environments (Field, 2000) 2 factors: three groups of participants go to the pub and have: no beer, or 2 pints, or 4 pints half of the participants are distracted and half are not distracted (controls) hence, a 2 x 3 between-subjects factorial design DV: creativity unbiased 3 rd parties rate the quality of limericks made up by each of our participants 46 23

24 application of between-subjects factorial ANOVA research questions: is there a main effect of alcohol consumption? does the quality of limerick you make up depend upon how many pints of beer you have had? is there a main effect of distraction? does the quality of limerick you make up depend on whether you were distracted or not? is there a consumption x distraction interaction? does the effect of distraction upon creativity depend upon consumption? (Or, does the effect of consumption upon creativity depend upon distraction?) 47 a combination of IV levels, e.g., 0 pints and distracted, is called a cell in most betweensubjects factorial designs there are n observations per cell the number of cells multiplied by n gives you N, the total number of observations (abn = N) Distraction Alcohol Consumption (pints) Distraction Cell Totals Controls Cell Totals Marginal Totals (B) Marginal Totals (A )

25 cell means are calculated these are the average of the n observations in each cell marginal means for each level of each factor are also calculated these are group means averaged over the levels of the other factor Distraction Alcohol Consumption (pints) Distraction Marginal Totals (B) Means Cell Totals Cell Means Controls the grand mean is the average of all N observations Cell Totals Cell Means Marginal Totals (A ) Means review of source / summary table MS = SS / df F = MS treat / MS error Summary Table Source df SS MS F sig A (cons) B (dist) AB Error Total p value if you want to know more about how the SS values were obtained from the data, look at the extra materials for Lecture

26 the results of this ANOVA show: a significant main effect of pints consumed no main effect of distraction a significant cons. x dist. interaction Summary Table Source df SS MS F sig A (cons) B (dist) AB Error Total no main effect of distraction rated creativity Alcohol consumed (pints) Distraction Controls 52 26

27 main effect of alcohol consumed rated creativity Alcohol consumed (pints) Distraction Controls 53 disordinal interaction rated creativity Alcohol consumed (pints) Distraction Controls 54 27

28 effect sizes 55 are significance tests so useful? researchers have uncritically focused on p values... problems with significance testing as a way of determining the importance of findings: use of an arbitrary acceptance criterion (α) results in a binary outcome (significant or non-significant) no information about the practical significance of findings a large p-value (non-significant) will eventually slip under the acceptance criterion as the sample size increases the magnitude of experimental effect,, or effect size,, has been proposed as an accompaniment (if not an outright replacement) to significance testing 56 28

29 magnitude of experimental effects the effect size gives you another way of assessing the reliability of the result in terms of variance can compare size of effects within a factorial design: how much variance explained by factor 1, factor 2, their interaction, etc. differentiating effect sizes (Cohen, 1973): 0.2 = small 0.5 = medium 0.8 = large 57 2 main approaches to estimating effect sizes in ANOVA eta-squared η 2 η 2 = SS SS effect SS total describes the proportion of variance in the sample s DV scores that is accounted for by the effect considered a biased estimate of the true magnitude of the effect in the population (tends to be larger than in reality) most commonly reported effect size measure because easily interpretable (like R 2 ) omega-squared ϖ 2 ϖ 2 = SS SS effect (df effect )MS error SS total + MS error describes the proportion of variance in the population s DV scores that is accounted for by the effect a less biased estimate of the effect size a more conservative estimate (smaller) difference between the two estimates depends on sample size and error variance 58 29

30 Summary Table from Slide 41 Source df SS MS F sig C (cons) D (distr) C x D Error Total eta-squared: η2 = SS effect SS SS total Consumption: = / =.37 (37% var) Distraction: = / =.02 (2% var) C x D: = / =.22 (22% var) number of pints consumed explains 37% of variance in DV scores (creativity ratings) distraction explains 2% of variance in DV scores interaction (distraction X number of pints consumed) explains 22% in DV scores 59 Summary Table from Slide 41 Source df SS MS F sig C (cons) D (distr) C x D Error Total omega-squared: ϖ 2 = SS SS effect (df effect )MS error SS total + MS error produces very similar but smaller (more conservative) estimates Consumption = [ (83.02)] / ( ) =.34 (34% var) [compared to 37%] Distraction = [ (83.02)] / ( ) =.01 (1% var) [compared to 2%] C x D = [ (83.02)] / ( ) 60 =.20 (20% var) [compared to 22%] 30

31 what s partial eta squared? compare means command in SPSS: asking for an ANOVA with effect size gives you eta squared proportion of total variance accounted for by the effect 2 η = SS SS effect total total variance = SS for all effects + SS error UNIANOVA, MANOVA, or GLM: asking for effect size gives you partial eta squared proportion of residual variance accounted for by the effect 2 partialη ( η 2 p ) = SS SSeffect + SS effect error residual variance = variance left over to be explained (i.e., not accounted for by any other IV in the model) 61 limitations of partial η 2 2 η SS = SS effect total 2 partialη ( η 2 p ) = SS SSeffect + SS effect error 1. in factorial ANOVA, [error + effect] is less than [total], so partial η 2 is more liberal or inflated SS effect1 = 4 SS effect2 = 4 SS intx = 4 SS error = 4 SS total = 16 effect size for factor 1: η 2 = 4 / 16 =.25 partial η 2 = 4 / (4 + 4) = 4 / 8 =

32 limitations of partial η 2 2. in factorial ANOVA, η 2 adds up to a maximum of 100%, but partial η 2 can add to > 100% SS effect1 = 4 SS effect2 = 4 SS intx = 4 SS error = 4 SS total = 16 (calculations as in previous slide) η 2 partial η 2 effect effect interaction.25.5 error.25.5 sum (!) hard to make meaningful comparisons with partial η 2 63 example with SPSS output Tests of Between-Subjects Effects Dependent Variable: I am dissatisfied with my performance recently Source Type III Sum of Squares df Mean Square F Sig. Partial Eta Squared Corrected Model a Intercept gendernum gpaestimate gendernum * gpaestimate Error Total Corrected Total a. R Squared =.367 (Adjusted R Squared =.165) partial η 2 : interaction between the two IVs accounts for 9% of the residual variance η 2 : interaction only accounts for 6% of the total variance (9.326 / ) ω 2 < η 2 < partial η

33 summary main effects and interactions are omnibus tests result from preliminary partitioning of variance look for any difference (anywhere) between the means statistical significance is not the be-all-and-end-all useful to estimate the size of effects gives us more information than statistical significance two approaches eta-squared (biased, but common) and omega-squared (less biased, but uncommon) partial eta-squared is an even more commonly reported effect size measure (output by SPSS) which is the portion of residual variance the effects account for 65 next week: Effect sizes, simple effects and follow-ups Readings: between-ps ANOVA I: omnibus tests (this lecture) Field (3 rd ed): Chapter 12 (pp ), Chapter 3 Field (2 nd ed): Chapter 10 (pp ), Chapter 2 between-ps ANOVA II: follow-up tests (next lecture) Field (3 rd ed): Chapter 12 (pp and pp ) Field (2 nd ed): Chapter 10 (pp and pp ) Howell (all eds): Chapter 13 (pp ) 66 33