STAT 5200 Handout #7a Contrasts & Post hoc Means Comparisons (Ch. 4-5)

Size: px

Start display at page:

Download "STAT 5200 Handout #7a Contrasts & Post hoc Means Comparisons (Ch. 4-5)"

Vivian Flynn
6 years ago
Views:

1 STAT 5200 Handout #7a Contrasts & Post hoc Means Comparisons Ch. 4-5) Recall CRD means and effects models: Y ij = µ i + ϵ ij = µ + α i + ϵ ij i = 1,..., g ; j = 1,..., n ; ϵ ij s iid N0, σ 2 ) If we reject H 0 : µ 1 =... = µ g based on F T rt ), it s natural to look more carefully at why How are µ i s not equal? We usually make pairwise comparisons: This involves two issues: Testing contrasts Dealing with multiple hypothesis Contrasts: A linear combination of parameters like µ i s or α i s) ψ = where w i µ i or ψ = w i α i w i = 0 This definition leads to some nice statistical properties In a CRD one-way ANOVA), all contrasts are estimable : i.e., they have unique least squares estimates and SE s) ˆψ = w iˆµ i = g w i ˆα i

2 For means model parameterization not so clean for effects model): SE of ˆψ) = V ar[ ˆψ] = ˆσ 2 wi 2 /n i We can use these to get 95% C.I. for ψ: ˆψ ± t.025,n g ) SE of ˆψ ) Equivalently, test H 0 : ψ = 0 using ˆψ SE of ˆψ) or, equivalently 2 ˆψ SE of ˆψ) We look at a different ψ in each pairwise comparison of treatments but ψ does not need to be a pairwise comparison Multiple Hypothesis Testing as in testing all possible pairwise treatment comparisons): Recall in significance testing: Type I error: Type I error rate: Type II error: Type II error rate: Power:

3 How to set tolerance for a Type I error when testing all K possible pairwise treatment comparisons? When we test H 0 at level α, this is an error rate If we test K H 0 s all true), each at level α = 0.05, how likely are we to reject any just by chance? P {reject at least one H 0 all true} = 1 P {reject none all true} = 1 P {don t reject #1 and don t reject #2 and... all true} = 1 P {don t reject H 0 H 0 true) K = 1 1 α) K So testing all H 0 s at α = 0.05 can make at least one Type I error quite likely! With multiple H 0 s, we need to consider meaningful error rates Type I error rates of major interest: for a family of K nulls H 01, H 02,..., H 0K ) 1. PCER = P reject H 0i H 0i true) for single i per-comparison error rate) 2. FWER = P reject at least one H 0i H 01,..., H 0K true) family-wise error rate, or experimentwise error rate) 3. FDR = # wrongly-rejected H 0i s ) / total # rejected H 0i s ) false discovery rate) 4. SFWER = P # wrongly-rejected H 0i s 1) strong family-wise error rate) 5. Simultaneous Confidence Intervals set confidence level for family of all K intervals)

4 Recommendations to control these error rates i.e., set tolerance for Type I errors): Error Rate All Pairwise Comparisons Other Specialized Comparisons PCER FWER FDR SFWER LSD plsd SNK REGWQ Simult. CI Bonferroni, Tukey Scheffé, Dunnett preferred NOTES: When asking many questions of data from an experiment, the honest thing to do is adjust for multiple testing. See example on last page of Handout #7.) Most criticisms of p-values can be deflected by understanding what a p-value is and isn t), and by appropriate handling of multiple testing.

5 Appendix: Multiple Comparison Procedures Ch. 5) For each of the error rates mentioned above, there are recommended multiple comparison procedures. The choice of a multiple comparison procedure MCP) depends on the error rate to be controlled and the type of comparisons of interest. The table above summarizes the textbook s recommendations. Each of these methods is described in Chapter 5 of the text, and briefly summarized in this appendix to this Handout #7. Bonferroni test each H 0i at level α/k best suited for small K 2 to 10 or so) since α/k gets too small for larger K; it s too conservative too hard to reject H 0 ) for larger K also controls SFWER a variation to control the FDR but not SFWER): Benjamini-Hochberg sort p-values: p 1) p 2)... p K) starting with largest p-value, work down, and reject H 0i if p j) j α/k for some j i well-suited for large K, but assumes all K tests are statistically independent Scheffé best suited when all possible contrasts are of interest so data snooping is allowed) general idea: For any contrast ψ = g w i µ i, compute ˆψ = g w iˆµ i = g w i Ȳ i Sampling distribution of ˆψ is based on F g 1,N g distribution note N g is d.f. for MSE) Reject H 0 : ψ = 0 only when ˆψ exceeds the Scheffé significant difference: SSD = g 1)Fg 1,N g MSE where F is upper α critical value of appropriate F distribution Downside: low power w 2 i n i

6 Tukey HSD) best suited when all possible pairwise mean comparisons special case of contrasts) are of interest general idea: Let Ȳmax and Ȳmin be the largest and smallest respectively) Ȳi among treatments i = 1,..., g, and define statistic q = Ȳmax Ȳmin MSE/n Sampling distribution of q is the studentized range distribution see tables on pages of text) depends on d.f. same as for MSE; ν in tables) and g number of factor levels; K in tables) Reject null H 0ij : µ i = µ j only when the difference between Ȳi and Ȳj exceeds the honest significant difference HSD): HSD = q 1 MSE + 1 ) 2 n i n j where q is the α critical value of appropriate studentized range distribution REGWQ REGWR) example of a step-down method, starting with most significant differences Sort factor levels so that Ȳ1) < Ȳ2) <... < Ȳg) For ordered means Ȳi) and Ȳj) i < j), stretch of means is j i + 1. For all comparisons involving stretch of k means like H 0 : µ i) =... = µ i+k 1) ), use same critical value. general idea for REGWQ author initials: Ryan-Einot-Gabriel-Welsch): Test stretch 1) to g), and stop if not significant. Otherwise, declare end means 1) and g) different, and zoom in to next smallest stretches Test stretches 1) to g 1) and 2) to g); for each, stop if not significant For each, otherwise declare end means different and zoom in to next smallest stretches Iterate until stop Declare means µ i) and µ j) significantly different if reject null for stretch i) to j) and if reject null for all stretches containing i) to j). [Aside: makes a closed procedure] Critical value q in HSD above) involves studentized range distribution and stretch size k and # factor levels g)

7 SNK Student-Newman-Keuls another step-down method similar to REGWQ, except critical value q in HSD above) involving studentized range distribution and stretch size k) does not involve # factor levels g LSD Fisher s least significant difference makes no adjustment for multiple comparisons; just do all pairwise mean t-tests with MSE as pooled variance estimate) Reject null H 0ij : µ i = µ j only when the difference between Ȳi and Ȳj exceeds the least significant difference LSD): LSD = t 1 MSE + 1 ) n i n j where t is the upper α/2 critical value of appropriate student t distribution with d.f. from MSE) plsd Fisher s protected least significant difference Do LSD tests only when F T rt test of H 0 : µ 1 =... = µ g is significant Does not give simultaneous confidence interval Controls FWER under complete null H 0 : µ 1 =... = µ g, not under a partial null in which some means are equal but others differ); i.e., provides no strong control of FWER, just weak control

8 Dunnett best suited for comparing a control treatment say factor level g) with other treatments factor levels 1,..., g 1) critical value based on Dunnett s t distribution see tables on pages of text) depends on d.f. same as for MSE; ν in tables) and g 1 number of factor levels - 1; K in tables) Reject null H 0i : µ i = µ g only when the difference between Ȳi and Ȳg exceeds the Dunnett significant difference DSD): DSD = d 1 MSE + 1 ) n i n g where d is the α critical value of appropriate Dunnett s t distribution Simultaneous confidence intervals Recall general C.I. for a parameter θ: ˆθ ± [ α Critical Value) SE of ˆθ)] Interpret: We are 1 α) 100% confident the interval contains its true value θ) For pairwise comparisons H 0 : µ i µ j = 0, build C.I. for µ i µ j : ˆµ i ˆµ j ) ± α Critical Value) SE[ˆµ i ˆµ j ]) ˆµ i ˆµ j = Ȳi Ȳj ; Critical Value depends on multiple comparison procedure; SE[ˆµ i ˆµ j ] = MSE ) 1 n i + 1 n j ) Reject H 0 if 0 / CI Equivalently, if Ȳi Ȳj > α Critical Value) MSE 1 n i + 1 n j ) Do C.I. for each µ i vs. µ j comparison of interest usually all pairwise); interpret as: For a family of K simultaneous intervals, we are 1 α) 100% confident all K intervals contain their true values µ i µ j )

Multiple Testing. Tim Hanson. January, Modified from originals by Gary W. Oehlert. Department of Statistics University of South Carolina

Multiple Testing. Tim Hanson. January, Modified from originals by Gary W. Oehlert. Department of Statistics University of South Carolina Multiple Testing Tim Hanson Department of Statistics University of South Carolina January, 2017 Modified from originals by Gary W. Oehlert Type I error A Type I error is to wrongly reject the null hypothesis