Multiple comparisons The problem with the one-pair-at-a-time approach is its error rate.

Transcription

1 Multiple comparisons The problem with the one-pair-at-a-time approach is its error rate. Each confidence interval has a 95% probability of making a correct statement, and hence a 5% probability of making an error. But there are 6 intervals, so the probability that they are all correct is much less than 95%, and the error rate for the family of intervals is much higher than 5%. The family error rate is actually 19.2% (see Table 13-5). 1 / 17 One Factor with More Than Two Levels Multiple comparisons

2 Sometimes the high family-wise error rate is not a problem. But suppose we ask Which level of hardwood gives paper that is significantly stronger than the other levels? Answering that question requires that we test each level against each other level, that is all 6 comparisons. We need a method that controls the error rate for the whole family of comparisons. 2 / 17 One Factor with More Than Two Levels Multiple comparisons

3 Tukey s Honest Significant Difference method does control the family-wise error rate. In R TukeyHSD(aov(Strength ~ factor(hardwood), paper)) The calculations are similar to the LSD method, but with LSD = 3.07 replaced by HSD = The confidence intervals are wider, but the conclusions are the same. 3 / 17 One Factor with More Than Two Levels Multiple comparisons

4 The HSD is calculated using the Studentized range distribution: HSD = q (.05,4,20 1 MS E + 1 ) 2 n i n i where q α,a,ν is the critical value for tail area α when the factor has a levels and MS E has ν degrees of freedom. The difference from the LSD is that t.025,20 is replaced by q.05,4,20 2. In the example, t.025,20 is computed as qt(.025, 20, lower.tail = FALSE) = 2.086, while q.05,4,20 2 is computed as qtukey(.05, 4, 20, lower.tail = FALSE) / sqrt(2) = / 17 One Factor with More Than Two Levels Multiple comparisons

5 Blocked Design The CRD is not always feasible. For example, it might be feasible to carry out only 12 strength tests in one day. We would then carry out the experiments in 2 blocks each of 12 runs, with 3 replicates of each treatment in each block. The 12 runs in each block should then be carried out in random order, and the design is an example of a randomized complete block design (RCBD). The analysis of measurements collected in a RCBD allows for a systematic difference between measurements made in different blocks. 5 / 17 One Factor with More Than Two Levels Blocked Designs

6 In other cases, the experimental units can be grouped into blocks, so that units within the same block are more similar than units in different blocks. Example: Fabric strength Question: what are the effects of 4 different chemicals on the strength of a fabric? Five samples of the fabric are chosen, and each is cut into 4 swatches. The 4 chemicals are randomly assigned to the 4 swatches from a given sample of fabric, applied, and measured for strength. 6 / 17 One Factor with More Than Two Levels Blocked Designs

7 This is a complete blocked design, because every treatment (chemical) is used in every block (sample). It is also a balanced design, because each treatment is used the same number of times in each block (namely, once). It is also randomized: a randomized complete block design (RCBD). The advantage: comparisons within blocks (samples) are not affected by inherent variations across blocks. 7 / 17 One Factor with More Than Two Levels Blocked Designs

8 Comparison with CRD Suppose we had used an unblocked design, by pooling the 20 swatches and randomly assigning the 5 replicates for each chemical to a given swatch. Then chemical A might be assigned to two swatches from sample 1, and none from sample 2. If sample 1 is inherently weaker than sample 2, this would make chemical A appear to weaken the fabric. Generally, differences from sample to sample would increase the noise level in the measured strengths. 8 / 17 One Factor with More Than Two Levels Blocked Designs

9 Analysis of data from a blocked design The analysis takes account of the variability from block to block. Specifically, we calculate a sum of squares for blocks: SS Blocks = n b (ȳ j ȳ ) 2, j=1 where now y i,j is the response for treatment i in block j. The corresponding ANOVA equation defines SS Errors : SS Total = SS Blocks + SS Treatments + SS Errors 9 / 17 One Factor with More Than Two Levels Blocked Designs

10 In R The ANOVA table: fabric <- read.csv("data/table csv"); fabricaov <- aov(strength ~ Sample + Chemical, fabric); summary(fabricaov) Confidence intervals for pairwise differences, with 95% family-wise confidence level: TukeyHSD(fabricAov, "Chemical") 10 / 17 One Factor with More Than Two Levels Blocked Designs

11 Special case: two levels The case where the factor has only two levels is often treated specially. Example: Shear strength of steel girders Context: Comparing different methods for measuring shear strength of steel plate girders; Experimental factor: Method; two levels, Karlsruhe and Lehigh; Blocking factor: Girder; the methods were each used to measure the strength of 9 girders; Response: Strength. 11 / 17 One Factor with More Than Two Levels Blocked Designs

12 In R girder <- read.csv("data/table csv") # as a blocked design: summary(aov(strength ~ Girder + Method, girder)) # paired t-test: with(girder, t.test(strength[method == "K"], Strength[Method == "L"], paired = TRUE)) # compare the paired differences with zero: with(girder, t.test(strength[method == "K"]- Strength[Method == "L"])) 12 / 17 One Factor with More Than Two Levels Blocked Designs

13 Statistical Models The observations in a randomized complete block design may be represented as where: y i,j = µ + τ i + β j + ɛ i,j, i = 1, 2,..., a; j = 1, 2,..., b, µ is an overall mean, that is a typical value for these measurements; τ i is the effect of the i th level of the factor; β j is similarly the effect of the j th block; ɛ i,j is a random error term, specific to this observation. This model is additive: the effects of the factor are assumed to be the same in each block. 13 / 17 One Factor with More Than Two Levels Statistical Models

14 The null hypothesis that the factor has no effect on the response is: H 0 : τ 1 = τ 2 = = τ a = 0. We could also test whether there was no difference among blocks: H 0 : β 1 = β 2 = = β b = 0, but that is not usually an important question. 14 / 17 One Factor with More Than Two Levels Statistical Models

15 The model y i,j = µ + τ i + β j + ɛ i,j is over-parametrized: if some constant is added to µ and subtracted from all the treatment effects τ i, the sum is not changed. That makes the parameters unidentifiable: they cannot be uniquely estimated. We avoid this problem by imposing a constraint on the treatment effects τ i : In the textbook: a i=1 τ i = 0; In R: τ 1 = 0; In JMP and SAS: τ a = 0. The block effects β j need a corresponding constraint. 15 / 17 One Factor with More Than Two Levels Statistical Models

16 So in R, with i = 1 and j = 1, y 1,1 = µ + ɛ 1,1 ; this implies that µ is the mean response for level 1 of the factor, the baseline level, in block 1, the baseline block. With i > 1 and j > 1, y i,j = µ + τ i + β j + ɛ i,j, so τ i is the difference between the mean response for level i and the baseline level, and similarly β j is a difference between blocks. Example: Parameter estimates for fabric strength summary(lm(strength ~ Sample + Chemical, fabric)) 16 / 17 One Factor with More Than Two Levels Statistical Models

17 Model for the CRD The model for a completely random design is similar, but with no block effects: y i,j = µ + τ i + ɛ i,j. The same constraint needs to be applied to the treatment effects τ i. Example: Parameter estimates for paper strength summary(lm(strength ~ factor(hardwood), paper)) 17 / 17 One Factor with More Than Two Levels Statistical Models