Week 14 Comparing k(> 2) Populations

Transcription

1 Week 14 Comparing k(> 2) Populations

2 Week 14 Objectives Methods associated with testing for the equality of k(> 2) means or proportions are presented. Post-testing concepts and analysis are introduced. In particular: 1 The principle of analysis of variance is used to derive the F test for comparing k(> 2) means. 2 The χ 2 test for comparing k(> 2) proportions is presented in its contingency table form and its relation to the F test is explained. 3 The rank-based Kruskal-Wallis test is presented. 4 The concepts of multiple comparisons and simultaneous CIs are introduced and methods for their implementation are presented.

3 Bonferroni Tukey s

4 The Statistical Model and Hypothesis X i1, X i2,..., X ini, i = 1,..., k, are independent samples. X ij = µ i + ɛ ij, Var(ɛ ij ) = σ 2 i. Write µ i = µ + α i, where µ = k 1 n i=1 µ i, and α i = µ i µ is the effect of population i (of factor level i). Of interest is the hypothesis of no effects (all α i = 0), or H 0 : µ 1 = µ 2 = = µ k vs H a : H 0 is false. The sample mean and sample variance from population i are: X i, S 2 i, i = 1,..., k.

5 The basic idea As in regression, variability is represented by the so-called sums of squares, or SS, and the ANOVA approach decomposes the total variability into components. For the comparison of k means, one of the components is called between groups variability and the other is called within groups variability. If the between groups variability is large compared to the within groups variability, the hypothesis of equal means is rejected.

6 Assume homoscedasticity, i.e. Var(ɛ ij ) = σ 2, for all i = 1,..., k, set N = n n k, and define: X = 1 N k n i X ij = 1 N i=1 j=1 k n i X i i=1 The overall mean X is the sample mean of all data. SST = k n i (X ij X ) 2 Total Sum of Squares i=1 j=1 SST is the numerator of sample variance of all data.

7 k n i SSE = (X ij X i ) 2 Error Sum of Squares i=1 j=1 SSE represents the within group variability SSTr = k n i (X i X ) 2 i=1 Treatment Sum of Squares SSTr represents the between group variability It can be shown that SST = SSTr + SSE.

8 Each SS has its own DF (recall N = n n k ): DF SST = N 1, DF SSE = N k, DF SSTr = k 1 Dividing each SS by its DF we obtain the mean squares: MSE = SSE SSTr, MSTr = N k k 1. MSE is the k-sample version of the 2-sample pooled variance: MSE = S 2 p = (n 1 1)S (n k 1)S 2 k n n k k Thus, MSE is an unbiased estimator of the common error variance σ 2.

9 These calculations are summarized in the ANOVA table: Source df SS MS F Treatment k 1 SSTr MSTr= SSTr k 1 Error N k SSE MSE= SSE N k Total N 1 SST Under H 0, and the assumption of normality F H0 = MSTr MSE F H0 F k 1,N k. Without normality the above holds approximately if all sample sizes are 30.

10 If H 0 does not hold, the statistic F H0 tends to take larger values. Thus, H 0 is rejected at level α if F H0 > F k 1,N k,α, where F k 1,N k,α denotes the (1 α)100-th percentile of the F distribution with k 1 and N k degrees of freedom.

11 Example The following data resulted from comparing the degree of soiling in fabric treated with three different mixtures of methacrylic acid. Mix 1: Mix 2: Mix 3: Test H 0 : µ 1 = µ 2 = µ 3 vs H a : H 0 is false, at α = 0.1.

12 Example (Continued) Solution. Because of the small sample sizes, we need to assume that the three populations are normal and homoscedastic. With this data, X 1 = 0.918, X 2 = 0.794, X 3 = and X = Also, the three sample variances are S1 2 = , S2 2 = and S2 3 = Thus, SSTr = 5( ) 2 + 5( ) 2 +5( ) 2 = SSE = 4S S S2 3 = It follows that, SST = = , MSTr = /(3 1) = ,

13 Example (Continued) MSE = /(15 3) = and F H0 = MSTr/MSE = The above calculation are summarized in the following ANOVA table Source df SS MS F Treatment k 1 = Error N k = Total N 1 = The rejection rule specifies that H 0 be rejected if F > F 2,12,0.1. Since F 2,12,0.1 = (found in R by qf(0.9,2,12)), H 0 is not rejected.

14 F and χ 2 Tests F and χ 2 distributions are connected by the following result: If Y F ν1,ν 2, with ν 2 large, then ν 1 Y χ 2 ν 1. For example, under H 0, k i=1 Q H0 = (k 1)F H0 = n i(x i X) 2 S 2 p χ 2 k 1. The χ 2 statistic for testing the equality of k proportions is essentially the Q H0 statistic adjusted to Bernoulli data.

15 Computational Form of Q H0 To adjust Q H0 to Bernoulli data, replace X i by p i, X by the pooled proportion of successes p, and S 2 p by p(1 p). Thus the test statistic for H 0 : p 1 = = p k is: Q H0 = k i=1 n i( p i p) 2, p(1 p) and H 0 is rejected at level α if Q H0 > χ 2 k 1,α. χ 2 k 1,α is found in R by qchisq(1-α, k-1). The p-value corresponding to Q H0 is found in R by 1-pchisq(Q H0, k-1).

16 The Contingency Table Form of Q H0 An alternative form of the test statistic Q H0, called the contingency table form, is more common: k 2 (O Q H0 = li E li ) 2, where i=1 l=1 O 1i = n i p i, O 2i = n i ( 1 p i ), E1i = n i p, E 2i = n i ( 1 p ). The algebraic equivalence of the two forms of Q H0 is shown on page 357 of the book. The contingency table form is not recommended for hand calculations. E li

17 Example (Pilot response time for different panel designs) The sample sizes, n i, and number of times, O 1i, that the response times were below 3 seconds for the four designs are as follows: n 1 = 45, O 11 = 29; n 2 = 50, O 12 = 42; n 3 = 55, O 13 = 28; n 4 = 50, O 14 = 24. Perform the test at α = Solution. We will use the computational form of Q H0. Here, each p i = O 1i /n i, so p 1 = , p 2 = 0.84, p 3 = and p 4 = Also, p = O 11 + O 12 + O 13 + O 14 n 1 + n 2 + n 3 + n 4 = 0.615, so that S 2 p = p(1 p) = Thus,

18 Example (Example continued) Q H0 = 45( ) ( ) ( )2 50( ) = Since > χ 2 3 (0.05) = 7.815, H 0 is rejected.

19 The extension of the two-sample rank sum test procedure to k samples, i.e., for testing is called Kruskal-Wallis test. H F 0 : F 1 = = F k, (4.1) Like the rank-sum test, the Kruskal-Wallis test uses the (mid-)ranks of the observations: Combine the observations, X i1,..., X ini, i = 1,..., k, from the k samples into an overall set of N = n n k observations, and arrange them from smallest to largest.

20 Let R ij denote the (mid-)rank of observation X ij, and set R i = n 1 i n i j=1 R ij, S 2 KW = 1 N 1 k n i i=1 j=1 ( R ij N + 1 ) 2. 2 Note that SKW 2 is the sample variance of the collection of all ranks. The Kruskal-Wallis TS is KW k = 1 S 2 KW k i=1 ( n i R i N + 1 ) 2. 2

21 If there are no ties a simpler expression holds: KW k = 12 N(N + 1) k i=1 ( n i R i N + 1 ) 2. 2 If the k populations have continuous distributions, the exact null distribution of KW k is known. For hand implementation, we require n i > 8, for all i, and use the approximate null distribution KW k χ 2 k 1. Thus, the RR at level α is KW k > χ 2 k 1,α, and the p-value is computed in R by 1-pchisq(KW k, k-1).

22 Example (Rank procedures for the flammability data) A flammability test was performed on six pieces from each of three types of fabric used in children s clothing. The response variable is the length of the burn mark. The data is given in Flammability.txt. Test the hypothesis that there is no difference in flammability among the three materials, at α = Solution The ranks, and the three rank averages are Ranks R i Material Material Material

23 Example (Example continued) The sample variance of the combined ranks is SKW 2 = Thus, KW k = [ ( ) 2 ( ) ( ) ] 2 = The R command 1-pchisq(4.823,2) yields a p-value of , so that H 0 cannot be rejected at level 0.05.

24 Bonferroni Tukey s Bonferroni Tukey s

25 Bonferroni Tukey s When H 0 : µ 1 = µ 2 = = µ k is rejected, it is not clear which of the µ i are significantly different. It would seem that this question can be addressed quite simply by making all pairwise comparisons. For example, if k = 5 and H 0 is rejected we can make CIs for all 10 pairwise differences, µ 1 µ 2,..., µ 1 µ 5, µ 2 µ 3,..., µ 2 µ 5,..., µ 4 µ 5, and if a CI does not include 0, the corresponding difference is declared significantly different from 0. Alternatively, we can test each of the 10 null hypotheses H i,j 0 : µ i = µ j, for 1 i < j 5, against the two-sided alternative. With some fine tuning this idea works.

26 Bonferroni Tukey s The fine-tuning is needed to assure that the overall, or experiment-wise error rate does not exceed α. The word experiment in the term experiment-wise error rate refers to all pairwise comparisons. The experiment-wise error rate is defined as the probability of at least one pair of means being declared different when all means are equal. The experiment-wise error rate is nothing but the type I error rate for the procedure that rejects H 0 : µ 1 = µ 2 = = µ k if at least one of the H i,j 0 is rejected, or if at least one of the ( k 2) CI for the pairwise differences does not include 0. Fine-tuning is needed because the experiment-wise error rate is different from the level of significance used for testing each H i,j 0.

27 Bonferroni Tukey s To appreciate how much the experiment-wise error rate can differ from the level of significance used for each H i,j 0, consider k = 5 and suppose the 10 tests are performed independently. (They are not really!) Then the probability that no H i,j 0 is rejected when in fact H 0 is true, is (1 α) 10, and thus the experiment-wise error rate is 1 (1 α) 10. If α = 0.05, this is 1 (1 0.05) 10 = In spite of the unrealistic independence assumption, the above calculation gives an fairly close approximation to the true experiment-wise error rate.

28 Bonferroni Tukey s Confidence intervals which control the experiment-wise error rate at a desired level α will be called (1 α)100% simultaneous confidence intervals. Thus, 10 CI are called 90% simultaneous confidence intervals if 90% of the time they all contain the true value of the respective difference or, equivalently, 10% of the time at least one of them does not contain the true value of the respective difference. The term multiple comparisons is used to refer to a test procedure that controls the experiment-wise error rate. Multiple comparisons can be done through simultaneous CI, or via hypothesis testing.

29 Bonferroni Tukey s We will see two methods of constructing simultaneous CIs. Bonferroni CIs: These use Bonferroni s inequality to give a lower bound on the level of the simultaneous CIs (or an upper bound on the experiment-wise error rate). Tukey CIs: The level of these simultaneous CIs is exact when sampling from normal homoscedastic populations. They can also be used as a good approximation with large samples from any distribution. Tukey s method can also be applied when sampling from skewed homoscedastic populations and have large sample sizes, as well as on the ranks with smaller sample sizes.

30 Bonferroni Tukey s Bonferroni Tukey s

31 Bonferroni Tukey s Bonferroni s CIs achieve the desired level by adjusting the level of the traditional CIs. Exact adjustment is not possible due to the dependence of the CIs. But Bonferroni s inequality guarantees that the intervals are conservative (i.e., the exact level of 90% Bonferroni CIs is higher than 90%). Bonferroni s inequality asserts that, the simultaneous coverage of m CIs, each of which has confidence level 1 α, is at least 1 mα. Equivalently, if each of m pairwise tests are performed at level α, the experiment-wise error rate is no greater than mα.

32 Bonferroni Tukey s For example, if we are comparing k means (or proportions) there are ( k 2) pairwise comparisons to be made. For k = 3, 4, 5 we have 3, 6, and 10 pairwise comparisons respectively. Assuming the pairwise 95% CIs are independently constructed, their simultaneous coverage and the corresponding Bonferroni lower bound, is (0.857, 0.85), (0.735, 0.7), (0.599, 0.5). The above was computed by the R command: a=0.05; (1-a)**3; 1-3*a; (1-a)**6; 1-6*a; (1-a)**10; 1-10*a)

33 Bonferroni Tukey s The above discussion leads to the following procedure for constructing (1 α)100% Bonferroni simultaneous CIs and multiple comparisons: For each of the m contrast construct a (1 α/m)100% CI. This set of m CIs are the (1 α)100% Bonferroni simultaneous CIs for the m contrasts. Bonferroni multiple comparisons at level α: If any of the m (1 α)100% Bonferroni simultaneous CIs does not contain zero, the corresponding contrast is declared significantly different from zero at experiment-wise level α. Bonferroni multiple comparisons can also be conducted with pairwise testing, without CIs. Each of the m pairwise tests is conducted at level α/m. Those rejected are declared significantly different at experiment-wise level α.

34 Bonferroni Tukey s Example Test, at α = 0.05, the null hypothesis that the four panel designs have no effect on whether or not the pilot reaction time is below 3 seconds (H 0 : p 1 = p 2 = p 3 = p 4 vs H a : H 0 is false) using Bonferroni multiple comparisons. Solution: We will construct 95% Bonferroni simultaneous CIs for the contrasts p 1 p 2, p 1 p 3, p 1 p 4, p 2 p 3, p 2 p 4, p 3 p 4. Because there are m = 6 contrasts, we construct (1 0.05/6)100% = 99.17% CIs for each of the above contrasts. The data are: n 1 = 45, O 11 = 29; n 2 = 50, O 12 = 42; n 3 = 55, O 13 = 28; n 4 = 50, O 14 = 24.

35 Bonferroni Tukey s Example (Continued) The CIs for each contrast were constructed according to p i (1 p i ) p i p j ± z αb /2 + p j (1 p j ) n i n j where α B = For example, the following R commands compute the CI for p 1 p 2 : n1= 45; n2=50; p1=29/n1; p2=42/n2 p1-p2 -qnorm(1-( )/2)*sqrt(p1*(1-p1)/n1+p2*(1-p2)/n2) p1-p2 +qnorm(1-( )/2)*sqrt(p1*(1-p1)/n1+p2*(1-p2)/n2) The second and third commands give , , respectively. The command prop.test(c(29,42), c(45,50), conf.level = , correct=f) gives the same result.

36 Bonferroni Tukey s Example (Continued) The resulting CIs can be presented in a table: Contrast 99.17% CI Contains zero? p 1 p 2 (-0.428, ) Yes p 1 p 3 (-0.124, 0.394) Yes p 1 p 4 (-0.101, 0.429) Yes p 2 p 3 (0.106, 0.555) No p 2 p 4 (0.129, 0.591) No p 3 p 4 (-0.229, 0.287) Yes Only p 2 p 3, and p 2 p 4 are significantly different from zero, at experiment-wise level α = 0.05, from p 3 and p 4. Thus, H 0 : p 1 = = p 4 was rejected due to significant differences between panel design 2 and panel designs 3 and 4.

37 Bonferroni Tukey s An organized way of presenting the multiple comparisons outcome consists of a) List the sample proportions (or sample means) in increasing order, and b) underline pairs that are not significantly different. Then pairs that are not underlined are significantly different. In the previous example, p 4 = 0.48, p 3 = 0.51, p 1 = 0.64 and p 2 = Thus, the multiple comparisons outcome can be presented as p 4 p 3 p 1 p

38 Bonferroni Tukey s Example The data in GradesTeachMeth.txt contain exams scores for students exposed to three different teaching methods (Exercises 5 and 6, Section 10.3). After reading the data into gr, the commands attach(gr); kruskal.test(score method) return a p-value of 0.021; thus H 0 : F 1 = F 2 = F 3 is rejected at α = Use the Bonferroni multiple comparisons procedure, based on the rank-sum test, to identify which methods differ significantly at α = Solution: We will perform multiple comparisons through testing. Because the desired experiment-wise error rate is 0.05, we will conduct each of the m = 3 pair-wise comparisons (A vs B, A vs C, and B vs C) at level 0.05/3 = If the p-value of one of these comparisons is smaller than , the corresponding methods are declared significantly different.

39 Bonferroni Tukey s Example (Continued) The command wilcox.test(score[1:16] method[1:16]) yields a p-value of for the A vs B comparison. The p-values for the other comparisons are obtained similarly. The results from all tree rank-sum tests are summarized in the following table: Comparison p-value Less than ? A vs B No A vs C Yes B vs C No Thus, methods A and C are significantly different at experiment-wise error rate α = 0.05, but methods A and B, as well as methods B and C are not significantly different.

40 Bonferroni Tukey s Bonferroni Tukey s

41 Bonferroni Tukey s This procedure requires normality (or all n i 30) and homoscedasticity. They are based on the studentized range distribution which is characterized by two degrees of freedom. The numerator degrees of freedom equals k (the number of populations). The denominator degrees of freedom equals the degrees of freedom for the SSE, i.e., N k, with N = n n k. Tables for the studentized range distribution are available but we will only use R output. The R commands for constructing them are given next.

42 Bonferroni Tukey s R Commands for Tukey s Simultaneous CIs Import the iron concentration data from http: //personal.psu.edu/acq/401/data/fedata.txt in the data frame fe. Then do: out=aov(fe$conc fe$ind) TukeyHSD(out) # lm instead of aov will NOT work here plot(tukeyhsd(out)) or TukeyHSD(out,conf.level=0.99) # for 99% simultaneous CIs plot(tukeyhsd(out,conf.level=0.99))