Linear Combinations. Comparison of treatment means. Bruce A Craig. Department of Statistics Purdue University. STAT 514 Topic 6 1

Transcription

1 Linear Combinations Comparison of treatment means Bruce A Craig Department of Statistics Purdue University STAT 514 Topic 6 1

2 Linear Combinations of Means y ij = µ + τ i + ǫ ij = µ i + ǫ ij Often study goal more than testing H 0 : all µ i equal Goal often a series of tests H 0 : L = c i µ i = L 0 Pairwise comparisons (if a = 3, test µ 1 = µ 2, µ 2 = µ 3, µ 1 = µ 3 ) Treatments vs control (if a = 4, test µ 1 = µ C, µ 2 = µ C, µ 3 = µ C ) Comparing combinations of treatments (e.g., 2µ 1 = µ 2 ) Assessing relationship between means of ordered trts Hypotheses may be planned or after the fact Very important to make the distinction STAT 514 Topic 6 2

3 Linear Combination of Means Can use our linear statistical model and properties of means and variances to construct t-test (F-test) of any linear combination L = c i y i. V( L) = V( c i y i. ) = c 2 i V(y i.) = MS E (c 2 i /n i ) t o = L L 0 V( L) Under H 0 : t o t N a STAT 514 Topic 6 3

4 Why Linear Combinations? These tests specifically address hypotheses of interest Overall F-test is jointly testing all possible alternatives Why include alternatives of no interest in your tests? - If overall error rate equal to α, error rate for single alternative will be < α - F test will reduce ( water down ) power of specific test - Can find individual tests significant while F not!! Issues with linear combinations - Multiple tests inflate overall Type I error rate - Not all tests of interest are independent of each other Will discuss procedures that control error rates STAT 514 Topic 6 4

5 Contrasts Linear combinations with coeff s that sum to zero Γ = c i µ i = 0 with c i = 0 H 0 : µ 1 = µ 2 c i = {1, 1,0,...,0} H 0 : 2µ 1 = µ 2 +µ 3 c i = {2, 1, 1,0,...,0} H 0 : 2µ 1 = µ 2 +µ 3 c i = { 1,0.5,0.5,0,...,0} Will estimate each Γ using C = c i y i. Recall under H 0 : t o = C/ V(C) t N a (Slide 3) STAT 514 Topic 6 5

6 Contrasts Also t 2 N a = F 1,N a so could consider F test Contrasts often presented in terms of Sum of Squares SS C = ( c i y i. ) 2 / (c 2 i /n i) If divide by MS E, simply t 2 o Can then compare to F 1,N a STAT 514 Topic 6 6

7 SAS Procedures (cont.sas) title1 Contrast Comparisons ; data one; infile c:\saswork\data\tensile.dat ; input percent strength time; proc glm data=one; class percent; model strength=percent; contrast C1 percent ; contrast C2 percent ; contrast C3 percent ; contrast C4 percent ; STAT 514 Topic 6 7

8 SAS Output Dependent Variable: STRENGTH Sum of Mean Source DF Squares Square F Value Pr > F Model Error Corrected Total R-Square C.V. Root MSE STRENGTH Mean Source DF Type I SS Mean Square F Value Pr > F PERCENT Contrast DF Contrast SS Mean Square F Value Pr > F C C C C STAT 514 Topic 6 8

9 Orthogonal Contrasts Suppose you have two contrasts {c i } and {d i } Orthogonal if c i d i = 0 (when n i constant) Can divide up SS Trt into a 1 orthogonal contrasts By Cochran s Thm comparisons independent Thus, previous four contrasts are independent STAT 514 Topic 6 9

10 Orthogonal Polynomial Contrasts Orthogonal contrasts used to study trend Should only be used if treatments quantitative (ordered) If equally-spaced and n i = n, the c i in Table IX Use software to determine coefficients in other situations Results in breakdown of polynomial regression µ i = β 0 +β 1 i +...+β a 1 i a 1 STAT 514 Topic 6 10

11 Using SAS to determine c i Often the levels of the trt are not equally spaced Can use Proc IML to determine coeffs proc iml; levels={ }; /* Consider these 5 levels */ print levels; coef=orpol(levels,3); /* Gives coefs up through cubic */ coef=t(coef); /* Puts coefs in rows instead of cols */ coef=coef[2:4,]; /* Eliminates the first row of coef matrix*/ print coef; /* 1st row linear, 2nd quadratic, 3rd cubic */ run; LEVELS COEF STAT 514 Topic 6 11

12 Testing Multiple Contrasts Let α now be P(at least one type I error) with α the significance level of a single test α known as the experimentwise or overall error rate α known as the comparisonwise error rate If m orthogonal (independent) contrasts α = 1 (1 α ) m α = 1 (1 α) 1/m Can control overall error rate α exactly Example: Consider m = 5 independent tests and want overall Type I error rate to be Using equation above, each test must use α = 1 (1.05) 1/5 = STAT 514 Topic 6 12

13 Testing Multiple Contrasts Scheffé s Method Protects against unplanned comparisons Set up 1 α simultaneous CI for all contrasts Overall error rate at most α low power Scheffé suggested using larger α level because of this Compare C to s C (a 1)Fα,a 1,N a Relationship with overall F test If the P-value of F-test is γ, then at the γ significance level, Scheffé method will find one contrast significant. c i = n i (y i. y.. ) = n iˆτ i STAT 514 Topic 6 13

14 Testing Multiple Contrasts Bonferroni s Method Recall P(A B) = P(A) + P(B) - P(A B) P(at least one type I error in m tests) mα Thus, for each test use α = α/m Designed for planned comparisons only Looks only at subset of contrasts Extremely conservative (low power) if m is large STAT 514 Topic 6 14

15 Comparison of Means Often only interested in pairwise comparisons Can be expressed as contrasts If comparing trt j and trt k 1 if i = j c i = 1 if i = k 0 otherwise STAT 514 Topic 6 15

16 Comparison of Means Pairwise comparisons a subset of all contrasts Want procedure that considers only this subset Trade-off between power and prob(type I error) No one best comparison method Methods vary in terms of protection Experimentwise error (overall Type I) Comparisonwise error (individual Type I) STAT 514 Topic 6 16

17 Pairwise Comparison Methods Least Significant Difference (LSD) Use MS E and its df to perform usual t-test y i. y j. MS E ( 1ni + 1nj ) t N a Does not control experimentwise error, controls α Protected Least Significant Difference (LSD) Only do LSD comparisons if F-test significant Provides a little control of overall error STAT 514 Topic 6 17

18 Pairwise Comparison Methods Tukey s (Tukey-Kramer) Method Uses studentized range distribution q instead of t Distribution based on y MAX y MIN in a trts Accounts for any possible pair being selected Controls overall experimentwise error rate α Reject if y i. y j. > q ( α,a,n a 1 MS E + 1 ) 2 n i n j The value q available in Table VII STAT 514 Topic 6 18

19 Pairwise Comparison Methods Newman-Keuls Test Performs Tukey s test for varying number of trts Start with p = a, then p = a 1, etc K p = q α (p,f) MS E /n Stop whenever difference not found significant Controls experimentwise error for all tests with m means When unequal sample size n = a/ 1/n i STAT 514 Topic 6 19

20 Pairwise Comparison Methods Duncan s Multiple Range Test Similar testing approach as Newman-Keuls Based on least significant range distribution Powerful does not control overall Type I Dunnett s Test Specifically designed for trts vs control situation Distribution based on MAX(y Control y Trt ) for a 1 trts Similar in approach to Tukey s test Controls experimentwise error rate STAT 514 Topic 6 20

21 title1 Means Comparison ; SAS Procedures data one; infile c:\saswork\data\tensile.dat ; input percent strength time; proc glm data=one; class percent; model strength=percent; means percent / alpha=.05 lines bon snk tukey; lsmeans percent / lines adjust=tukey; means percent / lines duncan lsd scheffe; means percent /dunnett; means percent / lsd clm; run; STAT 514 Topic 6 21

22 T tests (LSD) for variable: STRENGTH NOTE: This test controls the type I comparisonwise error rate not the experimentwise error rate. Alpha= 0.05 df= 20 MSE= 8.06 Critical Value of T= 2.09 Least Significant Difference= Means with the same letter are not significantly different. T Grouping Mean N PERCENT A B B B C C C STAT 514 Topic 6 22

23 Bonferroni (Dunn) T tests for variable: STRENGTH NOTE: This test controls the type I experimentwise error rate, but generally has a higher type II error rate than REGWQ. Alpha= 0.05 df= 20 MSE= 8.06 Critical Value of T= 3.15 Minimum Significant Difference= Means with the same letter are not significantly different. Bon Grouping Mean N PERCENT A A B A B B C C C C C STAT 514 Topic 6 23

24 Scheffe s test for variable: STRENGTH NOTE: This test controls the type I experimentwise error rate but generally has a higher type II error rate than REGWF for all pairwise comparisons Alpha= 0.05 df= 20 MSE= 8.06 Critical Value of F= Minimum Significant Difference= Means with the same letter are not significantly different. Scheffe Grouping Mean N PERCENT A A B A B B C C C C C STAT 514 Topic 6 24

25 Tukey s Studentized Range (HSD) Test for variable: STRENGTH NOTE: This test controls the type I experimentwise error rate, but generally has a higher type II error rate than REGWQ. Alpha= 0.05 df= 20 MSE= 8.06 Critical Value of Studentized Range= Minimum Significant Difference= Means with the same letter are not significantly different. Tukey Grouping Mean N PERCENT A A B A B B C C D C D D STAT 514 Topic 6 25

26 Student-Newman-Keuls test for variable: STRENGTH NOTE: This test controls the type I experimentwise error rate under the complete null hypothesis but not under partial null hypotheses. Alpha= 0.05 df= 20 MSE= 8.06 Number of Means Critical Range SNK Grouping Mean N PERCENT A B B B C C C STAT 514 Topic 6 26

27 Duncan s Multiple Range Test for variable: STRENGTH NOTE: This test controls the type I comparisonwise error rate, not the experimentwise error rate Alpha= 0.05 df= 20 MSE= 8.06 Number of Means Critical Range Duncan Grouping Mean N PERCENT A B B B C C C STAT 514 Topic 6 27

28 Dunnett s T tests for variable: STRENGTH NOTE: This tests controls the type I experimentwise error for comparisons of all treatments against a control. Alpha= 0.05 Confidence= 0.95 df= 20 MSE= 8.06 Critical Value of Dunnett s T= Minimum Significant Difference= Comparisons significant at the 0.05 level are indicated by *** Simultaneous Simultaneous Lower Difference Upper PERCENT Confidence Between Confidence Comparison Limit Means Limit *** *** *** STAT 514 Topic 6 28

29 General Testing Setting Consider m tests: H 1 0 vs H 1 a H 2 0 vs H 2 a. H m 0 vs H m a Consider that for m 0 of these tests H 0 is true and for m a of these tests H a true m 0 +m a = m STAT 514 Topic 6 29

30 General Testing Setting With m 0 and m a fixed but unknown, the given tests will allocate in the following way with V and T random variables DNR Reject Total H 0 True U V m 0 H a True T S m a W R m When m = 1, test designed to maximize E(S m a = 1) given E(V m 0 = 1) α STAT 514 Topic 6 30

31 General Testing Setting m > 1 Need to determine what to control DNR Reject Total Null True U V m 0 Alt True T S m a W R m Familywise Error Rate (FWER): Pr(V > 0) What Montgomery calls experimentwise error rate Controls the chance of there being a single false positive False Discovery Rate (FDR): E( V R > 0)Pr(R > 0) R Control the percent of rejected H 0 s that are wrong When m very large, trend towards controlling FDR STAT 514 Topic 6 31

32 Multiplicity Adjustment So far discussed approaches that adjust multiplier to SE Alter α level (e.g., Bonferroni) Use different distribution Conservative strong control of overall Type I error - avoids false positives Powerful able to pick up differences that exist - avoids false negatives All approaches try to strike some sort of balance STAT 514 Topic 6 32

33 Procedures Based on P-values Order the P-values from smallest to largest P 1,P 2,...,P m P (1),P (2),...,P (m) 1 Holm Refinement of Bonferroni Instead of using α = α m for all comparisons Continue to reject until P (k) > α/(m k +1) 2. False Discovery Rate (Benjamini and Hochberg) Continue to reject until P (k) > kα/m Both available in Proc Multtest in SAS STAT 514 Topic 6 33

34 False Discovery Rate Procedure 1: reject until P (k) > kα/m Known as a step-up procedure Depends on independence (although works well with some dependence among test statistics) Procedure 2: reject until P (k) > δ k [ ( )] 1/(m k+1) m δ k = 1 1 min 1, m k+1 α Known as a step-down procedure More powerful than Procedure 1 when number of tests is small and many hypotheses are clearly true STAT 514 Topic 6 34

35 False Discovery Rate ( ( Procedure 3: reject until P (k) > k m α/ 1 i)) Controls FDR under any dependence structure Not as powerful as first procedure Procedure 4: reject until P (k) > δ k ( ) m δ k = min 1, α (m k+1) 2 Controls FDR under any dependence structure More powerful than Procedure 3 when m small and most are not true STAT 514 Topic 6 35

36 Some Facts FDR FWER Equality when m = m 0 Newman-Keuls controls FDR There is range test by Ryan-Einoit-Gabriel-Welsch (REGWR) which is similar to Holm s control of FWER STAT 514 Topic 6 36

37 SAS Commands title1 Multiple Comparisons ; data one; infile U:\.www\datasets1\tensile.dat ; input percent strength time; proc multtest holm fdr data=one out=new1 noprint; class percent; contrast ; contrast ; contrast ; contrast ; contrast ; contrast ; contrast ; contrast ; contrast ; contrast ; test mean(strength); proc print data=new1; run; STAT 514 Topic 6 37

38 _contrast value se_ raw_p stpbon_p fdr_p Holm similar to FDR except 2 vs 5 and 3 vs 4 Holm similar to Bonferroni FDR similar to Newman-Keuls STAT 514 Topic 6 38

39 Background Reading Contrasts: Montgomery Orthogonal contrasts: Montgomery Scheffé method: Montgomery Pairwise mean comparisons: Montgomery Dunnett s method: Montgomery STAT 514 Topic 6 39