Topic 5. Mean separation: Multiple comparisons [ST&D Ch.8, except 8.3]

Transcription

1 5.1 Topic 5. Mean sepaation: Multiple compaisons [ST&D Ch.8, except 8.3] Basic concepts In the analysis of vaiance, the null hypothesis that is tested is always that all means ae equal. If the F statistic is not significant, we fail to eject H 0 and thee is nothing moe to do, except possibly edo the expeiment, taking measues to make it moe sensitive. If H 0 is ejected, then we conclude that at least one mean is significantly diffeent fom at least one othe mean. The oveall ANOVA gives no indication of which means ae significantly diffeent. If thee ae only two teatments, thee is no poblem; but if thee ae moe than two teatments, the poblem emains of needing to detemine which means ae significantly diffeent. This is the pocess of mean sepaation. Mean sepaation takes two geneal foms: 1. Planned, single degee of feedom F tests (othogonal contasts, last topic). Multiple compaison tests that ae suggested by the data (multiple compaison tests, Topic 5) itself (this topic). Of these two methods, othogonal F tests ae pefeed because they ae moe poweful than multiple compaison tests (i.e. they ae moe sensitive to diffeences than ae multiple compaison tests). As you saw in the last topic, howeve, contasts ae not always appopiate because they must satisfy a numbe of stict constaints: 1. Contasts ae planned compaisons, so the eseache must have a pioi knowledge about which compaisons ae most inteesting. This pio knowledge, in fact, detemines the teatment stuctue of the expeiment.. The set of contasts must be othogonal. 3. The eseache is limited to making, at most, (t 1) compaisons. Vey often, howeve, thee is no such pio knowledge. The teatment levels do not fall into meaningful goups, and the eseache is left with no choice but to cay out a sequence of multiple, unconstained compaisons fo the pupose of anking and disciminating means. The diffeent methods of multiple compaisons allow the eseache to do just that. Thee ae many such methods, the details of which fom the bulk of this topic, but geneally speaking each involves moe than one compaison among thee o moe means and ae paticulaly useful in those expeiments whee thee ae no paticula elationships among the teatment means. 5.. Eo ates Selection of the most appopiate multiple compaison test is heavily influenced by the eo ate. Recall that a Type I eo, occus when one incoectly ejects a tue H 0. The Type I eo ate is the faction of times a Type I eo is made. In a single compaison (imagine a simple t test) this is the value. When compaing thee o moe teatment means, howeve, thee ae at least two diffeent ates of Type I eo:

2 5. Compaison-wise Type I eo ate (CER) This is the numbe of Type I eos divided by the total numbe of compaisons Expeiment-wise Type I eo ate (EER) This is the numbe of expeiments in which at least one Type I eo occus, divided by the total numbe of expeiments Suppose the expeimente conducts 100 expeiments with 5 teatments each. In each expeiment thee is a total of 10 possible paiwise compaisons that can be made: Total possible paiwise compaisons (p) = t( t 1) Fo t = 5, p = (1/)*(5*4) = 10 i.e. T 1 vs. T, T 3,T 4,T 5 ; T vs. T 3,T 4,T 5 ; T 3 vs. T 4,T 5 ; T 4 vs. T 5 With 100 such expeiments, theefoe, thee ae a total of 1,000 possible paiwise compaisons. Suppose that thee ae no tue diffeences among the teatments (i.e. H 0 is tue) and that in each of the 100 expeiments, one Type I eo is made. Then the CER ove all expeiments is: The EER is CER= (100 mistakes) / (1000 compaisons) = 0.1 o 10% EER= (100 expeiments with mistakes) / (100 expeiments) = 1 o 100%. The EER is the pobability of making at least one Type I eo in the expeiment. As the numbe of means (and theefoe the numbe of possible compaisons) inceases, the chance of making at least one Type I eo appoaches 1. To peseve a low expeimentwise eo ate, then, the compaison-wise eo ate must be held extemely low. Convesely, to maintain a easonable compaison-wise eo ate, the expeiment-wise eo ate will inflate. The elative impotance of contolling these two Type I eo ates depends on the objectives of the study, and diffeent multiple compaison pocedues have been developed based on diffeent philosophies of contolling these two kinds of eo. In situations whee incoectly ejecting one compaison may jeopadize the entie expeiment o whee the consequence of incoectly ejecting one compaison is as seious as incoectly ejecting a numbe of compaisons, the contol of expeiment-wise eo ate is moe impotant. On the othe hand, when one eoneous conclusion will not affect othe infeences in an expeiment, the compaison-wise eo ate is moe petinent. The expeiment-wise eo ate is always lage than the compaison-wise eo ate. It is difficult to compute the exact expeiment-wise eo ate because, fo a given data set,

3 5.3 Type I eos ae not independent. But it is possible to compute an uppe bound fo the EER by assuming that the pobability of a Type I eo fo any single compaison is and is independent of all othe compaisons. In that case: Uppe bound EER = 1 - (1 - ) p t( t 1) whee p=, as befoe. So fo 10 teatments and = 0.05, the uppe bound of the EER is 0.9 (EER = 1 (1 0.05) 45 = 0.90 o 90%). The situation is moe complicated than this, howeve. Suppose thee ae 10 teatments and one shows a significant effect while the othe 9 ae appoximately equal. Such a situation is indicated gaphically below: x Y i. x x x x x x x x x Y Teatment numbe A simple ANOVA will pobably eject H 0, so the expeimente will want to detemine which specific means ae diffeent. Even though one mean is tuly diffeent, thee is still a chance of making a Type I eo in each paiwise compaison among the 9 simila teatments. An uppe bound on this pobability is computed by setting t = 9 in the above fomula, giving a esult of That is, the expeimente will incoectly conclude that two tuly simila effects ae actually diffeent 84% of the time. This is called the expeiment-wise eo ate unde a patial null hypothesis, the patial null hypothesis in this case being that the subset of nine teatment means ae all equal to one anothe. So we can distinguish between the EER unde the complete null hypothesis, in which all teatment means ae equal, and the EER unde a patial null hypothesis, in which some means ae equal but some diffe. Because of this fact, SAS subdivides the eo ates into the following fou categoies: CER = compaison-wise eo ate EERC = expeiment-wise eo ate unde a complete null hypothesis (standad EER) EERP = expeiment-wise eo ate unde a patial null hypothesis. MEER = maximum expeiment-wise eo ate unde any complete o patial null hypothesis Multiple compaisons tests

4 5.4 Statistical methods fo making two o moe infeences while contolling the Type I eo ates (CER, EERC, EERP, MEER) ae called simultaneous infeence methods. The mateial in this sectionis based pimaily on ST&D chapte 8 and on the SAS/STAT manual (GLM Pocedue). The basic techniques of multiple compaisons fall into two goups: 1. Fixed-ange tests: Those which povide confidence intevals and tests of hypotheses.. Multiple-ange tests: Those which povide only tests of hypotheses. To illustate the vaious pocedues, we will use the data fom two diffeent expeiments given in Table 4-1 (pevious class, equal eplication) and 5-1 (below, unequal eplication). The ANOVAs fo these expeiments ae given in Tables 4- and 5-. Table 5.1. Weight gains (lb/animal/day) as affected by thee diffeent feeding ations. CRD, with unequal eplications. Teatment N Total Mean Contol Feed-A Feed-B Feed-C Oveall Table 5-. ANOVA of data in Table 5-1. Souce of Vaiation df Sum of Squaes Mean Squaes F Total Teatment Exp. eo Fixed-ange tests These tests povide a single ange fo making all possible paiwise compaisons in expeiments with equal eplications acoss teatment goups (i.e. in balanced designs). Many fixed-ange pocedues ae available, and consideable contovesy exists as to which pocedue is most appopiate. We will pesent fou commonly used pocedues, moving fom the less consevative to the moe consevative: LSD, Dunnett, Tukey, and Scheffe. Othe paiwise tests ae discussed in the SAS manual The epeated t and least significant diffeence: LSD One of the oldest, simplest, and most widely misused multiple paiwise compaison tests is the least significant diffeence (LSD) test. The LSD is based on the t-test (ST&D 101); in fact, it is simply a sequence of many t-tests. Recall the fomula fo the t statistic: Y( ) s t whee s s Y ( ) Y ( )

5 5.5 This t statistic is distibuted accoding to a t distibution with ( 1) degees of feedom. The LSD test declaes the diffeence between means Y i and Y j of teatments i and j to be significant when: Y i Y j > LSD, whee 1 1 LSD t fo unequal (SAS calls this a epeated t test), df 1 LSD t fo equal (SAS calls this an LSD test), df Whee = pooled s and can be calculated by PROC ANOVA o PROC GLM. The above statistic is called the studentized ange statistic. The quantity unde the squae oot is called the standad eo of the diffeence, o SED. As an example, hee ae the calculations fo Table 4.1. Note that the significance level selected fo paiwise compaisons does not have to confom to the significance level of the oveall F test. To compae pocedues acoss the examples to come, we will use a common = Fom Table 4-1, = with 16 df. LSD t, df So, if the absolute diffeence between any two teatment means is moe than 0.143, the teatments ae said to be significantly diffeent at the 5% confidence level. As the numbe of teatments inceases, it becomes moe and moe difficult, just fom a logistical point of view, to identify those pais of teatments that ae significantly diffeent. A systematic pocedue fo compaison and anking begins by aanging the means in descending o ascending ode as shown below: Contol 4.19 HCl 3.87 Popionic 3.73 Butyic 3.64 Once the means ae so aanged, compae the lagest with the smallest mean. If these two means ae significantly diffeent, compae the next lagest mean with the smallest. Repeat this pocess until a non-significant diffeence is found. Label these two and any means in between with a common lowe case lette by each mean. Repeat the pocess with the next smallest mean, etc. Ultimately, you will aive at a mean sepaation table like the one shown below:

6 5.6 Table 5.5 Teatment Mean LSD Contol 4.19 a HCl 3.87 b Popionic 3.73 c Butyic 3.64 c Pais of teatments that ae not significantly diffeent fom one anothe shae the same lette. Fo the above example, we daw the following conclusions at the 5% confidence level: All acids educed shoot gowth. The eduction was moe sevee with butyic and popionic acid than with HC1. We do not have evidence to conclude that popionic acid is diffeent in its effect than butyic acid. When all the teatments ae equally eplicated, note that only one LSD value is equied to test all six possible paiwise compaisons between teatment means. This is not tue in cases of unequal eplication, whee diffeent LSD values must be calculated fo each compaison involving diffeent numbes of eplications. Fo the second data set (Table 5.1.), we find the 5% LSD fo compaing the contol with Feed B to be: LSD t, df The othe equied LSD's ae: A vs. Contol = A vs. B = A vs. C = B vs. C = C vs. Contol = Using these values, we can constuct a mean sepaation table: Teatment Mean LSD Feed B 1.45 a Feed A 1.36 b Feed C 1.33 b Contol 1.0 c Thus, at the 5% level, we conclude all feeds cause significantly geate weight gain than the contol. Feed B causes the highest weight gain; Feeds A and C ae equally effective.

7 5.7 One advantage of the LSD pocedue is its ease of application. Additionally, it is easily used to constuct confidence intevals fo mean diffeences. The 1- confidence limits of the quantity (µ A - µ B ) ae given by= (1 α) CI fo (µ A - µ B ) = ( YA YB ) LSD Because fewe compaisons ae involved, the LSD test is much safe when the means to be compaed ae selected in advance of the expeiment; although hadly anyone eve does this. The test is pimaily intended fo use when thee is no pedetemined stuctue to the teatments. If a lage numbe of means ae to be compaed and the ones compaed ae selected afte the ANOVA and the compaisons taget those means with most diffeent values, the actual eo ate will be much highe than pedicted. The LSD test is the only test fo which the compaison-wise eo ate equals α. This is often egaded as too libeal (i.e. too eady to eject H 0 ). It has been suggested that the EEER can be maintained at by pefoming the oveall ANOVA test at the level and making futhe compaisons if and only if the F test is significant (Fishe's Potected LSD test). Howeve, it was then demonstated that this assetion is false if thee ae moe than thee means. In those cases, a peliminay F test contols only the EERC, not the EERP Dunnett's Method In cetain expeiments, one may desie only to compae a contol with each of the othe teatments, such as compaing a standad vaiety o chemical with seveal new ones. Dunnett's method pefoms such an analysis while holding the maximum expeimentwise eo ate unde any complete o patial null hypothesis (MEER) to a level not exceeding the stated. In this method, a t* value is calculated fo each compaison. This tabula t* value fo detemining statistical significance, howeve, is not the Student's t but a special t* given in Appendix Tables A-9a and A-9b (ST&D p 64-65). Let Y 0 epesent the contol mean with 0 eplications, then: * 1 1 DLSD t fo unequal ( 0 i ), df 1 and * DLSD t fo equal ( 0 = i ), df Fom the seed teatment expeiment in Table 4-1, = with 16 df and the numbe of compaisons (p)= 3.

8 5.8 * By Table A-19b, t. 59.,16 DLSD t *, df (Note that DLSD= 0.15 > LSD= 0.14) This povides the least significant diffeence between a contol and any othe teatment. Note that the smallest diffeence between the contol and any acid teatment is: Contol - HC1 = = 0.3. Since this diffeence is lage than DLSD, it is significant; and all othe diffeences, being lage, ae also significant. The 95% simultaneous confidence intevals fo all thee diffeences ae computed as: (1 α) CI fo (µ 0 - µ i ) = ( Y0 Yi ) DLSD The limits of these diffeences ae, Contol - Butyic = 0.3 ± 0.15 Contol - HC1 = 0.46 ± 0.15 Contol - Popionic = 0.55 ± 0.15 We have 95% confidence that the 3 anges will include simultaneously the tue diffeences. When teatments ae not equally eplicated, as in the feed ation expeiment, thee ae diffeent DLSD values fo each of the compaisons. To compae the contol with Feed- C, fist note that t *. 517 (fom SAS; by Table A-9b, t* is.54 o.51 fo 0 and 0.05, 4 df, espectively): DLSD t *, df Since Y0 YC = 0.15 is lage than , the diffeence is significant. All othe diffeences with the contol, being lage than this, ae also significant Tukey's w pocedue Tukey's test was designed specifically fo paiwise compaisons. This test, sometimes called the "honestly significant diffeence test" (HSD), contols the MEER when the sample sizes ae equal. Instead of t o t*, it uses the statistic q, p, df that is obtained fom Table A-8. The Tukey citical values ae lage than those of Dunnett because the

9 5.9 Tukey family of contasts is lage (all possible pais of means instead of just compaisons to a contol). The citical diffeence in this method is labeled w: 1 1 w q, p, df fo unequal 1 w q, p,df fo equal Aside fom the new citical value, things looks basically the same as befoe, except notice that hee we do not multiply by a facto of because Table A-8 aleady includes the facto in its values. Fo example, fo p =, df = (equivalent to the standad nomal distibution Z), and α= 5%, the citical value is.77, which is equal to 1.96 *. Consideing the seed teatment data (Table 4.1): q 0.05,( 4, 16) = 4.05; and: w q ,( p, df ) (Note that w = > DLSD = > LSD = 0.143) By this method, the means sepaation table looks like: Table 4.1 Teatment Mean w Contol 4.19 a HCl 3.87 b Popionic 3.73 b c Butyic 3.64 c Like the LSD and Dunnett's methods, this test detects significant diffeences between the contol and all othe teatments. But unlike with the LSD method, it detects no significant diffeences between the HCl and Popionic teatments (compae with Table 5.5). This eflects the lowe powe of this test. Fo unequal, as in the feeding expeiment in Table 5.3, the contast between the Contol with Feed-C would be tested using: q 0.05,(4, ) = 3.93 w = ( )/ = Since Y Y = 0.15 is lage than , it is significant. As in the LSD, the only Cont C paiwise compaison that is not significant is that between Feed C ( Y ) and Feed A ( Y ). A C

10 Scheffe's F test fo paiwise compaisons Scheffe's test is compatible with the oveall ANOVA F test in the sense that it neve declaes a contast significant if the oveall F test is nonsignificant. Scheffe's test contols the MEER fo ANY set of contasts. This includes all possible paiwise and goup compaisons. Since this pocedue contols MEER while allowing fo a lage numbe of compaisons, it is less sensitive (i.e. moe consevative) than othe multiple compaison pocedues. The Scheffe citical diffeence (SCD) has a simila stuctue as that descibed fo pevious tests, scaling the citical F value fo its statistic: SCD 1 1 df TtF, df Tt, df fo unequal 1 SCD dfttf, df Tt, df fo equal Fo the seed teatment data (Table 4-1), = with df TR = 3, df = 16, and = 5 SCD 0.05= 3 * = (Note that SCD = > w = > DLSD = > LSD = 0.143) Again, if the diffeence between a pai of means is geate than SCD, that diffeence will be declaed significant at the given level, while holding MEER below. The table of means sepaations: F s Teatment Mean Contol 4.19 a HCl 3.87 b Popionic 3.73 b c Butyic 3.64 c When the means to be compaed ae based on unequal eplications, a diffeent SCD is equied fo each compaison. Fo the animal feed expeiment, citical diffeence fo the contast between the Contol and Feed-C is: SCD 0.05, (3, ) = 3 * ( ) = Since YCont YC = 0.15 is lage than , it is significant. Scheffe's pocedue is also eadily used fo inteval estimation: (1 α) CI fo (µ 0 - µ i ) = ( Y0 Yi ) SCD

11 5.11 The esulting intevals ae simultaneous in that the pobability is at least (1 α) that all of them ae tue simultaneously Scheffe's F test fo goup compaisons The most impotant use of Scheffe's test is fo abitay compaisons among goups of means. We use the wod "abitay" hee because, unlike the goup compaisons using contasts, goup compaisons using Scheffe's test do not have to be othogonal, no ae they limited to (t 1) questions. If you ae inteested only in testing the diffeences between all pais of means, the Scheffe method is not the best choice; Tukey's is bette because it is moe sensitive while contolling MEER. But if you want to "mine" you data by making all possible compaisons (paiwise and goup compaisons) while still contolling MEER, Scheffe's is the way to go. To make compaisons among goups of means, you fist define a contast, as in Topic 4: Q= ci Yi. with the constaint that c i 0 (o i ci 0 fo unequal ) We will eject the null hypothesis (H 0 ) that the contast Q= 0 if the absolute value of Q is lage than a citical value F S. This is the geneal fom fo Scheffe's test: Citical value F S df Tt F, dftt, df t i1 c i i Note that the pevious expessions fo Scheffe paiwise compaisons ( ) ae fo the paticula contast 1 vs. -1. If we want to compae the contol to the aveage of thee acid teatments in Table 4.1, the contast coefficients ae In this case Q is calculated by multiplying the coefficients fo the means of the espective teatments. t Q c i Y i 4.190(3) 3.868( 1) 3.78( 1) 3.640( 1) i1 The citical value F S 0.05, (3, 16) value fo this contast is: F S df Tt c 3 3(3.4) ( 1) ( 1) 5 ( 1) t i, df, df Tt i1 i F Since Q = > = Fs, we eject H 0. The aveage of the contol (4.190 mg) is significantly diffeent fom the aveage of the thee acid teatments (3.745 mg). Again, with Scheffe's method, you can test any conceivable set of contasts, even if they numbe moe than (t 1) questions and ae not othogonal. The pice you pay fo this feedom, howeve, is vey low sensitivity. Scheffe's is the most consevative method of compaing means; so if Scheffe's declaes a diffeence to be eal, you can believe it. Remembe that in these contasts we ae using means no totals.

12 Multiple-stage tests Befoe we stat: Multiple ange tests should only be used with balanced designs since they ae inefficient with unbalanced ones. The methods discussed so fa ae all "fixed-ange" tests, so called because they use a single, fixed value to test hypotheses and build simultaneous confidence intevals. If one fofeits the ability to build simultaneous confidence intevals with a single value, it is possible to obtain simultaneous hypothesis tests of geate powe using multiple-stage tests (MSTs). MSTs come in both step-up (fist compaing closest means, then moe distant means) and step-down vaieties (the evese); but only step-down methods, which ae moe widely used, ae available in SAS. The best known MSTs ae the Duncan and the Student-Newman-Keuls (SNK) methods. Both use the studentized ange statistic (q) and, hence, also go by the name multiple ange tests. With means aanged fom the lowest to the highest, a multiple-ange test povides citical distances o anges that become smalle as the paiwise means to be compaed become close togethe in the aay. Such a stategy allows the eseache to allocate test sensitivity whee it is most needed, in disciminating neighboing means. The idea of step-down MSTs these tests is this: The moe means (i.e. teatments) ae compaed, the smalle the pobability that they ae all the same. The geneal stategy is: Fist, the maximum and minimum means ae compaed paiwise using the lagest citical value since the compaison involves all the means. If this H 0 is accepted, the pocedue stops. Othewise, the analysis continues by compaing paiwise the two sets of nextmost-exteme means (i.e. μ 1 vs. μ t-1, and μ vs. μ t ) using a smalle citical value, because goups ae now smalle. This pocess is epeated with close and close pais of means until one eaches the set of (t 1) pais of adjacent means, compaed paiwise using the smallest citical value. The lage the ange of the anks, the lage the tabled citical point. t t-1 t- t- μ 1 μ μ 3 μ 4 t- t-1 Gaphical depiction of the geneal stategy of step-down MSTs. In this figue, the means ae aanged highest (μ 1 ) to lowest (μ 5 ). The significance levels of each of the 10 possible paiwise compaisons ae indicated by the. μ Duncan's multiple ange tests (Table A-7) The test is identical to LSD fo adjacent means in an aay but equies pogessively lage values fo significance between means as they ae moe widely sepaated in the aay. Howeve fo goups of two means uses the same value as LSD. It contols the CER at the level but it has a high type I eo ate (MEER). Its opeating chaacteistics appea simila to those of Fishe's unpotected LSD at level Since the last test is easie to compute, easie to explain, and applicable to unequal sample sizes,

13 5.13 Duncan's method is not ecommended by SAS. The highe powe of Duncan's method compaed to Tukey is, in fact, due to its highe Type 1 eo ate (Einot and Gabiel 1975). Duncan's test used to be the most popula method but many jounals no longe accept it. To compute Duncan citical anges (R p ), use the following expession, plugging in the appopiate values of the Studentized ange statistic (q ): Rp q p1, p, df The pocedue is to compute a set of citical values by using ST&D Table A-7.: Fo the seed teatment data in Table 4.1: P 3 4 q 0.05 (p, 16) R p Note that the citical diffeence fo p= is the same as the LSD test! The Student-Newman-Keuls (SNK) test Student-Newman-Keuls test (SNK) is moe consevative than Duncan's in that the Ttype I eo ate is smalle. This is because SNK simply uses as the significance level at all stages of testing, again stopping the analysis at the highest level of non-significance. Because is lowe than Duncan's vaiable significance values, the powe of SNK is geneally lowe than that of Duncan's test. SNK is often accepted by jounals that do not accept Duncan's test. The SNK test contols the EERC at the level but it behaves pooly in tems of the EERP and MEER (Einot and Gabiel 1975). To see this conside ten population means that cluste in five pais such that means within pais ae equal but thee ae lage diffeences between pais: μ 1 μ μ 3 μ 4 μ 5 μ 6 μ 7 μ 8 μ 9 μ 10 In such case, all subset homogeneity hypotheses fo thee o moe means ae ejected. The SNK method then comes down to five independent tests, one fo each pai, each conducted at the level. The pobability of at least one false ejection is As the numbe of means inceases, the MEER appoaches 1. Theefoe, the SNK method is not ecommended by SAS since it does not contol well the maximum expeiment wise eo ate unde any patial null hypothesis (e.g. the one in the figue above). The pocedue is to compute a set of citical values by using ST&D Table A-8. Fist compae the maximum and minimum means. If the ange is not significant

14 5.14 W p = q,(p, df) Fo unequal use the same coection as in Tukey ( ). Fo Table 4.1 data: p 3 4 q 0.05 (p, 16) Note that fo p= t W p = Tukey w W p and fo p= W p = LSD Table 5.9 Teatment Mean W p Contol 4.19 a HCl 3.87 b Popionic 3.73 c Butyic 3.64 c The REGWQ method A vaiety of MSTs that contol MEER have been poposed, but these methods ae not as well known as those of Duncan and SNK. An appoach developed by Ryan, Einot and Gabiel, and Welsh (REGW) sets: p = 1- (1-) p/t fo p < t-1 and p = fo p t-1. The REGWQ method pefoms the compaisons using a ange test. This method appeas to be among the most poweful step-down multiple ange tests and is ecommended by SAS fo equal eplication (i.e. balanced design). Assuming the sample means have been aanged in descending ode fomy 1 to Y k, the homogeneity of means Y i.,..., Y j, with i < j, is ejected by REGWQ if: Y i - Y j q( p ; p, df ) (Use Table A.8 ST&D)

15 5.15 Fo Table 5.1 data: p 3 4 p q p (p, 16) Citical value Fo p= t and p= t-1 the citical value is as in SNK, but is lage fo p < t-1. Note that the diffeence between HCl and popionic is significant with SNK but no significant with REGWQ ( < 0.145). Table 5.10 Teatment Mean F s Contol 4.19 a HCl 3.87 b Popionic 3.73 b c Butyic 3.64 c Conclusions and ecommendations Thee ae at least twenty othe paametic pocedues available fo multiple compaisons, not to mention the many non-paametic and multivaiate methods. Thee is no consensus as to which is the most appopiate pocedue to ecommend to all uses. One main difficulty in compaing the pocedues is the diffeent kinds of Type I eo ates used, namely, expeiment-wise vesus compaison-wise. All this is to say that the diffeence in pefomance of any two pocedues is likely due to the diffeent undelying philosophies of Type I eo contol than to the specific techniques used. To a lage extent, the choice of a pocedue is subjective and hinges on a choice between a compaison-wise eo ate (such as LSD) and an expeiment-wise eo ate (such as Tukey and Scheffe's test). Some suggested ules of thumb: 1. When in doubt, use Tukey. Tukey's method is a good geneal technique fo caying out all paiwise compaisons, enabling you to ank means and put them into significance goups, while contolling MEER.. Use Dunnett's (moe poweful than Tukey's) if you only wish to compae each teatment level to a contol. 3. Use Scheffe's if you wish to test a set of non-othogonal goup compaisons OR if you wish to cay out goup compaisons in addition to all possible paiwise compaisons. MEER will be contolled in both cases.

16 5.16 The SAS manual makes the following additional ecommendation: Fo contolling MEER fo all paiwise compaisons, use REGWQ fo balanced designs and Tukey fo unbalanced designs. One final point to note is that seveely unbalanced designs can yield vey stange esults, egadless of means sepaation method. To illustate this, conside the example on page 00 of ST&D. In this example, an expeiment with fou teatments (A, B, C, and D) have esponses in the ode A > B > C > D. A and D each have eplications, while B and C each have 11. The stange esult: The exteme means (A and D) ae found to be not significantly diffeent, but the intemediate means (B and C) ae.