Multiple Contrasts (Simulation)

Transcription

1 Chapter 590 Multple Contrasts (Smulaton) Introducton Ths procedure uses smulaton to analyze the power and sgnfcance level of two multple-comparson procedures that perform two-sded hypothess tests of contrasts of the group means. These are the Dunn-Bonferron test and the Dunn-Welch test. For each scenaro, two smulatons are run: one estmates the sgnfcance level and the other estmates the power. The term contrast refers to a user-defned comparson of the group means. The term multple contrasts refers to a set of such comparsons. An addtonal restrcton mposed s that the contrast coeffcents to sum to zero. When several contrasts are tested, the nterpretaton of the results s more complex because of the problem of multplcty. Multplcty here refers to the fact that the probablty of makng at least one ncorrect decson ncreases as the number of statstcal tests ncreases. Methods for testng multple contrasts have been developed to account for ths multplcty. Error Rates When dealng wth several smultaneous statstcal tests, both ndvdual-wse and experment wse error rates should be consdered.. Comparson-wse error rate. Ths s the probablty of a type-i error (rejectng a true H0) for a partcular test. In the case of the fve-group desgn, there are ten possble comparson-wse error rates, one for each of the ten possble pars. We wll denote ths error rate α c. 2. Experment-wse (or famly-wse) error rate. Ths s the probablty of makng one or more type-i errors n the set (famly) of comparsons. We wll denote ths error rate α f. The relatonshp between these two error rates when the tests are ndependent s gven by α f = ( α ) where C s the total number of contrasts. For example, f α c s 0.05 and C s 0, α f s There s about a 40% chance that at least one of the ten contrasts wll be concluded to be non-zero when n fact they are not. When the tests are correlated, as they mght be among a set of contrasts, the above formula provdes an upper bound to the famly-wse error rate. The technques descrbed below provde control for α f rather than α c. c C 590-

2 Techncal Detals The One-Way Analyss of Varance Desgn The dscusson that follows s based on the common one-way analyss of varance desgn whch may be summarzed as follows. Suppose the responses Y j n k groups each follow a normal dstrbuton wth means, 2,, k and unknown varance σ 2. Let n, n 2,, n k denote the number of subjects n each group. The control group s assumed to be group one. The analyss of these responses s based on the sample means and the pooled sample varance σ 2 n = Y = j= = Y n j k n ( Yj Y ) = j= k ( n ) = The F test s the usual method of analyss of the data from such a desgn, testng whether all of the means are equal. However, a sgnfcant F test does not ndcate whch of the groups are dfferent, only that at least one s dfferent. The analyst s left wth the problem of determnng whch of the groups are dfferent and by how much. To determne the mean dfferences that are most mportance, the researcher may specfy a set of contrasts. These contrasts, called a pror, or, planned, contrasts should be specfed before the expermental results are vewed. The Dunn-Bonferron procedure and the Dunn-Welch procedure have been developed to test these planned contrasts. The calculatons assocated wth each of these tests are gven below. 2 Contrasts A contrast of the means s a stated dfference among the means. The dfference s constructed so that t represents a useful hypothess. For example, suppose there are four groups, the frst of whch s a control group. It mght be of nterest to determne whch treatments are statstcally dfferent from the control. That s, the dfferences 2, 3, and 4 would be tested to determne f they are non-zero. Contrasts are often smple dfferences between two means. However, they may nvolve more than just two means. For example, suppose the frst two groups receve one treatment and the second two groups receve another + +. treatment. The contrast (dfference) that would be tested s ( ) ( ) Every contrast can be represented by the lst of contrast coeffcents: the values by whch the means are multpled. Here are some examples of contrasts that mght be of nterest when the experment nvolves four groups

3 Dfference Coeffcents 2 -,, 0, 0 3 -, 0,, 0 + +,, -, - ( 2) ( 3 4) ( ) ( 2 3 4) ( ) ( ) , -, -, , -, -, Note that n each case, the coeffcents sum to zero. Ths makes t possble to test whether the quantty s dfferent from zero. A lot s wrtten about orthogonal contrasts whch have the property that the sum of the products of correspondng coeffcents s zero. For example, the sum of the products of the last two contrasts gven above s 0(3) + (-)(-) + (-)(-) + (2)(-) = = 0, so these two contrasts are orthogonal. However, the frst two contrasts are not orthogonal snce (-)(-) + ()(0) + (0)() + (0)(0) = = (not zero). Orthogonal contrasts have nce propertes when the sample szes are equal. Unfortunately, they lose those propertes when the group sample szes are unequal or when the data are not normally dstrbuted. The procedures descrbed n ths chapter do not requre that the contrasts be orthogonal. Instead, you should focus on defnng a set of contrasts that answer the research questons of nterest. Dunn-Bonferron Test Dunn (964) developed a procedure for smultaneously testng several contrasts. Ths method s also dscussed n Krk (982) pages 06 to 09. The method conssts of testng each contrast wth Student s t dstrbuton wth degrees of freedom equal to N-k wth a Bonferron adjustment of the alpha value. That s, the alpha value s dvded by C, the number of contrasts smultaneously tested. The test rejects H0 f k = 2 σ c Y k = 2 c n t ( ) α / 2C, N k Note that ths s a two-sded test of the hypothess that c = 0 where c = 0. k = k = Dunn-Welch Test Dunn (964) developed a procedure for smultaneously testng several contrasts. Ths method, usng Welch s (947) modfcaton for the unequal varances, s dscussed n Krk (982) pages 00, 0, The method conssts of testng each contrast wth Student s t dstrbuton wth degrees of freedom gven below wth a Bonferron adjustment of the alpha value. That s, the alpha value s dvded by C, the number of contrasts smultaneously tested. The two-sded test statstc rejects H0 f k = k = c Y 2 2 c σ n t ( ) α / 2C, v' 590-3

4 where v' = k = 2 2 c σ n 4 4 c σ 2 n k 2 ( n ) = Defnton of Power for Multple Contrasts The noton of power s well-defned for ndvdual tests. Power s the probablty of rejectng a false null hypothess. However, ths defnton does not extend drectly when there are a number of smultaneous tests. The two defntons that we emphasze n PASS where recommended by Ramsey (978). They are any-contrast power and all-contrasts power. Other desgn characterstcs, such as average-contrast power and false-dscovery rate, are mportant to consder. However, our revew of the statstcal lterature resulted n our focus on these two defntons of power. Any-Contrast Power Any-contrast power s the probablty of detectng at least one of the contrasts that are actually non-zero. All-Contrasts Power All-contrast power s the probablty of detectng all of the contrasts that are actually non-zero. Smulaton Detals Computer smulaton allows us to estmate the power and sgnfcance level that s actually acheved by a test procedure n stuatons that are not mathematcally tractable. Computer smulaton was once lmted to manframe computers. But, n recent years, as computer speeds have ncreased, smulaton studes can be completed on desktop and laptop computers n a reasonable perod of tme. The steps to a smulaton study are. Specfy how each test s to be carred out. Ths ncludes ndcatng how the test statstc s calculated and how the sgnfcance level s specfed. 2. Generate random samples from the dstrbutons specfed by the alternatve hypothess. Calculate the test statstcs from the smulated data and determne f the null hypothess s accepted or rejected. The number rejected s used to calculate the power of each test. 3. Generate random samples from the dstrbutons specfed by the null hypothess. Calculate each test statstc from the smulated data and determne f the null hypothess s accepted or rejected. The number rejected s used to calculate the sgnfcance level of each test. 4. Repeat steps 2 and 3 several thousand tmes, tabulatng the number of tmes the smulated data leads to a rejecton of the null hypothess. The power s the proporton of smulated samples n step 2 that lead to rejecton. The sgnfcance level s the proporton of smulated samples n step 3 that lead to rejecton. Generatng Random Dstrbutons Two methods are avalable n PASS to smulate random samples. The frst method generates the random varates drectly, one value at a tme. The second method generates a large pool (over 0,000) of random values and then draws the random numbers from ths pool. Ths second method can cut the runnng tme of the smulaton by 70%! 590-4

5 As mentoned above, the second method begns by generatng a large pool of random numbers from the specfed dstrbutons. Each of these pools s evaluated to determne f ts mean s wthn a small relatve tolerance (0.000) of the target mean. If the actual mean s not wthn the tolerance of the target mean, ndvdual members of the populaton are replaced wth new random numbers f the new random number moves the mean towards ts target. Only a few hundred such swaps are requred to brng the actual mean to wthn tolerance of the target mean. Ths populaton s then sampled wth replacement usng the unform dstrbuton. We have found that ths method works well as long as the sze of the pool s the maxmum of twce the number of smulated samples desred and 0,000. Procedure Optons Ths secton descrbes the optons that are specfc to ths procedure. These are located on the Desgn, Contrasts, and Optons tabs. For more nformaton about the optons of other tabs, go to the Procedure Wndow chapter. Desgn Tab The Desgn tab contans most of the parameters and optons that you wll be concerned wth. Solve For Solve For Ths opton specfes the parameter to be solved for: power or sample sze (n). If you choose to solve for n, you must choose the type of power you want to solve for: any-contrast power or all-contrasts power. The value of the opton Power wll then represent ths type of power. Any-contrast power s the probablty of detectng at least one of the non-zero contrasts. All-contrast power s the probablty of detectng all non-zero contrasts. Note that the search for n may take several mnutes because a separate smulaton must be run for each tral value of n. You may fnd t qucker and more nformatve to solve for the Power for a range of sample szes. Test MC Procedure Specfy whch multple contrast procedure s to be reported from the smulatons. The choces are Dunn-Bonferron Test Ths s the most popular and most often recommended. Dunn-Welch Test Ths s recommended when the group varances are very dfferent. Smulatons Smulatons Ths opton specfes the number of teratons, M, used n the smulaton. As the number of teratons s ncreased, the runnng tme and accuracy are ncreased as well. The precson of the smulated power estmates are calculated usng the bnomal dstrbuton. Thus, confdence ntervals may be constructed for varous power values. The followng table gves an estmate of the precson that s acheved for varous smulaton szes when the power s ether 0.50 or The table values are nterpreted as 590-5

6 follows: a 95% confdence nterval of the true power s gven by the power reported by the smulaton plus and mnus the Precson amount gven n the table. Smulaton Precson Precson Sze when when M Power = 0.50 Power = Notce that a smulaton sze of 000 gves a precson of plus or mnus 0.0 when the true power s Also note that as the smulaton sze s ncreased beyond 5000, there s only a small amount of addtonal accuracy acheved. Power and Alpha Power Ths opton s only used when Solve For s set to Sample Sze (All-Contrast) or Sample Sze (Any-Contrast). Power s defned dfferently wth multple contrasts. Although many defntons are possble, two are adopted here. Any-contrast power s the probablty of detectng at least one non-zero contrast. All-contrasts power s the probablty of detectng all non-zero contrasts. As the number of contrasts s ncreased, these power probabltes wll decrease because more tests are beng conducted. Snce ths s a probablty, the range s between 0 and. Most researchers would lke to have the power at least at 0.8. However, ths may requre extremely large sample szes when the number of tests s large. FWER (Alpha) Ths opton specfes one or more values of the famly-wse error rate (FWER) whch s the analog of alpha for multple contrasts. FWER s the probablty of falsely detectng (concludng that the means are dfferent) at least one comparson for whch the true means are the same. For ndependent tests, the relatonshp between the ndvdual-comparson error rate (ICER) and FWER s gven by the formulas or FWER = - ( - ICER)^C ICER = - ( - FWER)^(/C) where '^' represents exponentaton (as n 4^2 = 6) and C represents the number of comparsons. For example, f C = 5 and FWER = 0.05, then ICER = Thus, the ndvdual comparson tests must be conducted usng a Type- error rate of 0.002, whch s much lower than the famly-wse rate of The popular value for FWER remans at However, f you have a large number of comparsons, you mght decde that a larger value, such as 0.0, s approprate. Sample Sze n (Sample Sze Multpler) Ths s the base, per group, sample sze. One or more values separated by blanks or commas may be entered. A separate analyss s performed for each value lsted here

7 The group samples szes are determned by multplyng ths number by each of the Group Sample Sze Pattern numbers. If the Group Sample Sze Pattern numbers are represented by m, m2, m3,, mk and ths value s represented by n, the group sample szes N, N2, N3,..., Nk are calculated as follows: N=[n(m)] N2=[n(m2)] N3=[n(m3)] etc. where the operator, [X] means the next nteger after X, e.g. [3.]=4. For example, suppose there are three groups and the Group Sample Sze Pattern s set to,2,3. If n s 5, the resultng sample szes wll be 5, 0, and 5. If n s 50, the resultng group sample szes wll be 50, 00, and 50. If n s set to 2,4,6,8,0, fve sets of group sample szes wll be generated and an analyss run for each. These sets are: As a second example, suppose there are three groups and the Group Sample Sze Pattern s 0.2,0.3,0.5. When the fractonal Pattern values sum to one, n can be nterpreted as the total sample sze of all groups and the Pattern values as the proporton of the total n each group. If n s 0, the three group sample szes would be 2, 3, and 5. If n s 20, the three group sample szes would be 4, 6, and 0. If n s 2, the three group sample szes would be (0.2)2 = 2.4 whch s rounded up to the next whole nteger, 3. (0.3)2 = 3.6 whch s rounded up to the next whole nteger, 4. (0.5)2 = 6. Note that n ths case, does not equal n (whch s 2). Ths can happen because of roundng. Group Sample Sze Pattern The purpose of the group sample sze pattern s to allow several groups wth the same sample sze to be generated wthout havng to type each ndvdually. A set of postve, numerc values (one for each row of dstrbutons) s entered here. Each tem specfed n ths lst apples to the whole row of dstrbutons. For example, suppose the entry s 2 and Grps = 3, Grps 2 =, Grps 3 = 2. The sample sze pattern used would be 2. The sample sze of group s found by multplyng the th number from ths lst by the value of n and roundng up to the next whole number. The number of values must match the number of groups, g. When too few numbers are entered, s are added. When too many numbers are entered, the extras are gnored. Equal If all sample szes are to be equal, enter Equal here and the desred sample sze n n. A set of g 's wll be used. Ths wll result n n = n2 = = ng = n. That s, all sample szes are equal to n

8 Effect Sze These optons specfy the dstrbutons to be used n the two smulatons. The frst opton specfes the number of groups represented by the two dstrbutons that follow. The second opton specfes the dstrbuton to be used n smulatng the null hypothess to determne the sgnfcance level (alpha). The thrd opton specfes the dstrbuton to be used n smulatng the alternatve hypothess to determne the power. Grps [A C] (Grps D I are found on the Data 2 tab) Ths value specfes the number of groups specfed by the H0 and H dstrbuton statements to the rght. Usually, you wll enter to specfy a sngle H0 and a sngle H dstrbuton, or you wll enter 0 to ndcate that the dstrbutons specfed on ths lne are to be gnored. Ths opton lets you easly specfy many dentcal dstrbutons wth a sngle phrase. The total number of groups g s equal to the sum of the values for the three rows of dstrbutons shown under the Data tab and the sx rows of dstrbutons shown under the Data 2 tab. Note that each tem specfed n the Group Sample Sze Pattern opton apples to the whole row of entres here. For example, suppose the Group Sample Sze Pattern was 2 and Grps = 3, Grps 2 =, and Grps 3 = 2. The sample sze pattern would be 2. Group Dstrbuton(s) H0 Ths entry specfes the dstrbuton of one or more groups under the null hypothess, H0. The magntude of the dfferences of the means of these dstrbutons, whch s often summarzed as the standard devaton of the means, represents the magntude of the mean dfferences specfed under H0. Usually, the means are assumed to be equal under H0, so ther standard devaton should be zero except for roundng. These dstrbutons are used n the smulatons that estmate the actual sgnfcance level. They also specfy the value of the mean under the null hypothess, H0. Usually, these dstrbutons wll be dentcal. The parameters of each dstrbuton are specfed usng numbers or letters. If letters are used, ther values are specfed n the boxes below. The value M0 s reserved for the value of the mean under the null hypothess. Followng s a lst of the dstrbutons that are avalable and the syntax used to specfy them. Each of the parameters should be replaced wth a number or parameter name. Dstrbutons wth Common Parameters Beta(Shape, Shape2, Mn, Max) Bnomal(P, N) Cauchy(Mean, Scale) Constant(Value) Exponental(Mean) Gamma(Shape, Scale) Gumbel(Locaton, Scale) Laplace(Locaton, Scale) Logstc(Locaton, Scale) Lognormal(Mu, Sgma) Multnomal(P, P2, P3,..., Pk) Normal(Mean, Sgma) Posson(Mean) TukeyGH(Mu, S, G, H) 590-8

9 Unform(Mn, Max) Webull(Shape, Scale) Dstrbutons wth Mean and SD Parameters BetaMS(Mean, SD, Mn, Max) BnomalMS(Mean, N) GammaMS(Mean, SD) GumbelMS(Mean, SD) LaplaceMS(Mean, SD) LogstcMS(Mean, SD) LognormalMS(Mean, SD) UnformMS(Mean, SD) WebullMS(Mean, SD) Detals of wrtng mxture dstrbutons, combned dstrbutons, and compound dstrbutons are found n the chapter on Data Smulaton and wll not be repeated here. Fndng the Value of the Mean of a Specfed Dstrbuton Most of the dstrbutons have been parameterzed n terms of ther means snce ths s the parameter beng tested. The mean of a dstrbuton created as a lnear combnaton of other dstrbutons s found by applyng the lnear combnaton to the ndvdual means. However, the mean of a dstrbuton created by multplyng or dvdng other dstrbutons s not necessarly equal to applyng the same functon to the ndvdual means. For example, the mean of 4 Normal(4, 5) + 2 Normal (5, 6) s 4*4 + 2*5 = 26, but the mean of 4 Normal (4, 5) * 2 Normal (5, 6) s not exactly 4*4*2*5 = 60 (although t s close). Group Dstrbuton(s) H Specfy the dstrbuton of ths group under the alternatve hypothess, H. Ths dstrbuton s used n the smulaton that determnes the power. A fundamental quantty n a power analyss s the amount of varaton among the group means. In fact, classcal power analyss formulas, ths varaton s summarzed as the standard devaton of the means. The mportant pont to realze s that you must pay partcular attenton to the values you gve to the means of these dstrbutons because they are fundamental to the nterpretaton of the smulaton. For convenence n specfyng a range of values, the parameters of the dstrbuton can be specfed usng numbers or letters. If letters are used, ther values are specfed n the boxes below. The value M s reserved for the value of the mean under the alternatve hypothess. A lst of the dstrbutons that are avalable and the syntax used to specfy them s gven above. Equvalence Margn Specfy the largest dfference for whch means from dfferent groups wll be consdered equal. When specfyng group dstrbutons, t s possble to end up wth scenaros where some means are slghtly dfferent from each other, even though they are ntended to be equvalent. Ths often happens when specfyng dstrbutons of dfferent forms (e.g. normal and gamma) for dfferent groups, where the means are ntended to be the same. The parameters used to specfy dfferent dstrbutons do not always result n means that are EXACTLY equal. Ths value lets you control how dfferent means can be and stll be consdered equal

10 Ths value s not used to specfy the hypotheszed mean dfferences of nterest. The hypotheszed dfferences are specfed usng the means (or parameters used to calculate means) for the null and alternatve dstrbutons. Ths value should be much smaller than the hypotheszed mean dfferences. Effect Sze Dstrbuton Parameters M0 (Mean H0) These values are substtuted for M0 n the dstrbuton specfcatons gven above. M0 s ntended to be the value of the mean hypotheszed by the null hypothess, H0. You can enter a lst of values usng the syntax or 0 to 3 by. M (Mean H) These values are substtuted for M n the dstrbuton specfcatons gven above. Although t can be used wherever you want, M s ntended to be the value of the mean hypotheszed by the alternatve hypothess, H. You can enter a lst of values usng the syntax or 0 to 3 by. Parameter Values (S, A, B, C) Enter the numerc value(s) of the parameters lsted above. These values are substtuted for the correspondng letter n all four dstrbuton specfcatons. You can enter a lst of values for each letter usng the syntax or 0 to 3 by. You can also change the letter that s used as the name of ths parameter usng the pull-down menu to the sde. Contrasts Tab Contrasts Contrasts These optons specfy the contrasts. You can specfy as many contrasts as are necessary, but a penalty s pad n terms of reduced power for each addtonal contrast. Thus, the number of contrasts should be lmted to those that are most mportant to the study. A contrast s a weghted average of the k (k = number of groups) group means n whch the weghts (coeffcents) sum to zero. Each successve coeffcent s appled to the correspondng group mean. For example, suppose k = 3 and the frst group s a control group. Two contrasts that mght be of nterest are - 0 and - 0. These are nterpreted as (-)Mean + ()Mean2 + (0)Mean3 and (-)Mean + (0)Mean2 + ()Mean3, respectvely. Notce that the coeffcents n each set sum to zero. Several predefned sets of contrasts are avalable or you can specfy your own. There s no set number of contrasts that must (or may) be specfed, but fewer contrasts result n hgher power and smaller requred samples szes. Possble entres are gven next. Indvdual Contrasts Enter a set of numbers, separated by blanks. One coeffcent must be entered for each group wth one set per box. Examples of vald contrasts are

11 Each Wth Frst Ths opton generates k- contrasts approprate for comparng each of the remanng groups wth the frst group. Ths mght be used when the frst group s a control group. If k = 4, the 3 contrasts are Each Wth Last Ths opton generates k- contrasts approprate for comparng each of the frst k- groups wth the last group. Ths mght be used when the last group s a control group. If k = 4, the 3 contrasts are Each Wth Next Ths opton generates k- contrasts approprate for comparng each group wth the next group. If k = 4, the 3 contrasts are Each Wth Remanng Each group mean s compared wth the average of those remanng to the rght. Suppose k=4, the 3 contrasts are Each Wth All Others Each group mean s compared wth the average of the other groups. Suppose k=4, the 4 contrasts are Progressve Splt The frst groups are compared to the last groups. The dvdng pont moves from left to rght. Suppose k=5, the 4 contrasts are

12 Optons Tab The Optons tab contans a random number pool sze opton. Random Numbers Random Number Pool Sze Ths s the sze of the pool of values from whch the random samples wll be drawn. Pools should be at least the maxmum of 0,000 and twce the number of smulatons. You can enter Automatc and an approprate value wll be calculated. If you do not want to draw numbers from a pool, enter 0 here. Example Power at Varous Sample Szes A study s beng planned to fnd the threshold level of a certan drug. Below ths threshold level, the response has lttle change. Once the threshold level s reached, there s a szeable jump n the mean response rate. Lttle change n the response occurs as the drug level s ncreased above the threshold. Scentsts beleve that the threshold level s between 3 and 7 ther best estmate, based on prevous studes, s 5. Prevous studes have shown that the standard devaton wthn a group s 3.0. In order to fnd the threshold, they desgn a study wth fve levels: 3.0, 4.0, 5.0, 6.0, and 7.0. Snce there s no trend n the mean value (only a sudden shft) as the dose level s ncreased, they decde to test the followng hypotheses: Dfference Coeffcents 2 -,, 0, 0, , -,, 0, , 0, -,, , 0, 0, -, Notce that ths set of hypotheses answers the queston drectly. An overall F-test would test the hypothess that at least one mean s dfferent, but t would not ndcate whch s dfferent. The queston mght be settled by consderng all possble pars, but there are ten pars, so ten hypothess tests would have to be consdered nstead of only four decreasng the power. Researchers want to detect a shft n the mean as small as 2.0. Hence, they want to study the power when the means are 0.0, 0.0, 2.0, 2.0, 2.0. They want to nvestgate sample szes of 0, 30, 50, and 70 subjects per group. They have no reason to assume that the varance wll change a great deal from group to group, so they decde to analyze the data usng the Dunn-Bonferron procedure. They set the FWER to Note that, based on these means, only the second of the four contrasts wll be sgnfcant, so the any-contrast power wll be the same as the all-contrast power

13 Setup Ths secton presents the values of each of the parameters needed to run ths example. Frst, from the PASS Home wndow, load the procedure wndow by expandng Means, then clckng on Multple Comparsons, and then clckng on. You may then make the approprate entres as lsted below, or open Example by gong to the Fle menu and choosng Open Example Template. Opton Value Desgn Tab Solve For... Power MC Procedure... Dunn-Bonferron Test Smulatons FWER (Alpha) n (Sample Sze Multpler) Group Sample Sze Pattern... Equal Grps... 2 Control Dstrbuton H0... Normal(M0 S) Control Dstrbuton H... Normal(M0 S) Grps Group 2 Dstrbuton(s) H0... Normal(M0 S) Group 2 Dstrbuton(s) H... Normal(M S) Equvalence Margn M0 (Mean H0)... 0 M (Mean H)... 2 S... 3 Contrasts Tab Contrasts... Each Wth Next Reports Tab All reports except Comparatve... Checked Annotated Output Clck the Calculate button to perform the calculatons and generate the followng output. Smulaton Summary Report Summary of Smulatons for Testng Multple Contrasts of 5 Groups MC Procedure: Dunn-Bonferron Test Group Total Any- Smpl. Smpl. All- S.D. of S.D. of Sm. Cont. Sze Sze Cont. Means Data Actual Target No. Power n N Power Sm H SD H FWER FWER M0 M S (0.05) [ ] (0.05) [ ] (0.009) [ ] (0.022) [ ] (0.022) [ ] (0.008) [ ] (0.08) [ ] (0.08) [ ] (0.009) [ ] (0.02) [ ] (0.02) [ ] (0.009) [ ] Pool Sze: Smulatons: Run Tme: seconds

14 Summary of Smulatons Report Defntons H0: the null hypothess that the contrast of the means s zero. H: the alternatve hypothess that the contrast of the means s not zero. Cont.: abbrevates 'Contrast'. Refers to a weghted average of the means whose weghts sum to zero. All-Cont. Power: the estmated probablty of detectng all unequal contrasts. Any-Cont. Power: the estmated probablty of detectng at least one unequal contrasts. n: the average of the group sample szes. N: the combned sample sze of all groups. Famly-Wse Error Rate (FWER): the probablty of detectng at least one zero contrast assumng H0. Target FWER: the user-specfed FWE. Actual FWER: the FWER estmated by the alpha smulaton. Sm H: the standard devaton of the group means under H. SD H: the pooled, wthn-group standard devaton under H. Second Row: provdes the precson and a confdence nterval based on the sze of the smulaton for Any-Contrast Power, All-Contrasts Power, and FWER. The format s (Precson) [95% LCL and UCL Alpha]. Summary Statements A one-way desgn wth 5 groups has an average group sample sze of 0.0 for a total sample sze of 50. Ths desgn acheved an any-contrast power of 0.36 and an all-contrast power of 0.36 usng the Dunn-Bonferron Test procedure for comparng each contrast of the group means wth zero. The target famly-wse error rate was and the actual famly-wse error rate was The average wthn group standard devaton assumng the alternatve dstrbuton s 3.0. These results are based on 2000 Monte Carlo samples from the null dstrbutons: N(M0 S); N(M0 S); N(M0 S); N(M0 S); and N(M0 S) and the alternatve dstrbutons: N(M0 S); N(M0 S); N(M S); N(M S); and N(M S). Other parameters used n the smulaton were: M0 = 0.0, M = 2.0, and S = 3.0. Ths report shows that a group sample sze of about 50 wll be needed to acheve 80% power or about 70 for 90% power. Any-Cont. Power Ths s the probablty of detectng any of the sgnfcant contrasts. Ths value s estmated by the smulaton usng the H dstrbutons. Note that a precson value (half the wdth of ts confdence nterval) and a confdence nterval are shown on the lne below ths row. These values provde the precson of the estmated power. All- Cont. Power Ths s the probablty of detectng all of the sgnfcant contrasts. Ths value s estmated by the smulaton usng the H dstrbutons. Note that a precson value (half the wdth of ts confdence nterval) and a confdence nterval are shown on the lne below ths row. These values provde the precson of the estmated power. Group Sample Sze n Ths s the average of the ndvdual group sample szes. Total Sample Sze N Ths s the total sample sze of the study. S.D. of Means Sm H Ths s the standard devaton of the hypotheszed means of the alternatve dstrbutons. Under the null hypothess ths value s zero. It represents the magntude of the dfference among the means. It s roughly equal to the average dfference between the group means and the overall mean. Note that the effect sze s the rato of Sm H and SD H. S.D. of Data SD H Ths s the wthn-group standard devaton calculated from samples from the alternatve dstrbutons

15 Actual FWER Ths s the value of FWER (famly-wse error rate) estmated by the smulaton usng the H0 dstrbutons. It should be compared wth the Target FWER to determne f the test procedure s accurate. Note that a precson value (half the wdth of ts confdence nterval) and a confdence nterval are shown on the lne below ths row. These values provde the precson of the Actual FWER. Target FWER Ths s the target value of FWER that was set by the user. M0 Ths s the value entered for M0, the group means under H0. M Ths s the value entered for M, the group means under H. S Ths s the value entered for S, the standard devaton. Error-Rate Summary for H0 Smulaton Error Rate Summary from H0 (Alpha) Smulaton of 5 Groups MC Procedure: Dunn-Bonferron Test Prop. (No. of Mean Type- No. of No. of Prop. Errors Mean Mn Max Sm. Zero Type- Type- > 0) Target Cont. Cont. Cont. No. Cont. Errors Errors FWER FWER Alpha Alpha Alpha Ths report shows the results of the H0 smulaton. Ths smulaton uses the H0 settngs for each group. Its man purpose s to provde an estmate of the FWER. No. of Zero Cont. Snce under H0 all means are equal, ths s the number of contrasts. Mean No. of Type- Errors Ths s the average number of type- errors (false detectons) per set (famly). Prop. Type- Errors Ths s the proporton of type- errors (false detectons) among all tests that were conducted. Prop. (No. of Type- Errors>0) FWER Ths s the proporton of the H0 smulatons n whch at least one type- error occurred. Ths s called the famlywse error rate. Target FWER Ths s the target value of FWER that was set by the user. Mean Cont. Alpha Alpha s the probablty of rejectng H0 when H0 s true. It s a characterstc of an ndvdual test. Ths s the average ndvdual alpha value over all of the contrasts

16 Mn Cont. Alpha Ths s the mnmum of all contrast alphas. Max Cont. Alpha Ths s the maxmum of all contrast alphas. Error-Rate Summary for H Smulaton Error-Rate Summary from H (Power) Smulaton of 5 Groups MC Procedure: Dunn-Bonferron Test (FDR) Prop. Prop. Prop. Prop. No. of Mean Mean Zero Non-0. Detect. Undet. All Any Zero/ No. of No. of that that that that Non-0 Non-0 Mean Mn Max Sm. Non-0 False False Prop. were were were were Cont. Cont. Cont. Cont. Cont. No. Cont. Pos. Neg. Errors Detect. Undet. Zero Non-0 Power Power Power Power Power 3/ / / / Ths report shows the results of the H smulaton. Ths smulaton uses the H settngs for each group. Its man purpose s to provde an estmate of the power. No. of Zero/Non-0 Cont. The frst value s the number of contrasts that were zero under H. The second value s the number of contrasts that were non-zero under H. Mean No. False Postves Ths s the average number of zero contrasts that were declared as beng non-zero by the testng procedure. A false postve s a type- (alpha) error. Mean No. False Negatves Ths s the average number of non-zero contrasts that were not declared as beng non-zero by the testng procedure. A false negatve s a type-2 (beta) error. Prop. Errors Ths s the proporton of type- and type-2 errors. Prop. Equal that were Detect. Ths s the proporton of the zero contrasts n the H smulatons that were declared as non-zero. Prop. Uneq. that were Undet. Ths s the proporton of non-zero contrasts n the H smulatons that were not declared as beng non-zero. Prop. Detect. that were Zero (FDR) Ths s the proporton of all detected contrasts n the H smulatons that were actually zero. Ths s often called the false dscovery rate. Prop. Undet. that were Non-0. Ths s the proporton of undetected contrasts n the H smulatons that were actually non-zero. All Non-0 Cont. Power Ths s the probablty of detectng all non-zero contrasts n the H smulaton. Any Non-0 Cont. Power Ths s the probablty of detectng any non-zero contrasts n the H smulaton

17 Mean, Mn, and Max Cont. Power These tems gve the average, the mnmum, and the maxmum of the contrast powers from the H smulaton. Detal Model Report Detaled Model Report for Smulaton No. Target FWER = 0.050, M0 = 0.0, M = 2.0, S = 3.0 MC Procedure: Dunn-Bonferron Test Hypo. Group Group Ave. Smulaton Type Groups Labels n/n Mean S.D. Model H0-2 A-A2 0/ N(M0 S) H0 3-5 B-B3 0/ N(M0 S) H0 All Sm= H -2 A-A2 0/ N(M0 S) H 3-5 B-B3 0/ N(M S) H All Sm= Detaled Model Report for Smulaton No. 2 Hypo. Group Group Ave. Smulaton Type Groups Labels n/n Mean S.D. Model H0-2 A-A2 30/ N(M0 S) H0 3-5 B-B3 30/ N(M0 S) H0 All Sm= H -2 A-A2 30/ N(M0 S) H 3-5 B-B3 30/ N(M S) H All Sm= Detaled Model Report for Smulaton No. 3 Hypo. Group Group Ave. Smulaton Type Groups Labels n/n Mean S.D. Model H0-2 A-A2 30/ N(M0 S) H0 3-5 B-B3 30/ N(M0 S) H0 All Sm= H -2 A-A2 30/ N(M0 S) H 3-5 B-B3 30/ N(M S) H All Sm= Detaled Model Report for Smulaton No. 4 Hypo. Group Group Ave. Smulaton Type Groups Labels n/n Mean S.D. Model H0-2 A-A2 70/ N(M0 S) H0 3-5 B-B3 70/ N(M0 S) H0 All Sm= H -2 A-A2 70/ N(M0 S) H 3-5 B-B3 70/ N(M S) H All Sm= Ths report shows detals of each row of the prevous reports. Hypo. Type Ths ndcates whch smulaton s beng reported on each row. H0 represents the null smulaton and H represents the alternatve smulaton. Groups Each group n the smulaton s assgned a number. Ths tem shows the arbtrary group number that was assgned. Group Labels These are the labels that were used n the ndvdual alpha-level reports. n/n n s the average sample sze of the groups. N s the total sample sze across all groups

18 Group Mean These are the means of the ndvdual groups as specfed for the H0 and H smulatons. Ave. S.D. Ths s the average standard devaton of all groups reported on each lne. Note that t s calculated from the smulated data. Smulaton Model Ths s the dstrbuton that was used to smulate data for the groups reported on each lne. Lst of Contrast Coeffcents Lst of Contrast Coeffcents Groups Contrasts A A2 B B2 B3 Con Con Con Con The contrasts are shown down the rows. The groups are shown across the columns. The coeffcents (weghts) are shown as the body of the table. Ths report shows values of the contrast coeffcents so you can double-check that they are what was ntended. Probablty of Rejectng Indvdual Contrasts Probablty of Rejectng Indvdual Contrasts. Smulaton No. Contrasts Alpha Power Con Con Con Con Alpha: probablty of rejectng hypothess that contrast s zero under alpha (H0) smulaton. Power: probablty of rejectng hypothess that contrast s zero under power (H) smulaton. Probablty of Rejectng Indvdual Contrasts. Smulaton No. 2 Contrasts Alpha Power Con Con Con Con Probablty of Rejectng Indvdual Contrasts. Smulaton No. 3 Contrasts Alpha Power Con Con Con Con Probablty of Rejectng Indvdual Contrasts. Smulaton No. 4 Contrasts Alpha Power Con Con Con Con Ths report shows alpha and ndvdual power for each contrast for each smulaton that was run. In ths example, only the second contrast was non-zero, so that s the only one whch has large values for the power

19 Plots Secton These plots gve a vsual presentaton of the all-contrasts power values and the any-contrast power values

20 Example 2 Comparatve Results Contnung wth Example, the researchers want to study the characterstcs of alternatve multple contrast procedures. Setup Ths secton presents the values of each of the parameters needed to run ths example. Frst, from the PASS Home wndow, load the procedure wndow by expandng Means, then clckng on Multple Comparsons, and then clckng on. You may then make the approprate entres as lsted below, or open Example 2 by gong to the Fle menu and choosng Open Example Template. Opton Value Desgn Tab Solve For... Power MC Procedure... Dunn-Bonferron Test Smulatons FWER (Alpha) n (Sample Sze Multpler) Group Sample Sze Pattern... Equal Grps... 2 Control Dstrbuton H0... Normal(M0 S) Control Dstrbuton H... Normal(M0 S) Grps Group 2 Dstrbuton(s) H0... Normal(M0 S) Group 2 Dstrbuton(s) H... Normal(M S) Equvalence Margn Desgn Tab (contnued) M0 (Mean H0)... 0 M (Mean H)... 2 S... 3 Contrasts Tab Contrasts... Each Wth Next Reports Tab Comparatve Reports... Checked Plots Tab Comparatve Any-Contrast Power Plot... Checked Comparatve All-Contrast Power Plot... Checked

21 Output Clck the Calculate button to perform the calculatons and generate the followng output. Numerc Results and Plots Power Comparson for Smultaneously Testng Multple Contrasts of 5 Groups Dunn Dunn Dunn Dunn Total Bonferron Welch Bonferron Welch Sm. Sample Target All-Cont. All-Cont. Any-Cont. Any-Cont. No. Sze Alpha Power Power Power Power Pool Sze: Smulatons: Run Tme: 5.53 mnutes. Famly-Wse Error-Rate Comparson for Smultaneously Testng Multple Contrasts of 5 Groups Total Dunn Dunn Sm. Sample Target Bonferron Welch No. Sze FWER FWER FWER These reports show the power and FWER of both multple contrast procedures. In these smulatons of groups from the normal dstrbutons wth equal varances, there s lttle dfference n the power of the two procedures

22 Example 3 Valdaton We could not fnd an artcle that gves power values for ths test, so we decded to valdate the procedure by comparng ts results to those of the one-way ANOVA procedure whch allows a sngle contrast to be tested. Usng the settngs of Example and usng the contrast 0, -,, 0, 0, we obtaned the followng powers: , , 0.93, and Setup Ths secton presents the values of each of the parameters needed to run ths example. Frst, from the PASS Home wndow, load the procedure wndow by expandng Means, then clckng on Multple Comparsons, and then clckng on. You may then make the approprate entres as lsted below, or open Example 3 by gong to the Fle menu and choosng Open Example Template. Opton Value Desgn Tab Solve For... Power MC Procedure... Dunn-Bonferron Test Smulatons FWER (Alpha) n (Sample Sze Multpler) Group Sample Sze Pattern... Equal Grps... 2 Control Dstrbuton H0... Normal(M0 S) Control Dstrbuton H... Normal(M0 S) Grps Group 2 Dstrbuton(s) H0... Normal(M0 S) Group 2 Dstrbuton(s) H... Normal(M S) Equvalence Margn M0 (Mean H0)... 0 M (Mean H)... 2 S... 3 Contrasts Tab Contrasts

23 Output Clck the Calculate button to perform the calculatons and generate the followng output. Numerc Results Summary of Smulatons for Testng Multple Contrasts of 5 Groups MC Procedure: Dunn-Bonferron Test Group Total Any- Smpl. Smpl. All- S.D. of S.D. of Sm. Cont. Sze Sze Cont. Means Data Actual Target No. Power n N Power Sm H SD H FWER FWER M0 M S (0.020) [ ] (0.020) [ ] (0.00) [ ] (0.09) [ ] (0.09) [ ] (0.00) [ ] (0.02) [ ] (0.02) [ ] (0.00) [ ] (0.007) [ ] (0.007) [ ] (0.00) [ ] Pool Sze: Smulatons: Run Tme: seconds. In each case, the confdence nterval ncludes the actual value. That s, s between and 0.37, s between 0.72 and 0.760, 0.93 s between 0.90 and 0.926, and s between and Ths valdates the procedure