ABSTRACT INTRODUCTION

Transcription

1 Fast Two-Sample Pemutation Tests, Even When One Sample is Lage, that Efficiently Maximize Powe Unde Cude Monte Calo Sampling J.D. Opdyke, Economic Consulting Goup Andesen, LLP, Boston, MA ABSTRACT I pesent a method fo quickly pefoming multiple nonpaametic two-sample pemutation tests on continuous data in SAS, even when one sample is lage. I maximize statistical powe (within the context of a cude Monte Calo appoach) by ovesampling dawing moe pemutation samples than desied, deleting duplicates, and then selecting the desied numbe of samples fom the emainde. I detemine the optimal numbe of samples to ovesample based on sampling pobability and the untime of a sampling pocedue (PROC PLAN). Implementing ovesampling with nealy optimal numbes of samples inceases stat-to-finish untime typically by only 5%, and always by less than 10%. Using telecommunications pefomance measuement data fom multiple souces with a wide ange of sample size pais, I benchmak stat-to-finish untime against a) anothe SAS pocedue (PROC MULTTEST), b) anothe SAS pogam witten fo the same pupose, and c) Cytel s PROC TWOSAMPL, with vey favoable esults. The elative benchmak speeds would be identical if applied to data fom andomized contolled studies. INTRODUCTION Pemutation tests ae as old as moden statistics, 1 and thei statistical popeties ae well undestood and thooughly documented in the statistical liteatue. 2 Though not always as poweful as thei paametic countepats that ely on asymptotic theoy, they sometimes have equal o even geate powe. 3 Often they can be used when asymptotic theoy falls shot (e.g. small samples and the Cental Limit Theoem), and when fully enumeated, they povide gatifyingly exact esults (as opposed to appoximations based on asymptotic theoy). 4 Most impotant, howeve, is thei eliance on vey few distibutional assumptions, 5 giving pemutation tests a much boade ange of application. Until ecently the majo dawback of pemutation tests has been thei high computational demands. Fully enumeating a pemutation test equies calculating the test statistic appopiate fo the hypotheses being tested fo evey possible two-sample 1 Pemutation tests wee advocated by one of the fathes of moden statistics, Si R.A. Fishe, as ealy as the 1920s. 2 Pesain (2001) and Mielke and Bey (2001) contain extensive bibliogaphies. 3 Fo just one example, see Andesen and Legende (1999). 4 In fact, it was Fishe who coectly chaacteized paametic tests elying on asymptotic theoy as mee appoximations to the exact esults of fully enumeated pemutation tests (Good (1994), p.4, citing Fishe). 5 Exchangeability of the data unde both the null and altenate hypotheses is the majo equiement of pemutation tests. Good (1994) states that a pemutation test equies only, that the undelying distibutions ae symmetic, and/o the altenatives ae simples [sic] shifts in value, though thee ae cases satisfying these citeia whee the basic, nonpaametic pemutation test discussed hee is not the most appopiate method and should not be elied upon (e.g. the Behens-Fishe poblem -- see Pesain (2001), Ch. 10). combination of the data points. 6 Then the value of the test statistic based on the oiginal two samples must be compaed to those based on all the pemutation samples to obtain a p-value 7 the esult of the test. 8 Dawing only a sample of all possible samples, as is typically done, still has been associated with pohibitive compute untimes. Recent advances in computing capacity and speed, howeve, inceasingly have elaxed this constaint. But efficient statistical code still is needed to most effectively exploit these advances and to ensue that the choice of method is diven as much by statistical theoy, and as little by technological constaint, as possible. The goal of the methods descibed below is to contibute to this effot. IMPLEMENTING PERMUTATION TESTS IN SAS Two pocedues in SAS can be used to pefom two-sample nonpaametic pemutation tests PROC MULTTEST 9 and PROC PLAN. The fome diectly samples the input dataset itself, while the late geneates a ecod-by-ecod list identifying those ecods on the input dataset to include in the samples. This list subsequently must be meged with the oiginal data to obtain the coesponding data points. In addition, PROC MULTTEST actually conducts the pemutation test and povides a p-value (assuming that, fo continuous data, a pooled-vaiance t-test is the appopiate test statistic); the esults of PROC PLAN, on the othe hand, must be manipulated manually to calculate the value of the test statistic associated with the oiginal sample pai, and then compae it to those associated with all the pemutation samples to obtain a p-value. Howeve, this entie pocess using PROC PLAN still is much faste than PROC MULTTEST unde most conditions, as shown in the benchmak section below, and it also povides moe flexibility in the definition of the test statistic. Howeve, PROC PLAN has a sample size constaint the poduct of the sum of the two sample sizes (n 1 + n 2) and the numbe of samples being dawn (T) cannot exceed 2 31 o the pocedue teminates. Yet this can be cicumvented by inseting calls to PROC PLAN in a loop which cycles oundup((n 1 + n 2)* T / 2 31 ) times, each loop dawing T * [oundup((n 1 + n 2)* T / 2 31 )] -1 samples until T samples have been dawn (see code in Appendix A). And this looping in and of itself does not slow the total untime of the pocedue. The elative speed of PROC PLAN when samples ae lage enough to equie such looping is at least 20 times faste than PROC MULTTEST. Anothe impotant issue egading the implementation of these pocedues is the sampling method: both PROC MULTTEST and PROC PLAN can pefom cude Monte Calo sampling without eplacement within a sample, as equied of a pemutation test, but neithe can avoid the possibility of dawing the same sample 6 Of couse, the sample sizes of all these possible combinations ae the same as the oiginal two samples. 7 The p-value is simply a popotion the pecentage of pemutation sample test statistic values at least as lage as that based on the oiginal data. 8 This pape addesses only the two-sample pemutation test, though its methods eadily can be applied to pemutation tests with moe complex study designs. 9 PROC MULTTEST is a vesatile pocedue with many functions I addess only its specific application to two-sample nonpaametic pemutation tests on continuous data.

2 moe than once. In othe wods, when dawing a sample of samples, both pocedues can only sample the entie samples with eplacement. This poblem of dawing duplicate samples, its effect on the statistical powe of the pemutation test, and a poposed solution that maximizes powe unde cude Monte Calo sampling 10 ae discussed below. DETERMINING THE NUMBER OF PERMUTATION SAMPLES Full enumeation of pemutation tests quickly becomes infeasible as sample sizes incease because the numbe of possible sample combinations becomes vey lage, even fo elatively small sample sizes. 11 Howeve, full enumeation is unnecessay as the pemutation test can be based on only a sample of all possible samples. Recognizing that the esulting p-value is simply an estimated popotion distibuted binomially, we can detemine its coefficient of vaiation (cv) 12 and confidence inteval as functions of the numbe of samples (T) dawn. Fo example, to achieve cv>0.10 with a p-value equal to the citical value of the test (p-value = α = 0.05), T = 1, This yields a 95% confidence inteval of within 0.01 of the estimated p-value, 14 which fo most pactical puposes is a sufficiently pecise estimate of the fully enumeated p-value. 15 Lage values of T will yield geate pecision, but the maginal inceases in pecision decease apidly and nonlinealy (see Gaph 1 below) These ae simple, unifom andom daws as opposed to moe complicated sampling algoithms, such as impotance sampling (see Mehta et. al. (1988)). ( n1+ n2)! n 11 The numbe of possible sample combinations is given by 1! n2!, which fo n 1 = 3 and n = 3 is just 20, but fo the slightly lage sample pai of n = 29 and n = 30, is a sizeable 59,132,290,782,430, The coefficient of vaiation is a unitless measue of elative spead. It is simply the standad eo of a statistic divided by its mean (see Za (1999), p. 40). 13 Solving fo ( 1 ( )) 0.05 ( ) p value p value T T cv = = = 0.10 p value 0.05 yields T=1,900, so to obtain cv>0.10, we must incease T to 1,901. This calculation also is pefomed in Efon and Tibshiani (1993), pp coefficient of vaiation Gaph 1: Pemutation p-value - cv and 1.96*standad eo by T (# of pemutation samples) fo p-value = ,000 1,200 1,400 1,600 1,800 2,000 2,200 2,400 2,600 2,800 3,000 3,200 3,400 T (# of pemutation samples) 1.96*se THE PROBLEM OF DRAWING DUPLICATE SAMPLES The above calculation assumes, howeve, that the T pemutation samples ae dawn without eplacement i.e. that no sample was dawn moe than once. 17 Moe impotantly, the pemutation test will lose statistical powe if thee ae any duplicate samples among these T samples. 18 Thus, maximizing powe equies a daw of a unique set of samples. This can be accomplished in diffeent ways, but the fastest when using PROC PLAN as descibed heein is to simply ovesample by dawing moe than T samples (say, samples), deleting any duplicate samples, and then andomly selecting T samples fom the emaining set. This appoach does not violate any statistical assumptions equied by a nonpaametic pemutation test specifically, that the distibution of all possible samples is unifom with (1) P ( Y y) = = 1 1 2! ( n + n ) 1,901 so it is a valid appoach fo obtaining T unique samples. Howeve, it bings up two inteelated questions, one egading sampling pobability, the othe egading computational efficiency: how many samples moe than T (-T) ae equied so that we ae left with at least T unique samples once duplicates ae deleted? = 2T? = 3T? And what is the untime tadeoff of dawing lage factos of T samples, which takes moe time simply because the numbe of samples is lage, but also educes the chance of coming up shot with less than T unique samples and having to edaw samples by calling PROC PLAN moe than once? Solving the poblem of minimizing expected untime addesses both of these questions below. cv *standad eo of p-value 14 This calculation assumes that both the sample size (in this case, T) is sufficiently lage and the value of the popotion (in this case, the p-value at p-value = α = 0.05) is sufficiently nonexteme (i.e. not close to 0 o 1) fo the binomially distibuted p-value to closely appoximate the nomal distibution. If these citeia ae satisfied, the 95% confidence inteval is defined by the ange of p-value ± 1.96 standad eos. T = 1,901 and p-value = 0.05 satisfy a vey stict of standad in the liteatue (npq>25, o T*pvalue*(1-p-value)>25 see Evans, Hastings, and Peacock (1993), p. 39) fo such an appoximation to be consideed valid. 17 As mentioned above, pemutation tests equie that each sample be dawn without eplacement i.e. with no item dawn moe than once within a sample. To maximize powe, the sample of pemutation samples also must be dawn without eplacement i.e. no entie sample among the T samples dawn may be dawn moe than once. 15 This appoximate level of pecision fo an estimate of a popotion was deemed excellent (fo T = 2,000) by Baun and Feng (2001) in thei ecent study of optimal pemutation tests. 16 An efficient altenative is to incease T only if the p-value is close to the citical value of the test. 18 Imagine, in the exteme, that the same sample pai is dawn all T times, o close to all T times. It would be impossible, o nealy so, to detect with a specified level of statistical confidence a diffeence between the two samples even if such a diffeence did, in fact, exist. In othe wods, it would be impossible to eject the null hypothesis of no diffeence even if thee was a diffeence, and the extent to which a test can coectly eject the null hypothesis is its level of statistical powe.

3 CHOOSING TO MINIMIZE EXPECTED RUNTIME Expected untime is the poduct of the time it takes to daw samples and the expected numbe of times samples must be dawn to obtain at least T unique samples. 19 Addessing the latte facto fist, the numbe of times samples must be dawn befoe at least T unique samples ae obtained follows the geometic distibution, which identifies the numbe of events occuing befoe the fist success: (2) ( ) ( ) ( s S = s = p p 1) P 1 whee p indicates the pobability of success (of obtaining at least T unique samples) fo each event (each call to PROC PLAN). The expected value of the geometic distibution is simply 1/p, and p is simply deived fom a geneal fom of the familia collecto s poblem. This poblem asks the question, How many samples ae equied, when sampling with eplacement, to obtain ( collect ) all samples fom the sampling distibution? o moe geneally, How many samples ae equied, when sampling with eplacement, to obtain T samples fom the sampling distibution? 20 The numbe of samples equied obviously will follow a pobability density function, which in fact, is the sum of geometic andom vaiables, and is pesented below: (3) ( n + n ) 1 2! j i ( 1 ) j! ( j i) P (# unique samples = j) = ( n1+ n2)! j! j! i= 0 ( n1 n2)! + n!( )! 1! n2! i j i whee = # of samples dawn and j The pobability of obtaining at least T unique samples is simply the cumulative pobability of obtaining T, T+1, T+2,, -1, and unique samples, as shown below: (4) ( n1+ n2)! j i ( 1 ) j! ( j i) p = P( j T) = ( n1+ n2)! j= T j! j! i= 0 ( n1 n2)! + n!( )! 1! n2! i j i whee T. Thus, the expected numbe of times samples must be dawn to obtain at least T unique samples is a function of the numbe of possible sample combinations and, and is povided below: (5) expected # of calls to PROC PLAN = 1 ( n1+ n2)! j i 1 ( 1 ) j! ( j i) = p ( n1+ n2)! j= T j! j! i= 0 ( n1 n2)! + n!( )! 1! n2! i j i Gaph 2 below illustates this functional elationship between p, 1/p, and fo n 1 = 68, n 2= 4, and T = 1,901: P(j>=1,901)=p 1/p = expected # calls ,901 1,902 1,903 Gaph 2: Pobability of at least T Unique Samples (p) and Expected # of calls to Poc Plan (1/p) by (fo n1=4, n2=68, and T=1,901) 1,904 1,905 1,906 1,907 1,908 1,909 1,910 1,911 1,912 p 1/p 1,913 1,914 1,915 1,916 1,917 1,918 1,919 1,920 1,921 Now to etun to the fist facto detemining expected sampling untime the time it takes PROC PLAN to daw a sample of samples. This is simply the untime of PROC PLAN as a function of, inteestingly, not the numbe of possible two-sample combinations, but athe the sum of the two sample sizes (n 1 + n 2), as well as the numbe of samples dawn,. This is shown in Gaph 3 below. 21 Real Time (seconds) Gaph 3: PROC PLAN Runtime by n 1+n 2 by , , , ,000 1,000,000 1,200,000 1,400,000 1,600,000 n 1+n 2 = 1,901 2,700 3, Investigating the possibility of dawing fewe than samples if PROC PLAN is called moe than once inceases the complexity of this poblem beyond the scope of this pape. Given the poximity to T of the nealy optimal values fo povided in Table 1 below, and the vey fast untime of PROC PLAN when the pobability of ecalling it is anything but infinitesimal, I suspect small, if any, gains in untime would esult fom such an inquiy, though I intend to exploe the issue. 20 Of couse, the sampling distibution hee is all possible two-sample combinations based on the two oiginal samples of data. Obviously, and (n 1 + n 2) ae coelated, but untime is vey well pedicted (adjusted R 2 = ) by the simple odinay least squaes multivaiate egession equation below: 21 PROC PLAN was un in SAS v.8.2 on a desktop PC with 2GB RAM and a 2GHz Pentium pocesso. Sample sizes wee geneated by assigning values of 3, 16, 27 to the smalle of the two samples, and, beginning at 100, assigning values by 100 incements to the lage sample up to 100,000, afte which point incements of 10,000 wee used up to 1.5 million (though the pogam has been un on sample pais as lage as 29 and 5,000,029). Thee values of wee used: 1,901, 2,700, and 3,500.

4 PROC PLAN Runtime = PPRT(n 1, n 2, ) = β 0 + β 1*(n 1 + n 2) + β 2* + β 3*(n 1 + n 2)* Nonlineaity at about (n 1 + n 2) = 65,500 and (n 1 + n 2) = 73,500 pompted the inclusion of dummy and inteaction tems, leading to the nea pefect pediction (adjusted R 2 = ) fo PPRT(n 1, n 2, ) pesented in Appendix B (see Gaph 4, which is simply a magnification of Gaph 3 up to (n 1 + n 2)=100,000). g(n 1, n 2,, T) (the poduct of 1/p in Gaph 2 and PPRT in Gaph 5 above) and demonstate an optimal, * = 1,908, fo T = 1,901, n 1 = 4, and n 2 = 68 (and numbe of possible sample combinations = 1,028,790) Gaph 6: Expected Runtime (1/p * each untime) by (fo n1=4, n2=68, and T=1,901) Real Time (seconds) Gaph 4: PROC PLAN Runtime by n 1+n 2 by Expected Runtime (seconds) ,901 1,902 1,903 1,904 1,905 1,906 1,907 1,908 1,909 1,910 1,911 1,912 1,913 1,914 1,915 1,916 1,917 1,918 1,919 1,920 1, ,000 20,000 30,000 40,000 50,000 60,000 70,000 80,000 90, ,000 Gaph 7 magnifies the elevant expected untime ange. n1+n2 = 1,901 2,700 3, Gaph 7: Expected Runtime (1/p * each untime) by (fo n 1=4, n 2=68, and T=1,901) Thus, expected untime g(n 1, n 2,, T) is the poduct of PROC PLAN Runtime and the expected numbe of calls to PROC PLAN: (6) expected untime = g(n 1, n 2,, T) = PPRT(n 1, n 2, ) * expected # of calls to PROC PLAN = PPRT(n 1, n 2, ) * 1 ( n1+ n2)! j i ( 1 ) j! ( j i) ( n1+ n2)! j= T j! j! i= 0 ( n1 n2)! + n!( )! 1! n2! i j i To get an intuitive feel fo as a function of n 1 and n 2 (fo a given T), note again that the second tem of (6) above is a combinatoial function of the sample sizes while the fist tem is meely a linea function of the sample sizes (see Gaph 5 below). Estimated Runtime (seconds) Gaph 5: Estimated PROC PLAN Runtime by (fo n1=4, n2=68, and T=1, based on PPRT in Appendix B) 1,901 1,902 1,903 1,904 1,905 1,906 1,907 1,908 1,909 1,910 1,911 1,912 1,913 1,914 1,915 1,916 1,917 1,918 1,919 1,920 1,921 The combinatoial tems in the second tem of (6) end up dominating as sample sizes incease, asymptotically conveging to 1.0 (one call to PROC PLAN) faste than the fist tem (each PROC PLAN untime) diveges. Hence, fo all but the smallest sample sizes, an optimal, in tems of expected untime (whee g/ = 0), will be faily close to T. Gaphs 6 and 7 below pesent Expected Runtime (seconds) ,901 1,902 1,903 1,904 1,905 1,906 1,907 1,908 * 1,909 1,910 1,911 1,912 1,913 1,914 1,915 1,916 1,917 1,918 1,919 1,920 1,921 Unfotunately, the high level of pecision needed to calculate numeic solutions fo * fo diffeent sample sizes (and T) equies use of a symbolic pogamming language, and thus, cannot be implemented on the fly in SAS. 22 Howeve, fo all pactical puposes, * need not be calculated fo evey combination of values of n 1 and n 2 nealy optimal can be calculated fo anges of the numbe of possible sample combinations because, as shown in Gaph 7, the maginal untime cost of dawing slightly lage than * is negligible (though the maginal untime cost of dawing smalle than * is elatively lage). Thus, if we ceate appopiate anges of the numbe of sample combinations, and fo the low end of each ange identify the lowest coesponding (n 1 + n 2) (because this gives us the lowest maginal incease in PPRT as inceases, and thus, the lagest *), then the coesponding low-end * will neve be lowe than any othe * fo any of the sample pais within that ange. In othe wods, though not optimal fo evey combination of sample sizes within the ange, the low-end * will be nealy optimal because it will be slightly lage (neve smalle) than all othe * fo sample size pais within that ange, and the maginal cost of being slightly lage than * is small, if not negligible. Table 1 below shows the values of used in the pogam the low-end * s fo diffeent anges of the numbe of possible sample combinations. 22 I used Mathematica v.4.1 to calculate p in Table 2 fom the cumulative distibution function (4) of the genealized collecto s poblem. This code is available upon equest. Good appoximations to (4), howeve, do exist see Read (1998) and Kuonen (2000).

5 TABLE 1: Nealy Optimal ( low-end *) Used in Pogam, Pobability (p) of Obtaining T = 1,901 Unique Samples, and Expected # of Calls to PROC PLAN (1/p) by Ranges of # of Possible Sample Combinations, C +! lowend p (lowe bound) * C < 10,626 C 1.0 (assuming C T) ( n1 n2 ) C = 10,626 52,360 52, , , , ,855 1,028,790 1,028,790 10,009,125 10,009,125 25,637,001 25,637, ,290, ,290,905 5,031,771,045 5,031,771,045 C POWER AT WHAT COST? 2, (appox.) 1, (appox.) 1, , , , , , , /p (lowe bound) A good metic fo the cost of employing ovesampling via Table 1 above in ode to maximize statistical powe is its stat-tofinish untime compaed to that associated with just dawing T samples and ignoing the duplicate sample poblem. Employing Table 1 inceased stat-to-finish untime typically by 5% but always by less than 10%. Maximizing powe aguably is woth this elatively small incease in untime. HOW FAST IS IT? RELATIVE SPEED SOME BENCHMARKS The stat-to-finish untime of the pogam using the ovesampling method with PROC PLAN as descibed above is fast elative to othe pogams and pocedues, as seen in Gaph 8 below: Gaph 8: Relative Stat-to-Finish Runtime (T = 1,901) a b c d e f g h whee a = PROC PLAN with ovesampling b = PROC MULTTEST, (n 1+n 2)<10,000, 1 Study pe Contol c = PROC MULTTEST, (n 1+n 2)<10,000, Many Study pe Contol d = PROC MULTTEST, 10,000<(n 1+n 2)<100,000 e = PROC MULTTEST, 100,000<(n 1+n 2)<150,000 f = PROC MULTTEST, 1,000,000<(n 1+n 2)<1,500,00 g = Cytel s PROC TWOSAMPL 24 h = looping in SAS 25 The only pocedue o pogam faste than PROC PLAN with ovesampling is PROC MULTTEST with small samples and one study goup pe contol goup. 26 On the one hand, smalle sample anges ae whee one is most likely to need pemutation tests. Howeve, this is whee the speed diffeential mattes the least in absolute tems even when compaing two hunded pemutation tests with these smalle sample sizes and a study/contol goup atio equal to one, PROC MULTTEST was neve moe than five minutes faste than PROC PLAN. So the tadeoff is seveal minutes pe un with PROC MULTTEST, vesus maximum powe and flexibility in the definition of the test statistic with PROC PLAN with ovesampling. In addition to the speed of PROC PLAN itself, a numbe of factos contibute to the speed of the entie pogam using PROC PLAN with ovesampling, including: Use of PROC APPEND to set two lage datasets togethe (one on top of the othe) wheneve possible. Judicious use of multiple PROC TRANSPOSE s to evaluate the summaized esults of the pemutation sampling. 24 Time did not pemit a thoough investigation into the effect of loweing the study/contol goup atio to one fo PROC TWOSAMPL, though ealy indications ae that its elative untime deceases somewhat, though not nealy as much as that of PROC MULTTEST. 25 See Jackson (1998) fo a copy of the SAS code employing looping to sample the pemutation samples. Bewae, howeve, that the code entes an infinite loop if the numbe of possible sample combinations fo a given sample pai is less than T. Also note that the code, unlike the appopiate definition of a pemutation test which includes ties in the numeato of the p-value, splits ties at the bounday afte assuming exactly one tie at the bounday (appaently in an attempt to make the test less statistically consevative). 23 Of couse, these altenatives do not delete duplicate samples and thus, fo the same numbe of samples T, have less powe than PROC PLAN with ovesampling to the extent they daw duplicate samples. 26 Fo (n + n ) beyond appoximately 10,000, the study/contol goup atio made less 1 2 of a diffeence in PROC MULTTEST s untime. The times epoted fo lage (n + n ) 1 2 ae fo a study/contol goup atio = 1, which would be the fastest time fo PROC MULTTEST.

6 Most test statistics can be constucted based on just one of the two samples in a pai and, if necessay, the pooled summay statistics of the pai. Thus, when conducting pemutation sampling, sample only the smalle of the two samples, but keep tack of which sample is used (study o contol) when constucting the test statistics based on the pemutation samples. Setting togethe the many output datasets fom PROC PLAN by enteing thei names into a long sting assigned to a global vaiable using CALL SYMPUT, and then placing the global vaiable in a set statement so that all the datasets ae set togethe all at once. Altenatives, such as cumulatively setting o even PROC APPENDing the datasets togethe one at a time in a loop, can consume a fai amount of untime. If the dataset is lage and contains a lage pecentage of ecods with the same esponse vaiable value (say, zeo), delete these ecods to avoid soting and late meging them with the PROC PLAN output list. Afte meging the emaining data with the PROC PLAN output and etaining all PROC PLAN ecods in the mege, eassign this value to the esponse vaiable when it is missing (i.e. when that ecod did not mege with the PROC PLAN output because it had been deleted). Most impotantly, if the data contains multiple study goups pe contol goup, thee is no need to output contol goup ecods multiple times fo each coesponding study goup when using PROC PLAN as outlined above. The oiginal data simply can be divided into two datasets one fo contol goup(s) and one fo study goups and each meged sepaately to the PROC PLAN output (then (PROC) APPENDed togethe afte the meges). Unless one constucts a sepaate dataset fo each pemutation test, both PROC MULTTEST and PROC TWOSAMPL equie contol goup ecods duplicated in the input dataset fo each study goup against which they ae being compaed. If the dataset is lage, this can take a temendous amount of sot-time as both pocedues equie soted input data. The same cannot be said fo the thee altenate methods. CONCLUSION I have pesented a method fo quickly pefoming multiple twosample nonpaametic pemutation tests on continuous data in SAS, even when one sample is lage. I also have pesented a pactical and efficient method fo maximizing powe (within the context of a cude Monte Calo appoach) with little incease in untime (5-10%). The pogam utilizing these two methods is at least seveal times faste than seveal widely available altenatives in almost all cicumstances, does not have the sample size constaints of all of these altenatives, and has moe flexibility in the definition of the test statistic used compaed with two of these altenatives (the two pocedues). Only the twosample test was exploed in this study, but the methods pesented ae flexible and can be adapted to moe complex study designs. To quote a medical study that empiically and favoably compaes pemutation tests to thei asymptotic theoy countepats, with continual impovements in computing speed thee seems no impotant agument emaining against pefeed use of this elementay but exact [when fully enumeated] and flexible method (Bullmoe et al. (1999), p.42). Use of the methods pesented in this pape hopefully will contibute to the effot of efficiently exploiting continual advances in computing speed and capacity so that the choice and implementation of statistical methods will be diven by applicable statistical theoy and not technological constaint. ABSOLUTE SPEED When un on data containing 220 sample pais whee the smalle sample was less than 30 obsevations but the lage sample was sometimes ove 64,000 obsevations, the untime of the pogam was 7 minutes, 45 seconds. 27 Fo data containing 6,682 sample pais whee the smalle sample was less than 30 obsevations but the lage was sometimes ove 5,000,000 obsevations, the untime was 8 hous, 36 minutes. The fome example obviously is moe typical of the contexts in which pemutation tests ae used, but the latte is instuctive because it is impotant to know the limits of the methods and softwae being elied upon. This study shows that the untime of PROC PLAN with ovesampling is not pohibitive even when applied to sample sizes as lage (if not fa lage) than would eve be used with pemutation tests All pogams wee un in SAS v.8.2 on a desktop PC with 2GB RAM and a 2GHz Pentium pocesso. 28 One such context is the telecommunications egulatoy aena. As a equiement of the Telecommunications Act of 1996, Regional Bell Opeating Companies (RBOCs), in ode to be pemitted to ente the long distance phone sevice maket, must open thei local phone sevice monopolies to competition. To initially establish competitive economic makets, this equies RBOCs to povide sevice to thei competitos customes that is equivalent to the sevice they povide to thei own customes. Poof of this sevice paity has been equied in the fom of thousands of statistical tests on hundeds of sevice pefomance metics (e.g. how quickly is a phone line installed; how quickly is a phone line epaied, etc.). When one of the two samples being compaed is small (typically that of the Competitive Local Exchange Caie s customes) and the Cental Limit Theoem cannot be elied upon, State Public Sevice Commissions often have equied the use of pemutation tests, even if the sample coesponding to the RBOC s own customes is millions of obsevations. Of the methods examined in this study, only PROC PLAN with ovesampling ealistically can be elied upon to handle such compaisons (fo continuous data pefomance metics) in a timely manne.

7 APPENDIX A *** The code below has been un, and its esults veified, many times. Tanslating file fomats, howeve, can always cause typos. Please notify the autho if you spot one.; *** set global vaiables; %let npemsampt=1901; %let seednum =1; %let seedoig =1; *** The PROC PLAN seed(s) can be set to -1 to key off the time of day (but they still must be incemented when PROC PLAN is looped below o the same daw of samples will esult). The seed geneated can be saved and eused if necessay fo veification / pocess contol puposes.; *** summaized data contains # study goup obs, # contol goup obs; *** use # of obs on summaized input data to obtain # of pemutation tests; poc summay data=sumdinpt; va nobscntl; output out=npemtst(dop=_type_) n=npems; data npemtst eo ; set npemtst(keep=npems _FREQ_); if npems~=_freq_ then output eo; else do; output npemtst; call symput('npemtst',compess(npems)); end; title "Missing NOBS Contol Goup"; poc pint data=eo; title " "; *** ceate pemutation samples one sample pai at a time with the maco makesamp; *** vaiables passed: expment = an identifie of the expeiment (any numbe of such "by vaiables" can be inseted into the maco) bigcomb = identifies which is lage: combins o (the size of the pemutation sample) nobsmal = min(nobstudy, nobscntl) combins = comb(sumofnobs, nobsmal) mincomb = min(combins, ) sumofnobs = sum(nobstudy, nobscntl) ncalls2pp = numbe of times to loop on PROC PLAN if nobstudy+nobscntl * size of daw > 2^31 topdaws = # of samples to daw in all but last loop lastdaw = # of samples to daw in last loop (takes can of modulus) smalle = identifies which sample - study o contol - is smalle and thus, coesponds with the pemutation samples ecount = counte fo ecod # of summaized data fed into makesamp... should count up to global vaiable npemtst %maco makesamp(expment =, bigcomb =, nobsmal =, combins =, mincomb =, sumofnobs =, ncalls2pp =, topdaws =, lastdaw =, smalle =, ecount = ); *** if combins <=, choose all sample combinations, then select T samples fom them; %if &bigcomb=0 %then %do; poc plan seed=&seednum; factos dawnum = &combins dataobsid = &nobsmal of &sumofnobs comb / nopint; output out = psamp&ecount; %if &combins>&npemsampt %then %do; poc plan seed=&seednum; factos dawnum = 1 dataobsid = &npemsampt of &combins andom / nopint; output out = choosmp; data choosmp(keep=dawnum); set choosmp(dop=dawnum); dawnum=dataobsid; poc sot data=choosmp; poc sot data=psamp&ecount; data psamp&ecount; mege psamp&ecount choosmp(in=inchoos) ; if inchoos then output psamp&ecount; data psamp&ecount(dop=dawnum2 temp); set psamp&ecount(dop=dawnum); etain dawnum2; temp=mod(_n_,&nobsmal); if _n_=1 then dawnum2=1; else if temp=1 then dawnum2=dawnum2+1; dawnum=dawnum2;

8 *** if combins >, check whethe PROC PLAN needs to be looped multiple times -- if not, simply select samples, delete duplicates, and keep T samples. If so, loop it fist to select samples. In eithe case, edaw samples if fewe than T unique samples ae dawn the fist time aound.; %if &bigcomb=1 %then %do; %edaw3: %if &ncalls2pp=1 %then %do; poc plan seed=&seednum; factos dawnum = &mincomb dataobsid = &nobsmal of &sumofnobs andom / nopint; output out = psamp&ecount; poc sot data=psamp&ecount; poc tanspose data=psamp&ecount out=temp pefix=stdy; va dataobsid; poc sot data=temp out=temp nodupkey; by stdy1-stdy&nobsmal; poc sot data=temp; set temp; if last.dawnum; call symput("uniqsampn",compess(dawnum)); %if &uniqsampn<&npemsampt %then %do; set temp; dawsize="&mincomb"; combins = "&sumofnobs" " choose " "&nobsmal"; nuniqdw=&uniqsampn; expeiment ="&expment"; seednum2=&seednum+1; call symput('seednum',compess(seednum2)); title "Fewe than &npemsampt Unique Pemutation Samples Selected fo &expment.: Redaw Pefomed"; poc pint data=temp; title " "; %goto edaw3; %if &uniqsampn>&npemsampt %then %do; data psamp&ecount; set psamp&ecount; if dawnum<=&npemsampt; %edaw4: %if &ncalls2pp>1 %then %do q=1 %to &ncalls2pp; %if &q<&ncalls2pp %then %do; poc plan seed=&q; factos dawnum = &topdaws dataobsid = &nobsmal of &sumofnobs andom / nopint; output out = ptemp&q; %if &q=&ncalls2pp %then %do; poc plan seed=&q; factos dawnum = &lastdaw dataobsid = &nobsmal of &sumofnobs andom / nopint; output out = ptemp&q; data psamp&ecount; set ptemp1; %do k=2 %to &q; data psamp&ecount; set psamp&ecount ptemp&k(in=in&k); if in&k then dawnum=dawnum+(&k-1)*&topdaws; poc sot data=psamp&ecount; poc tanspose data=psamp&ecount out=temp pefix=stdyn; va dataobsid; poc sot data=temp out=temp nodupkey; by stdyn1-stdyn&nobsmal; poc sot data=temp; set temp; if last.dawnum; call symput("uniqsampn",compess(dawnum)); %if &uniqsampn<&npemsampt %then %do;

9 set temp; dawsize="&mincomb"; combins = "&sumofnobs" " choose " "&nobsmal"; nuniqdw=&uniqsampn; expeiment ="&expment"; seednum2=&seednum+1; call symput('seednum',compess(seednum2)); title "Fewe than &npemsampt Unique Pemutation Samples Selected fo &expment.: Redaw Pefomed"; poc pint data=temp; title " "; %goto edaw4; %if &uniqsampn>&npemsampt %then %do; data psamp&ecount; set psamp&ecount; if dawnum<=&npemsampt; *** set lengths of any "by vaiables" in PROC PLAN output equal to those in the oiginal dataset fo a smooth mege of the PROC PLAN output with the oiginal dataset; data psamp&ecount(dop=seedback); length expeiment $10. whichsml $4.; set psamp&ecount; seedback="&seedoig"; call symput('seednum',compess(seedback)); expeiment ="&expment"; whichsml ="&smalle"; %mend; *** use nested maco to call PROC PLAN to geneate andom samples fo each ecod of the summaized input dataset; %maco loopingmaco(numloops); %do i=1 %to &numloops; set sumdinpt(keep=expeiment studynobs contlnobs ); if &i=_n_ then do; n=sum(studynobs,contlnobs); k=min(studynobs,contlnobs); combin=comb(n,k); dop n k; *** fo vesions of SAS 6.12 and olde, comb(,) will only go up to about 1*10E70 befoe teminating, so use the loop below; * combin=1; * do j=k to 1 by -1; * combin=combin*(n-j+1)/j; * end; * dop n k j; *** The cuent numeic size limit of SAS is 1.8*10E308 if one sample of the pai being tested is less than 30 obsevations, the lage sample would have to appoach 500,000,000,000 obsevations fo the numbe of possible twosample combinations to exceed this limit. Equal sample sizes of n 1 = n 2 = 514 appoach this limit (# combinations = *10E307), but one should examine closely whethe pemutation tests ae necessay and/o appopiate unde these cicumstances.; *** the 'table' below was calculated based on npemsampt=1901; if combin<10626 then nsamp=combin; else if combin<52360 then nsamp=2155; else if combin< then nsamp=1960; else if combin< then nsamp=1934; else if combin< then nsamp=1912; else if combin< then nsamp=1908; else if combin< then nsamp=1904; else if combin< then nsamp=1903; else if combin< then nsamp=1902; else if combin>= then nsamp=1901; call symput('ncomb',combin); x=min(combin,nsamp); call symput('mincomb',compess(x)); if combin=x then combbig=0; else if combin>x then combbig=1; call symput('combbig',compess(combbig)); nloops=ceil(x*sum(studynobs,contlnobs)/2**31); modplus=mod(&npemsampt,nloops); toploops=floo(&npemsampt/nloops); botmloop=toploops+modplus; call symput('ncalls',compess(nloops)); call symput('topndaw',compess(toploops)); call symput('lastndaw',compess(botmloop)); *** assign names of by vaiables; call symput('expmnt',expeiment); if studynobs<=contlnobs then whichsml="stdy"; else whichsml="cntl"; *** keep tack of which sample (size) coesponds to the pemutation samples fo calculating the test statistic; call symput('whichsml',compess(whichsml)); call symput('nsmalle', compess(min(studynobs,contlnobs))); call symput('sumnobs', compess(sum(studynobs,contlnobs))); output; end; % makesamp(expment =&expmnt, bigcomb =&combbig, nobsmal =&nsmalle, combins =&ncomb, mincomb =&mincomb, sumofnobs =&sumnobs, ncalls2pp =&ncalls, topdaws =&topndaw, lastdaw =&lastndaw,

10 %mend; smalle =&whichsml, ecount =&i ); %loopingmaco(&npemtst); *** now using CALL SYMPUT, place all the PROC PLAN output datasets (psamp1, psamp2, etc.) into a long sting in a global vaiable so they can all be set togethe at once (much faste than setting o appending them cumulatively in a loop). Then mege this entie PROC PLAN output dataset with the oiginal dataset by the appopiate by vaiables (e.g. expeiment) and the ecod id vaiable (above, dataobsid).; APPENDIX B PROC PLAN RunTime, PPRT(n 1, n 2, ), egession esults: Left hand side vaiable = untime seconds adjusted R 2 = Vaiable t value Key Paamete Estimate A B C D E F G H I J K L Vaiable Key Vaiable A Intecept B (n 1 + n 2) C D (n 1 + n 2) * E (dummy: (n 1 + n 2)<65.5K) F (dummy: (n 1 + n 2)<65.5K) * (n 1 + n 2) G (dummy: (n 1 + n 2)<65.5K) * H (dummy: (n 1 + n 2)<65.5K) * (n 1 + n 2) * I (dummy: 65.5K (n 1 + n 2) 73.5K) J (dummy: 65.5K (n 1 + n 2) 73.5K) * (n 1 + n 2) K (dummy: 65.5K (n 1 + n 2) 73.5K) * L (dummy: 65.5K (n 1 + n 2) 73.5K) * (n 1 + n 2) * REFERENCES Andesen, M.J. and P. Legende, An empiical compaison of pemutation methods fo tests of patial egession coefficients in a linea model, Jounal of Statistical Computation and Simulation, Vol. 62, No. 3, Rabe-Hesketh, Eic Taylo, and Michael J. Bamme, Global, Voxel, and Cluste Tests, by Theoy and Pemutation, fo a Diffeence Between Two Goups of Stuctual MR Images of the Bain, IEEE Tansactions on Medical Imaging, Vol. 18, No. 1, Baun, Thomas M. and Ziding Feng, Optimal Pemutation Tests fo the Analysis of Goup Randomized Tials. Jounal of the Ameican Statistical Association, Vol. 96, No. 456, Decembe, Evans, Mean, Nichols Hastings, and Bian Peacock, Statistical Distibutions, 2 nd ed., New Yok: John Wiley & Sons, Good, Phillip, Pemutation Tests, Spinge-Velag, New Yok, Efon, Badley and Robet Tibshiani, An Intoduction to the Bootstap, Chapman & Hall, London & New Yok, Affidavit of John Jackson, On Behalf of MCI-Woldcom, Befoe the Michigan Public Sevice Commission, Case No. U-11830, Novembe 18, 1998, ATTACHMENT A, Using Pemutation Tests to Evaluate the Significance of CLEC vs. ILEC Sevice Quality Diffeentials Kuonen, Diego, A Saddlepoint Appoximation fo the Collecto s Poblem, The Ameican Statistician, Vol. 54, No. 3, August Mehta, Cyus R., Nitin R. Patel, Palay Senchaudui, Impotance Sampling fo Estimating Exact Pobabilities in Pemutational Infeence, Jounal of the Ameican Statistical Association, Vol. 83, No. 404, Mielke, Paul W. and Kenneth J. Bey, Pemutation Methods: A Distance Function Appoach, Spinge-Velag, New Yok, Pesain, Fotunado, Multivaiate Pemutation Tests with Applications in Biostatistics, John Wiley & Sons, Ltd., New Yok, Read, K.L.Q., A Lognomal Appoximation fo the Collecto s Poblem, The Ameican Statistician, Vol. 52, No. 2, May Za, Jeold H., Biostatistical Analysis, 4 th ed., Uppe Saddle Rive, NJ: Pentice Hall, ACKNOWLEDGMENTS I owe special thanks to Gei Costanza, M.S., who povided me with a numbe of valuable insights. Any eos ae my own. CONTACT INFORMATION Please contact the autho with you questions and comments at: J.D. Opdyke Economic Consulting Goup - Andesen, LLP 225 Fanklin Steet, 13 th Floo Boston, MA Wok Phone: , Fax: jdopdyke@usa.net john.d.opdyke@andesen.com Bullmoe, Edwad T., John Suckling, Stephan Ovemeye, Sophia