ASYMPTOTIC TWO-TAILED CONFIDENCE INTERVALS FOR THE DIFFERENCE OF PROPORTIONS

Transcription

1 ASYMPTOTIC TWO-TAILED CONFIDENCE INTERVALS FOR THE DIFFERENCE OF PROPORTIONS Martín Andrés, A. *() ; Álvarez Hernández, M. () and Herranz Tejedor, I. () () Department of Biostatistics, Faculty of Medicine, University of Granada (Spain). () Department of Biostatistics, Faculty of Medicine, Complutense University of Madrid (Spain). ABSTRACT In order to obtain a two-tailed confidence interval for the difference between two proportions (independent samples), the current literature on the subject has proposed a great number of asymptotic methods. This paper assesses 80 classical asymptotic methods (including the best proposals made in the literature) and concludes that: () the best solution consists of adding 0.5 to all of the data and inverting the test based on the arcsine transformation; () a solution which is a little worse than the previous one (but much easier and even better when both samples are balanced) is a modification of the Adjusted Wald method proposed by Agresti and Caffo (usually adding z / / 4 to all of the data and then applying the classical Wald CI); (3) surprisingly, the classical score method is among the worst solutions, since it provides excessively liberal results. KEY WORDS: Arcsine transformation; Asymptotic confidence interval; Difference between proportions; Score method; Wald methods and adjusted Wald methods.. INTRODUCTION Obtaining a confidence interval (CI) for the difference d = p p between two proportions p i (for independent samples) is very common in many fields of science (Newcombe, 998). On some occasions, interest is focused on a one-tailed classical CI of d * Correspondence to: Department of Biostatistics, Faculty of Medicine. University of Granada. 807 Granada. Spain. amartina@ugr.es. Phone: Fax:

2 0, like in clinical trials of superiority or non-inferiority (in which the objective is to demonstrate that the treatment or drug is superior or non-inferior to the other, respectively). On most other occasions, the objective is to obtain a classical two-tailed CI L d U (which is the one covered in this paper). It is possible to obtain a classical two-tailed CI from the perspective of traditional methodology or through Bayesian methodology (which is more difficult to compute: Brown and Li, 005). In this paper, we focus only on the more common case of traditional perspective. Furthermore, the inference can be made in an exact or an approximate way. In the first case (exact inference), the real error of the CI ( * ) is never greater than the nominal error (); in the second case (approximate inference), * may be greater than ; in both cases, it is desirable that * is as close as possible to. As it is computationally intensive to obtain an exact CI, it is necessary to use specific computer programmes, it is not viable for moderately large sample sizes (Santner et al., 007), and normally leads to a conservative procedure (Agresti, 003), and researchers have devoted a great deal of attention to approximate methods. In this paper, we focus only on the more common case of approximate inference. Finally, the approximate solutions are based on the real distribution of the variables involved (the binomial distribution) or on their asymptotic distribution (usually the normal distribution). The first ones, sometimes referred to as quasi-exact (Chen, 00) or almost exact (Agresti, 003) inferences, require a certain computational intensity and can be based on the traditional p-value (Kang and Chen, 000; Chen, 00; Lin et al., 009) or on the midp-value (Newcombe, 998; Chen, 00; Agresti, 003; Agresti and Gottard, 007). The second ones, generally referred to as asymptotic inferences or with large samples, are normally simpler to apply and some of them are of great interest from a teaching/pedagogical point of view (Agresti and Caffo, 000); therefore, many researchers have devoted their efforts to proposing new procedures (Newcombe, 998; Agresti and Caffo, 000; Kang and

3 3 Chen, 000; Martín and Herranz, 003; Brown and Li, 005; Lecoutre and Faure, 007; Martín et al., 0, etc.) have not always been compared. In this paper, we focus only on the most common case of asymptotic inferences. In line with the aforementioned reasons, the objective of this paper is to propose new non-bayesian asymptotic methods to obtain a classical two-tailed CI and assess them comparatively with the best methods proposed in the literature in this field. Additionally, in the URL a free program may be obtained enabling the optimal methods selected in the paper (and the classic score method) to be applied.. CIs AND HYPOTHESIS TESTS In general (Agresti and Min, 00) it is advisable to obtain the two-tailed CIs through the inversion of the two-tailed test H: d=δ vs K: d δ (in order to ensure that the statistical inference is coherent). This means two things: first, that the definition of a procedure can be made from the perspective of the test or from the perspective of the CI. In this case, the perspective of the test will be adopted since, as will be seen in Section 3, it is this perspective which allows a more logical introduction of different procedures. The second conclusion is that assessing a CI procedure is equivalent to assessing its associated test procedure (if both are carried out to the same nominal error α). In order to assess a CI, it is customary to use the parameters of real coverage and average length; and in order to assess a test, it is customary to use the parameters of real error and power. However, real coverage and real error add up to (Section 4) and, moreover, the greater the power of the test the lesser the length of the CI which is obtained through its inversion. For this reason, the comparative assessment of different inference methods is carried out below based on the test which defines them. 3. PROCEDURES TO CARRY OUT THE INFERENCE

4 4 3.. Procedures based on the Z-statistic Let x i ~ B(n i ; p i ), i = or, be two independent binomial random variables and let y i = n i x i, Σx i = a, Σy i = a, Σn i = n = Σa i, p = x i /n i, q p, q i = p i, d = p p and i i i d p p. In order to test H: d= vs. K: d (with < < +) one can use different statistics, and one of the most common is the following Z-statistic: Z-Statistic: z=d p q p q Z n n () In order to use this statistic, it is necessary to estimate the unknown proportions p i, which can be done through various procedures (such as those indicated in this paper). Once we have estimated the p i (which may or not be a function of ), the test is performed in the classic way: comparing the value z Z which is obtained with z /, where z α/ is the 00 ( α/)th percentile of the standard normal distribution. Alternatively, the d ( L ; U ) will be obtained solving through the equation z Z z /. On some occasions, the solution will be explicit (more or less simple); in others, the solution will require an iterative procedure. The classical procedure found in elementary textbooks consists of substituting the unknown values p i with: Estimator W: piw pi =x i /n i, () in the Z statistic of expression (), which leads to the following ZW procedure (the classical Wald procedure) (from now on, the first expression refers to the test statistic and the second one to the CI that is obtained by inversion): pq pq z = d, CI : / / ZW dd z pq n n n ZW. (3) i i i Here, and as follows, we adopt the criteria of notating the procedure with the two letters that define it: Z (because of the statistic used) and W (because of the estimator assumed).

5 5 A more modern and effective procedure is that proposed by Newcombe (998). In this case, the unknown proportions p i are substituted with: Estimator N: pn u, pn l if d where li; ui pn l, pn u if d x z z x y / / i i i z / 4 ni ni z /, (4) which leads to the following ZN procedure (here, and in a similar way from now on, q in = p in ): z = ZN d p q p q n n N N N N, CI ZN : l l u u d z / n n d u u l l d z / n n. (5) In the previous expressions, (l i ; u i ) are the bounds of the ()-CI for p i obtained through the Wilson method (97). The theoretical justification for the ZN procedure can be seen in Zou and Donner (008). The two previous estimators (p iw and p in ) are not restricted to being compatible with the null hypothesis. When we require estimators restricted by H (such as those which are defined from here on in this paper), then p = p +δ and the only parameter to be estimated is p. From the conditioned point of view (that is, conditioned in the sense that the successes always add up to the observed value of a ), the estimators p ic are given by (Dunnett and Gent, 977): Estimator C: p C ( a n )/ n and pc ( a n )/ n, (6) although if p ic < 0 (> ) p ic = 0 (= ) must be used (Farrington and Manning, 990). By substituting p i with p ic in expression (), the ZC procedure is obtained, whose statistic is:

6 6 p CqC pcqc a a a a + if min, min,, n n n n n n d pcqc a a ZC where if or, V n n n p CqC a a if or. n n n z V= (7). The CI through the ZC procedure -which is the solution adopted by Wallenstein (997) and Martín and Herranz (003)- is provided by the two solutions for the second degree equation A=z n n n n +nn n A + B + C = 0, with /, B=z n n a a nn n d / and C=nnnd z / aa. The solution is valid only if the two values ( L, U ) thus obtained verify the first condition of expression (7), since then 0 p ic, but when one of the two bounds X fails, then the same one must be calculated for the values A=n z / +nn, B= z n a a n n d and / C=nn d z / aa (if X verifies the second condition) or A=nz / +nn, B=z / n a a nn d and C=nn d z / aa (if X verifies the third condition). In a more explicit manner: / / / nn n nnn nn n n z / n a a nn nz / pq CI ZC: d d z / z /, nn nz / nn nn n nn n a a nz / pq d z / z /, nnnz/ n n nn n n n a a z n n n 4nn aa pq i i nnnd z z, nnn ni (8) in which each one of the three previous expressions is valid subject to each one of the conditions of expression (7), respectively. From an unconditional perspective, the estimator of maximum likelihood p E of p

7 7 (estimator E) was devised by Mee (984) and is the solution to a cubic equation (Miettinen and Nurminen, 985). Substituting in expression () p and p with p E and pe p E, respectively, ZE procedure is obtained. The procedure has a disadvantage which is that the CI ZE does not have an explicit solution and must be obtained by iterative methods. The current version of ZE procedure (statistic z ZE ) was proposed by Mee (984), although Miettinen and Nurminen (985) proposed a very similar statistic to the previous one: n z / n. ZE stating that the statistic Finally, Peskun (993) used the criterion proposed by Sterne (954) when z Z would be significant when it is so for any value of p, that is, when z z every p. With this objective, he determined the minimum value of Z / z Z that is, the maximum value of its denominator and found that this is reached in: Estimator P: pp ( n n )/n and pp ( n n )/n, (9) Substituting p i with p ip in the z Z statistic in expression (), the ZP procedure is obtained. Now: 4 nn n ( d ) n z d z, IC : d n d z n z 4 / ZP ZP / / n 4nn nn n. (0) We have found that if we force 0 p ip as is the case of the conditional estimator p ic the resulting procedure is worse than the previous one. 3.. Procedures based on the X and A statistics Dunnet and Gent (977) proposed as an alternative to Z-statistic ( z Z ), the classical chi-square statistic (X-statistic) z n p p pq with p = p +. When p i is X i i i i i substituted by the p ic or p ie estimators, we obtain procedures XC (Dunnet and Gent, 977)

8 8 and XE respectively. As Nam (995) demonstrated that z z (the score method), the X statistic only leads to a procedure (XC) whose statistic is (CI XC requires iterative methods): XC i i ic ic ic ZE z n p p p q. () Herranz and Martín (008) proposed the following A-statistic based on the arcsine transformation (from now on p = p +): XE z A d nn n n where d sin p sin p and sin p sin p. () When p i is substituted by the p ic or p ie estimators, we obtain procedures AC (new) and AE (Herranz and Martín, 008), with statistics z AC and z AE, respectively. In both cases, the CIs (CI AC and CI AE respectively) are obtained through iterative methods. 4. SAMPLE DATA TO BE USED AND INFERENCE METHODS THEY PROVIDE As the Wald procedure ZW does not work well (Dunnet and Gent, 977; Hauck and Anderson, 986; Roebruck and Kühn, 995; Feigin and Lumelskii, 000; Brown and Li, 005; Xu and Kolassa, 007), several authors have tried to improve it using the simplicity of expressions (3). The improvements consist of the application of the Wald procedure not to the original data (x i ; y i ; n i ), but to the incremented data in the quantity h i, that is, the values (x i +h i ; y i +h i ; n i +h i ), which leads to what are normally called adjusted Wald methods. The increases h i =0.5 (Woolf, 955, in the context of parameter p /p ) and h i = (Agresti and Caffo, 000) are traditional. Other possibilities for increases are those suggested by Martín et al. (0) in a more general context from which they selected the two last shown below. Each increase leads to a different case, which will be enumerated as follows: Case 0: h i =0, Case : h i =0.5, Case : h i =, Case 3: h i = z / 4, and Case 4: z if p 0 if p Ii ˆ where I, I if d 4 0if p 0 0if p hi. (3) z S if p if p 0 i where S ˆ, S if d 4 0if p 0if p 0 /

9 9 The first two proposals are of a heuristic nature; the last two are obtained by making the centre of interval (3) for the incremented data the same as the centre of the CI for the score procedure (ZE) of the original data (Martín et al., 0). It must be noted that Case 3 produces the same results as Case 4 when the original observations are on the bounds of the sample space, that is, when x i 0 and x i n i (i); but if, for example, x = 0 and 0 < x < n then h = h = z / /4 in Case 3, but in Case 4 h = 3 z / /4 and h = z / /4 when the lower bound is obtained and h = h = z / /4 when the higher bound is obtained. The five previous increases can be applied to any of the eight procedures defined in the previous section (ZW, ZN, ZC, ZE, ZP, XC, AE and AC), which leads to 40 possible inference methods. For example, the ZW procedure will lead to five methods ZW0, ZW, ZW, ZW3 and ZW4 respectively. From these 40 methods, only eight have been proposed in the literature: ZW0 (the classical Wald procedure), ZW (Agresti and Caffo, 000), ZN0 (Newcombe, 998), ZC0 and XC0 (Dunnet and Gent, 977), ZE0 (Mee, 984), ZP0 (Peskun, 993) and AE0 (Herranz and Martín, 008). In the case of the A-statistic, the Anscombe (948) transformation (h i = 3/8) is used. The method is not included in this analysis since we have found that it does not improve upon the selection made in the following section. 5. ASSESSMENT OF THE PROPOSED METHODS 5.. Procedure to obtain and analyse results In order to make a comparative assessment of the 40 methods proposed, it is necessary to obtain certain parameters (parameters Δα, θ and F which are indicated below) which represent a synthesis of the quality of each method. To meet this objective it is necessary to carry out the following steps:

10 0 () Select one of the errors α=%, 5% or 0%, stressing the error of 5% (as it is the most frequent). () Select one of the pairs (n, n ) with n n and n i = 40, 60, 00. Cases of n > n are excluded since the null hypotheses H: p p = +δ and H : p p = δ are equivalent. (3) Select one of the values δ = 0, 0., 0., 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9 and Cases of δ<0 are excluded since the null hypotheses H: p p = +δ and H : q q = δ are equivalent. This exclusion, and that referred to in the previous paragraph, is justified by the fact that the statistic z exp of any of the methods cited takes the same value subject to H, H and H (Martín and Herranz, 00). (4) Construct the critical region CR={(x, x ) z exp z / }. (5) Calculate the real error (the size of the test) α * = Max0p Pr ( x, x p, ), where Pr (x, x p, ) = C (n, x )C (n, x ) p x ( p ) y ( p ) x ( p ) y and the CR increase Δα = α α * of the nominal error in relation to the real error. As the CI is obtained through the inversion of the test, each observation (x, x ) produces a CI for d given by CI(x, x ) = {δ 0 z exp 0 z / }, so that the real coverage will be * = n n 0p x x x p 0 x0 Min Pr (,, ) I( x, x ), where I(x, x )= if δci(x, x ) and I(x, x )=0 otherwise. As δci(x, x ) when z exp 0 z /, then I(x, x )= if (x, x )CR and therefore * = Min Pr ( x, x p, ) where CR refers to the 0 p CR complementary set of the set CR and * =α *. This means that Δα = α α * = ( α * ) ( α) = γ * γ, where γ= α is the nominal coverage (the nominal confidence of the interval). Therefore, if Δα>0, then both the test and the CI are conservative since the first (second) one has less (more) real error (coverage) than the nominal one; on the contrary, when Δα<0 then both the test and the CI are liberal.

11 Observe that value α * has been obtained by making exact calculations and not from simulations. (6) Calculate the value of θ = 00(the number of points in the CR set) / [(n +)(n +)] %, where (n +)(n +) is the total number of points in the sample space. The value of is a good indicator of the power of the test (Upton, 98; Martín and Silva, 994; Chen et al., 007), and was defined by Martin and Silva as long term power (as it is the average power of the test when the proportions p i are obtained from a uniform distribution). The long-term power allows us to make a global comparison of the power of two tests carried out at an error rate, which does not happen when we use the traditional power (since it depends on the alternative hypothesis that is considered). In order to achieve some economy of expression, from now on will just be referred to as power. (7) Determine whether the method fails, that is, if Δα %, % or 4% for α = %, 5% or 0%, respectively. This definition is introduced as it is to be wished that the asymptotic method provides a real error close to the nominal error, that is, that α * α or, in an equivalent manner, that Δα 0. Nevertheless, this implies that it is necessary to allow some discrepancy between the two. If, for example, α=5%, in some areas, it is traditional (Agresti and Coull, 998; Price and Bonett, 004; Martín et al., 0) to allow α * < 7%, that is, that the real coverage is greater than 93% or, equivalently, that Δα > %; in another case (Δα % or α * 7%), we could say that the test or the CI fails. If α * < 3% (Δα > +%), the test is very conservative, but does not give false significance, and this possible bad performance is made clear in the low value of θ. With all of this data a table like Table is constructed (which contains the results for the two methods that behave best). The data for the other 38 methods can be requested form the authors. With the objective of making an easy comparison of methods it is advisable to synthesize the data obtained by carrying out the following, final step.

12 (8) Calculate the total number of failures (F) and the average values of Δα ( ) and of θ ( ) in all of the combinations (n, n ). As the values δ have been omitted (for giving the same results as the values +δ), the previous calculations will be carried out assigning weight to case δ=0 and weight to the other cases. On some occasions, in order to check the stability of the conclusions, the same values for each combination (n, n ) will be calculated. Once the results are obtained, the selection of the optimal method is made subject to the following criteria: (a) Special importance will be given to the case α=5% (as it is the most frequent nominal error). (b) We will reject methods with an excessive number of failures (since they very often prove to be excessively liberal), showing preference for those methods which have fewer failures. (c) Among the methods that are still available, we choose those which have a closest to 0 (that is, those methods whose mean error is close to the nominal one). If there is a tie, conservative methods ( > 0) are preferred to liberal ones ( < 0), so that the significance is reliable. (d) Among the methods that are still available, those with the greatest power are preferred. In this case, it is necessary to take into account that if the method X is more liberal than Y, then Y X and it is to be expected that Y X (which does not mean that X is a better method than Y). 5.. Selection of the optimal method Table shows the values of F, and (α = 5%) for the best methods for each group (AE, ZW4, ZP0, ZW3, ZW and ZC0) from the 40 methods studied and for the three

13 3 classical methods (ZN0, ZE0 and ZW0). The methods excluded perform much worse (since they have between 8 and 0 failures). The following observations are made: (i) The classical ZW0 method performs very badly (since it almost always fails), whilst the ZE0 score method performs badly (it has too many failures and is quite liberal). Something similar happens in the case of the well-known ZN0 method (although it has much fewer failures and is somewhat less liberal). (ii) The ZP0 method should be discarded since, although it has no failures, it is conservative and has little power. (iii) The ZW, ZW3 and ZC0 methods are similar and occupy an intermediate position (they only have two failures, although they are somewhat liberal on average), and they are preferable to the first two as they provide CIs which are simpler to calculate. (iv) The two best methods are AE and ZW4 (they do not have failures, are slightly conservative on average and have good power). The detailed results for these two methods can be seen in Table, in which it can be observed that the AE method is clearly better than the ZW4 method in general, and the opposite happens when both samples are balanced. The previous conclusions hold in general for α=% and α=0%. Consequently, we can conclude that the optimal method is AE, although the ZW4 method (which is much easier to carry out) is only a little worse (is actually better when n =n ). In particular, the values of (F,, ) for α = % in the two selected methods AE and ZW4 are (0, 0.08, 8.89) and (0, 0.3, 8.93), respectively; now method AE is only slightly better than method ZW4. Additionally, it can be observed that the results of Table are coherent with what one would expect. Thus, the ZW and ZW3 methods are similar since the value of h i for ZW3 is.96 /4 (the value of h i for ZW). Furthermore, the value of Δα for the ZP0 method is greater than for the ZE0 method, which is to be expected given that ZP0 method is based on

14 4 the lowest possible value of the statistic obtained by the ZE0 method. Moreover, it is interesting to compare our results with those found in the current literature. According to Roebruck and Kühn (995) the ZE0 method is better than the ZC0 method, but according to Wallenstein (997) the opposite happens; our results show the second scenario. For Wallenstein (997), the ZC0 method is better than the ZP0 method; according to our results, this is true from the point of view of power, but not regarding the number of failures and the average error. For Newcombe (998) the ZN0 method is better than the ZE0 method, which is line with our results. The reason for these discrepancies (and for others that occur in the literature) is that many authors tend to emphasize the values of * ( p ) Pr ( x, x p, ) and thus calculate its average and its minimum value in the CR different combinations of p, δ, n and n. Here emphasis is placed on the value Min ( p ) for which we calculate its average for different combinations of δ, n and n * * p * (values which are then subtracted from α to obtain Δα). The first, ( p ), is the real error of the test in the value p ; the second, α *, is the size of the test, which is what must be considered (Barnard, 947; Chan, 998; Agresti and Min, 00). Finally, it is important to highlight the surprisingly poor performance of the much reputed ZE0 method, which is in line with the findings of Chan (998), Newcombe (998) and Santner et al. (007). What is even more surprising is the fact that the one of the two best methods (ZW4) is the approach of the ZE0 method Statistics with continuity correction Martín and Herranz (004) demonstrated that the ZE0 method improved if a continuity correction (cc in the following) was applied as this reduced its liberality. Their argument is applicable to any statistic Z or A based on expressions () or (), respectively -the case of the X statistic is omitted due to its poor performance- which become:

15 5 z z Zc Ac d c pq i i / ni if d c / n nn if n n where c 0 if d c / n nn if n n nn d c nn if d c = where c 0 if c n nn d If the two methods AE and ZW4 selected in the previous section are based on expression (4) instead of expressions () or (), another two AEc and ZW4c methods with cc are obtained. Table 3 shows the new results. It is observed that cc is not useful to improve the performance of the methods selected previously. We have also found that this selection remains when comparing the 35 methods with cc and the 35 methods without cc. (4) 6. ANALYSIS OF THE SPECIFIC CASE OF THE TEST FOR = 0 A complementary question is the test for δ = 0, that is, the classical test for homogeneity for two independent proportions. Its traditional contrast statistic is normally expressed as ( x y x y ) n/ nn aa, which corresponds to the current ZE0 method. exp As now p ic p ie, then the ZE, ZC and XC procedures, on the one hand, and the AC and AE procedures, on the other hand, are the same. Table 4 shows the results of the best methods from among the 80 methods studied (40 with cc and 40 without cc) for =5%: the ZE0, ZE, ZE3, ZE4 and AE methods without cc and the ZE0c, ZEc, ZE3c, ZE4c and AEc methods with cc (all of them without failures). Then the results are provided for each combination (n, n ), since there are strong variations from one case to another. It can be observed that, although there is no unanimous selection, in general the best method is ZE0c (although AEc is almost equal) when the sample sizes are different, as otherwise the ZE3c method is usually more appropriate.

16 6 7. EXAMPLE Rodary et al. (989) studied the response to the chemotherapy and radiation therapy in a randomized clinical trial on nephroblastoma. The response rates were 0.943=83/88 (x /n ) and =69/76 (x /n ) in the chemotherapy and radiation group, respectively. Table 5 contains the results obtained by the two best asymptotic methods (AE and ZW4), the traditional asymptotic method (ZE0c) and two exact methods (based on two different order statistics). With regard to the asymptotic methods, it can be seen that l(zw4) = < l(ae) = 0.77, which accords with method ZW4 being the one to choose when n and n are similar, and that l(ae) = 0.77 l(ze0c) = 0.770, which accords with the fact that method ZE0c does not perform badly in relatively large samples and for values of close to 0. With regard to the exact methods, it can be seen that l(exact AE) = < l(exact ZY) = 0.899, which suggests that the new order E can be a good alternative to the order ZY for constructing the exact CI. Note that the exact CI (order AE) is very similar to the asymptotic CI of the AE method. 8. CONCLUSIONS In applied statistics, it is often necessary to obtain a two-tailed confidence interval for the difference d = p p between two proportions (independent samples). As exact methods are computationally very intensive and require special computer programmes, the literature in this field has shown great interest in approximate methods, especially in asymptotic methods (as they are generally simpler to apply). In this paper, 80 methods have been assessed (including the most relevant examples in the literature). There are several conclusions. Firstly, the classical ZE0 score method is, surprisingly, the worst in terms of performance (although, if it is applied with the appropriate continuity correction, it is better if

17 7 it is used to carry out the classical test of homogeneity of two proportions for samples of a different size). Secondly, the optimal method is AE (adding 0.5 to all of the data and using the statistic based on the arcsine transformation), whose application requires iterative methods. Alternatively, we can use the very simple ZW4 method (the Wald CI applied to the increased data usually in z / 4) which is only a little worse than the previous one and is better than / AE when both samples are of a similar size. In the URL a free program may be obtained enabling the optimal methods selected in the paper (and the classic score method) to be applied. 9. ACKNOWLEDGMENTS This research was supported by the Spanish Ministry of Education and Science, grant number MTM (co-financed by the European Regional Development Fund). REFERENCES Agresti, A. (003). Dealing with discreteness: making exact confidence intervals for proportions, differences of proportions, and odds ratios more exact. Statistical Methods in Medical Research, 3-. DOI: 0.9/ sm3ra. Agresti, A. and Caffo, B. (000). Simple and effective confidence intervals for proportions and difference of proportions result from adding two successes and two failures. The American Statistician 54 (4), DOI: 0.307/ Agresti, A. and Coull, B. A. (998). Approximate Is Better than "Exact" for Interval Estimation of Binomial Proportions. The American Statistician 5 (), 9-6. DOI: 0.307/ Agresti, A. and Gottard, A. (007). Nonconservative exact small-sample inference for discrete data. Computational Statistics & Data Analysis 5, DOI:0.06/j.csda

18 8 Agresti, A. and Min, Y. (00). On small-sample confidence intervals for parameters in discrete distributions. Biometrics 57, DOI: 0./j X x. Anscombe, F.J. (948). The transformation of Poisson, binomial and negative-binomial data. Biometrika 35, DOI: 0.093/biomet/ Barnard, G.A. (947). Significance tests for tables. Biometrika 34, DOI: 0.307/3357. Brown, L. and Li, X. (005). Confidence intervals for two sample binomial distribution. Journal of Statistical Planning and Inference 30, DOI: 0.06/j.jspi Chan, I.S.F. (998). Exact tests of equivalence and efficacy with a non-zero lower bound for comparative studies. Statistics in Medicine 7, DOI: 0.00/(SICI) ( ). Chen, X. (00). A quasi-exact method for the confidence intervals of the difference of two independent binomial proportions in small sample cases. Statistics in Medicine, DOI: 0.00/sim.053. Chen, L-A; Hung, H-N and Chen, C-R. (007). Maximum average-power (MAP) tests. Communications in Statistics-Theory & Methods 36, DOI: 0.080/ Dunnett, C.W. and Gent, M. (977). Significance testing to establish equivalence between treatments, with special reference to data in the form of tables. Biometrics 33, DOI: 0.307/ Farrington C.P. and Manning, G. (990). Test statistics and sample size formulae for comparative binomial trials with null hypothesis of non-zero risk difference or non-unity relative risk. Statistics in Medicine 9, DOI: 0.00/sim Feigin, P.D. and Lumelskii, Y.P. (000). On confidence limits for the difference of two binomial parameters. Communications in Statistics-Theory & Methods 9 (), 3-4. DOI: 0.080/ Hauck, W.W. and Anderson, S. (986). A comparison of large-sample confidence interval methods for the difference of two binomial probabilities. The American Statistician 40 (4), DOI: 0.307/ Herranz Tejedor, I. and Martín Andrés, A. (008). A numerical comparison of several unconditional exact tests in problems of equivalence based on the difference of proportions. Journal of Statistical Computation and Simulation 78 (), DOI: 0.00/ sim Kang, S-H and Chen, J.J. (000). An approximate unconditional test of non-inferiority between

19 9 two proportions. Statistics in Medicine 9, DOI: 0.00/ ( ). Lecoutre, B. and Faure, S. (007). A note on new confidence intervals for the difference between two proportions based on an Edgeworth expansion. Journal of Statistical Planning and Inference 37, DOI:0.06/j.jspi Lin, Y., Newcombe, R.G., Lipsitz, S. and Carter, R.E. (009). Fully specified bootstrap confidence intervals for the difference of two independent binomial proportions based on the median unbiased estimator. Statistics in Medicine 8, DOI: 0.00/sim Martín Andrés, A. and Herranz Tejedor, I. (003). Unconditional confidence interval for the difference between two proportions. Biometrical Journal 45 (4), DOI: 0.00/ bimj Martín Andrés, A. and Herranz Tejedor, I. (004). Asymptotical tests on the equivalence, substantial difference and non-inferiority problems with two proportions. Biometrical Journal 46 (3), DOI: 0.00/bimj Martín Andrés, A. and Herranz Tejedor, I. (00). On the inferences for linear functions of two proportions. JP Journal of Biostatistics 4(3), Martín Andrés, A. and Silva Mato, A. (994). Choosing the optimal unconditioned test for comparing two independent proportions. Computational Statistics and Data Analysis 7, DOI: 0.06/ (94) Martín Andrés, A., Álvarez Hernández, M. and Herranz Tejedor, I. (0). Inferences about a linear combination of proportions. Statistical Methods in Medical Research 0, DOI: 0.77/ Mee, R.W. (984). Confidence Bounds for the difference between two probabilities. Biometrics 40 (4), Miettinen, O. and Nurminen, M. (985). Comparative analysis of two rates. Statistics in Medicine 4, 3-6. DOI: 0.00/sim Nam, Jun-Mo (995). Sample size determination in stratified trials to establish the equivalence of two treatments. Statistics in Medicine 4, DOI: 0.00/sim Newcombe, R.G. (998). Interval estimation for the difference between independent proportions: comparison of eleven methods. Statistics in Medicine 7, DOI: 0.00/ (SICI) ( ). Peskun, P.H. (993). A new confidence interval method based on the normal approximation for the difference of two binomial probabilities. Journal of the American Statistical Association

20 0 88 (4), DOI: 0.307/ Price, R. M. and Bonett, D. G. (004). An improved confidence interval for a linear function of binomial proportions. Computational Statistics & Data Analysis 45 (3), DOI: 0.06/S (03) Rodary, C., Com-Nougue C. and Tournade, M. (989). How to establish equivalence between treatments: A one-sided clinical trial in paediatric oncology. Statistics in Medicine 8(5), DOI: 0.00/sim Roebruck, P. and Kühn, A. (995). Comparison of tests and sample size formulae for proving therapeutic equivalence based on the difference of binomial probabilities. Statistics in Medicine 4, DOI: 0.00/sim Santner, T.J., Pradhan, V., Senchaudhuri, P., Mehta, C.R. and Tamhane, A. (007). Smallsample comparison of confidence intervals for the difference of two independent binomial proportions. Computational Statistics & Data Analysis 5, DOI: 0.06/ j.csda Sterne, T.E. (954). Some remarks on confidence of fiducial limits. Biometrika 4 (/), DOI: 0.093/biomet/ Upton, G.J.G. (98). A comparison of alternative tests for the comparative trial. Journal of the Royal Statistical Society A 45 (), DOI: 0.307/9843. Wallenstein, S. (997). A non-iterative accurate asymptotic confidence interval for the difference between two proportions. Statistics in Medicine 6, DOI: 0.00/(SICI) ( ). Woolf, B. (955). On estimating the relation between blood group and disease. Annals of Human Genetics 9 (4), DOI: 0./j tb0348.x. Wilson, E.B. (97). Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association,, 09-. DOI: 0.307/ Xu, L. and Kolassa, J. E. (007). Testing the difference of two binomial proportions: Comparison of continuity corrections for saddlepoint approximation. Communications in Statistic-Theory and Methods, 37(4), 3-8. DOI: 0.080/ Zou, G. and Donner, A. (008). Construction of confidence limits about effect measures: A general approach. Statistics in Medicine 7, DOI: 0.00/sim.3095.

21 Table Increase in nominal error regarding real size * ( = * : first entry) and % of points in the CR ( : second entry) for the two optimal methods ZW4 and AE in the indicated values of, n and n when a two-tailed test is made for H: d= at an error rate of =5%. n n ZW4 AE n n ZW4 AE Note: In none of the cases there is a failure ( <%).

22 Table Number of failures (F), average increase in error ( ) and average % of points in the CR ( ) for a selection of the 40 methods without cc compared (=5%) Method F AE ZW ZP ZW ZW ZC ZN ZE ZW

23 3 Table 3 Number of failures (F), average increase in error ( ) and average % of points in the CR ( ) for the two methods selected (with and without cc) (=5%) Method F AE ZW AEc ZW4c

24 4 Table 4 Average increase in error ( : first entry) and average % of points in the CR ( : second entry) for a selection of the 40 methods compared with and without cc. (=5%; =0) Method ZE0 ZE0c ZE ZEc ZE Method ZE3c ZW4 ZW4c AE AEc

25 5 Table 5 CIs (95%) for the difference of proportions d = p p for the Rodary et al. (989) data x /n = 83/88 and x /n = 69/76. Inference Method CI Length l AE ().90; Asymptotic ZW4 ().05; ZE0c (3).87; Exact ZY (4).35; AE (5).95; Notes: () Optimal asymptotic method: adding 0.5 to all of the data and solving iteratively the equation z =.96, with z given by the expression () for p i = p ie. () Alternative method to the previous one (which AE AE is simpler than, almost as reliable as the previous method and is recommended when the sample sizes are similar, as happens in this case): adding to all of the data the values given in expression (3) and applying the second expression (3). As 0<x i <n i, the data must be increased in the quantity.96/4=0.96, and the second expression (3) must be applied to the two samples x /n = 83.96/89.9 and x /n = 69.96/77.9. (3) Classical score method with cc (not very reliable): iteratively resolve the equation z AEc =.96, with z given by the first expression (3) for p i = p ie. (4) Optimal exact method (Herranz and Martín, 008): programme http// bioest/sdg.exe, selecting test SG and order ZY. (5) Exact method based on the order given by the statistic AE. AEc