Proceedings of he Annual Meeing of he American Saisical Associaion Augus 5-9 00 CONFIDENCE INTERVAL FOR TE DIFFERENCE IN BINOMIAL PROPORTIONS FROM STRATIFIED X SAMPLES Peng-Liang Zhao John. Troxell ui Quan Michael Lee and James A. Bolognese Merck Research Laboraories P.O. Box 000 RY 33-0 Rahay NJ 07065 ey Words: Weighed Average Approaches Likelihood Approaches Coverage Probabiliy Absrac Confidence inervals for he difference of binomial proporions beeen o reamens in sraified x samples are usually obained using he normal approximaion for a eighed average of he differences over all sraa. For his approach differen eighs such as Cochran-Manel-aenszel (CM inverse of variance (INVAR and minimum risk (MR can be used. Alernaively under he assumpion of a common difference in binomial proporions across sraa confidence inervals can be consruced using likelihood approaches including he profile likelihood (PL mehod simple asympoic (SA mehod and Wilson score (WS mehod. In his paper e firs compare he coverage probabiliies of hese 6 differen mehods ihou and ih a coninuiy correcion. A simulaion sudy shos ha for he case of even raes less han 0% in one or boh reamen groups he CM eighed average mehod ih he coninuiy correcion has he bes coverage probabiliy; hile all oher mehods can have an inadequae coverage probabiliy in some parameer seings even for large sample sizes. For he case of even raes beeen 0% and 90% he CM mehod and he PL mehod have similar performances and are he bes for small and mid-large sample sizes and all mehods have similar coverage probabiliies for large sample sizes. To oher eighed average mehods he sample size (SS eigh and equal (EQ eigh ere also compared ih he CM mehod in a differen sample size configuraion and he CM mehod is beer han or similar o he SS and EQ mehods.. Inroducion In clinical rials he parameer of ineres in many siuaions is he difference of he binomial response raes beeen o reamens. For example he raes of clinical adverse experiences beeen he es herapy and placebo are usually compared o evaluae he safey of he es herapy and in his case he difference of he raes can be he focal poin since i is easy o inerpre and meaningful for mos clinicians. When combining daa from several sudies or from paien subgroups such as age groups gender prior herapies or disease saus sraificaion for adusing for hese facors is frequenly used. This aricle considers he inerval esimaion for he difference of binomial proporions based on sraified x samples. Confidence inervals for he difference of binomial proporions beeen o reamens from sraified x samples are commonly obained using he normal approximaion for a eighed average of he differences over all sraa. For his eighed average approach he o mos popular used eighs are he Cochran- Manel-aenszel eigh (Cochran 95 Manel and aenszel 959 and Inverse of Variance eigh (Radhakrishna 965. Recenly he Minimum Risk eigh as proposed by Mehrora and Railkar (000. These hree eighing mehods esimae he same parameer if he rue underlying differences of binomial raes beeen o reamens are consan across sraa. If his homogeneiy assumpion does no hold hoever hey esimae differen parameers and some are more appropriae and meaningful han ohers. For his reason he performances of hese hree eighing mehods can only be compared under he homogeneiy condiion. The esing for homogeneiy of rae differences across sraa is no he opic of his paper bu can be conduced using he mehod given by Lui and elly (000. Under he homogeneiy assumpion confidence inervals for he common difference of binomial proporions can be consruced alernaively using likelihood approaches including he profile likelihood mehod simple asympoic mehod and Wilson score mehod. The performances of hese 6 mehods menioned above have no been ell sudied ye. In his paper e use simulaion o compare he coverage probabiliies of hese 6 differen mehods. To oher eighed average mehods he sample size eigh and equal eigh can have he same eigh as he Cochran- Manel-aenszel mehod for cerain sample size seings. ere e also compare he sample size eigh and equal eigh ih he oher eighed average mehods in a differen sample size configuraion.. Weighed Average Approaches Throughou his paper e assume ha paiens are randomly assigned ino o reamen groups ( i = 0 and here are sraa (. Le n i be he number of paiens and x i be he number of responders in sraum and reamen i. Le p i be he rue
responder rae and p i = xi / ni be he observed responder rae in sraum and reamen i. Le N i n i = = denoe he oal number of paiens in reamen i. Le = p be he rue rae difference and = p p 0 be he observed rae difference beeen he o reamens in sraum. We are ineresed in he overall reamen difference hich does no necessarily need he homogeneiy of he bu does require ha here be no qualiaive reamen-by-sraum ineracion. In he eighed average approaches he overall reamen difference is usually defined by and esimaed by = = = = here is he eigh for sraum ih =. The = ( α 00% confidence inerval for is given by ± z V( α / = ( here z c is he (-c percenile of he sandard normal disribuion and V( = ( p ( p / n + ( p ( p / n. ( 0 0 0 Coninuiy correcions are commonly used o deal ih small sample sizes for binomial daa. We use he coninuiy correcion facor suggesed by Mehrora and Railkar (000. This coninuiy correcion facor is approximaely 3/8 imes ha used in Manel and aenszel (959 since he laer as found o be overly conservaive. The ( α 00% confidence inerval for ih his coninuiy correcion is given by 3 n 0 n / ( ± zα V +. (3 6 = = 0 n + n The o mos popular eighs are he Cochran- Manel-aenszel (CM eigh and Inverse of Variance (INVAR eigh. The CM eigh firs discussed by Cochran (95 and hen by Manel and aenszel (959 is proporional o he harmonic mean of he sample sizes in each sraum: = ( n0 + n / ( n0 + n. ( = The INVAR eigh sudied by Radhakrishna (965 is proporional o he inverse of he variance of he observed rae difference in each sraum: = V /( V = (5 here V ( = V is given by (. The CM mehod gives more eigh o sraa ha have more paiens. The INVAR mehod pus more eigh on sraa ha have greaer precision o esimae reamen differences. Recenly Mehrora and Railkar (000 proposed he Minimum Risk (MR eigh. The argeed overall reamen difference for he MR mehod is f ( p p ih f = = 0 ( n + n / ( n + n. (6 f = The MR eigh is derived by minimizing he mean squared error loss: E( f. For he formula of he MR eigh see Mehrora and Railkar (000. To oher eighs are he Sample Size (SS Weigh = f (here f is as above and he Equal (EQ Weigh = /. For prognosic sraificaion facors (such as sex and age he SS eigh f represens he fracion of paiens ha enered sraum in he arge paien populaion. The SS mehod also gives more eigh o sraa ha have more paiens bu in a differen manner from he CM mehod. When n0 c * n 0 0 = = ih a consan c for all he SS and CM mehods have he same eigh. The EQ mehod reas all sraa equally. When he number of paiens across sraa are he same in each reamen group he EQ and CM mehods have he same eigh. The raionale and heoreical meris of hese eighing mehods have been discussed in Mehrora and Railkar (000 and Mehrora (00. The INVAR and MR eighs involve he esimaed variance V. If he observed variance V is zero in a sraum he INVAR and MR eighs can no be compued. To overcome his problem he procedure of adding 0.5 has been commonly used (Mehrora 00. Tha is for he esimaion of he variance V in ( e replace = 0 ih p i + (0.5 / ni and p i = ih p i (0.5 / ni. This adding 0.5 correcion is applied only o he esimaion of he variances V bu no o he observed rae difference = p p 0. So e refer his procedure as variance correcion hroughou his paper. Alhough his variance correcion is no required for he CM SS and EQ eighs i can improve he coverage probabiliy hen he sample sizes are small. The performances of hese eighing mehods can only be compared under he condiion ha he are consan across sraa. The comparisons ere made ihou and ih he coninuiy correcion as given in p i
( and (3. The variance correcion as used for he INVAR and MR mehods regardless of he sample sizes and for he CM SS and EQ mehods hen he sample size N or N as less han 00. 3. Likelihood Approaches The likelihood approaches are derived under he assumpion of a common difference across sraa. Tha is e assume ha = p for =.... Under his assumpion he parameers are reduced o θ = ( p 0 p0... p0 and he likelihood funcion is G( θ here F( x = F = 0 n 0 n x p 0 F( x n p 0 + (7 x n x ( x n p = p ( p. For he purpose of compuaion i is necessary o se 0 0 =. I can be verified ha here exiss an unique maximum likelihood esimaor (MLE * * * * θ = ( p p... p * 0 0 0 for (7. This MLE can be compued using he Neon-Raphson ieraion mehod or a search algorihm. We used he search algorihm for our simulaion sudy since i is more reliable for he lo even rae case. Le ( ~ p ~ p ~ 0( 0 (... p0 ( be he MLE for (7 for a given value. Then he maximum likelihood equaion for ~0 p ( becomes a cubic equaion and has an unique closed-form soluion (see Mieinen and Nurminen 985. The firs likelihood approach discussed here is he profile likelihood (PL mehod (also called he likelihood raio mehod in he seing of he hypohesis ~ es. Le θ ( = ( ~ p ( ~ p (... ~ p (. The 0 0 0 ( ( α 00% confidence limis L based on he PL mehod are he values of such ha * ~ [log( G ( θ log( G( θ ( ] = z. (8 α / The confidence limis L and are in he inervals * ( and * ( respecively and can be obained numerically using he bisecion mehod. The second likelihood approach is he Wald-ype simple asympoic (SA mehod. The SA confidence inerval is consruced using * and is esimaed variance. Based on asympoic heory e have proved * ha he variance of can be esimaed by * V( = / = i= 0 ( p * i ( p * i / n i (9 * * * here p = p0 +. So he ( α 00% confidence inerval for based on he SA mehod is * * ± zα / V(. (0 The hird likelihood approach is he Wilson Score (WS mehod. Wilson (97 proposed he score mehod for consrucing he confidence inerval for he single proporion of one binomial sample. To apply he WS mehod o our case he idea is o consruc he * confidence inerval using and is variance under he rue value assuming ha p for all. Le ~ p ( = ~ p ( + and 0 ~ ~ ~ p ( ( ( 0 (. 0 0 i pi n + n V = i= ni n + n ( * I can be proved ha he variance of under he given rue value can be esimaed by ~ [ ( ]. ( ~ V ( = / V = So he ( α 00% confidence limis ( L based on he WS mehod are he values of such ha * ~ ( / V ( = zα /. (3 The soluions L and of (3 can be compued numerically using he bisecion mehod. The PL and WS mehods proposed by Mieinen and Nurminen (985 are differen from he ones sudied in his paper. Their PL mehod uses he unresriced MLE β = ( p 0... p0 p... p insead of replaces z α / by α * θ and χ in (8. oever heir PL mehod does no have confidence limis for some daa. Their WS mehod defines he ( α 00% confidence limis L o be he values of such ha ( [ ( p = p ~ ] /( V ( 0 = zα / = This mehod requires choice of he eigh. The eigh hey suggesed depends on he value of and he compuaion becomes more complex. Therefore heir WS mehod is no evaluaed here. Noe ha for he unsraified case ( = he PL and WS mehods of Mieinen and Nurminen and our proposed PL and WS mehods become he same respecively. The performances of he PL SA and WS mehods given in (8 (0 and (3 are evaluaed ihou and ih he coninuiy correcion. The same coninuiy correcion facor (3/6[/ ( n n /( n + n ] as in = he previous secion is used. The coninuiy correced versions of he confidence inervals from (8 (0 and (3 are obained by subracing and adding he 0 0.
coninuiy correcion facor from he loer limi and o he upper limi respecively.. Simulaion Resuls.. Simulaion Mehods We compare he performances of differen mehods discussed in Secions and 3 under he seing of a common difference across sraa (i.e. p for all. The rue underlying parameers are p 0 p0... p0 and. The simulaion sudy as conduced under he folloing seings for sraa sample sizes and parameers.. Number of sraa: =.. Toal sample size per reamen: N 0 = N = N = 0 00 00 500 000. Number of paiens per reamen in sraum : n 0 n = = N * here =(0.5 ( 3 0.5 0.5 0.5 and (0. 0. 0.35 0.35 are he o cases represening he balanced and unbalanced sraum frequencies. 3. The rue reamen difference: = 0 0.05 0. 0... Three cases are considered regarding he range of he rue ( p 0 p0 p0 p0 : (a. Thiry differen ( p 0 p0 p0 p0 randomly chosen from 0 o 0.. (b. Teny differen ( p 0 p0 p0 p0 randomly chosen from 0. o 0.5. (c. Teny differen ( p 0 p0 p0 p0 randomly chosen from 0. o 0.7. The same chosen ( p 0 p0 p0 p0 as used for differen mehods N ( and. The case (a is he lo even rae case represening he ypical siuaion in analyses of clinical adverse experience daa from clinical rials. For (b and (c he raes in boh reamen groups ill be beeen 0. and 0.9. The SS and CM mehods have he same eigh for he above sample size seings. For ( =(0.5 0.5 0.5 0.5 he EQ and CM mehods also have he same eigh. Comparisons of he SS and EQ mehods ih he CM mehod are discussed in secion.3. ere and in secion. e focus on he CM INVAR and MR mehods as ell as he 3 likelihood mehods. For each se of N ( ( p 0 p0 p0 p0 and he coverage probabiliy as esimaed by simulaing 0000 cases of sraified x samples for he eighed average approaches and simulaing 000 cases for he likelihood approaches as he laer require much longer compuaion ime. When he rue coverage is greaer han 0.9 he sandard error of he esimaed coverage probabiliy is abou 0.007 for he eighed average approaches and abou 0.0075 for he likelihood approaches. The rue coverage is considered o be less han if he esimaed coverage is less han and 0.97 (3 sandard error limi off for he eighed average approaches and he likelihood approaches respecively. For each N ( and he esimaed individual coverage probabiliies corresponding o differen ( p 0 p0 p0 p0 and heir firs quarile median and hird quarile are ploed. These plos are similar o he box plo bu do no have hiskers... Resuls The confidence inerval (CI ih he coninuiy correcion has a beer coverage probabiliy han ha ihou he coninuiy correcion. ence e presen he resuls only for he coninuiy correced CIs. The resuls are generally similar beeen ( = (0.5 0.5 0.5 0.5 and ( = (0. 0. 0.35 0.35. Because of he limied space here e only sho he resuls for case (a ih ( = (0.5 0.5 0.5 0.5 in Figures o 5. The figures for (b and (c can be obained from auhors per reques. For he case (a he CM mehod has he bes coverage in he sense ha he coverage of is 95% CI is closer o he arge han he ohers (Figures o 5. The coverage of he 95% CI for he CM mehod is a leas 0.98. The INVAR and MR mehods do no have an adequae coverage even for some large sample sizes (see N=00 o 000 and = 0.05 0. for INVAR; see N=500 000 and = 0 0.05 for MR. For N=0 and = 0 o 0. he coverage for he hree eighed average approaches is conservaive (ell above due o he variance correcion. As discussed previously he INVAR and MR mehods need his correcion o handle zero variance. For he CM mehod ihou his correcion he coverage for N=0 is beeen o 0.997 for = 0 and beeen 0.9 and 7 for = 0.05 o 0.. Alhough a conservaive coverage is no good i is probably beer han an inadequae one. Therefore e recommend using he variance correcion for he CM mehod hen N is less han 00. Among he likelihood approaches he PL mehod is he bes. Comparing ih 0.97 (3 sandard error limi off he PL mehod has an adequae coverage for = 0. and 0. hen N 00. oever he coverage of he PL mehod can be loer han 0.97 for = 0 (see N=500 in Figure ; he coverage is abou 0.98. The SA and WS mehods have a very poor coverage for = 0 ih almos all N and for = 0.05 ih N 00. One explanaion is ha he esimaed variances in formulas (9 and ( are no good for small N and lo raes because hey involve he inverse of he individual
variances. In summary he CM mehod is he bes among hese 6 mehods for he case (a. For he cases (b and (c by comparing he resuls of 0000 simulaions for he 3 eighed average mehods and also by comparing he resuls of 000 simulaions for all 6 mehods e can summarize he resuls as follos. When N 00 he CM and PL mehods (and he WS mehod for case (c perform similarly and are he bes. When N 00 all 6 mehods have similar coverage probabiliies..3. Addiional Simulaion Resuls We also compared he SS and EQ mehods ih he CM mehod under he folloing arifical sample size seing. The oal sample size for he reamens is: * N0 = N = N = 0 00 00 500 000. The number of paiens in he sraa for he reamens is: n n n = (0. 0. 0. 0. *(N / ( 0 0 03 n0 ( n n n n = (0. 0. 0.35 0.35 * N. For he lo even rae case (a he CM mehod is sill he bes for all N and he SS and EQ mehods also can have an inadequae coverage. For cases (b and (c he CM mehod is eiher beer han or similar o he SS and EQ mehods. Deailed figures are available from auhors. 5. Discussion and Conclusion We have compared he coverage probabiliies of 8 differen mehods under he condiion ha he rue rae differences are consan across sraa. The likelihood approaches require his homogeneiy assumpion and involve complex compuaions ye hey do no have a beer coverage han he CM eighed average mehod. Therefore he likelihood approaches are no recommended. The eighed average approaches excep he INVAR mehod do no require he homogeneiy assumpion and sill define and esimae a meaningful overall reamen difference as long as here is no qualiaive reamen-by-sraum ineracion. The simulaion sudy has shon ha in he case of even raes less han 0% in one or boh reamen groups he CM mehod is he bes and has a coverage probabiliy very close o or above he arge ; hile all oher mehods can have an inadequae coverage. For he case of he even raes beeen 0% and 90% he CM mehod is sill he bes among he eighed average approaches hen N 00 and all mehods have similar coverage probabiliies hen N 00. In pracice he choice of an appropriae eigh for he eighed average approaches can be made based on he above findings and he sraificaion facor (prognosic versus non-prognosic. For prognosic sraificaion facors (such as sex and age a meaningful overall reamen difference is naurally defined by = ( p p ih he SS eigh = 0 = f. The choice is beeen he SS eigh and MR eigh since boh are argeing o esimae his overall reamen difference. Since paiens are randomly assigned ino reamen groups equally or in a cerain raio in clinical rials i is expeced ha n0 = c * n ih a consan c for all ould be nearly rue and he SS eigh is very close o he CM eigh. So for prognosic sraificaion facors in general e recommend using he SS mehod o obain he confidence inerval. When he even raes are beeen 0% and 90% and he sample sizes are large he MR mehod can be used since i usually has a (slighly shorer confidence inerval idh han he SS mehod. If he SS and CM eighs are very differen simulaion evaluaions of he SS and MR mehods for he real siuaion are needed. For non-prognosic sraificaion facors (such as sudy cener or differen sudies he definiion of he overall reamen difference is more subecive (unless he homogeneiy condiion holds and depends on he choice of he eigh. Based on he performance in general e recommend using he CM mehod o obain he confidence inerval. When he even raes are beeen 0% and 90% and he sample sizes are large he INVAR eigh can be used if he homogeneiy assumpion holds because he INVAR mehod has he shores confidence inerval idh under he homogeneiy condiion. Acknoledgmen The auhors are graeful o Prof. Sco Zeger for suggesing he PL mehod and moivaing his research. The auhors also hank Drs. Devan Mehrora Ji Zhang and Thomas Capizzi for heir very helpful suggesions. References. Cochran W.G. (95. Some mehods for srenghening he common chi-square ess. Biomerics 0 7-5.. Manel N. and aenszel W. (959. Saisical aspecs of he analysis of daa from rerospecive sudies of disease. Journal of he Naional Cancer Insiue 79-78. 3. Radhakrishna S. (965. Combinaion of resuls from several x coningency ables. Biomerics 86-98.. Mehrora D. V. and Railkar R. (000. Minimum risk eighs for comparing reamens in sraified binomial rials. Saisics in Medicine 9 8-85.
5. Lui.-J. and elly C. (000. A revisi on ess for homogeneiy of he risk difference. Biomerics 56 309-35. 6. Mieinen O. S. and Nurminen M. (985. Comparaive analysis of o raes. Saisics in Medicine 3-6. 7. Mehrora D. V. (00. Sraificaion Issues ih binary endpoins. Drug Informaion Journal o appear. 8. Wilson E. B. (97. Probable inference he la of succession and saisical inference. Journal of he American Saisical Associaion 09-. Coverage Probabiliy for 95% CIs.00 Figure 3 N=00 = 0 = 0.05 = 0. = 0. Figures o 5 Empirical Coverage Probabiliy For 95% CIs Wih Coninuiy Correcion ( =(0.5 0.5 0.5 0.5 (a 30 Differen ( p 0 p0 p0 p0 from 0 o 0. ( 0000 Simulaions for CM INVAR and MR ( 000 Simulaions for PL SA and WS Coverage Probabiliy for 95% CIs.00 Figure N=0 = 0 = 0.05 = 0. = 0. Coverage Probabiliy for 95% CIs.00 3 5 6 3 5 6 3 5 6 3 5 6 CM ( INVAR ( MR (3 PL ( SA (5 WS (6 Figure N=500 = 0 = 0.05 = 0. = 0. 3 5 6 3 5 6 3 5 6 3 5 6 CM ( INVAR ( MR (3 PL ( SA (5 WS (6 Figure 5 N=000 = 0 = 0.05 = 0. = 0. Coverage Probabiliy for 95% CIs.00 3 5 6 3 5 6 3 5 6 3 5 6 CM ( INVAR ( MR (3 PL ( SA (5 WS (6 Figure N=00 = 0 = 0.05 = 0. = 0. 3 5 6 3 5 6 3 5 6 3 5 6 CM ( INVAR ( MR (3 PL ( SA (5 WS (6 Coverage Probabiliy for 95% CIs.00 3 5 6 3 5 6 3 5 6 3 5 6 CM ( INVAR ( MR (3 PL ( SA (5 WS (6 Noe: The coverage of he SA mehod for N=0 and = 0.05 is beeen 0.5 and 0.76 and hence does no appear in Figure.