TESTING FOR HOMOGENEITY IN COMBINING OF TWO-ARMED TRIALS WITH NORMALLY DISTRIBUTED RESPONSES

Transcription

1 Sankhyā : The Indian Journal of Statistics 2001, Volume 63, Series B, Pt. 3, pp TESTING FOR HOMOGENEITY IN COMBINING OF TWO-ARMED TRIALS WITH NORMALLY DISTRIBUTED RESPONSES By JOACHIM HARTUNG and DO GAN ARGAÇ University of Dortmund, Germany SUMMARY. We deal with testing for homogeneity of mean differences in a series of independent two-armed trials with normally distributed outcomes assuming heteroscedastic variances both among the two arms in each trial and between the various trials. We consider tests for investigating this hypothesis under these nonstandard circumstances using classical analysis of variance techniques and concavity arguments. By way of simulation, the performance of the tests with respect to the attained significance levels and power is compared. The proposed tests keep their level and attain reasonable power. 1. Introduction Consider a series of independent experiments, each consisting of two arms with normally distributed outcomes. A question of interest is to test whether all the experiments share a common effect. This hypothesis is called the homogeneity hypothesis. According to Chalmers (1991), Hardy and Thompson (1998) and Thompson and Sharp (1999), the question of homogeneity is an important part of any meta-analysis. Testing for homogeneity may also be seen as testing for the presence of treatment by centre interaction. The hypothesis of homogeneity corresponds to testing the homogeneity of the means in the one-way model of analysis of variance. The goal of the present paper is to propose and extend some known or refined tests in the one-way classification to the case of independent two-armed experiments with normally distributed responses. This is done using a classical momentmatching approach due to Satterthwaite (1946) and Patnaik (1949). However, the approach presented is not limited to normal outcomes, and is general enough to deal with other type of response variables (see Whitehead and Whitehead (1991) and Lipsitz et al. (1998)). Paper received November 2000 ; Revised May AMS (1991) subject classification. Primary 62J10; secondary 62P10. Key words and phrases. Heteroscedastic variances, homogeneity, meta-analysis, two-armed experiments, refined Cochran test, refined Welch test, Behrens-Fisher problem.

2 Homogeneity in combined two-armed trials 299 The main result of the paper is that the proposed extensions of the modified Brown-Forsythe test and of the approximate ANOVA F-test keep their level and attain reasonable power in most cases considered in the simulations. The same holds with the proposed refined Welch test which however in some constellations attains considerably higher power. In section 2, the one-way model is briefly reviewed and the new tests are developed in sections 3 and 4. Their finite sample properties are investigated by way of simulation in section 5. Finally, three real data examples are discussed in detail to demonstrate the application of the new tests. 2. Approved Tests in the One-Way ANOVA Model We review briefly the one-way classification of analysis of variance and the tests which may be used to test for homogeneity of the means. Let y ij be the observation on the j-th subject of the i-th population, i = 1,..., K and j = 1,..., n i, K 2 and n i 2, y ij = µ i + e ij = µ + β i + e ij ; i = 1,..., K, j = 1,..., n i, (2.1) where µ is the common mean for all the K populations, β i is the effect of population i with i=1 β i = 0, and e ij, i = 1,..., K, j = 1,..., n i, are error terms which are assumed to be mutually stochastically independent and normally distributed with E(e ij ) = 0, V ar(e ij ) = σ 2 e,i ; i = 1,..., K, j = 1,..., n i. We consider the homogeneity hypothesis H 0 : µ 1 = = µ K, and the following test statistics are used for testing this hypothesis: (i) ANOVA F-test. The F-test, F, is given by F = N K K 1 i=1 n i(ȳ i. ȳ.. ) 2 i=1 (n, (2.2) i 1)s 2 e,i with N = i=1 n i, ȳ i. = j=1 y ij/n i, ȳ.. = i=1 n iȳ i. /N, and s 2 e,i = n i j=1 (y ij ȳ i. ) 2 /(n i 1). This test was originally meant to test for equality of population means under variance homoscedasticity and has in this case under H 0 an F distribution with K 1 and N K degrees of freedom, denoted by F K 1, N K. The null hypothesis H 0 is rejected at level α if F exceeds the corresponding (1 α)- quantile, i. e. if F > F K 1, N K;1 α. (ii) Cochran test. The statistic K K C = w i (ȳ i. h j ȳ j. ) 2, (2.3) i=1 where w i = (s 2 e,i /n i) 1, h i = w i / k=1 w k, was proposed by Cochran (1937), and then modified by Welch (1951), confer also Cochran (1954). j=1

3 300 joachim hartung and do gan argaç Cochran s test is the standard test for testing homogeneity in meta-analysis, confer Whitehead and Whitehead (1991), and Normand (1999). Under H 0, the Cochran statistic is distributed asymptotically as a central χ 2 -variable with K 1 degrees of freedom. Reject H 0 at level α if C > χ 2 K 1;1 α. (iii) Welch test. The Welch test is given by W = i=1 w i(ȳ i. j=1 h jȳ j. ) 2 (K 1) + 2 (K 2) (K + 1) 1 K i=1 (n i 1) 1 (1 h i ) 2, (2.4) and Welch (1951) proposed to approximate its distribution using an F-variable. Under H 0, the statistic W has an approximate F distribution with K 1 and ν W degrees of freedom, where K 2 1 ν W = 3 K i=1 (n i 1) 1 (1 h i ). (2.5) 2 The hypothesis H 0 is rejected at level α if W > F K 1,νW ;1 α. (iv) Brown-Forsythe test. This test is also known as the modified F-test and is given by i=1 B = n i (ȳ i. ȳ.. ) 2 i=1 (1 n. (2.6) i/n)s 2 e,i Brown and Forsythe (1974) use a Satterthwaite approximation to derive the null distribution of the statistic B. When H 0 is true, B is distributed approximately as an F variable with K 1 and ν degrees of freedom where [ K ] 2 i=1 (1 n i/n)s 2 e,i ν = i=1 (1 n i/n) 2 s 4 e,i /(n i 1). (2.7) We reject H 0 at level α if B > F K 1,ν;1 α. (v) modified Brown-Forsythe test. Mehrotra (1997) developed the following test B i=1 = n i(ȳ i. ȳ.. ) 2 i=1 (1 n (2.8) i/n)s 2 e,i in an attempt to correct a flaw in the original Brown-Forsythe test. The flaw in the Brown-Forsythe testing procedure, as identified by Mehrotra (1997), is in the specification of the numerator degrees of freedom. Specifically, Brown-Forsythe used K 1 numerator degrees of freedom whereas Mehrotra (1997) used a Box (1954) approximation to obtain the numerator degrees of freedom, ν 1, where [ K i=1 (1 n i/n)s 2 e,i ν 1 = [ K i=1 s4 e,i + K ] 2 i=1 n is 2 e,i /N (2.9) K 2 i=1 n is 4 e,i /N ] 2

4 Homogeneity in combined two-armed trials 301 and the denominator degrees of freedom ν is given in (iv) above. Under H 0, B is distributed approximately as an F variable with ν 1 and ν degrees of freedom. The null hypothesis H 0 is rejected at level α if B > F ν1,ν;1 α. (vi) approximate ANOVA F-test. Asiribo and Gurland (1990) based their test on F i=1 = n i(ȳ i. ȳ.. ) 2 i=1 (1 n. (2.10) i/n)s 2 e,i Note that F equals B = B. Under H 0, the test statistic F is distributed approximately as an F variable with ν 1 and ν 2 degrees of freedom where ν 1 is as given in (v) above, [ K i=1 (n i 1)s 2 e,i ν 2 = i=1 (n, (2.11) i 1)s 4 e,i and H 0 is rejected at level α if F > F ν1,ν 2 ;1 α. We notice that the numerator degrees of freedom for F and B are equal. The difference between the two test procedures is in the denominator degrees of freedom in the unbalanced case. The tests mentioned above may be divided roughly into two groups, the ANOVA F-test type statistics (Brown-Forsythe test, modified Brown-Forsythe test, approximate ANOVA F-test), which use a Satterthwaite type approximation to approximate the null distribution of the statistics, and the Cochran-Welch type tests (Cochran test, Welch test), which use an asymptotic null distribution and a refined approximation of the asymptotic null distribution based on a power series approach for small sample sizes. ] 2 3. Refined Cochran and Welch Tests Next, we will propose refined versions of the Cochran and Welch statistics, which are based on the following observation, where a denotes the transpose of a vector a R K. We omit its proof. Lemma 3.1 Cochran s test statistic C(w) = i=1 w i(ȳ i. j=1 w jȳ j. / j=1 w j) 2, given in (2.3), is a montonically increasing and concave function of the weights w = (w 1,..., w K ). Let now n i 4 and ω = (ω 1,..., ω K ) with ω i = (σ 2 e,i /n i) 1, i = 1,..., K. Since the estimators (n i 1)s 2 e,i /σ2 e,i are distributed like a χ2 n i 1 variable, we obtain the expectation of w i = (s 2 e,i /n i) 1 as E(w i ) = n i 1 n i 3 1 σ 2 e,i /n i = n i 1 n i 3 ω i, i = 1,..., K. From that observation, we can get unbiased weights by taking w i = (n i 3)(n i 1) 1 w i and a refined version of Cochran s test, C = C(w ) = C((w 1,..., w K ) ). Thus by Lemma 3.1 and by applying Jensen s inequality, we derive C(w ) C(w), and E {C(w ) ȳ 1.,..., ȳ K. } C (E(w )) = C(ω), (3.1)

5 302 joachim hartung and do gan argaç where the expectation is with respect to the weights w, and note also that the weights w 1,..., w K and the means ȳ 1.,..., ȳ K. are independent. A second refined statistics is now developed by applying the approximation suggested by Welch (1951) on the refined Cochran s statistic C to give a refined Welch type statistic W. We remark that a similar approach is used by Böckenhoff and Hartung (1998) for correcting the variance estimator in the context of testing for significance of the overall treatment effect in a homogeneous fixed effects model of meta-analysis. 4. Extension to Two-armed Trials Now consider K 2 independent experiments each with two arms. Let X ij denote the j-th observation in the first arm of the i-th experiment and let Z ik denote the k-th observation in the second arm of the i-th experiment, i = 1,..., K, j = 1,..., m i, k = 1,..., l i. Note that we do not assume that the number of observations in the two arms of each experiment are balanced. The observations of the first arm are assumed to be normally distributed, i. e. X ij N(η Xi, σx 2 i ), and also for the second arm we assume Z ik N(η Zi, σz 2 i ). We assume that the observations X 11, Z 11,..., X ij, Z ik,..., X KmK, Z KlK are mutually stochastically independent. Let X i = m i j=1 X ij/m i and SX 2 i = m i j=1 (X ij X i ) 2 /(m i 1) denote the mean and variance, respectively, of the first arm in the i-th trial, and Z i = l i k=1 Z ik/l i and SZ 2 i = l i k=1 (Z ik Z i ) 2 /(l i 1) denote the mean and variance, respectively, of the second arm in the i-th trial. It follows from the assumptions that X i N(η Xi, σ 2 X i /m i ) and Z i N(η Zi, σ 2 Z i /l i ), (4.1) and the variances are distributed as S 2 X i σ2 X i m i 1 χ2 m i 1 and S 2 Z i σ2 Z i l i 1 χ2 l i 1. (4.2) The means and variances are also independent. The difference of the means X i Z i is an unbiased estimator of the treatment effect η Xi η Zi, and is distributed as X i Z i N(η Xi η Zi, σ 2 X i /m i + σ 2 Z i /l i ). (4.3) Now to make use of the one-way ANOVA model in section 2, let y i = X i Z i, µ i = η Xi η Zi, and σi 2 = σ2 X i /m i + σz 2 i /l i. We are interested in testing H 0 : µ 1 = = µ K. Note that in the one-way model of analysis of variance, the variance s 2 e,i is distributed as σe,i 2 χ2 n i 1/(n i 1), that is a scaled χ 2 -distribution with n i 1 degrees of freedom. To extend the tests presented in the previous sections, the main task is to take a suitable estimator ˆσ i 2 of σi 2 above and to define an approximate χ 2 ν i -distribution for ν i ˆσ i 2/E(ˆσ2 i ), i = 1,..., K. According to Satterthwaite (1946) and Patnaik (1949), the first two moments of the involved distributions are equated, yielding ν i = 2{E(ˆσ i 2)}2 /var(ˆσ i 2). An

6 Homogeneity in combined two-armed trials 303 unbiased estimator of σi 2 is given by ˆσ2 i = S2 i = S2 X i /m i + SZ 2 i /l i. For the variance, we obtain var(si 2) = (1/m2 i ) 2σ4 X i /(m i 1) + (1/li 2) 2σ4 Z i /(l i 1), which we estimate by its best unbiased estimator provided by Hartung and Voet (1986), (1/m 2 i ) 2S4 X i /(m i + 1) + (1/li 2) 2S4 Z i /(l i + 1), i = 1,..., K. We are now in the position to extend the presented tests to two-armed experiments. We replace in the formulas of section 2 and 3 the degrees of freedom n i 1 by ν i just derived, i. e. n i = ν i + 1 and N = i=1 ν i + K, the mean ȳ i. is replaced by y i = X i Z i and ȳ.. by i=1 (ν i +1)y i / i=1 (ν i +1), and s 2 e,i /n i is interchanged by Si 2, or s2 e,i by (ν i + 1)Si 2. For illustration, we will only show the transformed version of the ANOVA F-test, F = (N K) i=1 n i(ȳ i. ȳ.. ) 2 /{(K 1) i=1 (n i 1)s 2 e,i } = i=1 ν i K 1 i=1 (ν i + 1) {y i i=1 (ν i + 1)y i / i=1 (ν i + 1) i=1 ν i(ν i + 1)Si 2 } 2 = F X, and the corresponding critical value is given by F K 1,. The extended ν i;1 α i=1 versions of the test statistics will be denoted with a subscript X. In order not to exclude cases with small degrees of freedom, we have replaced ν i by max(3, ν i ) in the following computations with the extended refined Cochran and Welch tests, CX and W X. 5. Simulations By way of simulation ( runs for each constellation) the performance of the tests is investigated for K = 6 and K = 12 experiments to be combined, where for K = 12 the patterns of K = 6 are replicated once in order to demonstrate this effect of doubling with respect to K. The analogous effect of doubling the sample sizes of each trial is also illustrated in Designs C, D, and E of Table 1. Using different constellations, given in Table 1, of the sample sizes, m i and l i, and the standard deviations, σ Xi and σ Zi, we obtained the simulated actual significance levels. The refined Welch test is the only test that keeps the level in all cases, see Tables 2 and 3. The modified Brown-Forsythe test and the approximate ANOVA F-test are liberal in case of Design B, but keep the level otherwise. Besides Welch s test, the remaining tests cannot be recommended generally since they do not attain acceptable levels in most cases. Further constellations with nearly identical sample sizes in both arms of each experiment are simulated within the discussions of the clinical trials considered in the next section, where also some power results are provided.

7 304 joachim hartung and do gan argaç Table 1. Sample Designs. K=6 K=12 Design Arm i A 1 m i σ Xi l i once replicated σ Zi B 1 m i the designs σ Xi l i for K = 6 σ Zi C 1 m i σ Xi l i σ Zi C 1 m i σ Xi l i σ Zi D 1 m i σ Xi l i σ Zi E 1 m i σ Xi l i σ Zi D 1 m i σ Xi l i σ Zi E 1 m i σ Xi l i σ Zi Table 2. Actual Simulated Significance Levels for the Designs of Table 1, (nominal level 5%) for K=6. ˆα% Design F X C X CX W X WX B X BX FX A B C C D E D E

8 Homogeneity in combined two-armed trials 305 Table 3. Actual Simulated Significance Levels for the Designs of Table 1, (nominal level 5%) for K=12. Design F X C X CX W X WX B X BX FX A B C C D E D E ˆα% 6. Real Data Examples To illustrate the application of the various homogeneity tests discussed above, we give three examples, two from the area of meta-analysis of clinical trials and one from a pilot study for a multicentre trial. We have considered two meta-analyses with different constellations concerning sample sizes and observed means and variances. Such situations are comparable to usual multicentre studies. Therefore, we present an example of a small multicentre trial which is not of the size of a usual metaanalysis. We include power calculations ( runs each) of our tests here in the examples because the observed means give realistic hints how to choose the alternative, which otherwise would be artificial and difficult to motivate with so many possibilities of setting the parameters involved. As alternatives, we have chosen the observed means and fractions of the observed means, respectively. In the discussion of the examples, we focus on the three tests approved in the simulations in the previous section: the extended refined Welch test W X, the extended modified Brown-Forsythe test B X, and the extended approximate ANOVA F-test F X. Example 1. The data are taken from Li, Shi and Roth (1994), confer Table 4. In eight randomized controlled trials the effectiveness of a new drug, amlodipine, in the treatment of angina was examined. The response variable, the change in work capacity, was compared for patients who received either the drug or placebo. The change in work capacity is the ratio of the exercise time after the patient receives the intervention (drug or placebo) to before the patient receives the intervention. The logarithms of the observed changes are assumed to be approximately normally distributed.

9 306 joachim hartung and do gan argaç Table 4. Change in work capacity in the treatment of angina, data taken from Li, Shi and Roth (1994). Amlodipine Placebo Study Sample Standard Sample Standard size Mean deviation size Mean deviation The variance estimators differ considerably, hence the assumption of variance homoscedasticity seems not to be justified. We perform now the tests presented above to investigate the homogeneity question among the studies. The results are summarized in Table 5. Table 5. Value of the test statistics and corresponding critical values (level α = 5%) for the data of Table 4. Statistic Value of the statistic Critical value F X W X WX B X BX FX C X CX Only the Brown-Forsythe test rejects the null hypothesis. For illustration, we simulated the empirical level and power using the sample sizes and the variance estimators given in Table 4, treating the variance estimators as the true values. As alternatives, the observed means and a half of them were taken, respectively. We obtained the results given in Table 6. Table 6. Empirical level and power values for the data of Table 4, (nominal level 5%). α = 5% F X C X C X W X W X B X B X F X Empirical level Alternative 1/2 observed means observed means The extended refined Welch test, the extended modified Brown-Forsythe test and the extended approximate ANOVA F-test attain similar and acceptable levels. The extended refined Welch test is more powerful compared to the other two tests.

10 Homogeneity in combined two-armed trials 307 Example 2. The data are taken from Normand (1999), reproduced in Table 7. This second example lists nine studies where specialist inpatient stroke care (Specialist care) was compared to the conventional non-specialist care (Routine management) in general medical wards. The length of stay during the acute hospital admission is used as the response variable. The central hypothesis of interest is whether specialist stroke unit care will result in a shorter length of hospitalization compared to routine management. Table 7. Care for stroke patients from nine studies, data taken from Normand (1999). Specialist care Routine management Study Sample Standard Sample Standard size Mean deviation size Mean deviation Performing the tests in this example, yields the results given in Table 8. Table 8. Value of the test statistics and corresponding critical values (level α = 5%) for the data of Table 7. Statistic Value of the statistic Critical value F X W X WX B X BX FX C X CX All of the tests reject the null hypothesis of homogeneity. The simulation results concerning the attained level and the power are provided in Table 9. Table 9. Empirical levels and power values for the data of Table 7, (nominal level 5%). α = 5% F X C X C X W X W X B X B X F X Empirical level Alternative 1/4 observed means /3 observed means We observe here that the extended refined Welch test, the extended modified Brown-Forsythe test and the extended approximate ANOVA F-test have the same

11 308 joachim hartung and do gan argaç empirical level, but in this case the refined Welch test is considerably more powerful. We have also simulated the power with the observed means and in this case all of the tests attain a power value of 100%. With the alternative chosen as half of the observed means, the ANOVA F-test has a power value of 94.9, the power of the remaining tests ranges between 98.4% 100%. Example 3. To give an impression of the performance of our tests in a study with few unbalanced heteroscedastic centres with small total sample size and one particularly small centre, we consider the following: In a pilot study for a multicentre trial on the treatment of stroke, the effect of a calcium antagonist versus placebo is compared. The response is measured on a validated positively directed stroke scale, ranging from The data of the baseline corrected mean treatment effects after 28 days of treatment are presented in Table 10. Table 10. Multicentre pilot study on the treatment of stroke. Drug Placebo Centre Sample Standard Sample Standard size Mean deviation size Mean deviation Berlin Göttingen Rendsburg Ulm The values of the test statistics and the corresponding critical values are given in Table 11. Table 11. Value of the test statistics and corresponding critical values (level α = 5%) for the data of Table 10. Statistic Value of the statistic Critical value F X W X WX B X BX FX C X CX Only the extended Brown-Forsythe and extended modified Brown-Forsythe tests, and the extended approximate ANOVA F-test do reject the null hypothesis. The empirical levels and power values are summarized in Table 12. The extended modified Brown-Forsythe test and the extended approximate ANOVA F-test attain acceptable empirical levels. Observe that the extended refined Welch test is liberal which seems to contradict the acceptance of the null hypothesis in this case, but note that, in general, the behaviour of a test in a particular example cannot be predicted from simulations. Also, the extended refined Welch test is more powerful than the other two tests, but this is largely due to its liberality witnessed here in this example.

12 Homogeneity in combined two-armed trials 309 Table 12. Empirical level and power values for the data of Table 10, (nominal level 5%). α = 5% F X C X C X W X W X B X B X F X Empirical level Alternative 1/2 observed means observed means Acknowledgement. We thank a referee for the careful reading of the first draft of the manuscript and his/her useful suggestions. The support of the German Research Community (DFG, Sonderforschungsbereich 475, Project: Meta-Analysis in Biometry and Epidemiology ) is gratefully acknowledged. Reference Asiribo, O. and Gurland, J. (1990). Coping with variance heterogeneity, Communications in Statistics, Series A, 19, Böckenhoff, A. and Hartung, J. (1998). Some corrections of the significance level in metaanalysis, Biometrical Journal, 40, Box, G.E.P. (1954). Some theorems on quadratic forms applied in the study of analysis of variance problems, I. Effect of inequality of variance in the one way classification, Annals of Mathematical Statistics, 25, Brown, M.B. and Forsythe, A.B. (1974). The small sample behavior of some statistics which test the equality of several means, Technometrics, 16, Chalmers, T.C. (1991). Problems induced by meta-analyses, Statistics in Medicine, 10, Cochran, W.G. (1937). Problems arising in the analysis of a series of similar experiments, Journal of the Royal Statistical Society, Supplement, 4, (1954). The combination of estimates from different experiments, Biometrics, 10, Hardy, R.J. and Thompson, S.G. (1998). Detecting and describing heterogeneity in metaanalysis, Statistics in Medicine, 17, Hartung, J. and Voet, B. (1986). Best invariant unbiased estimators for the mean squared error of variance component estimators, Journal of the American Statistical Association, 81, Li, Y., Shi, L., and Roth, H.D. (1994). The bias of the commonly-used estimate of variance in meta-analysis, Communications in Statistics, Series A, 23, Lipsitz, S.R., Dear, K.B.G., Laird, N.M., and Molenberghs, G. (1998). Test for homogeneity of the risk difference when data are sparse, Biometrics, 54, Mehrotra, D.V. (1997). Improving the Brown-Forsythe solution to the generalized Behrens- Fisher problem, Communications in Statistics, Series B, 26, Normand, S.T. (1999). Meta-analysis: Formulating, evaluating, combining, and reporting, Statistics in Medicine, 18, Patnaik, P.B. (1949). The non-central χ 2 - and F-distributions and their applications, Biometrika, 36, Satterthwaite, F.E. (1946). An approximate distribution of estimates of variance components, Biometrics Bulletin, 2, Thompson, S.G. and Sharp, S.J. (1999). Explaining heterogeneity in meta-analysis: A comparison of methods, Statistics in Medicine, 18,

13 310 joachim hartung and do gan argaç Welch, B.L. (1951). On the comparison of several mean values: An alternative approach, Biometrika, 38, Whitehead, A. and Whitehead, J. (1991). A general parametric approach to the meta-analysis of randomized clinical trials, Statistics in Medicine, 10, Joachim Hartung and Do gan Argaç Department of Statistics University of Dortmund Vogelpothsweg 87 D Dortmund Germany s: hartung@statistik.uni-dortmund.de argac@statistik.uni-dortmund.de