Effect of investigator bias on the significance level of the Wilcoxon rank-sum test

Transcription

1 Biostatistics 000, 1, 1,pp Printed in Great Britain Effect of investigator bias on the significance level of the Wilcoxon rank-sum test PAUL DELUCCA Biometrician, Merck & Co., Inc., 1 Walnut Grove Dr., Horsham, PA 19044, USA paul delucca@merck.com DAMARAJU RAGHAVARAO Professor of Statistics, Temple University, Department of Statistics, Broad and Montgomery Ave. Philadelphia, PA 191, USA v5558e@vm.temple.edu SUMMARY When using subjective ordered categorical variables to measure the efficacy of an active treatment versus placebo in a double-blind clinical trial setting, bias may be introduced into the response variables when investigators become partially or totally unblinded to treatment assignment due to characteristic side effects. The investigators may alter the classification of a patient s response to treatment based on perceived treatment assignment. The introduction of bias leads to a considerable increase in the actual significance level of the Wilcoxon rank-sum test. Keywords: Clinical trial; Nominal level; Perceived groups; Subjective categorical measure. 1. INTRODUCTION The randomized, double-blind design is the standard design used for clinical trials as part of the New Drug Application NDA process, for post-approval marketing trials where a new compound is compared with an existing standard treatment, and other areas of medical research of new therapies. Of the advantages of the randomized, double-blind design, none is more important than the protection it provides against the introduction of bias into the evaluation of treatment effect by the random allocation of patients to treatment and the masking of treatment assignment from investigators and patients. Even though efforts are made to limit bias through randomization and blinding, it is possible for bias to be introduced in the setting of randomized, double-blind clinical trials. It is possible that characteristic side effects of a study drug may reveal a patient s treatment assignment. This is especially true when an active drug is compared with an inactive placebo. Greenberg and Fisher 1994 provide a commentary on the unblinding of double-blind clinical trials in studies involving psychotropic drugs. They state that all past studies of antidepressant effectiveness are open to question. The degree to which psychotropic drugs exceed placebo in their therapeutic potency is uncertain. The authors state that the problem extends to many other drug trial areas as well, such as the use of AZT for treating AIDS. The effect of investigator bias on the power and level of the two-sample Z-test has been described by DeLucca et al In this article, we examine the effect of bias on the significance level of the Wilcoxon rank-sum test when bias is introduced through the informed guessing of treatment assignment. In particular, in the setting of a randomized, double-blind clinical trial in which response to treatment is c Oxford University Press 000

2 108 PAUL DELUCCA AND DAMARAJU RAGHAVARAO measured using subjective ordered categorical variables, we show that the consequence of the introduction of bias is to considerably inflate the actual significance level of the test and, consequently, the power of the test.. GENERAL APPROACH Consider a double-blind clinical trial in which an active treatment t is being compared with placebo c. The investigator is asked to classify the patient s condition at the end of the study using a subjective ordered categorical response variable. We assume that the ordered categories represent intervals of an underlying but unobservable continuous distribution. We further assume that the effect of treatment is manifested in a shift in the unobservable continuous distribution of the active group from the placebo group. If unintended unblinding of treatment assignment has occurred due to characteristic side effects, then the investigator s evaluation of treatment effect may be biased. Let l ij correspond to the probability the investigator will classify a patient from the ith actual group as being in the jth perceived group, where i = t, c and j = t, c. The probability the investigator forms no opinion for patients in the ith actual group is 1 l it l ic. In many circumstances, it is reasonable to assume that l tt > l tc and l cc > l ct. We will also assume the perception of being in the active placebo treatment group results in a shift in the response variable one category better worse. If the investigator forms no opinion of treatment assignment, then no shift of the response variable occurs. More extreme shifts or stochastic models for shifts might also be used. We will consider the case in which the response variable consists of five ordered categories. Consider the following 5 table in which response to treatment is measured without bias and the sample sizes in the active and placebo groups are n and m, respectively. Response to treatment no bias Much Slightly Slightly Much worse worse No change improved improved Treatment group Total Active Placebo y 1 p 1 x 1 1 y p x y 3 p 3 x 3 3 y 4 p 4 x 4 4 Note: y s and x s are the frequencies. p s and s are the probabilities. y 5 p 5 n x 5 5 m When bias is introduced into the response variable, as described previously, the cell probabilities will be shifted and are represented by p j active and j placebo for j = 1,,...,5. The shifted cell probabilities in the active treatment group can be easily calculated by: p j = l tt p j l tt l tc p j + l tc p j+1, j =, 3, 4; p 1 = 1 l tt p 1 + l tc p ; p 5 = l tt p l tc p 5. The shifted cell probabilities in the placebo group can be calculated in a similar manner. If we consider the Mann Whitney form of the Wilcoxon rank-sum statistic we know that in the absence of bias: W XY = [ number of paris X i, Y j with Xi < Y j ] + 1 [ number of pairs Xi, Y j with Xi = Y j ].

3 Investigator bias effect on the level of the Wilcoxon rank-sum test 109 Now, W XY = W s 1 nn + 1, where W s is the usual Wilcoxon rank-sum statistic and E W XY = pmn, where p = prx < Y, which in the absence of investigator bias may be defined as p = 1 i p i p i + j p j i< j p i. Under the null hypothesis and in the absence of investigator bias p = 0.5. When investigator bias is introduced p = prx < Y, with modified response variables X and Y,isno longer 0.5. One can explore the relationship between p and p in a number of ways. If the cell probabilities without investigator bias are known, then p is a deterministic function of the l ij s, p j s, and j s. That is, p can be computed exactly as p = 1 i p i p i + j p j i< j p i. However, there are many different configurations of the cell probabilities in the 5 table that result in the same value of p, but yield different values of p. In addition, it is unlikely that the cell probabilities will be known for p to be computed exactly. Therefore, we now explore the relationship between p and p for selected l tt, l tc, l ct, and l cc using simple linear regressions of p on p as a way to estimate p for a given p, say p = 0.5, regardless of the configuration of the cell probabilities. We start by constructing all possible configurations of the cell probabilities for a 5 table with the following restrictions to avoid extreme observed cell frequencies: max p j, j = 0.6, min p j, j = 0.1. Different table configurations are generated by altering the cell probabilities in 0.1 increments. We compute p for each table configuration. Using the quantities l tt, l tc, l ct, and l cc defined previously, we introduce bias into the cell probabilities for each table configuration and then compute p. Values of 0., 0.3, and 0.4 are selected for l tt and l cc and values of 0.1, 0., and 0.3 are selected for l tc and l ct, with l tt > l tc and l cc > l ct. For each of the 36 combinations of l tt, l tc, l ct, and l cc, we perform a linear regression of p on p. To determine the effect of bias on the significance level of the Wilcoxon rank-sum test we obtain p from the regression equations when p = 0.5. The linear equations fit with very large R values min R EFFECT ON THE LEVEL OF SIGNIFICANCE OF THE TEST Consider testing the null hypothesis of no treatment effect p = 0.5 versus the alternative hypothesis that the active treatment is superior to placebo p > 0.5. The normal approximation to the Mann Whitney form of the Wilcoxon rank-sum test is based upon the test statistic Z = W XY EW XY varwxy = W XY pmn, mnm + n + 1 with p = 0.5 and the critical region for a one-sided α-level test given by Z > z α, where z α is the 100α upper percentile point of the standard normal distribution. The significance level of the Wilcoxon rank-sum test is given by mn p 1 Level = 1 z α mnm + n + 1, where is the cumulative distribution function of a standard normal variable. In the absence of investigator bias p is equal to 0.5. In the presence of investigator bias, the actual level of the test α is obtained by replacing p by p in the equation above. 1 1

4 110 PAUL DELUCCA AND DAMARAJU RAGHAVARAO Table 1. Actual level when nominal level is 0.05, nominal level when actual level is 0.05, and p for selected values of l tt, l tc, l ct, l cc when n = 100 and m = 100 l tt l tc l ct l cc p Actual level Nominal level

5 Investigator bias effect on the level of the Wilcoxon rank-sum test 111 Interest lies not only in evaluating the effect of bias on the actual level of the test, α, but also in the nominal α-level required to preserve the actual level, which is usually Straightforward algebra gives mn p 1 α = 1 z α + mnm + n + 1. When the cell probabilities are known, p, obtained from the deterministic function, can be substituted for p and the effect of bias on the actual level α of the nominal α-level test can be calculated exactly. When the cell probabilities are unknown, p, obtained from the linear regression equations when p = 0.5, is used to determine the effect of bias on the actual level α of the nominal α-level test. Table 1 gives p obtained from the regression equations when p = 0.5, the actual level α of the test when nominal α = 0.05 is used, and the nominal α-level when the actual level is α = 0.05 for a one-sided test with n = 100, m = 100, and selected values of l tt, l tc, l ct, and l cc. One can see from Table 1 that in the presence of moderate investigator bias detailed above, the actual level of the test is considerably larger than the nominal 0.05 level. p and the actual level of the test are monotonically increasing and the nominal level is monotonically decreasing as the difference between l cc and l ct increases, for fixed differences between l tt and l tc, and as the difference between l tt and l tc increases, for fixed differences between l cc and l ct. That is, as the investigator is better able to correctly identify patients in the active and placebo groups, the actual level increases and the nominal level required to preserve the actual level at 0.05 decreases. 4. CONCLUSION In this article we demonstrated that the introduction of investigator bias into subjective response variables measured on an ordered categorical scale can considerably inflate the actual level of the Wilcoxon rank-sum test. Failure to guard against the introduction of bias may lead to inflated α-levels, depending on the values of l tt, l tc, l ct, and l cc. When limitations to blinding exist, one may use the methods here to explore the possible magnitude of bias and consider using a smaller nominal level to reduce the effect of bias on the actual level of the test. 1 ACKNOWLEDGEMENTS The authors thank Professor Zeger for his helpful comments in improving the original submission of this paper. REFERENCES DELUCCA, P., RAGHAVARAO, D. AND ALTAN, S Effect of investigator bias on the power and level of the two-sample Z-test. Journal of Biopharmaceutical Statistics 9, GREENBERG, R. P. AND FISHER, S Seeing through the double-masked design: a commentary. Controlled Clinical Trials 15, [Received July, Revised September 9, 1999 and October 13, 1999]