On the Inefficiency of the Adaptive Design for Monitoring Clinical Trials

Size: px

Start display at page:

Download "On the Inefficiency of the Adaptive Design for Monitoring Clinical Trials"

Willis Chandler
6 years ago
Views:

1 On the Inefficiency of the Adaptive Design for Monitoring Clinical Trials Anastasios A. Tsiatis and Cyrus Mehta tsiatis/ Inefficiency of Adaptive Designs 1

2 OUTLINE OF TOPICS Hypothesis testing problem Fixed sample vs group-sequential vs adaptive test A general framework for considering both adaptive and traditional tests Definition of Optimal sequential tests Generalization of the Neyman-Pearson Theorem Why adaptive tests are inefficient Example Concluding remarks Inefficiency of Adaptive Designs 2

3 Hypothesis Testing Problem Interest will focus on testing for treatment difference Let Z 1,...,Z N be iid p(z,δ), where δ denotes treatment difference For example, Z i = X i Y i, where X i denotes response for ith individual on treatment A, and Y i denotes response for ith individual on treatment B X i N(µ A,σ 2 =1/2) Y i N(µ B,σ 2 =1/2) Hence Z i N(δ, 1), where δ = µ A µ B Problem: Test H 0 : δ 0 vs. H A : δ>0 Inefficiency of Adaptive Designs 3

4 Fixed Sample Test For fixed sample size N, the Neyman-Pearson theorem states that among all level α tests, the test which maximizes power against the alternative δ>0 is given by the rejection region R opt = {z 1,...,z N : N i=1 where C is chosen so that P 0 (R opt )=α p(z i,δ) p(z i, 0) >C}, In our example, this is equivalent to rejecting H 0 when S N N z α, where S N = N i=1 Z i N = (z α + z β ) 2 δa 2, where δ A denotes the clinically important difference that we wish to detect with power 1 β Inefficiency of Adaptive Designs 4

5 Drawbacks Doesn t allow for the possibility of stopping early to reject or accept the null hypothesis Need to specify clinically important alternative Inefficiency of Adaptive Designs 5

6 Group-Sequential Tests Interim analyses are conducted after n 1 <...<n K = N observations with the possibility of stopping the trial at each analysis Stop and reject H 0 after n j observations if S nj C ju or accept H 0 if S nj C jl,s nj = n j i=1 Z i C jl <C ju,j=1,...,k 1 C KL = C KU The boundaries C jl,c ju,j =1,...,K and the maximum sample size N are chosen so that the test has specified level α and power 1 β to detect the clinically important difference δ A Inefficiency of Adaptive Designs 6

7 Traditional designs for clinical trials dictate that we specify the clinically important treatment difference that needs to be detected with specified power using a test at a specified level of significance The difficulty is that often we don t have a good idea on what constitutes a clinically important difference Adaptive designs, which estimate the treatment difference using the observed data and adaptively change the design according to the estimated treatment difference, have a great deal of appeal Inefficiency of Adaptive Designs 7

8 Example of an Adaptive Design Choose initial alternative δ A =.466 We adapt a two-stage design (n 1 =25,n 2 = 50) A standard two stage group-sequential test would be After n 1 = 25 observations stop to reject H 0 if S 25 > (z =2.77) accept H 0 if S 25 < 2.19 (z =0.44) After n 2 = 50 observations (final analysis) reject H 0 if S 50 > (z =1.96) accept H 0 otherwise This test has level.025 and 90% power to detect δ =.466 Inefficiency of Adaptive Designs 8

9 Adaptation If 2.19 S estimate ˆδ = S 25 /25 take ˆn 2 2n 1 (δ A /ˆδ) 2 to correspond roughly to the sample size needed to detect the difference ˆδ with 90% power at the.025 level of significance In our example we take ˆn 2 to be a multiple of n 1 =25uptoa maximum of N = 250. Note that ˆn 2 is random and a function of S n1 To maintain type I error we reject when S 25 + ( ) 1/2 25 (Sˆn2 ˆn 2 25 S 25) > Inefficiency of Adaptive Designs 9

10 Some Notation Let N denote maximum sample size to be considered Let Z N denote the sample space, i.e. all realizations {z 1,...,z N } Let Z j denote the sample space restricted to the first j observations in our sample i.e. all realizations {z 1,...,z j } Let R Z N denote the rejection region and A = R C denote the acceptance region A level α test has the property that P 0 (R) =α Inefficiency of Adaptive Designs 10

11 Stopping Rules (Group-Sequential and Adaptive tests) Decisions to stop the study can be made after n 1 <...<n K = N observations The sample space Z N is partitioned into a sequence of mutually exclusive and exhaustive rejection and acceptance regions (R j, A j ) Z nj, j =1,...,K if (z 1,...,z nj ) R j,thenh 0 is rejected after j observations if (z 1,...,z nj ) A j,thenh 0 is accepted after j observations R= R 1... R K A= A 1... A K Inefficiency of Adaptive Designs 11

12 A level α sequential test has the property that P 0 (R) = K P 0 (R j )=α j=1 Inefficiency of Adaptive Designs 12

13 Example using Adaptive Test Decisions for stopping can be made after n j =25j observations j =1,...,10 R 1 = S 25 > A 1 = S 25 < 2.19 For j =2,...,10 R j =(S 25 Γ j,t j > 13.84) A j =(S 25 Γ j,t j 13.84) where Γ 2 = [11.65, 13.84], Γ 10 =[2.19, 5.49) Γ j = [11.65 (2/j), {2/(j 1)}),j =3,...,9 T j = S 25 +(j 1) 1/2 (S 25j S 25 ) Inefficiency of Adaptive Designs 13

14 Operating Characteristics Sequential tests (including adaptive) allow possibility of stopping early and either rejecting or accepting null hypothesis The operating characteristics can be summarized by Rejection spending function Acceptance spending function β(j, δ) =P δ (R 1 )+...+ P δ (R j ) φ(j, δ) =P δ (A 1 )+...+ P δ (A j ) These are monotone functions in j for j =1,...,K for all δ β(k, δ)+φ(k, δ) =1forallδ A level α test when β(k, 0) = α Inefficiency of Adaptive Designs 14

15 Optimality If we have two competing level α tests for H 0 : δ 0 versus H A : δ>0, with rejection-acceptance regions (R 1j, A 1j ), j =1,...,K and (R 2j, A 2j ), j =1,...,K, then we say that test 1 is uniformly better than test 2 if β 1 (j, δ) β 2 (j, δ) for all j =1,...,K and all δ>0 φ 1 (j, δ) φ 2 (j, δ) for all j =1,...,K and all δ 0 Inefficiency of Adaptive Designs 15

16 Generalization of Neyman-Pearson Theorem Define the α and θ spending functions as α(j) =β(j, 0) θ(j) =φ(j, 0) For prespecified α and θ spending functions, define the sequential likelihood ratio test for testing the simple null hypothesis δ = 0 against the simple alternative δ = δ 1 as the test which rejects at the first time j when n j i=1 p(z i,δ 1 ) p(z i, 0) C uj and accepts at the first time j when n j i=1 p(z i,δ 1 ) p(z i, 0) C lj, Inefficiency of Adaptive Designs 16

17 Generalization of Neyman-Pearson Theorem Theorem 1: The optimal sequential test for the simple null hypothesis H 0 : δ =0 versus the simple alternative H 1 : δ = δ 1 with a specified α(θ)-spending function is the sequential likelihood ratio test By optimal, we mean that for any other sequential test with the same α and θ spending function, we have β LR (j, δ 1 ) β(j, δ 1 ), j =1,...,K Inefficiency of Adaptive Designs 17

18 If we go back to our example of testing equality in the mean of normally distributed responses between two treatments, the sequential likelihood ratio test is equivalent to rejecting at the first j when S nj C uj and accepting at the first j when S nj C lj The test above does not depend on the specific alternative δ 1 > 0, hence it is uniformly more powerful for δ>0thanany other sequential test with the same α and θ spending function Inefficiency of Adaptive Designs 18

19 By reversing the role of rejection and acceptance we can also show that the above test also is uniformly better in accepting the null hypothesis φ LR (j, δ) φ(j, δ), j=1,...,k,δ <0 Consequently, among all sequential tests with a prespecified α and θ spending function, the likelihood ratio sequential test defined above is uniformly best Inefficiency of Adaptive Designs 19

20 Inferiority of the Adaptive Design Any level α adaptive test, no matter how complex, has an induced α and θ spending function as well as rejection and acceptance functions β(j, δ) and φ(j, δ) for j = 1,...,K and for all δ We can always construct a likelihood ratio sequential test with the same α and θ spending function as that for the adaptive design By the main theorem we know that the likelihood ratio sequential test must be uniformly better than the adaptive design Returning to our example of an adaptive design, the alpha-theta function is given by Inefficiency of Adaptive Designs 20

21 Table 1: Alpha Theta Functions j α(j) θ(j) j α(j) θ(j) j α(j) θ(j) Inefficiency of Adaptive Designs 21

22 Figure 1: Comparison of Standard Group Sequential ( ) and Adaptive (---) Designs H0_Rejection_Probability µ=0.36 µ=0.22 µ=0.08 H0_Acceptance_Probability µ=0.08 µ=0.22 µ= Information Fraction Information Fraction H0_Rejection_Probability µ= 0.08 µ= 0.15 H0_Acceptance_Probability µ= 0.08 µ= Information Fraction Information Fraction Inefficiency of Adaptive Designs 22

23 Conclusions Although on the surface adaptive designs have a great deal of appeal, after examining these designs more carefully we find that they are uniformly inferior to standard sequential tests Perhaps a better strategy would be for the statisticians and collaborators to choose a range of possible treatment differences that include A larger treatment difference believed to be plausible δ init A minimally acceptable treatment difference, perhaps corresponding to maximum sample size N Search for standard sequential designs with specified power to detect the minimally acceptable difference and have high probability of rejecting H 0 as early as possible for the plausible range of treatment differences Inefficiency of Adaptive Designs 23

Partitioning the Parameter Space. Topic 18 Composite Hypotheses

Topic 18 Composite Hypotheses Partitioning the Parameter Space 1 / 10 Outline Partitioning the Parameter Space 2 / 10 Partitioning the Parameter Space Simple hypotheses limit us to a decision between one