BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS: COMPARISONS BETWEEN BAYESIAN AND EM ANALYSES

BLINDED EVALUATIONS OF EFFECT SIZES IN CLINICAL TRIALS: COMPARISONS BETWEEN BAYESIAN AND EM ANALYSES IBRAHIM TURKOZ, PhD Janssen Research & Development LLC, NJ BASS XX Nov, 013 BASS XX 1

Outline Introduction Background Literature Review Objective Methods EM Algorithm Bayesian Approaches Model Selection Simulations Set-up Results Application Discussions and Conclusions Questions Nov, 013 BASS XX Outline

Background Clinical trials can have a major impact on public health Required information for designing a clinical trial is not fully available Ongoing monitoring of accumulating data is necessary Interim analyses are common Usually costly Requires unblinding treatment assignments Statistical implications We will investigate the use of the accumulated data for purposes of modifying aspects of the study design IN BLINDED SETTING Nov, 013 BASS XX Introduction 3

Background, Motivating Question, MDD Trial [1] Prior AD Treatment Additional AD Treatment 1. Mahmoud A.R., Pandina J.G., Turkoz I., Kosik-Gonzalez C., Canuso M.C, Kujawa J.M., and Gharabawi G.M. Risperidone for treatment-refractory major depressive disorder. Annals of Internal Medicine, 007; 147: 593-60. Nov, 013 BASS XX Introduction 4

Background, Motivating Question, MDD Trial [1] Active + AD Treatment as usual Prior AD Treatment Additional AD Treatment RANDOMIZATION for those who did not respond Placebo + AD Treatment as usual Placebo + AD Treatment as usual 6 weeks long, but the primary time point was Week 4 1. Mahmoud A.R., Pandina J.G., Turkoz I., Kosik-Gonzalez C., Canuso M.C, Kujawa J.M., and Gharabawi G.M. Risperidone for treatment-refractory major depressive disorder. Annals of Internal Medicine, 007; 147: 593-60. Nov, 013 BASS XX Introduction 5

Background, Motivating Question, MDD Trial [1] Active + AD Treatment as usual Prior AD Treatment Additional AD Treatment RANDOMIZATION for Patients with Caroll Depression Score >0 And CGI-S4 Placebo + AD Treatment as usual Placebo + AD Treatment as usual 6 weeks long, but the primary time point was Week 4 Outcome Parameters Clinician Rated HRSD-17 (primary efficacy scale) CGI-S Patient Reported Most Troubling Symptoms (MTS) or PaRTS-D Patient Global Impression of Severity PGIS 1. Mahmoud A.R., Pandina J.G., Turkoz I., Kosik-Gonzalez C., Canuso M.C, Kujawa J.M., and Gharabawi G.M. Risperidone for treatment-refractory major depressive disorder. Annals of Internal Medicine, 007; 147: 593-60. Nov, 013 BASS XX Introduction 6

Background, Motivating Question, MDD Trial Primary Time Point? Week 4 or Week 6? Primary Endpoint? HRSD-17 or PaRTS-D Nov, 013 BASS XX Introduction 7

Background, Motivating Question, Schizoaffective Trial Psychosis Mania Depression Nov, 013 BASS XX Introduction 8

Background, Motivating Question, Schizoaffective Trial PANSS 30-10 Psychosis Mania Depression YMRS 0-60 HAM-D-1 0-5 Nov, 013 BASS XX Introduction 9

Background, Motivating Question, Schizoaffective Trial [1] Psychosis Mania Depression n=95 PBO Study 300 N=311 n=16 PALI PR 3-1 mg/d (start at 6 mg; flexible) BL Wk 6 1. Canuso CM, Lindenmayer JP, Kosik-Gonzalez C, Turkoz I, Carothers J, Bossie CA, Schooler NR. A randomized, double-blind, placebo-controlled study of dose ranges of paliperidone extended-release in the treatment of subjects with schizoaffective disorder. J Clin Psychiatry. 010 May;71(5):587-98. Nov, 013 BASS XX Introduction 10

Background, Motivating Question, Schizoaffective Trial PRIMARY ENDPOINT SECONDARY ENDPOINTS Psychosis Depression HAM-D-1 Changes in PANSS Total Score at Week 6 Mania YMRS Nov, 013 BASS XX Introduction 11

Background, Motivating Question, Schizoaffective Trial PRIMARY ENDPOINT SECONDARY ENDPOINTS Psychosis Depression HAM-D-1 Changes in PANSS Total Score at Week 6 FOR REGISTRATION PURPOSES Mania YMRS Nov, 013 BASS XX Introduction 1

Background, Motivating Question, Schizoaffective Trial Nov, 013 BASS XX Introduction 13

Background, Motivating Question, ADHD Trial 1 1. Wigal SB, Wigal T, Schuck S, Brams M, Williamson D, Armstrong RB, and Starr HL. Academic, Behavioral, and Cognitive Effects of OROS Methylphenidate on Older Children with Attention-Deficit/Hyperactivity Disorder. J Child Adolesc Psychopharmacol. 011 April; 1(): 11 131. Nov, 013 BASS XX Introduction 14

Background, Motivating Question, ADHD Trial 1 1. Wigal SB, Wigal T, Schuck S, Brams M, Williamson D, Armstrong RB, and Starr HL. Academic, Behavioral, and Cognitive Effects of OROS Methylphenidate on Older Children with Attention-Deficit/Hyperactivity Disorder. J Child Adolesc Psychopharmacol. 011 April; 1(): 11 131. Nov, 013 BASS XX Introduction 15

Background It is important to evaluate secondary endpoints to differentiate your drug from other drugs in market place. Nov, 013 BASS XX Introduction 16

Background It is important to evaluate secondary endpoints to differentiate your drug from other drugs in market place. Another symptom, Functionality, Medication Satisfaction, Certain Adverse Reactions, Cognition, Etc. Nov, 013 BASS XX Introduction 17

Background It is important to evaluate secondary endpoints to differentiate your drug from other drugs in market place Another symptom, Functionality, Medication Satisfaction, Certain Adverse Reactions, Cognition, Etc. Failure to consider important outcomes can limit the conclusions of clinical trials. Nov, 013 BASS XX Introduction 18

Background Examination of appropriate statistical methods to test secondary endpoints requires careful consideration Fixed sequence gate-keeping procedures [1,,3,4] are widely used because of their ease of application and interpretation in many circumstances These gate-keeping procedures often require prospective ordering of null hypotheses for secondary endpoints ***Can we use information that is available to order the null hypotheses based on interim effect sizes without breaking the blind? 1. Demitrienko A., Tamhane A., Bretz F. Multiple testing problems in pharmaceutical statistics 010; Chapman&Hall/CRC Biostatistics Series. Westfall P.H., Krishen, A., Optimally weighted, fixed sequence and gatekeeper multiple testing procedure. J of Statistical Planining and Inference. 99 (001) 5-40 3. Weins, B., Dmitrienko, A. (005). The fallback procedure for evaluating a single family of hypotheses. Journal of Biopharmaceutical Statistics 15, 99 94. 4. Bretz F., Maurer W., Brannath W., Posch M., A graphical approach to sequentially rejective multiple test procedures, Stat in Med, 009; 8:586 604. Nov, 013 BASS XX Introduction 19

Literature Review EM algorithm to estimate the within group variance for sample size reestimation without unblinding at interim stages has been suggested [1,,3] Enrollment order of subjects and the randomization block sizes to estimate the within group variance have also been used [4] Risk of unblinding the treatment effect from blinded inferences based on knowledge of the randomization block size for continuous and binary outcomes [5] The recent draft FDA guidance on adaptive designs [6] discusses possible study design modifications such as selection and/or order of secondary endpoints in addition to sample size re-estimation Blinded treatment effects for survival endpoints [7] were also examined 1.Gould AL, Shih WJ. Sample size re-estimation without unblinding for normally distributed outcomes with unknown variance. Communications in Statistics Theory and Methods 199; 1: 833 853.. Shih WJ. Sample size reestimation in clinical trials. In Biopharmaceutical sequential statistical applications, Peace K (ed.). Marcel Dekker: New York, 199; 85 301. 3. Shih WJ. Sample size reestimation for triple blind clinical trials. Drug Information Journal 1993; 7: 761 764. 4. Xing B, Ganju J. A method to estimate the variance of an endpoint from an on-going blinded trial. Statistics in Medicine 005; 4: 1807-1814. 5. Miller F., Friede T, Kieser T. Blinded assessment of treatment effects utilizing information about randomization block length. Statistics in Medicine 009; 8: 1690-1706. 6. Guidance for industry: Adaptive design clinical trials for drugs and biologics. http://www.fda.gov/downloads/drugs/guidancecomplianceregulatoryinformation/guidances/ucm01790.pdf 7. Xie J, Quan H, Zhang J. Blinded assessment of treatment effects for survival endpoint in an ongoing trial. Pharmaceutical Statistics 01. Nov, 013 BASS XX Introduction 0

Objective Instead of formal interim analyses on unblinded data, use the information from blinded data Assess the feasibility of estimating magnitude of treatment effect on various secondary endpoints in ongoing trial Primary objective is to compare posterior ordering of the signalto-noise ratios (effect sizes), d /, using available data points Nov, 013 BASS XX Introduction 1

Model ijk k k ik ik ijk k y 0 z, ~ N(0, ) ii.. d. Y ( i 1,..., n, j 0,1, k 1,,..., K), is the response variable for the k endpoint from j ijk j treatment group on subject i th th Y Y i0k N 0k k i1k ~ N( 1k, k ) ~ (, ) z 0k k ik the mean placebo response magnitude of the treatment effect latent binary treatment assignment We assume that lower 1k 0k k k Y ~ N(, ) y scores are better and the average effect of the treatment is, k Nov, 013 BASS XX Methods

Model For a two component univariate normal mixture model, it is appropriate to specify the complete likelihood function for a given response in the form : p( y, z θ) p( y1 θ1) (1 ) p( y0 θ0) ; =p(z 1) 0 0 θ1 1 0 where, θ (, ); (, ); is randomization weight. The signal-to-noise ratio for endpoint k, d /, ( k 1,..., K) k k k ***Our objective is to compare the EM and Bayesian algorithm induced posterior ordering of the signal-to-noise ratios of endpoints, P( d d... d y) 1 k Nov, 013 BASS XX Methods 3

EM Algorithm p( y, z θ) p( y θ ) (1 ) p( y θ ) 1 1 0 0 ; =p(z 1) MMLEs are the most common way to estimate parameters of interest; however, MMLEs often do not have closed form solutions. EM seeks the MMLE by iteratively applying Expectation and Maximizing steps. E-Step, E{log p( y, z θ) y, θˆ () t } M-Step, the conditional expectation is maximized with respect to θ. This yields the new estimates ˆ( t1) ( t1) for θ and a distribution for z. Nov, 013 BASS XX Methods 4

EM Algorithm Conditional expectation of latent treatment assignment at iteration ( t 1) is: ˆ P( z 1, ) (, ) (1 ) (,. ) () t ( ) ( ) ( 1) () ˆ t ˆ t t ˆ t N( y 0, ) i y θ ( t) ˆ( ) ( ) ( t) ˆ t ˆ t ˆ ˆ ( t) N y 0 N y 0 In the M step, the mean parameters ˆ, ˆ, and variance parameter ˆ at iteration are ( t 1) : 0 n n ( t1) ˆ( t) ( t1) ˆ( t) ( t1) P zi y θ yi P zi y θ yi ˆ 0 ( t1) i 1 ˆ( t1) i 1 0 n n ( t1) ˆ( t) ( t1) ˆ( t) P( zi 0 y, θ ) P( zi 1 y, θ ) i1 i1 ( 0, ) ( 1, )( ) ˆ,, ˆ n (t 1) i1 ( 1, ˆ )( ) ( 0, ˆ P z y θ y P z y θ )( y ) n ( t1) ( t) ( t1) ( t1) ( ) ( t 1) ˆ t ˆ i i 1 i i 0 Nov, 013 BASS XX Methods 5

Bayesian Approach The likelihood function of the mixture distribution and the prior distributions of θ and z are combined to obtain the joint posterior distribution, which is assumed to contain all the information about the unknown parameters. Using the Bayes theorem, the posterior distribution satisfies: p( z, θ y) p( y z, θ) p( z θ) p( θ) where, p( y z, θ) is the likelihood pz ( θ) is the prior probability of latent treatment assignment p( θ) is the prior distribution of θ Nov, 013 BASS XX Methods 6

Bayesian Approach p( z, θ y) p( y z, θ) p( z θ) p( θ) The conditional distribution of the treatment allocation is proportional : p( z y, θ) p( y z, θ) this reduces to latent treatment allocation probability in EM. Prior Distributions: j ~ N(, ), in order to reduce sensitvity For simplicity purposes, we have 0 ~ N( y0, s0 ), 1 0 N y 1 s1 ~ (, ), Gamma(, ) Nov, 013 BASS XX Methods 7

Bayesian Approach Regardless of the structure of the prior distribution, the posterior distribution does not have a simple closed form. In order to make inferences about the unknown parameters, MCMC methodologies are employed to generate samples from the posterior distribution. The joint posterior probability distribution of these parameters satisfies : 0 0 0 p( z,,, y) p( y z,,, ) p( z) p(,, ) n 1 1 exp 1 ( y i 0 ) exp ( y i 0) z 1 i zi0 0 y1 0 y0 s1 s0 1 ( ) ( ) 1 exp exp exp( ) Nov, 013 BASS XX Methods 8

Bayesian Approach Regardless of the structure of the prior distribution, the posterior distribution does not have a simple closed form. In order to make inferences about the unknown parameters, MCMC methodologies are employed to generate samples from the posterior distribution. The joint posterior probability distribution of these parameters satisfies : 0 0 0 p( z,,, y) p( y z,,, ) p( z) p(,, ) n 1 1 exp 1 ( y i 0 ) exp ( y i 0) z 1 i zi0 0 y1 0 y0 s1 s0 1 ( ) ( ) 1 exp exp exp( ) Likelihood for treatment Nov, 013 BASS XX Methods 9

Bayesian Approach Regardless of the structure of the prior distribution, the posterior distribution does not have a simple closed form. In order to make inferences about the unknown parameters, MCMC methodologies are employed to generate samples from the posterior distribution. The joint posterior probability distribution of these parameters satisfies : 0 0 0 p( z,,, y) p( y z,,, ) p( z) p(,, ) n 1 1 exp 1 ( y i 0 ) exp ( y i 0) z 1 i zi0 0 y1 0 y0 s1 s0 1 ( ) ( ) 1 exp exp exp( ) Likelihood for placebo arm Nov, 013 BASS XX Methods 30

Bayesian Approach Regardless of the structure of the prior distribution, the posterior distribution does not have a simple closed form. In order to make inferences about the unknown parameters, MCMC methodologies are employed to generate samples from the posterior distribution. The joint posterior probability distribution of these parameters satisfies : 0 0 0 p( z,,, y) p( y z,,, ) p( z) p(,, ) n 1 1 exp 1 ( y i 0 ) exp ( y i 0) z 1 i zi0 0 y1 0 y0 s1 s0 1 ( ) ( ) 1 exp exp exp( ) Prior for treatment arm Nov, 013 BASS XX Methods 31

Bayesian Approach Regardless of the structure of the prior distribution, the posterior distribution does not have a simple closed form. In order to make inferences about the unknown parameters, MCMC methodologies are employed to generate samples from the posterior distribution. The joint posterior probability distribution of these parameters satisfies : 0 0 0 p( z,,, y) p( y z,,, ) p( z) p(,, ) n 1 1 exp 1 ( y i 0 ) exp ( y i 0) z 1 i zi0 0 y1 0 y0 s1 s0 1 ( ) ( ) 1 exp exp exp( ) Prior for Placebo arm Nov, 013 BASS XX Methods 3

Bayesian Approach Regardless of the structure of the prior distribution, the posterior distribution does not have a simple closed form. In order to make inferences about the unknown parameters, MCMC methodologies are employed to generate samples from the posterior distribution. The joint posterior probability distribution of these parameters satisfies : 0 0 0 p( z,,, y) p( y z,,, ) p( z) p(,, ) n 1 1 exp 1 ( y i 0 ) exp ( y i 0) z 1 i zi0 0 y1 0 y0 s1 s0 1 ( ) ( ) 1 exp exp exp( ) Prior for Sigma In computations, =3 and small scale hyperparameter ( 0.01) are assumed. Nov, 013 BASS XX Methods 33

BASS XX Bayesian Approach Nov, 013 Methods 34 0 1 0 1 1 0 0 1 0 1 0 1 0 1 The posterior conditional distributions of parameters of interest in equation are given by : 1 ~,, 1 1 1 1 ( ) ~ i i i i i i y z y y s s N n n s s s s y z y s N 1 1 1 1 1 0 1 3 1,, 1 1 ( ) ~ i i i i i i i n z z s s y z

Bayesian Approach with Covariate We assume a trial with stratified randomization. The sample is divided into subsamples on the basis of stratifaction factor (covariate: additional medication usage, yes and no groups). Let g i denote the known strata membership of the subject i. g strata and 0 if it does not. The joint posterior distribution then takes the form: i 1 if observation belongs to the yes n 1 1 p( z, 0, yes, no, y) e xp ( y i 0 g ) i zi1 gi0,1 1 ( 0 g y1 ) i g i ( 0 y0) exp ( y i 0) exp exp z 0 0,1 1 i g s s i 0 1 1 exp( ) Nov, 013 BASS XX Methods 35

Bayesian Approach with Covariate We assume a trial with stratified randomization. The sample is divided into subsamples on the basis of stratifaction factor (covariate: additional medication usage, yes and no groups). Let g i denote the known strata membership of the subject i. g strata and 0 if it does not. The joint posterior distribution then takes the form: i 1 if observation belongs to the yes n 1 1 p( z, 0, yes, no, y) e xp ( y i 0 g ) i zi1 gi0,1 1 ( 0 g y1 ) i g i ( 0 y0) exp ( y i 0) exp exp z 0 0,1 1 i g s s i 0 1 1 exp( ) Nov, 013 BASS XX Methods 36

Bayesian Approach with Covariate Conditional posterior probability treatment assignment at each iteration is given by : 1 exp ( yi 0 yes ), Yes strata 1 1 exp ( y i 0 yes ) exp ( y i 0) Pz ( i 1) 1 exp ( y i 0 no), No strata 1 1 exp ( yi 0 no) exp ( y i 0) Nov, 013 BASS XX Methods 37

Bayesian Approach with Covariate The posterior distribution for parameters of interest are computed to be : yi zi ( yes no) y ( y 0 1, yes yes ) ( y1, no ) no i 1 s0 s1 1 0 ~ N,, n 1 n 1 s0 s1 s0 s 1 yes ~ ( yi 0) zi ( yi 0) zi i1, g 1 0 y 1, yes i1, g 0 y 1, no s N 1 1 s1 1,, no ~ N,, zi zi zi zi i1, g 1 1 i1, g 1 1 i1, g 1 i1, g 1 s1 s 1 s1 s 1 ~ ( yi 0 zi yes ) ( yi 0 zi i1, g 1 i1, g n 3 no ). Nov, 013 BASS XX Methods 38

Bayesian Approach with Covariate and a Power Prior Denote the historical data by D 0 0 0 0 0 approaches no barrowing from D0 0 and current data by D [0,1] to control the influence of the historical data on the current study 1 approaches full barrowing from D [1,,3] The basic idea is to use the power 0 parameter to control the influence of the historical data on the current study Let assume LD ( ) is the past data likelihood 0 LD ( ) is the current data likelihood 0 0 0 0 p D L D p p (, ) ( ) 0 ( ) ( ) is the posterior given past likelihood, Then, the full likelihood is proportional to p( D, ) L( D ) 0 0 1. Ibrahim J, Chen MH. Power prior distributions for regression models. Statistical Science 000; 15:46-60.. Duan Y, Ye K, Smith EP. Evaluating water quality using power priors to incorporate historical information. Environmetrics 006; 17: 95-106. 3. Hobbs BP, Carlin BP, Mandrekar S, and Sargent DJ. Hierarchical commensurate and power prior models for adaptive incorporation of historical information in clinical trials. Biometrics, 011; 67:1047-1056. Nov, 013 BASS XX Methods 39

Bayesian Approach with Covariate and a Power Prior The joint posterior distribution can be extended to include the prior for Think of n 0 0 The result takes the form : as the historical effective sample size. 0 n 1 1 p( z, 0, y, n,, 0 y) exp ( y zi 1 g 1 p, no ( 0 no 1, no) 1 p, no exp s 1 s1 ) 1 n 1 p, yes ( 0 0 yes y1, yes ) n1 p, yes 0 exp ( y i 0) exp s z 0 1 s i 1 n y n 0 0 1 n0 p ( 0 0 y0) n0 p 0 exp 1 exp( ) s 0 s0 a1 b1 0 (1 0) Prior for Power Parameter i 0 g Nov, 013 BASS XX Methods 40

Bayesian Approach with Covariate and a Power Prior The posterior distribution of parameters of interest is computed to be: 1 a b1 ( 0 0 y0) nop ( 0 yes y1, yes ) n1 p, yes 0 0 (1 0) exp s0 s1 ( 0 0 no y1, no) exp s 1 n 1 p, no 0 yi zi ( yes no) y0n0 p ( y 0 1, yes yes ) n1 p, yes ( y 0 1, no no) n1 p, no 0 1 0 1 1 ~ i s s N, n n0 p n 0 1p n 0 n 0 p n 0 1p 0 s0 s1 s0 s1 Nov, 013 BASS XX Methods 41

Bayesian Approach with Covariate and a Power Prior The posterior distribution of parameters of interest is computed to be: yes no ( yi 0) zi i1, g1 ( 0 y1, yes ) n1 p, yes 0 s 1 1 ~ N,, zi zi i1, g 1 n1 p, yes 0 i1, g 1 n1 p, yes 0 s1 s 1 ~ N i1, g ( y i ) z 0 i ( y ) n 0 1, no 1 p, no s 1 z i i i1, g n1 p, no i1, g n1 p, no s1 s1 Nov, 013 BASS XX Methods 4 0 0 0 ( yi 0 zi yes ) ( yi 0 zi no) i1, g 1 i1, g n 3 ~. 1,, z

Model Selection By contrast with frequentist analogues, Bayesian model comparison distinguishes which likelihood and prior combinations better fit the data. These measures of performance can be used to choose a single best model or improve estimation via model averaging. [1] The deviance information criterion (DIC) provides a natural measure of performance in this setting. Models with smaller DIC should be preferred to models with larger DIC. DIC p D, d = D D( θ), since pd D D( θ) = D( ˆ 0, ) D( ˆ 0, ) Expectation over (, ) Expectation over ( ˆ, ˆ ) 0 0 1. D.J. Spiegelhalter, N.G. Best, B.P. Carlin, A. Linde. Bayesian measures of model complexity and fit (with discussion). Journal of the Royal Statistics Society, Series B (Statistical Methodology)00; 64: 583 639. Nov, 013 BASS XX Methods 43

Model Selection 0 ˆ0 ˆ DIC D(, ) D(, ) ( y 0 ) ( 0, ) i zi D E0,, this is a posterior expectation over ( 0, ),average after all the iterations i 1 ˆ ˆ averaged after all the iterations updates at each iteration 0 updates at each iteration z i latent treatment assignment at each iteration ˆ ˆ ( y ˆ0 ) ( ˆ0, ) i zi D Ez, this is a posterior expectation over z, it is using averaged parameters i i i 1 ˆ ˆ 0 averaged after all the iterations ˆ averaged after all the iterations ˆ zi averaged after all the iterations latent treatment assignment at each iteration Nov, 013 BASS XX Methods 44

MCMC Gibbs Sampling An important feature of the Gibbs sampler is that each simulated posterior parameter value is always accepted The main drawback of the Gibbs sampler in this setting is the lack of mixing We employed a Metropolis-Hastings algorithm with an acceptance-rejection sampling This involved subsampling 10% of the treatment and placebo group observations and proposing a label switch 1) Draw random candidate samples fromfor switching treatment ) Generate u from the Uniform (0,1) distribution (1 ( pz ( i 1)) i 3) If, u make label change; otherwise return to step 1. pz ( 1) i i Nov, 013 BASS XX Methods 45

Simulation Set-Up Scenario d1 d d3 N 1 0.5 = 0.5 = 0.5 170 0.8 > 0.5 > 0.3 170 3 1.0 > 0.5 > ~0 170 4 1.0 > 0.5 > ~0 7 5 1.0 > 0.5 > ~0 468 Nov, 013 BASS XX Simulations 46

Simulation Set-Up Sample Size and Power Computations using a Two-Sample t-test with Two-Sided Type I Error=0.05 Nov, 013 BASS XX Simulations 47

Simulation Set-Up treatment arms with a 1:1 treatment allocation ratio, k 3 uncorrelated endpoints, y, y, and y, 1 3 No missing data, For each case 0 data sets are generated, l ( l 1,...,0), Different prior for each of the endpoints. Scenario d 1 d d3 1 0.5 = 0.5 = 0.5 170 0.8 > 0.5 > 0.3 170 3 1.0 > 0.5 > 0 170 4 1.0 > 0.5 > 0 7 N 5 1.0 > 0.5 > 0 468 Generate Input Data Sets: 0( k, l) ~ N( Y0k, s0k), 1 k l 0 k l k l N Y1 k s1k IG k bk k l (, ) (, ) (, ) ~ (, ), ~ (, ), 1,,3, 1,...,0. Without loss of generality, we assumed that 0k 0k 1k Y 10, s s 10 Nov, 013 BASS XX Simulations 48

Simulation Set-Up 0 N 1 Each true input clinical trial data set is based on ~ (10,10 ) and with normal distribution with variance 10 and corresponding mean value to generate effect sizes for a given scenario. Initial values of (,, ) for simulations are based on the true input data set parameters and the initial 0 0 power parameter estimate is given as 0.5 and 0.5. Posterior inferences are based on 1000 iterations. The first 00 iterations are considered to be the burn in period for both EM and Bayesian algorithms. th Every 4 point after the burn in period is stored for the Bayesian simulations to reduce correlations to emulate thinning process. Eight hundred simulation results in EM and 00 results in Bayesian methods are summarized. Posterior parameter estimates are summuarized. Empirical probability of each triplet combination of effect size ordering is examined. In each case for a given data set, the last 0 simulation results are kept for effect size orderings. For each of the study data sets, a total of 0 0 0 8000 possible triplet combinations of orderings are available. For 0 different study data sets, a total of 160,000 possible orderings are available. Nov, 013 BASS XX Simulations 49

Simulation Results Posterior Probability of Effect Size Ordering by Scenario Nov, 013 BASS XX Simulations 50

Simulation Results Posterior Distribution of Effect Sizes by Scenario, Averaged Over 0 Data Sets Nov, 013 BASS XX Simulations 51

Simulation Results Posterior Distribution of Effect Sizes by Scenario Nov, 013 BASS XX Simulations 5

Simulation Results, Scenario 1 Nov, 013 BASS XX Simulations 53

Simulation Results, Scenario 1 Nov, 013 BASS XX Simulations 54

Simulation Results, Scenario 1 Nov, 013 BASS XX Simulations 55

Simulation Results, Scenario 1 Nov, 013 BASS XX Simulations 56

Simulation Results, Scenario 1 Nov, 013 BASS XX Simulations 57

Simulation Results, Scenario Nov, 013 BASS XX Simulations 58

Simulation Results, Scenario Nov, 013 BASS XX Simulations 59

Simulation Results, Scenario 3 Nov, 013 BASS XX Simulations 60

Simulation Results, Scenario 3 Nov, 013 BASS XX Simulations 61

Simulation Results, Scenario 4 Nov, 013 BASS XX Simulations 6

Simulation Results, Scenario 4 Nov, 013 BASS XX Simulations 63

Simulation Results, Scenario 5 Nov, 013 BASS XX Simulations 64

Simulation Results, Scenario 5 Nov, 013 BASS XX Simulations 65

MDD Trial Primary Time Point? Week 4 or Week 6? Primary Endpoint? HRSD-17 or PaRTS-D Nov, 013 BASS XX Application 66

MDD Trial Results Nov, 013 BASS XX Application 67

MDD Trial Results Nov, 013 BASS XX Application 68

MDD Study Simulations Decision Rule 1: Calculate the posterior probability that the HRSD 17 effect size at Week 4 is larger than that of Week 6 * * Wk4 Wk6 P( d ( H ) d ( H ) Y) Decision Rule : Calculate 95% Credible Intervals for Week 4 and Week 6 effect sizes Wk4 * Wk 4 Wk 4 Wk6 * Wk6 Wk6 d( H ) d ( H ) d( H ) d( H ) d ( H ) d( H ) Priors for HRSD 17 placebo and treatment means at Week 4 ( H ) were assumed to be : 04 HWk4 N Y0 s0 14( HWk4) ~ N Y1 s1 ( ) ~ ( 15, 7 ), ( 1, 7 ) Wk4 Nov, 013 BASS XX Application 69

MDD Study Simulations Summary of Effect Sizes for HRSD-17 and PaRTS-D At Week 4, EM Results were Not Available Posterior Ordering of Effect Sizes for HRSD-17 Week 4 Week 6 EM 0.0% 100.0% Bayesian 47.7% 5.3% Nov, 013 BASS XX Application 70

MDD Study Simulations DIC Scores on HRSD-17 and PaRTS-D Nov, 013 BASS XX Application 71

Discussion and Conclusion We investigated whether or not the researchers can obtain reliable estimates of effect sizes without knowing treatment assignments. Instead of relying on the clinician s subjective evaluations, the suggested methodologies provide numerical assistance. We showed how to order secondary null hypotheses while the study is ongoing, without unblinding the treatments, without losing the validity of the testing procedure, and with maintaining the integrity of the trial. In our simulations, we used prior distributions whose hyper parameters reflect the true values of the model parameters. In MDD analyses, we used prior distributions whose hyper parameters reflect the original study design elements. Nov, 013 BASS XX Discussion 7

Discussion and Conclusion The EM algorithm frequently failed to generate posterior parameter estimates. Both the Bayesian and the EM algorithms overestimated treatment differences and standardized effect sizes. The Bayesian algorithms performed better than existing EM algorithm counterparts in ordering the standardized effect sizes. With the Bayesian algorithm, the posterior probability for identifying the ground-truth ordering increased both as a function of the effect size differences and as a function of the sample size. For large sample sizes, the proportion of times the true ordering was selected was high (above 35%) and the variability of standardized effect sizes was low. With the EM algorithm, the probability for identifying the ground-truth ordering was low (sometimes near zero) and the variability of standardized effect sizes was high. Nov, 013 BASS XX Discussion 73

Discussion THANK YOU! QUESTIONS? Nov, 013 BASS XX Discussion 74