Results of a simulation of modeling and nonparametric methodology for count data (from patient diary) in drug studies Erhard Quebe-Fehling Workshop Stat. Methoden für korrelierte Daten Bochum 24-Nov-06 Report on work in collaboration with Luda Rekeda, David Smith and José Pinheiro
Outline Project background and counts from diary Original nonparametric analysis Motivation for change and objective of simulation Simulation methodology: Data generation Primary methods compared Simulation results Discussion and conclusion 2 EQF Simulation of modeling and nonparametric methodology for counts from diary / Bochum 24Nov06
Background Emselex / Enablex / darifenacin (DAR) Drug to treat Overactive Bladder (OAB) Approved in EU (Emselex ) and US (Enablex ) 2004 and other markets The clinical development program which formed the basis for approval had been performed by another company The clinical endpoint considered is a count variable, the number of incontinence episodes experienced by a patient in a trial European Public Assessment Report including scientific discussion available under http://www.emea.eu.int/humandocs/humans/epar/emselex/emselex.htm 3 EQF Simulation of modeling and nonparametric methodology for counts from diary / Bochum 24Nov06
Diary data, primary variable Pivotal studies conducted Parallel studies, randomized and double-blind Daily electronic diary keeping noting date and time 1-2 weeks at baseline, Week 2, Week 6, Week 12 Primary variable (Absolute) change from baseline to Week 12 in number of incontinence episodes (IE) Secondary variables (amongst others) Micturition frequency Number of urgency episodes 4 EQF Simulation of modeling and nonparametric methodology for counts from diary / Bochum 24Nov06
Original nonparametric analysis Estimates and tests for treatment difference (Stratified) Hodges-Lehmann estimate (weighted median of Walsh differences) for treatment difference Wilcoxon rank-sum test (initially in Development Phase 3) Stratified Wilcoxon rank-sum test, stratified by baseline severity (later) 3 Baseline strata in mean # IE/week: below 14, between 14 and 21, above 21 Last Observation Carried Forward (of weekly means) was applied 5 EQF Simulation of modeling and nonparametric methodology for counts from diary / Bochum 24Nov06
Reasons for and limitations of nonparametric methodology Reasons for nonparametric analysis Skewed distribution Some patient data would show very large changes from baseline Limitations of nonparametric analysis Only a few strata can be incorporated Hodges-Lehmann estimate relatively difficult to understand 6 EQF Simulation of modeling and nonparametric methodology for counts from diary / Bochum 24Nov06
Motivation for wish to change methodology to modeling approach Reasons Better incorporation of covariates (more and not necessarily as strata) Better fit of data leading to more powerful tests and more precise estimates Which model? Distributions based on the Poisson distribution are natural ones for counts So this is the origin for alternative models considered 7 EQF Simulation of modeling and nonparametric methodology for counts from diary / Bochum 24Nov06
Motivation / objective of simulation Compare modeling approach and nonparametric methodology Try to show superiority of modeling approach over nonparametric methodology Focus on power 8 EQF Simulation of modeling and nonparametric methodology for counts from diary / Bochum 24Nov06
Simulation methodology (SM 1): Data generation Use data of placebo-treated patients from the 3 pivotal fixed dose Phase 3 studies A resampling approach was used to produce simulated datasets with two treatment arms: placebo and active drug. About 170 patients were assigned to each treatment group The only change induced in the data was a reduction in the number of IE for the half of the patients assigned to active drug; the remaining characteristics of the data (e.g., drop-out times, baseline covariates) remained the same For each given percentage reduction, the resampling procedure was repeated 1,000 times 9 EQF Simulation of modeling and nonparametric methodology for counts from diary / Bochum 24Nov06
SM (2): Data generation (2) and methods comparison The simulations were carried out adopting percentage reductions of 0%, 5%, 10%,, 50% of the DAR group compared to the placebo group at Week 12 When looking over time the Week 12 reduction was assumed to be the maximal change. The reduction at Week 2 and Week 6 was assumed to be 80% and 95% of the change at Week 12, respectively Several analysis methods run on each dataset generated (1000 times) Calculate p-value in each case for treatment comparison at Week 12, Power = % of p-values < 0.05 10 EQF Simulation of modeling and nonparametric methodology for counts from diary / Bochum 24Nov06
SM (3): Primary methods (tests) compared Nonparametric tests Wilcoxon test Van Elteren test (Stratified Wilcoxon) as used previously on Change from bsl to Week 12, with carry-forward Models based on the Poisson distribution PAE GLMM Normal model (with Identity link) NIM = ANCOVA in principle Remark: Treatment only used as factor in the models due to difficulty to incorporate further factors into nonpar analysis 11 EQF Simulation of modeling and nonparametric methodology for counts from diary / Bochum 24Nov06
(Stratified) Wilcoxon test Testing Rank test / test based on U-statistics Statistic approximately normally distributed when sample sizes become large Estimation (Stratified) Hodges-Lehmann estimate (Weighted) median of Walsh differences 12 EQF Simulation of modeling and nonparametric methodology for counts from diary / Bochum 24Nov06
Models based on the Poisson distribution for weekly means PAE - Model on weekly mean number of IE Distributions based on the Poisson but over-dispersed w.r.t. that distribution Autoregressive Covariance structure assumed Empirically based GEE (Generalized Estimation Equations) estimate Combined model of Baseline, Week 2, Week 6, Week 12 weekly means (no carry-forward) Estimation and Testing GEE see next slide 13 EQF Simulation of modeling and nonparametric methodology for counts from diary / Bochum 24Nov06
Estimation and testing with GEEs Estimation Starting point score equations (generalized from likelihood score equations) Algorithm called Iteratively (re)weighted least squares equal to MLE in normal case with identity link Testing Contrast statement in SAS So, Generalized Score Statistic (mentioned for SAS Proc Genmod) LR tests in mentioned special cases 14 EQF Simulation of modeling and nonparametric methodology for counts from diary / Bochum 24Nov06
Poisson model based daily counts GLMM Model based on daily counts Generalized Linear Mixed Model Hierarchical combined model of Baseline, Week 2, Week 6, Week 12 daily values (no carry-forward) Random effects are used to account for individual differences Estimation Restricted penalized quasi-likelihood (REPQL), based on iteratively reweighted restricted maximum likelihood estimation for an approximate linear mixed-effects model Testing Wald-type t-test based on the asymptotic (normal) distribution of the fixed effects estimates 15 EQF Simulation of modeling and nonparametric methodology for counts from diary / Bochum 24Nov06
ANCOVA (Analysis of Covariance) NIM Normal distribution with identity link Normal distribution Identity link Model based GEE (Generalized Estimation Equations) estimate Combined model of Baseline, Week 2, Week 6, Week 12 weekly means (no carry-forward) Model on absolute changes Estimation and Testing LS estimates and tests = ML estimates and tests 16 EQF Simulation of modeling and nonparametric methodology for counts from diary / Bochum 24Nov06
Simulation results Power comparison NIM PAE GLMM WILCOXON Van Elteren 100 80 Power (%) 60 40 20 0 0 10 20 30 40 Average reduction in number of episodes (%) 17 EQF Simulation of modeling and nonparametric methodology for counts from diary / Bochum 24Nov06
Discussion (1) Although PAE and NIM are best in the mid-range of simulated %changes there has been little difference in power between nonpara- metric methods and those methods using models based on the Poisson distribution The results of this simulation using data from placebo-treated patients in the pivotal fixed dose studies are in contrast to the results of other applications of modeling based on the Poisson distribution in the DAR project where there appeared to be a clear benefit GLMM used in response to EMEA (EU) for Study 1001 PAE used as post-hoc analysis for a Phase IV study However, these results do not harm either as they don t contradict other analyses Especially to be noted that (nonparametric) Wilcoxon and stratified Wilcoxon-test show similar power in this simulation; so baseline appeared not to be important 18 EQF Simulation of modeling and nonparametric methodology for counts from diary / Bochum 24Nov06
Discussion (2) Possibly the mix of the 3 somewhat different studies has led to distributions with heavy tails in the two treatment groups (especially many patients with relatively large outliers in data); nonparametric methods should be good in such cases In the Poisson modeling there are perhaps too many approximations meeting the distributional assumption is counterbalanced by approximations in the calculations GLMM has possibly more approximations than PAE? Simulation methodology was perhaps in favor of nonparametric methodology; e.g. with the implemented benefit over time, LOCF doesn t make a big difference and there is not much advantage for modeling All models based on the Poisson distribution need to include a second parameter to handle the displayed over-dispersion w.r.t. Poisson 19 EQF Simulation of modeling and nonparametric methodology for counts from diary / Bochum 24Nov06
Discussion (3) Cleaner simulations from theoretical models where the corresponding analysis methods should be best could be useful but are beyond the scope of this exercise Other points to be generally considered (less relevant in this application) Models could have incorporated more factors and covariates than just treatment and baseline Relative (% reduction over placebo) versus absolute reduction Handling of missing values (carry-forward) 20 EQF Simulation of modeling and nonparametric methodology for counts from diary / Bochum 24Nov06
Conclusion In a simulation using the IE count data of placebo-treated patients in the three pivotal fixed dose Phase 3 studies there has been little difference in power between nonparametric methods and Poisson models The analyses based on distributions representing over-dispersion w.r.t. Poisson appear to be reasonable However, the extra benefit in power based on the Poisson-based methods compared to the nonparametric analysis cannot be justified for these data 21 EQF Simulation of modeling and nonparametric methodology for counts from diary / Bochum 24Nov06
Recommendations Further research required Starting from scratch: go for modeling With established nonparametric methodology: keep nonparametric methodology as primary and perform modeling in supplementary fashion 22 EQF Simulation of modeling and nonparametric methodology for counts from diary / Bochum 24Nov06