A comparison of estimation methods in nonlinear mixed effects models using a blind analysis

Transcription

1 A comparison of estimation methods in nonlinear mixed effects models using a blind analysis Pascal Girard, PhD EA 3738 CTO, INSERM, University Claude Bernard Lyon I, Lyon France Mentré, PhD, MD Dept Biostatistics, INSERM U738, University Paris VII, Paris France Project objectives Compare various recently developed and classical algorithms for non-linear mixed effect models using ML parametric estimate: Bias Precision Standard Error estimates PAGE 05 Pampeluna 2

2 The simulation model: one compartment, 1 st order absorption reparametrized to avoid flip-flop With CP Dose * Ka V *( Ka-Ke) () t = ( exp( - Ke t) - exp( - Ka t) )*exp( ε ) V = θ 1 EXP(η 1 ) KE = θ 2 EXP(η 2 ) KA = KE+ θ 3 EXP(η 3 ) θ 1, θ 2, θ 3 fixed effect parameters, greater than 0, for V, KA and KA-KE; η 1, η 2, η 3 are random effects representing between individual variability normally distributed, with a block VARIANCE matrix Ω : (η 1, η 2, η 3 ) ~ N ( 0,0,0), 2 ϖ 1 ϖ 21 ϖ 31 ϖ 31 ϖ 32 2 ϖ 3 ε random normally distributed effect representing within residual variability with ε ~N(0, σ 2 ). ϖ ϖ ϖ PAGE 05 Pampeluna 3 Simulation design simulated datasets individuals (ID) / dataset 4 samples / ID at times 1, 4, 12, 24 hrs + random shift: ε 1 ~ N(0, SD=0.25) at 1 and 4 hrs ε 2 ~ N(0, SD=2) at 12 and 24 hrs Missing points at random P(missing)=0.25 PAGE 05 Pampeluna 4

3 Parameters used for simulation V Ke Ka-Ke Value CV IIV 47% 81% 15% Corr IIV V Ke Ka-Ke V Ke Ka-Ke CV Resid = 25% PAGE 05 Pampeluna 5 Example of 4 simulated datasets datasim.txt Run = 1 datasim.txt Run = 2 Cp Cp Time datasim.txt Run = Time datasim.txt Run = 4 Cp Cp Time Time PAGE 05 Pampeluna 6

4 Blinded analysis Exact model was given to each participant who ignored the true simulation parameters Suggested initial parameter estimates provided only for fixed effects: V = 30 (true=27.2) Ke = 0.3 (true = 0.232) Ka-Ke =0.5 (true=0.304) Data sets blinded and send on 7th March 2005; Results received between March 25 th and June 10 th 05 PAGE 05 Pampeluna 7 Algorithms involved in the comparison Approximate Likelihood SAS proc nlmixed FO approximation NONMEM V and VI FOCE and (Splus 6.2) Lindstrom MJ, Bates DM. Nonlinear mixed effects models for repeated measures data. Biometrics 1990; 46: ITBS (MW\Pharm & MulTifit) Mentré F, Gomeni R. A two-step iterative algorithm for estimation in nonlinear mixed-effect models with an evaluation in population pharmacokinetics. J Biopharm Stat 1995; 5: Exact Likelihood 3 Stochastic methods PEM MCPEM SAEM One method based on approximated integral of exact likelihood SAS proc nlmixed Adaptive Gaussian Quadrature: 3 points per dimension (27 points in all) used to integrate the multidimensional exact likelihood PAGE 05 Pampeluna 8

5 Compilation of results: Relative error, bias and Root Mean Square Error For Ω, computation of 3 1 Det( Ω with true=1 est Ω true ) Relative Estimation error (RER) for each simulation presented as boxplots: RER= (Est True) / True Wilcoxon test for testing H0: RER=0 vs. H1: RER 0 Bias % = mean(est True) / True Precision: relative Root Mean Square Error (RMSE) : 2 ( Est True) ) 2 / % RMSE = mean True Results send out to participants on June 6 th PAGE 05 Pampeluna 9 RESULTS

6 All algorithms converged and provided estimates for all simulated data sets except FOCE (NONMEM and ) 1 st try 1 2 nd try NONMEM Version V 2 NONMEM Version VI 2 (Beta) Only Est. step successful % 88% % 49% Est and COV successful % 70% 47% 49% 1. Biased estimates with 1 st set of results 2 nd set of results were 2. repeated search with perturbed initial estimates Almost perfect matching on failed run# 7, 9, 28, 45, 47, 53, 68, 69, 76, 80, 90 PAGE 05 Pampeluna 11 V true= Pval Wilcox= N= NM V NM VI MWPharm MultiFit SAS FO SAS GQ MCPEM PEM SAEM Ke true= Pval Wilcox= N= NM V NM VI MWPharm MultiFit SAS FO SAS GQ MCPEM PEM SAEM Ka-Ke true= NM V NM VI MWPharm MultiFit SAS FO SAS GQ MCPEM PEM SAEM Pval Wilcox= N= PAGE 05 Pampeluna 12

7 om11 V true= Pval Wilcox= N= NM V NM VI MWPharm MultiFit SAS FO SAS GQ MCPEM PEM SAEM om22 Ke true= Pval Wilcox= N= NM V NM VI MWPharm MultiFit SAS FO SAS GQ MCPEM PEM SAEM om33 Ka-Ke true= NM V NM VI MWPharm MultiFit SAS FO SAS GQ MCPEM PEM SAEM Pval Wilcox= N= PAGE 05 Pampeluna 13 Cor12 V.Ke true= Pval Wilcox= N= NM V NM VI MWPharm MultiFit SAS FO SAS GQ MCPEM PEM SAEM Cor13 V.KaKe true= Pval Wilcox= N= NM V NM VI MWPharm MultiFit SAS FO SAS GQ MCPEM PEM SAEM Cor23 Ke.KaKe true= NM V NM VI MWPharm MultiFit SAS FO SAS GQ MCPEM PEM SAEM Pval Wilcox= N= PAGE 05 Pampeluna 14

8 Det(OM.est*inv(OM.true))**(1/3) true= Pval Wilcox= N= NM V NM VI MWPharm MultiFit SAS FO SAS GQ MCPEM PEM SAEM S**2 true= Pval Wilcox= 0.01 N= NM V NM VI MWPharm MultiFit SAS FO SAS GQ MCPEM PEM SAEM PAGE 05 Pampeluna 15 Bias % for fixed effects +<5% +5-10% % +>20% TH1 (V=27.2) TH2 (Ke=0.23) TH3 (Ka-Ke=0.30) NM V NM VI * ** MWPharm MultiFit SAS FO SAS GQ MCPEM PEM SAEM * NM VI based on 49 successful runs out of ** 2 nd chance and 88 successful runs out of PAGE 05 Pampeluna 16

9 Bias % for variance omega**2 +<20% % +50-% +>% om V (tr=0.22) om Ke (tr=0.65) om33 (tr=0.02) NM V NM VI * ** MWPharm MultiFit SAS FO SAS GQ MCPEM PEM SAEM * NM VI based on 49 successful runs out of ** 2 nd chance and 88 successful runs out of PAGE 05 Pampeluna 17 Bias % for correlation coefficients +<20% +20-% +-500% +>500% V-Ke (tr=0.91) V-KaKe (tr=0.33) Ke-KaKe (tr=0.01) NM V NM VI * ** MWPharm MultiFit SAS FO SAS GQ MCPEM PEM SAEM * NM VI based on 49 successful runs out of ** 2 nd chance and 88 successful runs out of PAGE 05 Pampeluna 18

10 Bias % for det( Ω.inv(Ω tr ) ) and S 2 +<10% % % +>50% det( OM.inv(OMtr) ) (true=1) S**2 (true=0.06) NM V NM VI * ** MWPharm MultiFit SAS FO SAS GQ MCPEM PEM SAEM * NM VI based on 49 successful runs out of ** 2 nd chance and 88 successful runs out of PAGE 05 Pampeluna 19 RMSE Results

11 RMSE % for fixed effects <10% 10-20% 20-50% >50% TH1 (V=27.2) TH2 (Ke=0.23) TH3 (Ka-Ke=0.30) NM V NM VI * ** MWPharm MultiFit SAS FO SAS GQ MCPEM PEM SAEM * NM VI based on 49 successful runs out of ** 2 nd chance and 88 successful runs out of PAGE 05 Pampeluna 21 RMSE % for variance omega**2 <20% 20-50% 50-% >% om V (tr=0.22) om Ke (tr=0.65) om33 (tr=0.02) NM V NM VI * ** MWPharm MultiFit SAS FO SAS GQ MCPEM PEM SAEM * NM VI based on 49 successful runs out of ** 2 nd chance and 88 successful runs out of PAGE 05 Pampeluna 22

12 RMSE % for correlation coefficients <20% 20-% -500% >500% V-Ke (tr=0.91) V-KaKe (tr=0.33) Ke-KaKe (tr=0.01) NM V NM VI * ** MWPharm MultiFit SAS FO SAS GQ MCPEM PEM SAEM * NM VI based on 49 successful runs out of ** 2 nd chance and 88 successful runs out of PAGE 05 Pampeluna 23 RMSE % for det( Ω.inv(Ω tr ) ) and S 2 <20% 20-50% 50-% >% det( OM.inv(OMtr) ) (true=1) S**2 (true=0.06) NM V NM VI * MWPharm MultiFit SAS FO SAS GQ MCPEM PEM SAEM * NM VI based on 49 successful runs out of ** 2 nd chance and 88 successful runs out of PAGE 05 Pampeluna 24

13 Comparison between 1 st set and 2 nd one for : «Because we were surprised with the blinded results regarding NLME, we went back to the code we used to produce the fits and realized that we had specified the model in a unstable way that created convergence problems in NLME. We re-defined the model formulation in S+ and re-ran the analyses. The fits were considerably more stable than before.» Hsu & Pinheiro V Ke Ka-Ke st set 2nd set 1st set 2nd set 1st set 2nd set om11 V om22 Ke om33 Ka-Ke st set 2nd set st set 2nd set st set 2nd set PAGE 05 Pampeluna 25 Use of non-parametric ranks to classify all methods Rank TH1 (V=27.2) NM V NM VI * ** MWPharm MultiFit SAS FO SAS GQ MCPEM PEM 0.9 SAEM Quote = rank k= 1 10 Quote k * NM VI is beta version: only 49% convergence ** : only 88% convergence with 2 nd try PAGE 05 Pampeluna 26

14 Classification of methods based on their sum of ranks for bias and RMSE SAEM PEM BIAS SAS GQ PEM 84 SAS GQ SAEM PEM MCPEM NM V NM VI* SAEM RMSE ** MWPharm MultiFit MCPEM SAS FO SAEM 89 MCPEM 78 PEM 73 SAS GQ 70 NM V 64 NM VI 59 MWPharm 58 MultiFit SAS FO 17 PAGE 05 Pampeluna 27 CPU Times Adj_time=time_one_run GHz SAEM 1 PEM MCPEM SAS GQ SAS FO MultiFit MWPhar m Nlme NM VI NM V 3 GHz CPU time for 1 run (sec) 6 Adjusted CPU times SAEM, 32 PEM, 612 MCPEM, 360 SAS GQ, 19 SAS FO, 2 MultiFit, 13 MWPharm, 40, 11 NM VI, 6 NM V, Time (sec) 1. A compiled version would probably be considerably faster than actual implementation in Matlab for PEM and SAEM 2. For a successful nls + run 3. For a successful NONMEM run (EST & COV) PAGE 05 Pampeluna 28

15 Conclusions Methods without approximation of the likelihood (either stochastic methods or GQ) show better performances than others with an approximation. Reduction of BIAS and RMSE with stochastic methods is obtained only with slight CPU time increase for SAEM FO in SAS shows, as expected, large bias FOCE methods, either implemented in or in NONMEM have either convergence problems or computational ones for covariance matrix with these data sets Future NONMEM VI does not solve NM V issues but rather seems to be even more stringent than version V Datasets will be made available soon on a web site there are a number of buts and ifs when you make comparisons of this sort (Niclas) PAGE 05 Pampeluna 29 Acknowledgements for accepting to participate into this blind comparison (in alphabetical order) Serge Guzy (MCPEM, Xoma corp., USA) Niclas Jonsson (NONMEM FOCE V & VI. Univ. Uppsala, Sweden) Marc Lavielle (SAEM, Univ. Paris Sud Orsay, FR) Bob Leary (PEM; Pharsight, USA) José Pinheiro and Chyi-Hung Hsu (Splus. Novartis, USA) Hans Proost (MW\Pharm and MultiFit. Univ. Groningen, NL) Russ Wolfinger (SAS PROC NLMIXED, USA) PAGE 05 Pampeluna 30