SIGNAL RANKING-BASED COMPARISON OF AUTOMATIC DETECTION METHODS IN PHARMACOVIGILANCE

Transcription

1 SIGNAL RANKING-BASED COMPARISON OF AUTOMATIC DETECTION METHODS IN PHARMACOVIGILANCE A HYPOTHESIS TEST APPROACH Ismaïl Ahmed 1,2, Françoise Haramburu 3,4, Annie Fourrier-Réglat 3,4,5, Frantz Thiessard 4,5,6, Carmen Kreft-Jais 7, Ghada Miremont-Salamé 3,4,5, Bernard Bégaud 3,4,5, Pascale Tubert-Bitter 1,2 1: Inserm, U780, Villejuif - 2: Univ Paris-Sud, Villejuif - 3: Inserm, U657, Bordeaux - 4: Pellegrin Hospital, Bordeaux - 5: Université Victor Segalen Bordeaux 2, Bordeaux - 6: Inserm, U593, Bordeaux - 7: Afssaps, Saint-Denis Statistics for health registers and linked databases Open University, May 2009

2 Introduction (1) Pharmacovigilance : post-marketing surveillance Objectives Detection : identification as early as possible of ADRs non observed during clinical trials (rare, latent, affecting sub-groups) Characterization of new risks Data sources : spontaneous reporting systems Spontaneous reporting data : Features Life-size capture of events in the exposed population Under-reporting, unknown baseline and exposure incidence 2 / 30

3 Introduction (2) Framework Signals Very large databases (spontaneous reports) Incorporating a statistical tool for signal detection Statistical associations within the database (automatic detection) Potential adverse drug reactions (ADRs) 3 / 30

4 Introduction (3) Pharmacovigilance database Large contingency table crossing all the drugs (D) and all the adverse events (AE) French database : 672 D 820 AEs (80 % of the cells are empty) Adverse event i Other adverse events Drug j n ij Other Drugs Automatic signal detection methods Frequentist methods : Proportional Reporting Ratio (UK) Reporting Odds Ratio (Netherlands) Bayesian methods : Gamma Poisson Shrinkage (USA) Bayesian Confidence Propagation Neural Network (WHO) 4 / 30

5 Introduction (4) Limits of the current methods Thresholds for the statistics of interest are arbitrarily chosen Do not take into account the multiple comparisons Proposed Approach : To account for the multiple comparisons for the choice of a threshold Through the use recent False Discovery Rate (FDR) approaches Leads to alternative statistics of interest P-values for the frequentist methods posterior probabilities of the null hypothesis for the Bayesian methods 5 / 30

6 Outline Description of the current methods Extension to the multiple comparison setting Simulation study Application to the French Data Discussion 6 / 30

7 Some notations For a particular couple (AE i, D j ) Drug j Other Drugs Adverse event i n ij n i j n i. Other adverse events n īj n ī j n ī. n.j n. j n n ij : Number of reports involving AE i and D j n i. : Marginal count involving AE i n.j : Marginal count involving D j n : Total number of AE-D pairs counts 7 / 30

8 Frequentist methods Reporting Odds Ratio (ROR) van Puijenbroek et al For the adverse event-drug pair (i, j) ˆψ ij = nijnī j n īj n i j ln( ˆψ ij) is assumed to follow a normal distribution with variance : A signal is generated if cvar{ln( ˆψ ij)} = n ij n ī j n īj n i j ln( ˆψ ij) 1.96 var{ln( ˆψ ij)} 1/2 > 0 Proportional Reporting Ratio (PRR) Evans et al Same idea but with the relative risk as association measure of interest 8 / 30

9 Bayesian Gamma Poisson Shrinkage (GPS) DuMouchel (1999) Poisson - 2 gamma mixture model n ij e ij, λ ij Pn(λ ij e ij) avec e ij = λ ij ŵ Ga(ˆα 1, ˆβ 1) + (1 ŵ) Ga(ˆα 2, ˆβ 2) ni. n.j n where (ŵ, ˆα 1, ˆα 2, ˆβ 1, ˆβ 2) maximizes the marginal likelihood ˆw fbn {n ij; α 1, β 1/(β 1 + e ij)} + (1 w) f Bn {n ij; α 2, β 2/(β 2 + e ij)} Q ij Association measure λ ij = λ ij n ij, e ij λ ij w ij Ga(ˆα 1 + n ij, ˆβ 1 + e ij) + (1 w ij) Ga(ˆα 2 + n ij, ˆβ 2 + e ij) Signal generation Q 0.05 (λ ij) > 2 Szarfman et al. (2002) 9 / 30

10 Bayesian Confidence Propagation Neural Network (BCPNN) (1) Bate et al. (1998), Noren et al. (2006) Multinomial-Dirichlet model (n ij, n i j, n īj, n ī j) Mu(n, p ij, p i j, p īj, p ī j) with (p ij, p i j, p īj, p ī j) Di(α ij, α i j, α īj, α ī j) The hyperparameters depend on the cell counts The posterior distribution of (p ij, p i j, p īj, p ī j) is also a Dirichlet : (p ij, p i j, p īj, p ī j) Di(γ ij, γ i j, γ īj, γ ī j) with γ kl = α kl + n kl In particular p ij Be(γ ij, γ i j + γ īj + γ ī j) p i. = p ij + p i j Be(γ ij + γ i j, γ īj + γ ī j) p.j = p ij + p īj Be(γ ij + γ īj, γ i j + γ ī j) 10 / 30

11 Bayesian Confidence Propagation Neural Network (BCPNN) (2) Bate et al. (1998), Noren et al. (2006) Association measure IC ij = log 2 p ij p i. p.j! Ratio of beta distributions No analytic form Signal generation Q (IC ij) > 0 Interpolation model built from Monte Carlo simulations : Noren et al. (2006) 11 / 30

12 Description of the current methods Extension to the multiple comparison setting Simulation study Application to the French Data Discussion 12 / 30

13 False Discovery Rate and Pharmacovigilance Automatic signal detection methods are data mining tools Extension to the hypothesis testing framework relying on the recent developments in multiple comparison statistical field detection thresholds based on statistical criteria False Discovery Rate (Benjamini and Hochberg (1995)) E(proportion of false discoveries among the generated signals) used in the genomic data analysis adapted to massive comparisons and exploratory analysis 13 / 30

14 Frequentist methods - Proposed approach (1) New statistic of interest : P-values e.g ROR : for each cell, we want to test H 0ij : ψ ij ψ 0 «ln( The corresponding P-values p ij = 1 Φ ˆψ ij ) ln(ψ 0 ) var[ln( ˆψ ij )] 1/2 where Φ denotes the standard normal cdf The current decision rule corresponds to choose ψ 0 = 1 and generate signals for cells with p ij Exactly the same idea for the PRR method Alternative : mid-p-values from the Fisher s exact test (midrfet) 14 / 30

15 Frequentist methods - Proposed approach (2) FDR estimation P-values are assumed to follow a mixture of two distributions F (p) = π 0 F 0(p) + (1 π 0) F 1(p) F 0 (p) is the cdf of p under the null hypothesis F 1 (p) is the cdf of p under the alternative hypothesis For a P-value rejection region [0, γ] with γ ]0, 1] : FDR(γ) = π0f0(γ) F (γ) The main difficulty is to estimate π 0 qvalue Storey (2003), LBE Dalmasso et al. (2005) They are based on few distribution assumptions They provide an upper bound of the FDR They were developped for single null hypotheses uniform distribution of the p-values under H 0 (F 0 (γ) = γ) In our case the null hypothesis is one-sided The distribution of the p-value is not uniform But we can use those procedures on p = 1 2 p / 30

16 Bayesian methods - Proposed approach (1) New statistic of interest : posterior probability of the null hypothesis For each cell, we want to test H 0ij : λ ij R 0 for the GPS model p ij H 0ij : R 0 for the BCPNN model p i.p.j and thus to calculate the posterior probability of H 0ij Pr(λ ij R 0) for the GPS model Pr(IC ij ln(r 0)) for the BCPNN model The current decision rules correspond to R 0 = 2 and Pr(λ ij R 0) 0.05 for the GPS model R 0 = 1 and Pr(IC ij ln(r 0 )) for the BCPNN model 16 / 30

17 Bayesian methods - Proposed approach (2) : FDR estimation Based on the bayesian decision theory framework - Müller et al. (2004) Status z ij {0, 1} Decision d ij {0, 1} FDR FDP = P ij (1 zij)dij FDR = E[FDP] Pij dij Bayesian FDR estimation E[FDP data] = P ij vij dij P ij dij where v ij = P r(z ij = 0 data) is the posterior Pr. of H 0ij Pr(λ ij R 0) for the GPS model Pr(IC ij ln(r 0 )) for the BCPNN model d ij = 1 [vij α] 17 / 30

18 Description of the current methods Extension to the multiple comparison setting Simulation study Application to the French Data Discussion 18 / 30

19 Simulation study Data generation Model : n ij Mu(n, p ij ) from the French database p ij p i. w Di(n i.) p w.j Di(n.j ) = p ij = rw ij pw i. pẉ j P ij rw ij pw i. pẉ j log(rij w ) Lo(0, 0.5) From p ij n ij s and the true marginal probabilities : p i. = P j p ij real status of the cells according to ψ ij, and ψ 0 for the frequentist methods R ij = p ij p i. p and R 0 for the bayesian methods.j Simulation plan 500 simulated datasets The FDRs are calculated for cells with n ij 3 Simulation for ROR, midrfet, GPS and BCPNN methods Results are presented for {ψ 0, R 0} = 1 and 2 p.j = P i p ij 19 / 30

20 Simulation results - Comparison of the methods True FDRs (Monte Carlo estimation) ψ 0 = 1, R 0 = 1 ψ 0 = 2, R 0 = midrfet ROR BCPNN GPS Average number of generated signals (a) Average number of generated signals (b) 20 / 30

21 Simulation results - midrfet True and estimated FDRs ψ 0 = 1, R 0 = 1 ψ 0 = 2, R 0 = FDR FDR estimate Average number of generated signals (a) Overestimation of the FDR (as expected) Average number of generated signals (b) 21 / 30

22 Simulation results - ROR True and estimated FDRs ψ 0 = 1, R 0 = 1 ψ 0 = 2, R 0 = FDR FDR estimate Average number of generated signals (a) Normal approximation Underestimation Average number of generated signals (b) 22 / 30

23 Simulation results - GPS True and estimated FDRs ψ 0 = 1, R 0 = 1 ψ 0 = 2, R 0 = FDR FDR estimate Average number of generated signals (a) Average number of generated signals (b) 23 / 30

24 Simulation results - BCPNN True and estimated FDRs ψ 0 = 1, R 0 = 1 ψ 0 = 2, R 0 = FDR FDR estimate Average number of generated signals (a) Underestimation Average number of generated signals (b) 24 / 30

25 Simulation results - Comparison of the methods True and estimated FDRs ψ 0 = 1, R 0 = 1 ψ 0 = 2, R 0 = midrfet ROR BCPNN GPS Average number of generated signals (a) Average number of generated signals (b) 25 / 30

26 Description of the current methods Extension to the multiple comparison setting Simulation study Application to the French Data Discussion 26 / 30

27 Application to the French database ψ 0 = 1, R 0 = 1 ψ 0 = 2, R 0 = midrfet ROR BCPNN GPS Number of generated signals (a) Current decision rules Method Sig. FDR ROR BCPNN GPS Number of generated signals (b) Based on the FDR : e.g GPS Sig. R 0 = R 0 = / 30

28 Description of the current methods Extension to the multiple comparison setting Simulation study Application to the French Data Discussion 28 / 30

29 Discussion Extension of the current methods to the multiple comparison framework No modification of the model New decision rules The FDR is calculated within the database Spontaneous reporting database several sources of bias true associations in the database may not reflect the situation in the population It is a measure for evaluating and comparing the performances of the automatic signal detection methods Close performances for all the automatic methods The GPS model provides better FDR estimates 29 / 30

30 References I. Ahmed et al. FDR estimation for frequentist pharmacovigilance signal detection methods. Biometrics, In press. I. Ahmed et al. Bayesian pharmacovigilance signal detection methods revisited in a multiple comparison setting. Statistics in Medicine, In press. A. Bate et al. A bayesian neural network method for adverse drug reaction signal generation. European Journal of Clinical Pharmacology, 54(4) : , Jun Y. Benjamini and Y. Hochberg. Controlling the false discovery rate : a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B, 57(1) : , C. Dalmasso et al. A simple procedure for estimating the false discovery rate. Bioinformatics, 21(5) : , Mar W. DuMouchel. Bayesian data mining in large frequency tables, with an application to the fda spontaneous reporting system. The American Statistician, 53(3) : , P. Müller et al. Optimal sample size for multiple testing : the case of gene expression microarrays. Journal of The American Statistical Association, 99 : , G. N. Norén et al. Extending the methods used to screen the who drug safety database towards analysis of complex associations and improved accuracy for rare events. Statistics in Medicine, 25(21) : , J. D. Storey. The positive false discovery rate : A bayesian interpretation and the q-value. The Annals of Statistics, 31(6) : , A. Szarfman et al. Use of screening algorithms and computer systems to efficiently signal higher-than-expected combinations of drugs and events in the US FDA s spontaneous reports database. Drug Safety, 25(6) : , E. P. van Puijenbroek et al. A comparison of measures of disproportionality for signal detection in spontaneous reporting systems for adverse drug reactions. Pharmacoepidemiology and Drug Safety, 11(1) :3 10, / 30