Estimation of Optimally-Combined-Biomarker Accuracy in the Absence of a Gold-Standard Reference Test

Estimation of Optimally-Combined-Biomarker Accuracy in the Absence of a Gold-Standard Reference Test L. García Barrado 1 E. Coart 2 T. Burzykowski 1,2 1 Interuniversity Institute for Biostatistics and Statistical Bioinformatics (I-Biostat) 2 International Drug Development Institute (IDDI) 1 / 26

Outline Problem setting Accuracy definition Optimal combination of biomarkers Absence of gold-standard reference Bayesian latent-class mixture model Naive prior definition Controlled prior definition Simulation study Data and model Results Conclusions 2 / 26

Problem setting Outline Problem setting Accuracy definition Optimal combination of biomarkers Absence of gold-standard reference Bayesian latent-class mixture model Naive prior definition Controlled prior definition Simulation study Data and model Results Conclusions 3 / 26

Problem setting Problem setting Establish accuracy of a combination of biomarkers in the absence of a gold-standard reference test Area under the Receiver Operating Characteristics (ROC) curve (AUC) as measure of accuracy Choose combination of biomarkers that maximizes AUC Imperfect reference test leads to biased estimates of accuracy => To this end a Bayesian latent-class mixture model will be proposed 4 / 26

Problem setting Accuracy definition Area under the Receiver Operating Characteristics curve Classification example Classification example ROC curve 0.0 0.1 0.2 0.3 0.4 Control Disease 0.0 0.1 0.2 0.3 0.4 Control Disease Se1 0.0 0.2 0.4 0.6 0.8 1.0 AUC=0.98 AUC=0.76 2 0 2 4 2 0 2 4 0.0 0.2 0.4 0.6 0.8 1.0 Biomarker value Biomarker value 1 Sp 5 / 26

Problem setting Optimal combination of biomarkers Data assumptions and notation Underlying true biomarker distribution Mixture of two K -variate normal distributions by true disease status (D) Y D=0 N K (µ 0, Σ 0 ) Y D=1 N K (µ 1, Σ 1 ) Se: Unknown sensitivity of the reference test (T) Sp: Unknown specificity of the reference test (T) θ: Unknown true prevalence of disease in the data set Reference test is imperfect Conditionally on true disease status, misclassification independent of biomarker value Ignoring will UNDERESTIMATE performance of biomarker 6 / 26

Problem setting Optimal combination of biomarkers AUC of optimal combination of biomarkers According to Siu and Liu (1993) the linear combination maximizing AUC is of the form: a Y D=0 N(a µ 0, a Σ 0a) a Y D=1 N(a µ 1, a Σ 1a) For which: a (Σ 0 + Σ 1) 1 (µ 1 µ 0 ) Area Under the ROC Curve: AUC OptComb = Φ { } ((µ 1 µ 0 ) (Σ 0 + Σ 1) 1 (µ 1 µ 0 )) 1 2 This is all under the assumption of a gold standard reference test. We propose to extend this to the imperfect reference test case. 7 / 26

Problem setting Optimal combination of biomarkers AUC of optimal combination of biomarkers According to Siu and Liu (1993) the linear combination maximizing AUC is of the form: a Y D=0 N(a µ 0, a Σ 0a) a Y D=1 N(a µ 1, a Σ 1a) For which: a (Σ 0 + Σ 1) 1 (µ 1 µ 0 ) Area Under the ROC Curve: AUC OptComb = Φ { } ((µ 1 µ 0 ) (Σ 0 + Σ 1) 1 (µ 1 µ 0 )) 1 2 This is all under the assumption of a gold standard reference test. We propose to extend this to the imperfect reference test case. 8 / 26

Problem setting Absence of gold-standard reference Underlying versus observed data Ignoring misclassification in imperfect reference test will lead to bias of estimated accuracy: True distributions VS observed data Observed Control (T=0) Observed Disease (T=1) True Control (D=0) True Disease (D=1) In example: conditionally independent misclassification Misclassification in reference test causes skewed observed distributions Goal: retrieve accuracy of true underlying biomarker by observed data 2 0 2 4 6 Biomarker value 9 / 26

Bayesian latent-class mixture model Outline Problem setting Accuracy definition Optimal combination of biomarkers Absence of gold-standard reference Bayesian latent-class mixture model Naive prior definition Controlled prior definition Simulation study Data and model Results Conclusions 10 / 26

Bayesian latent-class mixture model Full data likelihood L(µ 0, µ 1, Σ 0, Σ 1, θ, Se, Sp Y, T, D) ( N { = θse t i (1 Se) (1 t i ) 1 (2π)K Σ EXP 1 } ) d i 1 2 (Yi µ 1 ) Σ 1 1 (Y i µ 1 ) i=1 ( { (1 θ)(1 Sp) t i Sp (1 t i ) 1 (2π)K Σ EXP 1 } ) (1 d i ) 0 2 (Yi µ 0) Σ 1 0 (Y i µ 0 ) 11 / 26

Bayesian latent-class mixture model Naive prior definition Naive prior definition Hyperprior θ Uniform(0.1,0.9) Priors D i Bernoulli(θ) (Observation i: 1,...,N) µ kj N(0,10 6 ) (Disease indicator j: 0, 1; Biomarker k: 1,...,K) Σ 1 j Wish(S,K) (Disease indicator j: 0, 1) with S = VarCov-matrix of observed control group Se = Sp Beta(1,1)T(0.51, ) [Non-informative] OR Se = Sp Beta(10,1.764706)T(0.51, ) [Informative] 12 / 26

Bayesian latent-class mixture model Naive prior definition Se/Sp Beta(10,1.764706) Prior Mean = 0.85 Var = 0.009988479 Equal-tail 95%-probability interval: 0.6078-0.9834 Informative Se/Sp prior Density 0 1 2 3 4 0.5 0.6 0.7 0.8 0.9 1.0 Se or Sp 13 / 26

Bayesian latent-class mixture model Naive prior definition Implied prior AUC Density 0 10 20 30 40 50 Implied AUC prior 0.0 0.2 0.4 0.6 0.8 1.0 AUC Prior specification is used commonly (e.g. O Malley and Zou (2006)) Uninformative mixture component priors lead to prior point mass distribution centred at 1 for AUC Extremely informative prior for component of interest! 14 / 26

Bayesian latent-class mixture model Controlled prior definition Controlled prior definition (AUC) AUC OptComb { } = Φ ((µ 1 µ 0 ) (Σ 0 + Σ 1 ) 1 (µ 1 µ 0 )) 1 2 { } = Φ ((µ 1 µ 0 ) L L(µ 1 µ 0 )) 1 2 With L = the Cholesky factor of (Σ 0 + Σ 1 ) 1 Set = L(µ 1 µ 0 ) N K (κ, Ψ) µ 0k N(0, 10 6 ) (k: 1,...,K) µ 1 = L 1 + µ 0 15 / 26

Bayesian latent-class mixture model Controlled prior definition Implied priors AUC For κ = 0 0, σi = 0.7 and ρ ij = 0.6 0 [i,j: 1,...,K] AUC Kappa=(0.0,0.0,0.0),Sd=(0.7,0.7,0.7),Cor=(0.6,0.6,0.6) Simulated AUC fy 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Expansion appr. Less informative prior distribution for AUC Prior on gives control over informativeness AUC prior Similar issue for Σ 0 and Σ 1 (Wei, Y. & Higgins, P.T., 2013) 0.5 0.6 0.7 0.8 0.9 1.0 Y 16 / 26

Simulation study Outline Problem setting Accuracy definition Optimal combination of biomarkers Absence of gold-standard reference Bayesian latent-class mixture model Naive prior definition Controlled prior definition Simulation study Data and model Results Conclusions 17 / 26

Simulation study Data and model 100 data sets for 3 biomarkers 4 data generating settings R = I K vs R I K Σ 0 = Σ 1 vs Σ 0 Σ 1 Parameter settings N = 100, 400 or 600 θ = 0.5 Se = Sp = 0.85 Mixture component parameters set such that: AUC 1 = AUC 2 = AUC 3 = 0.75 AUC OptimalCombination R=IK = 0.88 AUC OptimalCombination R IK = 0.78 Analysis model: General model fitting 2 unstructured covariance matrices 18 / 26

Simulation study Results AUC Results (Average of median posterior AUC) In cell: Average of median posterior AUC (Standard Error) [Number of converged fits] Data Gen. Sample Size Model Se/Sp True VarCov Corr N100 N400 N600 prior AUC Σ 0 = Σ 1 ρ 0 GS 0.784 0.689 (0.039) [100] 0.695 (0.022) [100] 0.694 (0.018) [100] Σ 0 Σ 1 ρ = 0 GS 0.879 0.756 (0.037) [100] 0.762 (0.019) [100] 0.760 (0.015) [100] Σ 0 Σ 1 ρ 0 GS 0.784 0.693 (0.040) [100] 0.697 (0.022) [100] 0.696 (0.018) [100] Gold Standard model fit leads to significant underestimation 19 / 26

Simulation study Results AUC Results (Average of median posterior AUC) Data Gen. Model Sample Size VarCov Corr Se/Sp True prior AUC N100 N400 N600 Σ 0 = Σ 1 ρ 0 GS 0.784 0.689 (0.039) [100] 0.695 (0.022) [100] 0.694 (0.018) [100] Σ 0 = Σ 1 ρ 0 NINF 0.784 0.812 (0.056) [18] 0.803 (0.034) [61] 0.796 (0.030) [60] Σ 0 Σ 1 ρ = 0 GS 0.879 0.756 (0.037) [100] 0.762 (0.019) [100] 0.760 (0.015) [100] Σ 0 Σ 1 ρ = 0 NINF 0.879 0.879 (0.041) [89] 0.878 (0.022) [100] 0.882 (0.019) [100] Σ 0 Σ 1 ρ 0 GS 0.784 0.693 (0.040) [100] 0.697 (0.022) [100] 0.696 (0.018) [100] Σ 0 Σ 1 ρ 0 NINF 0.784 0.797 (0.047) [72] 0.787 (0.030) [100] 0.785 (0.025) [100] Proposed model with controlled prior leads to unbiased estimates Non-convergence issues for low information data Small sample size Correlation Over-specified VarCov model Increased sample size increases precision 20 / 26

Simulation study Results AUC Results (Average of median posterior AUC) Data Gen. Model Sample Size VarCov Corr Se/Sp True prior AUC N100 N400 N600 Σ 0 = Σ 1 ρ 0 GS 0.784 0.689 (0.039) [100] 0.695 (0.022) [100] 0.694 (0.018) [100] Σ 0 = Σ 1 ρ 0 NINF 0.784 0.812 (0.056) [18] 0.803 (0.034) [61] 0.796 (0.030) [60] Σ 0 = Σ 1 ρ 0 INF 0.784 0.791 (0.053) [33] 0.790 (0.033) [83] 0.782 (0.027) [86] Σ 0 Σ 1 ρ = 0 GS 0.879 0.756 (0.037) [100] 0.762 (0.019) [100] 0.760 (0.015) [100] Σ 0 Σ 1 ρ = 0 NINF 0.879 0.879 (0.041) [89] 0.878 (0.022) [100] 0.882 (0.019) [100] Σ 0 Σ 1 ρ = 0 INF 0.879 0.870 (0.045) [96] 0.876 (0.022) [100] 0.879 (0.019) [100] Σ 0 Σ 1 ρ 0 GS 0.784 0.693 (0.040) [100] 0.697 (0.022) [100] 0.696 (0.018) [100] Σ 0 Σ 1 ρ 0 NINF 0.784 0.797 (0.047) [72] 0.787 (0.030) [100] 0.785 (0.025) [100] Σ 0 Σ 1 ρ 0 INF 0.784 0.786 (0.049) [83] 0.784 (0.029) [100] 0.783 (0.025) [100] Inclusion of prior Se and Sp information No effect on point estimate or precision Positive effect on number of converged data sets 21 / 26

Conclusions Outline Problem setting Accuracy definition Optimal combination of biomarkers Absence of gold-standard reference Bayesian latent-class mixture model Naive prior definition Controlled prior definition Simulation study Data and model Results Conclusions 22 / 26

Conclusions Conclusions Bayesian latent-class mixture model: Takes unknown true disease status into account Incorporates information from reference test while acknowledges imperfectness Provides estimates of accuracy of the reference test Simulation study Model is able to retrieve true AUC when: sample size is adequate sensible non-informative prior is considered Careful prior specification Complex function of uninformative prior distributions => informative prior => biased estimates Controlled prior specification is proposed 23 / 26

Conclusions Further considerations Sensitivity to misspecified Se/Sp prior distribution Extend to incorporate non-normally distributed biomarkers Evaluate impact of conditional independence assumption 24 / 26

Conclusions References O Malley, A.J., Zou, K.H.: Bayesian multivariate hierarchical transformation models for ROC analysis. Statistical Medicine. 25, 459 479 (2006) Su, J.Q., Liu, J.S.: Linear combinations of multiple diagnostic markers. Journal of the American Statistical Association. 88, 1350 1355 (1993) Wei, Y, Higgins, P.T.: Bayesian multivariate meta-analysis with multiple outcomes. Statistics in Medicine (2013) doi: 10.1002/sim.5745 25 / 26

Conclusions Thank you for your attention! 26 / 26