Assessing Test Quality from EQA/PT Surveys. James O. Westgard University of Wisconsin Medical School Sten A. Westgard Westgard QC, Inc.

Transcription

1 Assessing Test Quality from EQA/PT Surveys James O. Westgard University of Wisconsin Medical School Sten A. Westgard Westgard QC, Inc. Madison, WI

2 Issues to consider Is analytical quality better than needed for the intended clinical use? In this age of evidence-based medicine, where s s the evidence to address this issue? What evidence of quality can be provided from EQA/PT survey results?

3 Background Touchstone Test Methodology Identify exemplars that represent important clinical and diagnostic applications Quality required defined by available CLIA acceptability criteria for proficiency testing Regulated analytes where laboratories are required to participate and report PT results to CMS PT data available from different survey programs that represent a broad scope of testing sites Ref: Westgard JO, Westgard SA. The quality of laboratory testing today. Am J Clin Pathol 2006;125:

4 Estimates of Quality from PT National Test Quality (NTQ) Sigma = TEa CLIA CLIA /CV group National Method Quality (NMQ) Sigma = WtAv [(TEa CLIA Bias Local Method Quality (LMQ) Sigma = WtAv (TEa CLIA Bias msubgp )/CV msbgp ] CLIA )/CV msubgp

5 US PT Programs Accessed PT Prog #Labs Type of Laboratory AAFP 296 Family practice office labs MLE 577 Internist office labs AAB 1498 Clinics and small hospitals API 2647 Small/mod hospital labs CAP 4240 Mod/large hospital labs Total 9258 Approx 36,000 total labs enrolled in PT today

6 Cholesterol Quality by PT Program CLIA TEa=10% (85.1% of US Labs) PTProg Labs Mean NTQ NMQ LMQ AAFP σ 2.01σ 2.54σ MLE AAB API CAP Wt.Av σ 3.02σ 3.67σ

7 Cholesterol Quality (NTQ) AAFP Probability for Rejection (P) MLE API AAB CAP NY Sigma Scale P P N R 1.0 fr ed /2of3 /R /3 /6 3s 2s 4s 1s x /2 /R /4 3s 2s 4s 1s s s s /2 2s /R 4s s s s Systematic Error (SE, multiples of s)

8 Cholesterol Quality Today Sigma Scale Probability for Rejection (P) NTQ NMQ LMQ P P N R 1.0 fr ed /2of3 /R /3 /6 3s 2s 4s 1s x /2 /R /4 3s 2s 4s 1s s s s /2 2s /R 4s s s s Systematic Error (SE, multiples of s)

9 Analytical Quality Today! TEST QR NTQ NMQ LMQ Cholesterol 10% 2.88 σ 3.02 σ 3.67 σ Calcium 1.0 mg/dl Glucose 10% Prothrombin Time 15% INR 20% Fibrinogen 20%

10 Glycohemoglobin Clin Lab News March 2002

11 NACB Guidelines for GHb Clin Chem 2002;48: Precision Goal CV max 3% to 5% Bias Goal 0% via NGSP standardization Clinical quality for intended use Treatment goals should be based on ADA recommendations, which include maintaining GHb concentrations <7%, and reevaluation of the treatment regiment for GHb values >8%.

12 Updated NACB Guidelines, Version 1107, accessed at AACC website, Nov 27, 2007 Treatment goals should be based on the ADA recommendations which include maintaining GHb concentrations <7% and in individual patients as close to the non-diabetic range as safely possible. small changes in GHb (e.g., ± 0.5%GHb) over time may reflect assay variability rather than a true change in glycemic status.

13 Quality Required for Intended Clinical Use Change from 7.0 to 8.0 %Hb corresponds to clinical decision interval of 14.3% (1/7) New NACB guidelines suggest interpreting changes of over 0.5 %Hb for treatment, which would correspond to approx. 7% (0.5/7) CAP criterion for acceptable performance in proficiency testing is 15% No reference given

14 Glycated Hemoglobin (GHb) More than 30 different methods available Ion-exchange columns, HPLC, Immunoassay, Affinity chromatography Some differences in specificity and susceptibility to interferences Significant differences in sample volume, time of analysis, cost All US commercial methods certified by NGSP National Glycohemoglobin Standardization Program Standardized to provide comparable results

15 NGSP ( US National Glyco-hemoglobin Standardization Program Purpose is to calibrate methods to be sure values are equivalent to DCCT values Diabetes Control and Complications Trial Manufacturers follow CLSI guidelines to demonstrate performance of their methods Report results equivalent to HbA1c values Purpose is to establish traceability and hopefully comparability of all methods

16 Traceability True Value Definitive Method IFCC Primary Reference Material Method Validation?? Traceability Reference Method NGSP Manuf. Secondary Reference Material Method Validation & EQA CAP Observed Value Field Method YOU Manuf. Control Material Internal Quality Control

17 Master Equations Master equations used to convert IFCC reference values to national reference methods in US, Japan, and Sweden Maintained by a global network of 14 reference labs Recent study demonstrates stability of calibration materials and international reference system See The IFCC Reference Measurement System for HbA1c: A 6-year 6 Progress Report. Clin Chem 2008;54:

18 CAP PT Program Pooled fresh whole blood samples 3 samples per survey 3 surveys per year Serves large US laboratories Approx participants Grading criterion TEa of 15% Data from 2007 GH2 A03 sample NGSP assigned value of 7.60 %Hb

19 What is quality of GHb Test? What measures might be used? Observed trueness? Observed precision and accuracy? Calculated sigma-metric? metric? Estimated total error? Estimated measurement uncertainty? Significant difference between serial tests?

20 ISO Trueness and Accuracy Traceable Value Observed Mean X X X X X X X X X Observed Dispersion X X X Bias Trueness Error of a result Accuracy

21 Trueness and Bias

22 Precision

23 Sigma-metrics metrics (TEa=15%) LOCAL Method Quality NATIONAL Method Quality

24 GHb Quality 2007 TEa=15%, CAP Survey 2806 labs NMQ Sigma Scale LMQ P P N R 1.0 fr ed Probability for Rejection (P) /2of3 /R /3 /6 3s 2s 4s 1s x /2 /R /4 3s 2s 4s 1s s s s /2 2s /R 4s s s s Systematic Error (SE, multiples of s)

25 GHb Quality 2007 TEa=10%, CAP Survey 2806 labs NMQ LMQ Sigma Scale P P N R 1.0 fr ed Probability for Rejection (P) /2of3 /R /3 /6 3s 2s 4s 1s x /2 /R /4 3s 2s 4s 1s s s s /2 2s /R 4s s s s Systematic Error (SE, multiples of s)

26 Total Error vs Uncertainty Traceable Value Observed Mean X X X X X X X X X Observed Dispersion X X X Bias Zs Total Error Bias + Zs Expanded Combined Uncertainty Z (Bias 2 + s 2 ) 1/2

27 Reference Change Value Fraser C. Biological Variation: From Principles to Practice. AACC Press, What difference between two test results is statistically significant? RCV = 2 1/2 *Z*[CV 2 A + CV 2 I ] 1/2 Where 2 represents the two test results Z is a multiplier related to the expected probability CV A is the analytical CV CV I is the intra-individual individual or within-subject CV

28 TE, MU, RCV Reference Change Value Total Error Meas. Uncert.

29 GHb Measures of Quality NGSP Reference Value 7.60 %Hb Trueness 0.06 %Hb National Method Quality 4.17 sigma (TEa 15%) Results correct within 0.68 %Hb (Total Error) Results in doubt up to 0.68 %Hb (Uncertainty) Reference Change Value = 1.11 %Hb Fraser s medically significant change between two measurements (CVwithinsubject = 4.1%, WtAv CV=3.1%)

30 How explain this to customers who believe all methods give same results?

31 THE BIG QUESTION Should you report measurement uncertainty or manage analytical quality?

32 How should you manage quality? Define quality required for intended use Evaluate precision and bias of method Select appropriate QC procedures Use quality-planning models Analytical model expands Total Error to include QC Clinical model expand analytical model to include pre-analytic variables such as within-subject biologic variation Methodology published in 1991 Clin Chem 1991;37: ; 661; 37:

33 Quality-Planning Models or Error Budgets Conventional Total Error Budget Stable Imprecision Stable Inaccuracy Analytical Quality Planning Budget Stable Imprecision Stable Inaccuracy QC Safety Margin Clinical Quality Planning Budget Preanalytic factors Stable Imprecision Stable Inaccuracy Same bottom line, but different expenses QC Safety Margin Different bottom line and different expenses

34 Why need QC safety margin? Probability for Rejection (P) Sigma Scale P P N R 1.0 fr ed Desirable Error Detection /2of3 /R /3 /6 3s 2s 4s 1s x /2 /R /4 3s 2s 4s 1s s s s /2 2s /R 4s s s s Systematic Error (SE, multiples of s) Safety Margin of 1.7s to 3.5s Needed

35 How assure GHb test will be correct within 15%? Analytical Quality Planning Budget 15.0% Stable Imprecision Stable Inaccuracy QC Safety Margin 3.0% 0.0%???

36 Use 1 with 2.5s N=2 Probability for Rejection (P) TE a =15.00 % (SE) Process metric (Sigma-scale) P fr P ed N R /2 /R /4 3s 2s 4s 1s s /2 /R 3s 2s 4s /2 3s 2s s s 1 3s /2 2s Sigma = s /2 2s /R ΔSE 4s crit = Control metric (ΔSE, multiples of s)

37 How assure GHb test will correctly classify a patient? Clinical Quality Planning Budget 14.3% Preanalytic factors Stable Imprecision Stable Inaccuracy QC Safety Margin 4.1% 3.0% 0.0%???

38 Need multirule QC w/n=6 D int =14.30 % (SE) Process metric (Sigma-scale) Sigma = 3.62 ΔSE crit = 1.97 Probability for Rejection (P) P fr P ed N R s /2of3 2s /R 4s /3 1s /6 x s /2of3 2s /R 4s /3 1s Control metric (ΔSE, multiples of s)

39 How assure quality with minimum QC? CV=2.0%, Bias=0%, and QC w/n=2 and 2.5s control limits