A Discussion of Measurement of Uncertainty for Life Science Laboratories

Transcription

1 Webinar A Discussion of Measurement of Uncertainty for Life Science Laboratories 12/3/2015 Speakers Roger M. Brauninger, Biosafety Program Manager, A2LA (American Association for Laboratory Accreditation), Frederick, Maryland Roger M. Brauninger, Biosafety Program Manager has been employed with the American Association for Laboratory Accreditation since February He provides assistance in the day-to-day laboratory accreditation operations for laboratories in the Environmental, Biological, and Chemical fields and Reference Material Producers and serves as staff contact to organizations dealing with food and drug safety, biodefense and threat monitoring in the Biological and Chemical fields. Objectives At the conclusion of this program, participants will be able to: Identify the types of analytical methods which require an estimate of measurement uncertainty Discuss ways the laboratory can use commonly collected laboratory data to calculate an acceptable estimate of uncertainty and when it needs to be recalculated Continuing Education Credit CE credit is not available for this program, but you will receive a Certificate of Attendance. Evaluation/Certificate of Attendance Process Continuing education credit is not available for this program. Individuals who successfully complete the program and evaluation by 10/20/2016 will receive a Certificate of Attendance. Go to to complete the evaluation. 1. After you complete the evaluation, you will automatically receive the Certificate of Attendance. 2. Type or print your name on the certificate. Archived Program The archived streaming video will be available within two days. Anyone from your site can register view the program. URL: Comments, opinions, and evaluations expressed in this program do not constitute endorsement by APHL. The APHL does not authorize any program faculty to express personal opinion or evaluation as the position of APHL. The use of trade names and commercial sources is for identification only and does not imply endorsement by the program sponsors. This program is copyright protected by the speaker(s) and APHL. The material is to be used for this APHL program only. It is strictly forbidden to record the program or use any part of the material without permission from the author or APHL. Any unauthorized use of the written material or broadcasting, public performance, copying or re-recording constitutes an infringement of copyright laws. ACKNOWLEDGEMENT These webinars are supported by a Cooperative Agreement funded by FDA, through which APHL, AFDO and AAFCO are working to build an Integrated Laboratory System to Advance the Safety of Food and Animal Feed. For more information about ISO/IEC Accreditation go to: This project was 100% funded with federal funds from a federal program of $1,100,000. The Association of Public Health Laboratories (APHL) sponsors educational programs on critical issues in laboratory science. For more information, visit

2 Simple Measurement Uncertainty: Roger Brauninger American Association for Laboratory Accreditation Frederick, Maryland Topics Why do we need to determine MU? When do we need to recalculate? How large is too large? How do we calculate MU? RSS LCS Replicates and MPN 2 Measurement uncertainty Helps determine the quality of the testing. Measurable basis to determine improvements. Estimates variability of testing over time: materials, equipment, environment, and people; Tells the range of values that are about equally likely to appear as the result of the same measurement procedure on a particular sample. 3 1

3 ISO/IEC Testing laboratories shall have and shall apply procedures for estimating uncertainty of measurement. Reasonable estimation shall be based on knowledge of the performance of the method and Shall make use of, for example, previous experience and validation data. 4 A Few Terms Variance: Average of the squared differences from the Mean. Standard Deviation (SD): Square root of the Variance. Relative Standard Deviation (RSD): SD divided by the Mean x 100) Standard Uncertainty (u): Total of the combined uncertainty contributors (or the single contributor, if there is only one component). Expanded Uncertainty (U): Combined uncertainty contributors (or the single contributor, if there is only one component), multiplied by the coverage factor 95% confidence interval, assuming normal distribution. 5 Students t test coverage factor (k)= 6 2

4 Measurement uncertainty s Essence: There are two major factors to consider Repeatability describes the variability of results when doing the testing sequentially; time after time under the same conditions. Reproducibility describes the variability across different testing conditions; changes in time, personnel, environment, and equipment. Both must be considered to get accurate picture of process variability. 7 Sources of Measurement Uncertainty: Variation in method performance Different analysts Sampling effects Different weighing/pipetting techniques Differences in platting techniques Background Effects 8 Sources of Measurement Uncertainty: of microbiological test methods? Variation in controls/rm Storage Conditions Sample or Matrix Effects Living positive/process control Unable to count the CFU directly Unable to determine viable CFU directly Different sources of CRMs/RMs Purity differences Uncertainties differ 9 3

5 Sources of Measurement Uncertainty: Variation in media/reagents/equipment Each batch of media prepared Different technician, Different lot # of media, Different autoclave cycle Each reagent solution preparation Reagent Purity Variation in equipment Sensitivity LOD Detectors 10 Sources of Measurement Uncertainty of microbiological test methods? Variation in incubation conditions Method allows range in incubation time Method allows range in incubation temperature Extractions times, solvents, temps, matrices Organisms are unpredictable! 11 When to Consider Recalculating MU? When something in the process changes New analyst(s) New equipment New method for testing or for preparing control New Reagents/ Positive controls Annually? 12 4

6 How large an MU estimate is too large? Ask yourself if it is Acceptable to the lab if need to report? Acceptable to the customer/regulator if need to report? Reflective of laboratory capability? 13 A few Measurement uncertainty estimation variations. Root Sum Squared Method LCS (e.g., SD x k )Method Using Data Generated From Replicates and Most Probable Number (MPN) Method 14 Root Sum Squared Method 15 5

7 Root Sum Squared Method: Microbiology Lab Control data: SD[or RSD*] x k Methods Values obtained for RSS must be in the same unit The formula requires the data to be linear. But microbiological data is logarithmic Therefore MUST convert to base 10 (or natural base) ` *NOTE: Because it is not relative to the concentration of the sample, when using SD of data sets, the uncertainty value is added/ subtracted., thus it would be the same across all concentrations for that method. When using RSD of data sets, the uncertainty value is multiplied, as the value is being compared and is relative to the data produced. 16 Root Sum Squared Method A Hypothetical Example: a = uncertainty due to analyst (5%) b = uncertainty due to equipment (1%) c = uncertainty due to media prep (4%) n = the rest (<1%) 17 Hypothetical example using Root Sum Squared Method Root Sum Squared Method Bottoms Up approach Must Identify all contributors Must assign a value u = 18 6

8 Hypothetical example using Root Sum Squared Method u = (0.05) 2 + (0.01) 2 + (0.04) 2 + (0.01) 2 u = u = Ue = 2 x = = 13.2% 19 e.g., count of 120 CFU Step 1 Transform the CFU value to log 10 value. (e.g., 120 CFU = (log 10 )) Step 2 Calculate the SD of the log 10 values. SD= Step 3 Apply the coverage factor (95%, k=2) to the SD to obtain the relative Ue. (e.g., rue = or 13.2%). Step 4 The log value is multiplied by rue (e.g., Ue= x = ) Step 5 The Ue is added and subtracted from the log value. (e.g., ( ) = and ( ) = Step 6 - Convert the log value for the sample measurement back to CFU for the reported result. This is accomplished by taking the anti-log of each of the endpoints of the interval (anti-log of x = 10 x ). (e.g., = 64.1 = 64 CFU and = = 225 CFU) Uncertainty interval = CFU 20 LCS Data (e.g., SD x k Method) 21 7

9 Chemical: Aflatoxin example Values of method from the spiked recoveries of the aflatoxin results were plotted on control charts and the standard deviations then determined Alfatoxin (ppb) Statistics Sample 1 10 Mean SD RSD MU (SD x k) MU (RSD x k) % Laboratory Control Data with Same Target Value (e.g., 100 CFU) Raw Data (actual CFU Log 10 Value Difference from Mean Difference Squared recovered) (Log 10 Value) VAR= X= SD= Microbial Laboratory Control Data Using the Standard Deviation (SD) Step 1 Transform the CFU values to log 10 value. Step 2 Calculate the SD of the log 10 values. SD= Step 3 Apply the coverage factor (95%, k=2) to the SD to obtain Ue. The Ue in this example is If using the student t-tables, with n=20 (19 degrees of freedom), the coverage factor k would be 2.09, which would provide an expanded uncertainty of

10 Microbial Laboratory Control Data Using the Standard Deviation Step 4 To calculate the uncertainty for any subsequent laboratory result using SD x k, the result is first converted to log base 10 value (log 10 ), and the expanded uncertainty of is added and subtracted from the log value. Step 5 - Convert the log value for the sample measurement back to CFU for the reported result. This is accomplished by taking the anti-log of each of the endpoints of the interval (anti-log of x = 10 x ). For example, estimating the uncertainty using SD x k for a result of 150 CFU: 150 in log 10 = Adding and subtracting from gives an interval from to ; transforming back to counts: =32.10, and = Therefore the uncertainty interval is 32 to 701 CFU. 25 Microbial Laboratory Control Data Using the Relative Standard Deviation Step 1 Transform the CFU values to log 10. Step 2 Calculate the mean and relative standard deviation (RSD) of the log 10 values. The percentage SD (or relative SD), in log units: / = , or 17.8%; Step 3 Apply the coverage factor to the RSD to obtain the relative expanded uncertainty. For 95% coverage we use a coverage factor of k=2. The relative expanded uncertainty is thus 2 x =0.3550, or 35.5%. Note: If using the student t-tables, with n=20 (19 degrees of freedom), the coverage factor k would be 2.09, which would provide an relative expanded uncertainty of or 37.10%. 26 Microbial Laboratory Control Data Using the Relative Standard Deviation Step 4 To estimate MU for any subsequent laboratory result using RSD x k: The result is first converted to the log 10 value, Multiplied by (35.5% converted to decimal) and The expanded uncertainty is added and subtracted from the log value. Step 5 To obtain CFU for the reported result Convert the log value for the sample measurement back. by taking the anti-log of each of the endpoints of the interval (anti-log of x = 10 x ). 27 9

11 Microbial Laboratory Control Data Using the Relative Standard Deviation For estimating MU using RSD x k for a result of 150 CFU: 150 in log 10 = and the relative expanded uncertainty in log counts is x = Add and subtract from gives an interval from to ; Transform back to counts: =25.33, and = Therefore the uncertainty interval is 25 to 889 CFU. NOTE: UM can also remain as a % and the lab can express upper and lower control limits as the mean (in CFU) ± 3 x RSD. 28 Microbial Replicates But Not all are the same 29 Sources of uncertainty reflected in different types of replicates Type of Replicate Source Reproducibility Recovery True Plate Random error X X X X Counting error X X X X Dilutions X X X Environment X X Equipment X X Analyst X X 30 10

12 Reproducibility Replicates Results derived from laboratory control samples, not test samples. Sources of uncertainty are reflected in the differences between replicates. The difference between each pair of log counts is converted into a variance (squared difference). These variances are pooled and converted into an estimate of the combined standard uncertainty. 31 Reproducibility Replicates Raw Data Log 10 Raw Data (actual Log 10 Difference between Difference (actual CFU Value CFU recovered) Value Replicates between recovered) Second Replicate X= (Log 10 Value) Replicates First Replicate Squared VAR= SD= Column 1 Column 2 Column 3 Column 4 Column 5 Column Reproducibility Replicates Step 1 Transform the raw data by taking the log base 10 value (log 10 ) of the data (column 2, 4). Step 2 Calculate the overall mean of 40 results in columns 2 and 4, this is Step 3 Calculate the difference between the transformed replicates (column 5) then square the differences between the transformed replicates (column 6). Step 4 Add the differences together (column 6) and divide by 2n, where n = the total number of pairs of duplicates (for this example n = 20) to get

13 Reproducibility Replicates Step 5 Take the square root of the result in step 4; this equals the pooled reproducibility standard deviation, which is Step 6 Convert this standard deviation into a relative standard deviation (RSD) by dividing by the mean (1.9219) from Step 2, which produces an RSD of (e.g., = ) Step 7 Apply the coverage factor (k=2 for 95% coverage) to the RSD to get the estimate of the expanded uncertainty, (Note this is a log 10 value). (e.g., Ue= 2 x = ) If using the student t-tables, with n=20 (19 degrees of freedom), k would be 2.09, which would provide an relative expanded uncertainty of Reproducibility Replicates Step 9 To calculate the uncertainty for any subsequent laboratory result, the subsequent result is first converted to the log base 10 value (log 10 ), multiplied by and then this expanded uncertainty is added and subtracted from the log 10 value. Step 10 To estimate the MU of a sample, convert the log value for the sample measurement back to CFU for the reported result. This is accomplished by taking the anti-log of each of the endpoints of the interval (anti-log of x = 10 x ). For example, using a result of 150 CFU: 150 in log 10 = The relative Ue (rue) in log counts is ( x ). Add and subtract rue from gives an interval from to ; transformed back to counts: =90.97, and = Therefore the uncertainty interval is 90 to 248 CFU. 35 Recovery replicates The same amount of inoculum plated both with/without matrix. The sources of uncertainty are reflected in the difference between different sets of replicates. A single SD is calculated as the estimate of combined standard uncertainty

14 Recovery replicates CFU Inoculated Log 10 Value CFU recovered in Spike Log 10 Value % Recovery of Log Values X=97.0% SD=3.6% Column 1 Column 2 Column 3 Column 4 Column 5 30, , , , , , , , , , , , , , , , , , , , , , , , , , , , , Mean % Recovery = 97.0% SD = 3.6% 37 Recovery replicates Step 1 Transform the CFU values (columns 1, 3) to log base 10 value (log 10 ) (columns 2, 4). Step 2 Calculate the % recovery of the log 10 values by dividing column 4 by column 2 and multiplying by 100 (column 5). Step 3 Calculate the mean and standard deviation (SD) of the % recovery of the log 10 values. The mean recovery is 97.0% and the SD of % recovery is 3.6%. The SD is an estimate of the combined standard uncertainty, which can be used as a relative uncertainty. 38 Recovery replicates Step 4 For reporting purposes, apply the coverage factor to the SD to obtain Ue. For 95% coverage we use a coverage factor of k=2. The expanded uncertainty in this example is thus 7.2%. (If using the student t-tables, with n=20 (19 degrees of freedom), the coverage factor k would be 2.09, which would provide an expanded uncertainty of 7.5%.) Step 5 Because the recovery is expressed as a percentage, when calculating the expanded uncertainty for a sample, this percentage needs to be multiplied by the log 10 value in order to estimate the uncertainty in log units. For example, for a result of 150 CFU: 150 in log 10 = The Ue in log counts is ( x 0.072). Add and subtract Ue from gives an interval from to ; transforming back to counts: =104.6, and = Therefore the uncertainty interval is 104 to 216 CFU

15 Most Probable Number data 40 Measurement uncertainty MPN LCS results should be used provided these are an appropriate matrix and concentration. If all method steps are included, can use SD or RSD. If <20 LCS results, the CF should be the appropriate t statistic for 95% confidence for the associated DF. The estimate of combined uncertainty is then expanded using the formula: MPN MU(LCS) = k x SD, where k equals 2 (for 95% confidence). 41 Most Probable Number, FDA Bacteriological Analytical Manual For 3 tubes each at 0.1, 0.01, and g inocula, the MPNs per gram and 95 percent confidence intervals. Pos. tubes Conf. lim. Pos. tubes Conf. lim. MPN/g MPN/g Low High Low High < , , , , >

16 Laboratory Control Data with Same Target Value (e.g., 100 CFU) MPN value Predicted CFU Log 10 Value (Pos. Tubes) SD= Measurement uncertainty MPN Step 1 Locate the MPN predicted CFU based on the number of positive tubes Step 2 Transform the CFU values to log 10 value. Step 2 Calculate the SD of the log 10 values. SD= Step 3 Apply the coverage factor (95%, k=2) to the SD to obtain Ue. The Ue in this example is (Students t table with n=20 = Ue = Measurement uncertainty MPN Step 4 To calculate the uncertainty for any subsequent laboratory result using SD x k, the result is first converted to log base 10 value (log 10 ), and the expanded uncertainty of is added and subtracted from the log value. Step 5 - Convert the log value for the sample measurement back to CFU for the reported result. This is accomplished by taking the anti-log of each of the endpoints of the interval (anti-log of x = 10 x ). For example, estimating the uncertainty using SD x k for a result of 160 CFU: 160 in log 10 = Adding and subtracting from gives an interval from to ; transforming back to counts: =41.27, and = Therefore the uncertainty interval is 41 to 621 CFU

17 Acknowledgements A2LA G108 Guidelines for Estimating Uncertainty for Microbiological Counting Methods By Dawn Mettler and Dan Tholen And personal communication 46 Questions / Comments 47 For Further Information Contact: Roger M. Brauninger Phone: rbrauninger@a2la.org American Association for Laboratory Accreditation 5301 Buckeystown Pike, Suite 350 Frederick, MD Comments/concerns about this program? webinar@aphl.org 16