Unit 4. Statistics, Detection Limits and Uncertainty. Experts Teaching from Practical Experience

Transcription

1 Unit 4 Statistics, Detection Limits and Uncertainty Experts Teaching from Practical Experience

2 Unit 4 Topics Statistical Analysis Detection Limits Decision thresholds & detection levels Instrument Detection Limits vs. Method Detection Limits Kinectrics Inc

3 Unit 4 Topics Uncertainty Requirements Guide to Uncertainty in Measurement (GUM) Uncertainty budgets Sampling uncertainty Kinectrics Inc

4 Unit 4 Learning Objectives Describe the responsibilities of the user of the standards with regard to Detection Limits Identify the guidance on analysis of samples and reporting of results Describe the changes in the guidance on estimating and reporting uncertainties Kinectrics Inc

5 Unit 4a Statistics Experts Teaching from Practical Experience

6 Interpretation of Survey Results MARSSIM (Chapter 8) recommends the following sequence of steps: 1. Review the Data Quality Objectives (DQOs) and Sampling Design 2. Conduct a Preliminary Data Review 3. Select the Tests 4. Verify the Assumptions of the Tests 5. Draw Conclusions from the Data More detail is given in EPA QA/G-9S Kinectrics Inc

7 Review the DQOs and Sampling Design Review the objectives of the study Translate the objectives into statistical hypotheses Translate the objectives into limits on Type I & Type II errors Review the sampling design & note any special features or potential problems Kinectrics Inc

8 Conduct a Preliminary Data Review Review Quality Assurance reports Look for problems or anomalies Calculate basic statistical quantities Calculate percentiles, measures of central tendency, dispersion and, if the data involves two variables, the correlation coefficient Graph the data Select graphical representations that illuminate the structure of the data Kinectrics Inc

9 Select the Statistical Methods Select the statistical method Follow the Decision Tree from EPA QA/G-9S Identify assumptions underlying the test List the key underlying assumptions such as distributional form, dispersion, independence, etc Note any sensitive assumptions where relatively small deviations could jeopardize the validity of the test Kinectrics Inc

10 Verify the Assumptions Determine approach for verifying assumptions Review (or develop) a statistical model for the data Select the methods for verifying the assumptions Perform tests of assumptions Adjust for distributional assumptions (if warranted) Perform the calculations required for the tests Determine corrective actions Kinectrics Inc

11 Verify the Assumptions Determine corrective actions (if required) Determine if data transformation will correct the problem If data is missing, explore collecting more data or using theoretical justification Consider robust procedures or nonparametric hypothesis tests Kinectrics Inc

12 Draw Conclusions from the Data Perform the statistical procedure Perform & document the statistical tests Identify outliers and recalculate if necessary Draw study conclusions If the null hypothesis is rejected, draw conclusions & document If the null hypothesis is not rejected, verify limits on decision errors then draw conclusions & document Interpret the results Kinectrics Inc

13 Draw Conclusions from the Data Evaluate performance of the sampling design Evaluate the statistical power of the design over the full range of parameter values Kinectrics Inc

14 Unit 4b Detection Limits Experts Teaching from Practical Experience

15 Definitions Both N & N caution that terminology and the definitions used for these concepts (non-detect level & detection limit) are not always consistent. It is the responsibility of the user to understand, document, and justify the detection limit reported by any laboratory engaged to perform analyses. Kinectrics Inc

16 Definitions Organization or Discipline L C L D CSA N288.4/5 Non-detect Level (L C ) Detection Limit (L D ) Health Physics (ANSI/HPS N ) Occupational Hygiene (AIHA, IOHA) Environmental Analytical Chemistry Cmte. of ACS Critical Level (CL) Limit of Detection (LOD) Limit of Detection (LOD) IUPAC Detection Decisions (L C ) Lower Limit of Detection (LLD) Limit of Quantification (LOQ) Limit of Quantification (LOQ) Minimum Detectable Value or Detection Limit (L D ) ISO 11929:2010 Decision Threshold (y*) Detection Limit (y # ) Kinectrics Inc

17 Definitions Non-detect level the level below which quantitative results are not obtained from the measurement system or analysis method selected. The non-detect level is the smallest value of the measurand for which the probability of a wrong conclusion that the measurand is present when it actually is not present ( error of the first kind or a false positive error ) does not exceed a specified probability, α. Excerpt from Clause 3.1 of CSA N and N Kinectrics Inc

18 Definitions Detection limit the level (relative to background) above which an effect can confidently be measured. The detection limit is the smallest value of the measurand for which the probability of a wrong conclusion that the measurand is not present when it actually is present ( error of the second kind of a false negative error ) does not exceed a specified probability, β. Excerpt from Clause 3.1 of CSA N and N Kinectrics Inc

19 Definitions Non-detect Level (L C ) for Type I Error (α) Detection Limit (L D ) for Type II Error (β) From IUPAC Recommendations 1995 Kinectrics Inc

20 Definitions CSA N Annex D, Table D.1 Formulae for non-detect (critical) level and detection limit Kinectrics Inc

21 CSA N The level below which quantitative results are not obtained from the measurement system or analysis method selected is called the non-detect level. This level shall be defined and its derivation should be documented. Measurements below this level are often reported as being less than some value. Paraphrased from Clause of CSA N Kinectrics Inc

22 CSA N The treatment of results that are less than the nondetect level for the measurement shall be defined and documented. The results of measurements that are below the nondetect level may be reported as not detected or as being below a less than value. The values of the non-detect level should be documented and reported. Paraphrased from Clause 8.3 of CSA N Kinectrics Inc

23 CSA N Quantitative numerical values should be reported rather than less than some value or non-detect level. The requirement for numerical values needs to be considered within the context of laboratory capabilities and the proximity of the result to any applicable benchmark value. Paraphrased from Clause of CSA N Kinectrics Inc

24 Low Count Rates The user is cautioned that these formula might not be appropriate in all situations, particularly in lowlevel counting. (see Strom & MacLellan, Health Physics, 81(1), 2001). Kinectrics Inc

25 Instrument vs. Method Detection Limits A note following Clause of N warns that: some laboratories might report an instrument detection level that is often much lower than the detection level of the method (which includes any required sample preparation). It is the responsibility of the user to understand, document, and justify the detection limit reported by any laboratory engaged to perform analyses. Both standards generally assume (but often do not always explicitly state) that the detection level is the Method Detection Level Kinectrics Inc

26 Unit 4b Summary and Review Detection Limits Non-detect level & detection limit Instrument Detection Limits vs. Method Detection Limits Learning Objective Describe the responsibilities of the user of the standards with regard to Detection Limits Kinectrics Inc

27 Unit 4c Uncertainty Experts Teaching from Practical Experience

28 Uncertainty Uncertainty - a quantitative expression of error that results from incomplete knowledge or information about a parameter or value. Measurement Uncertainty (from JCGM 200:2012 International Vocabulary of Metrology): non-negative parameter characterizing the dispersion of the quantity values being attributed to a measurand, based on the information used Statistical uncertainty - that component of uncertainty which arises from imprecision. Systematic uncertainty - that component of uncertainty which arises from biases. Excerpt from Clause 3.1 of CSA N Type A uncertainty - determined by repeated measurement Type B uncertainty - determined in any other manner Kinectrics Inc

29 Requirements N288.4: The uncertainty associated with each measured or calculated value should be estimated. Excerpt from Clause of CSA N N288.5: The uncertainty associated with results of effluent monitoring measurements and any dose estimates derived from them should be discussed in the report. Excerpt from Clause of CSA N Both: The uncertainty should take into account both sampling and measurement errors. Sampling errors cannot always be quantified but they shall be kept to a minimum by design of the monitoring program. Kinectrics Inc

30 Introduction to Uncertainty Two introductory texts on uncertainty: Kinectrics Inc

31 Guide to Uncertainty in Measurement The Guide to Uncertainty in Measurement (GUM) was prepared by the Joint Committee for Guides in Metrology (JCGM) Originally published in 1995 (JCGM 100:1995) Minor revision published in 2008 (JCGM 100:2008) Available for download from the BIPM website at The 1995 version of the GUM was adopted by ISO & IEC as ISO/IEC Guide 98-3:2008 the 2008 revision has not yet been adopted by ISO/IEC Kinectrics Inc

32 Evaluation of Measurement Data The JCGM Working Group on Uncertainty in Measurement is preparing additional guidance on the Evaluation of Measurement Data : Guide to the expression of uncertainty in measurement (JCGM 100:2008) Propagation of distributions using a Monte Carlo method Supplement 1 to the GUM (JCGM 101:2008) Extension to any number of output quantities Supplement 2 to the GUM (JCGM 102:2011) Introduction to the Guide to the expression of uncertainty in measurement and related documents (JCGM 104:2009) Other documents & supplements are in preparation Kinectrics Inc

33 Further Guidance UKAS M3003 (Ed. 2, Jan. 2007) The Expression of Uncertainty and Confidence in Measurement Provides an introduction to the subject with examples of the application of the guidance given in the GUM Information/Pubs-Technical-Articles/Pubs- List/M3003.pdf The Eurachem/CITAC Guide (Ramsey & Ellison, 2007) Measurement of uncertainty arising from sampling: A guide to methods and approaches Provides additional guidance on the estimation of uncertainty due to sampling and sample preparation Kinectrics Inc

34 Uncertainty Budgets The American Society for Quality (ASQ) defines an uncertainty budget as: a statement of measurement uncertainty, of the components of that measurement uncertainty, and of their calculation and combination A National Research Council template for Uncertainty Budgets (an Excel spreadsheet) is available at: Kinectrics Inc

35 Uncertainty Budgets Example: Calculation of activity by liquid scintillation counting Symbol A = {(C-B) x exp(-λt) x R x S} / {V x T x ε x P ϒ } Symbol A Activity V Volume C Gross Counts T Time B Background Counts ε Efficiency exp(-λt) Decay Factor P ϒ Emission Probability R Random Summing Correction S Self-absorption Correction Kinectrics Inc

36 Uncertainty Budgets Component Value Uncertainty Distribution Divisor Relative Uncertainty Gross Counts Normal (1σ) % Background Counts Normal (1σ) % Decay % Normal (1σ) % R % Normal (1σ) % S % Normal (1σ) % Volume (l) Rectangular (half-range) % Live Time (s) Rectangular (half-range) % Efficiency % Normal (2σ) % Emission Probability % Normal (2σ) % ACTIVITY (Bq/l) Normal (2σ) 11.49% Kinectrics Inc

37 Rectangular Distribution P(x) = 1/2a, for (b a) < x < (b+a) 0, otherwise E(x) = Mean = b E(x 2 ) = b 2 + a 2 /3 σ 2 = E(x 2 ) E(x) 2 = a 2 /3 σ = a/ 3 Kinectrics Inc

38 Sampling Uncertainty The standards do not address sampling uncertainty in detail. Several reviews have concluded that, in general: Uncertainty due to sampling is greater than uncertainty due to analysis (often an order of magnitude or more greater), Heterogeneity is the leading contributor to random sampling uncertainty, and Non-representative sampling is the leading contributor to systematic sampling uncertainty. Kinectrics Inc

39 Sampling Uncertainty Kinectrics Inc

40 Sampling Uncertainty Sampling Uncertainty should be included but it may be difficult (or impossible) to quantify. Possible approaches include: Empirical Methods Duplicate samples Multiple protocols Collaborative trial Sampling proficiency test Reference sampling method/target Modeling Cause & Effect Modeling Gy s Sampling Theory Kinectrics Inc

41 Sampling Uncertainty Component Estimated Method Sampling Analytical Precision Bias Precision Bias Duplicate Samples (Single sampler using a single sampling protocol) Multiple Protocols (Single sampler using multiple sampling protocols) Collaborative Trial (Multiple samplers using the same sampling protocol) Sampling Proficiency Test (Multiple samplers each using a different sampling protocols) Yes No Yes Between protocols Yes No * (may be estimated by including Certified Reference Materials) No * (may be estimated by including Certified Reference Materials) Between samplers Yes Yes Between samplers & protocols Yes Yes Kinectrics Inc

42 Model Uncertainty Model uncertainty is an assessment of the degree of confidence that a mathematical model is a "correct representation of the system. Model uncertainty includes: An estimate of the uncertainty due to the structure of the model (structure uncertainty); and An estimate of uncertainty in each of the parameters used in the risk assessment equations (parameter uncertainty). Kinectrics Inc

43 Model Uncertainty Estimates of the uncertainties in the N288 models are generally not available: Excerpt from CSA N Clause Conservatism is introduced into the current model by selecting conservative values for food, water, soil, and air intake rates for the representative person, typically at the 95th percentile level. Kinectrics Inc

44 Model Uncertainty Estimates of the uncertainties in the N288 models are generally not available: IAEA TRS 472 (Handbook of Parameter Values for the Prediction of Radionuclide Transfer in Terrestrial and Freshwater Environments) Estimations of uncertainty about each such value were carried out by applying statistical analysis, where possible... In some cases, the values were given without a statement of uncertainty or a range, because of the limited data available. Kinectrics Inc

45 Model Uncertainty Estimates of the uncertainties in the N288 models are generally not available: NCRP Report 164 (Uncertainties in Internal Dose Assessment) The dose limits that are recommended by the International Commission on Radiological Protection (ICRP) for regulatory purposes are based on the use of values of dose per unit intake that are to be applied without any consideration of uncertainty. Kinectrics Inc

46 Reporting Uncertainty The number of significant figures quoted in a result shall not imply a degree of precision greater than that warranted by the sources of uncertainty. Uncertainty should be rounded to one significant figure The least significant figure in the result should correspond to the significant figure in the uncertainty (e.g., 1900 ± 200 Bq/l). More significant figures should be carried for calculation steps than what is reported in the final stage. Paraphrased from Clause 9.3 of CSA N and CSA N Kinectrics Inc

47 Unit 4c Summary and Review Uncertainty ISO Guide to Uncertainty in Measurement (GUM) Uncertainty budgets Sampling uncertainty Reporting uncertainty Learning Objectives Identify the guidance on analysis of samples and reporting of results Describe the changes in the guidance on estimating and reporting uncertainties Kinectrics Inc