General Accreditation Guidance. Estimating and reporting measurement uncertainty of chemical test results

Transcription

1 General Accreditation Guidance Estimating and reporting measurement uncertainty of chemical test results January 018

2 opyright National Association of Testing Authorities, Australia 013 This pulication is protected y copyright under the ommonwealth of Australia opyright Act NATA s accredited facilities or facilities seeking accreditation may use or copy this pulication or print or this pulication internally for accreditation purposes. Individuals may store a copy of this pulication for private non-commercial use or copy a reasonale portion of this pulication in accordance with the fair dealing provisions in Part III Division 3 of the opyright Act You must include this copyright notice in its complete form if you make a copy of this pulication. Apart from these permitted uses, you must not modify, copy, reproduce, repulish, frame, upload to a third party, store in a retrieval system, post, transmit or distriute this content in any way or any form or y any means without ress written authority from NATA.

3 Guidelines for estimating and reporting measurement uncertainty of chemical test results Introduction Those making decisions ased on test results need to know if the results are sufficiently reliale for the intended purpose. Every test or caliration result is suject to uncertainty. Estimates of measurement uncertainty (MU) provide information aout the reliaility of results. MU is an important part of a reported result and it may e argued that a result is incomplete unless accompanied y an estimate of MU. ompetent laoratories evaluate and monitor the performance of their test methods and are aware of the uncertainty associated with the results reported to customers. ISO/IE , Section 5.4.6, requires caliration and testing laoratories to have and apply procedures to estimate the uncertainty of their measurements. Furthermore, Section of the standard states that test reports shall include information regarding MU when a customer instructs the facility to provide the information, when it is relevant to the validity or application of test results, or when it affects compliance to a specification limit. This Technical Note is directed primarily to chemical testing laoratories ut the general principles also apply to other areas of testing. Drawing on authoritative references, it aims to define MU, lain why laoratories should estimate it, and then provide general guidance on the estimation and reporting of MU. More detailed, specific information is provided in the cited references. There are several approaches to estimates of uncertainty that are statistically sound. The methods descried here are not the only ones that are acceptale. However uncertainty is determined, estimates must not give a wrong impression of MU. This Technical Note provides guidance only. What is Measurement Uncertainty? Measurement uncertainty has een defined as a parameter associated with the result of a measurement that characterises the dispersion of values that could reasonaly e attriuted to the measurand or more lately as a non-negative parameter characterising the dispersion of quantity values eing attriuted to a measurand, ased on the information used 3 Note 1: The measurand is defined as the particular quantity suject to measurement 3 For example, vapour pressure of a given sample of water at 0, mass percentage fat in a sample of cheese, or mass fraction (mg/kg) of a pesticide in a sample of apples. ASTM 4 descries MU as an estimate of the magnitude of systematic and random measurement errors that may e reported along with the measurement result. Whilst this definition perhaps does not properly emrace the concept of uncertainty it does provide some guidance regarding the factors contriuting to MU and therey some clues on how it may e estimated. Both definitions make the important point that MU is associated with a test result not a test method. However, the MU associated with the results produced y application of a test method is a key performance characteristic of the method. Some regulators suggest the fitness for purpose of a test method to e judged solely on the MU associated with results 5. The true value is the result we would otain if we made a perfect measurement. However no measurement can e perfect, so the true value can never e known. onsidering the ISO definition of MU, the dispersion of values that could reasonaly e attriuted to the measurand implies that the test result, effectively a range taking the estimated MU into account, will encompass the true value with a particular proaility which depends on the level of confidence provided y the estimate of MU. By convention, in most instances an anded uncertainty is estimated, applying a coverage factor to the comined standard uncertainty to provide a level of confidence of approximately 95%. January 018 Page 3 of 13

4 Laoratories estimating MU in this manner should e 95% confident that the reported result - with MU will include the true value. MU may e estimated to provide different levels of confidence though the use of different coverage factors. ISO Guide to the Expression of Uncertainty of Measurement (GUM) definitions Standard uncertainty, u(x ) i Uncertainty of the result, x, of a measurement ressed as a standard deviation. i omined standard uncertainty, u (y) c Standard uncertainty of the result, y, of a measurement when the result is otained from the values of a numer of other quantities, equal to the positive square root of a sum of terms, the terms eing the variances or covariances of these other quantities weighted according to how the measurement result varies with these quantities. overage factor, k Numerical factor used as a multiplier of the comined standard uncertainty in order to otain an anded uncertainty. Expanded uncertainty, U Quantity defining an interval aout the result of a measurement that may e ected to encompass a large fraction of the distriution of values that could reasonaly e attriuted to the measurand. Note : The ISO GUM refers to the ISO/IE Guide to the Expression of Uncertainty of Measurement (eference ) Note 3: An anded uncertainty is calculated from a comined standard uncertainty and a coverage factor, using U = k x u c Why estimate measurement uncertainty? For facilities seeking accreditation to ISO/IE 1705, the gli answer to this question is ecause the Standard requires you to do so! However, there are good scientific reasons for the ISO Standard to stipulate this requirement. A good estimate of MU is necessary for facilities and their customers to: ensure results are fit for purpose, make informed decisions, and provide information that laoratories can use to improve their test methods. Traceaility is another important property of a test result, particularly if it is to e used for legal or regulatory purposes. Metrological traceaility is defined as property of a measurement result wherey the result can e related to a reference through a documented unroken chain of calirations, each contriuting to the measurement uncertainty' 3 The last phrase of this definition emphasises that link etween MU and traceaility. Further information aout Metrological Traceaility can e found in Policy ircular 11. How may facilities estimate MU? There are a numer of different approaches to estimating MU. In deciding upon the approach to use, laoratories should consider the nature of the test, the purpose of the test, the information availale on which to ase an estimate, how test results will e used and the risk associated with decisions ased on test results. If a well-recognised test method specifies limits to the values of the major sources of MU and the form of presentation of calculated results, a facility is considered to have satisfied the requirements of ISO/IE 1705, with respect to the estimation of MU, y following the test method and reporting instructions (ISO/IE 1705, Section , Note ). Nevertheless, laoratories in this situation are advised to estimate MU since erience has shown that the limits imposed y standard methods are not always sufficient to ensure results are fit for purpose. If a facility s estimate of MU is ased on the stated reproduciility for a standard method then the facility should ensure that the estimate covers all sources of uncertainty pertaining to their measurements. January 018 Page 4 of 13

5 The ISO/IE 1705 Standard (Section ) also recognises that the nature of some test methods may preclude vigorous, metrologically and statistically valid calculation of MU. In such cases, laoratories are still required to make a reasonale estimate of MU ased on, for example, professional judgement and erience, knowledge of method performance and validation data. Such estimates must not give a wrong impression of MU. The many different approaches for the estimation of MU may include components derived from what may e roadly categorised into either ottom-up or top-down calculations. The fundamental metrological ottom-up approach descried in the ISO/IE Guide to the Expression of Uncertainty in Measurement, commonly called the GUM, comines the uncertainties associated with all individual operations of an analytical procedure to calculate the comined standard uncertainty and, after multiplication y a coverage factor, the anded uncertainty (usually the 95% confidence range). The GUM sets down general principles and guidance for estimating the uncertainty associated with single steps of analytical procedures (weighings, volumes etc.), calculated concentrations of standard solutions, spiked concentrations or relatively simple tests. For more complex test procedures it can e laorious, even if spreadsheets are used for calculations. Horwitz 6 considers the approach inappropriate for chemical tests ecause it ignores the fact that the uncertainties associated with some of the numerous factors influencing test results tend to cancel out. Furthermore, it is argued that chemical test results are often influenced y factors that overwhelm the uncertainties considered in the GUM approach. 6,7 An advantage of the GUM approach is that it provides a clear understanding of the analytical operations that contriute significantly to MU, allowing the analyst to focus on improving these operations if required to reduce the MU associated with test results. However, erienced analysts are most likely ale to identify the critical steps of analytical procedures independent of any estimates of MU. Eurachem/ITA 8 provides guidance on how the GUM approach may e applied to chemical measurements. It also includes examples using a top-down approach. A top-down approach utilises data from method validation, intra-laoratory Q and/or inter-laoratory studies. The use of such data, include all contriutions to uncertainty. Numerous references, aimed at providing chemical testing facilities with practical guidance and examples of simple top-down approaches to estimating MU, are availale A top-down approach is generally more practical than the GUM approach for estimating the MU of chemical test results. However it is the facility s decision to use the method most appropriate for their circumstances and supported y the availale data. MU may not e the same over the range of concentrations tested. In this case, the laoratory must estimate the MU at the levels commonly tested, including levels close to the limit of detection, or regulatory limit, as appropriate. The following su-sections provide examples of how a top-down approach may e applied to estimate MU. A systematic procedure for otaining fit-for-purpose estimates of MU is presented in Figure 1 and a practical example of how MU may e estimated utilising data on ias and precision is shown in Appendix 1. Estimating measurement uncertainty from reproduciility studies Note: A facility may estimate MU ased on the etween-laoratory reproduciility reported for a standard method, however the facility should first ensure that they are ale to achieve the within-laoratory repeataility stated for the method. The standard deviation determined from inter-laoratory studies under reproduciility conditions (i.e. no variation to the method, including susampling at the laoratory level), s, may e used as an approximation of the comined uncertainty associated with a result. 11 This estimate is then douled to give the anded uncertainty (95% onfidence Interval). u s, and U = s where: u = comined standard uncertainty s = standard deviation under reproduciility conditions U = anded uncertainty, 95% confidence interval January 018 Page 5 of 13

6 In the asence of data from inter-laoratory studies on a particular method, the reproduciility standard deviation may e estimated from an equation proposed y Horwitz. The Horwitz equation 14-16, s = , was empirically derived y plotting reproduciility standard deviation vs concentration, (in g/g) for more than 7000 inter-laoratory studies. This equation may e used to estimate s and U at different concentrations. It may also e used to check the validity of estimates of s from inter-laoratory studies. The equation predicts that lowering analyte concentration y two orders of magnitude will doule the etweenlaoratory relative standard deviation, SD, associated with a test result. However, advances in analytical chemistry with the introduction of techniques such as isotope-dilution mass spectrometry, have provided the capaility to achieve very low limits of quantitation with less uncertainty than predicted y the Horwitz equation. Thompson and Lowthian 17 have reported that laoratories tend to out-perform the Horwitz function at concentrations elow 10 µg/kg. σ = 0. * c 0.0 * c 0.01* c if c if 1.0*10 1.*10 7 if c c where c = concentration, (e.g. the assigned value ressed as a dimensionless mass ratio 1ppm 10-6 or % 10 - ) requires c to e dimensionless mass ratio, eg.1ppm 10-6 or 1% If results are corrected for ias, or the ias is small, estimates of MU ased on s calculated using the Horwitz equation may suffice where there is no requirement for a more rigorous estimate of the uncertainty associated with results. Estimation of MU from within-laoratory data of ias and precision A reasonale estimate of MU may e gained from information on the ias and precision associated with a test result. This information may e gained from participation in suitale inter-laoratory and/or intralaoratory studies and used to estimate MU Suitale inter-laoratory studies are not always availale and more often than not, laoratories are required to estimate MU from data generated in-house. Ideally, precision and ias should e determined within the same analytical run as the sample(s) analysed. This is generally impractical and cost-prohiitive for most chemical tests. However, if suitale Q samples (matrix-matched ertified eference Materials [Ms], other suitale eference Materials [Ms] or spiked samples) are run within each sample atch to ensure the test method is consistently operating within acceptale control limits, then the data generated may e used to evaluate oth intermediate reproduciility (within-laoratory precision under reproduciility conditions) and average ias over a given period of time or numer of sample atches. Precision and ias should e evaluated at the concentration(s) most relevant to the purpose of the test. Note 4: Estimates of MU may also e gained from the evaluation of precision and ias during initial validation of a method. onsidering a model for the result of a chemical test conducted under the aove conditions: y = y true + + e (Equation 1) where: y = oserved measurement, uncorrected for ias; y true = true result; = ias, defined as y - y true, the difference etween the mean of a large numer of oserved results ( y ) and the true result; and e = random error for within-laoratory reproduciility conditions, s. L January 018 Page 6 of 13

7 Expression for comined standard uncertainty The comined standard uncertainty of y may e estimated as: u (y) = s L + u (Equation ) where: u (y) = comined standard uncertainty of y s L u = standard deviation of results otained under within-laoratory reproduciility conditions = standard uncertainty associated with ias. Estimating ias and the uncertainty associated with ias In practice, the true result is not known and it is necessary to ase estimates of ias on the ected result for the analysis of a M, or other suitale sample. = y - y where: y is the ected result (Equation 3) u = u( y ) + u(y ) (Equation 4) where: u( y ) = standard uncertainty of oserved result; u( y ) = standard uncertainty of ected result. If the oserved result is taken to e the mean result from replicate analyses of a M or other suitale sample, performed over several months, the standard uncertainty may e taken as the standard deviation of the mean of the oserved result. i.e. u( y ) = s L (Equation 5) n where n = numer of replicates performed. The standard uncertainty of the ected result may e known from the certified value of a M, the characterisation of a secondary reference material, estimated from the uncertainties associated with spiking, or estimated in some other way, as the particular situation demands. Once and u are estimated according to Equations 3 and 4, it is important to test whether or not the ias, taking into account the uncertainty associated with its measurement, is significant. If > t (0.05, n-1)u, where t is the Student-t value at n-1 degrees of freedom, then ias is significant and steps must e taken to account for ias when calculating results or estimating MU (see elow). If the average ias is ased on 0 or more measurements, can e simply compared with u in order to check if the average ias is significant. Estimating comined standard uncertainty and anded uncertainty If ias is not significant, the comined standard uncertainty may e calculated from Equation. If u (y) is ased on sufficient data the anded uncertainty, U, may e calculated using a coverage factor of to give an approximate level of confidence of 95%. U (95% confidence interval) = k u (y) = u (y) (Equation 6) Note 5: The Eurachem/ITA Guide 8, Section 8.3, recommends that for most purposes k is set at. The Guide states that this value may e insufficient where the comined uncertainty is ased on statistical oservations with relatively few degrees of freedom (less than aout six) and further recommends that in such cases k e set equal to the two-tailed value of the Student s t for the numer of degrees of freedom associated with the oservations and the level of confidence required (normally 95%). It should e noted that aout 19 degrees of freedom are required for k to e less than.1. If ias is significant and ased on reliale data such as the analysis of a M, then the measurement result should always e corrected for ias. Where excessive ias is detected, action should e taken to investigate, and if possile, eliminate the cause of the ias. 11 Such actions should at least reduce ias to acceptale levels efore a facility proceeds with analyses of test samples. January 018 Page 7 of 13

8 If results are not corrected for significant ias, the MU should e enlarged to ensure that the true result is encompassed y the reported confidence interval. A numer of approaches have een descried for taking significant ias into account when estimating MU. The est approach for handling significant ias is to eliminate or minimise the ias. O Donnell and Hiert 18 critically evaluated different approaches and concluded the est estimate of the anded uncertainty to e: U = k u (y) + run, where run is the run ias Note 6: The symols used in the aove equation are consistent with those used in this Technical Note, rather than those used y O Donnell and Hiert. O Donnell and Hiert specify run ias in their recommended equation for enlarging the estimate of MU, and provide lanation for using this approach. A reasonale estimate of MU may e otained if average total ias (when the ias is small), evaluated as descried aove, is used in place of run ias. U = k u (y) + (Equation 7) Failure to include the uncertainty of the estimation of ias in estimates of comined uncertainty or failure to correct for significant ias or enlarge the uncertainty of the result to account for it, invalidates oth the test result and the estimated MU. If ias can e quantified and is consistent, then it is preferale to correct the final result for ias, rather than and the uncertainty to include ias. Presenting a result x + the uncertainty assumes that x is in the middle of a normal distriution of results. Expanding the uncertainty to include a large, known ias changes the distriution, and gives a false impression of the result eing in the middle of a wider range. Laoratories should ensure that MU is estimated (at least) at the concentrations most relevant to the purpose of the test, for example regulatory, legal or specification levels. eality checks for estimates of MU Having completed an uncertainty estimation, the estimate should e suject to a reality check. A reality check can e as simple as considering whether the MU is consistent with the knowledge and erience of the facility. MU estimations reported y other facilities (using the same approach to estimate the MU, for the same analyses and using similar equipment) will not e exactly the same as the facility s own estimate, ut they should not vary excessively. Data from collaorative trials, inter-laoratory reproduciility and proficiency program results will all provide useful information for comparison. 19 eporting Measurement Uncertainty The Eurachem/ITA Guide 8 states that unless otherwise required, the result x should e stated together with the anded uncertainty U, calculated using a coverage factor, k =. The recommended form for reporting a result is: (esult): x ± U (units). The coverage factor used to calculate U should e stated, for example; 'DDT: 3.5 ± 0.14 mg/kg The reported uncertainty is an anded uncertainty calculated using a coverage factor of which gives a level of confidence of approximately 95%. The value and its uncertainty should oth e reported in the same units. esults reported as x ± (a percentage of x) are not recommended. The value of a result and its uncertainty should not e reported with an excessive numer of significant figures. It is rarely, if ever, necessary to report a chemical test result with the uncertainty stated to more than two significant figures. esults should e rounded to e consistent with their uncertainty. January 018 Page 8 of 13

9 It is recognised that some Laoratory Information Management Systems may not e ale to readily report MU in the recommended format. Other presentations are acceptale providing the essential elements of the recommended format are clearly covered in the test report. For further information regarding MU, contact your lient oordinator or the Sector Manager, Life Sciences: Neil Shepherd, in NATA s Melourne office on (03) or Neil.Shepherd@nata.com.au eferences 1. ISO/IE 1705 (005) General requirements for the ompetence of aliration and Testing Laoratories. ISO/IE Guide 98-3 (008) Uncertainty of Measurement-Part 3: Guide to the Expression of Uncertainty in Measurement (GUM 1995) 3. ISO/IE Guide 99 (007), International Vocaulary of Metrology-Basic and general concepts and associated terms, (VIM3) 4. ASTM International (005) Form and Style of ASTM Standards 5. E Directive 001//E 6. Horwitz, W (003), J. of AOA International, 86, Eurola (00) Measurement Uncertainty in Testing, Technical eport No. 1/00 8. Eurachem/ITA (00), Eurachem/ITA Guide, Quantifying Uncertainty in Analytical Measurement, nd Edition: Barwick V.J. and Ellison S. L.. (000), Protocol for uncertainty evaluation from validation data, VAM, eport No. LG/VAM/1998/ ISO (004), Guidance for the use of epeataility, eproduciility and Trueness Estimates in Measurement Uncertainty Estimation, ISO/TS Magnusson B., Naykki T., Hovind H. and Krysell M. (003), Handook for alculation of Measurement Uncertainty in Environmental Laoratories, NODTEST eport T Eurola (007) Measurement uncertainty revisited: Alternative approaches to uncertainty evaluation Technical eport 1/007, Horwitz W. Kamps L.. and Boyer K. W (1980), J. Assoc. Off. Anal. hem., 63, Horwitz W. (198), Anal. hem. 54, 67A 76A 16. Boyer K. W., Horwitz W. and Alert. (1985), Anal. hem., 57, Thompson M and Lowthian P J (1997), Journal of AOA International, 80(3), O Donnell G. E. and Hiert D. B. (005) Analyst, (130), oyal Society hemistry (003), Is my Uncertainty Estimate ealistic? Analytical Method ommittee Technical Brief No. 15 January 018 Page 9 of 13

10 Figure 1 k s L s U u u c Y y ias coverage factor Std Dev of results under within-la reproduciility conditions Std Dev of results under inter-la reproduciility conditions anded uncertainty uncertainty of ias comined standard uncertainty true result test result Standard method with pulished s Yes No Data on ias () and precision (s L) availale? Yes Estimate and u No Horowitz equation: pulications, erience, judgement Investigate: refine procedures to achieve reported performance; estimate MU from own data No La demonstrated to follow method within pulished performance characteristics Is significant? Is >t(0.05,n-1)u No Yes Yes Assess other factors, u c(other), u c =u c (other)+s U=u c Yes Do any other factors significantly contriute to MU? Is result corrected for ias? No U=Ku c+ No u c=s U(95% I)= S u c =s L +u U=ku c Yes Y=y- u c =s L +u U=ku c eality heck January 018 Page 10 of 13

11 Appendix 1: Estimation of MU from within-laoratory data on ias and precision Scenario A facility has validated a method for the determination of residues of the pesticide, chlorpyrifos in tomatoes and has applied the method on customer samples sumitted over a period of eighteen weeks. A matrixmatched certified reference material (MX 001) otained from the NMI was used during the validation process and henceforth once a fortnight to supplement spiked samples routinely used for within-atch Q. The certified value for chlorpyrifos in the M is ±0.031 mg/kg (95% confidence range). The facility initially estimated the MU of chlorpyrifos results ased solely on validation data ut now wishes to update its estimate, taking into account the Q data generated during routine analysis of customer samples. The facility analysed seven replicates of the M during method validation and a further nine replicates during routine analyses. The following data was generated via these processes: Validation data: mean mg/kg, standard deviation ( s r ) mg/kg, V 13.0% Q data: mean mg/kg, standard deviation ( s ) 0.08 mg/kg, V 1.3% ontrol charts plotted for oth the M and spiked samples run as Q samples demonstrate the test method to e under statistical control. The average result was consistent for oth validation and Q analyses, however, as ected, the results of the Q tests conducted under intra-laoratory reproduciility conditions were less precise than those for method validation conducted under near repeataility conditions. Both the % recovery (negative ias) and precision were considered acceptale for the test. It was considered important to estimate the MU associated with results close to 0.5 mg/kg, the maximum residue limit (ML) for chlorpyrifos in tomato. No dout with the ML in mind, the M produced y the NMI has a certified value for chlorpyrifos closely matching the ML. Estimation of MU Data The mean of all results is the est estimate of the likely atch-to-atch result. This value (the average of all validation and Q replicate measurements) is the average oserved result, y = mg/kg; mean recovery = 79.3% The standard deviation of the Q results est reflects the imprecision of results produced under normal test conditions; s = s = 0.08 mg/kg The ected result, L y, is the certified value of the M; ± mg/kg The standard uncertainty of the ected result u y is otained y halving the anded uncertainty (95% confidence range) stated for the M. u = 0.031/ = mg/kg y Estimation of ias and the uncertainty of ias Bias ( ) = y - y (Equation 3) = = mg/kg This is the average ias and the est availale estimate of the ias that might apply to any individual test result. Uncertainty of ias (from equation 4), u = u y u y January 018 Page 11 of 13

12 Uncertainty of the oserved result (Equation 5), u y = s L n = 0.08/3 = 0.07 mg/kg u = u y u y = = mg/kg Is the ias significant? Is > t (0.05, n-1) u? Here, n = 9 since the uncertainty of the oserved result is ased on nine replicate tests From t tales (critical values for two-tailed student t-tests), t (0.05, 8) =.306 Is >.306 x 0.031? Is > 0.071? Answer YES, so ias IS significant It is necessary for the facility to correct results for ias (preferale alternative) or enlarge their estimate of MU to account for uncorrected significant ias. Estimating comined standard uncertainty and anded uncertainty (i) results corrected for ias If the facility decided to correct results at or aout the ML for the negative ias, the correction could e achieved y simply adding the ias to the measured value, assuming the ias to e constant over a narrow concentration range. There is uncertainty associated with this correction, and it is therefore necessary to include the uncertainty of ias in the comined standard uncertainty, irrespective of whether or not results are corrected for ias. Assuming a raw test result = 0.35 mg/kg. orrected result = 0.45 mg/kg omined standard uncertainty, (from Equation ) u y = s u = L 0 = mg/kg Expanded uncertainty (Equation 6); U (95% confidence interval) = u = 0.18 mg/kg y The facility would report the result as 0.45 ± 0.18 mg/kg, noting that the reported uncertainty is an anded uncertainty calculated using a coverage factor of to give a level of confidence of approximately 95%. (ii) results not corrected for ias Expanded uncertainty (Equation 7): U uc ( y) = = 0.8 mg/kg The facility would report the result as 0.35 ± 0.8 mg/kg. When the anded uncertainty is enlarged in example (ii) to account for uncorrected significant ias, the reported result encompasses a 95% confidence range, which is wider than justified on the lower side of the measured value: the uncertainty allows for oth negative and positive ias although only negative ias is present. The result corrected for ias in example (i) provides the customer with a etter estimate of the true result and a etter asis for decision-making, although as ected when the true value is close to a limit, the result is equivocal with respect to compliance with the ML. January 018 Page 1 of 13

13 AMENDMENT TABLE The tale elow provides a summary of changes made to the document with this issue. Section Amendment Entire document This document replaces the former Technical Note 33. The document has een renamed to reflect the new accreditation criteria documentation structure. January 018 Page 13 of 13