Quality Assurance of Monitoring/Surveillance Data

June 12, 2013 Quality Assurance of Monitoring/Surveillance Data Takanori UKENA, Ph.D. takanori_ukena@nm.maff.go.jp MAFF Ministry of Agriculture Forestry and Fisheries Food Safety and Consumer Affairs Bureau 1

Contents Importance of Quality Assurance 1. Laboratory Management, ISO/IEC 17025 2. Use of Validated Analytical method 3. Internal Quality Control and Proficiency Testing 4. Measurement Uncertainty 5. Evaluation of Analytical Results Experience of MAFF Japan 2

3 Importance of Quality Assurance

4 What is Quality? degree to which a set of inherent characteristics fulfils requirement". (ISO 9000) In case of analysis delivery of reliable data/results within an agreed span of time under agreed conditions, at agreed costs, and with necessary aftercare".. The agreed conditions should include a specification as to the precision and accuracy of the data

5 Quality Management Quality Assurance Quality Control (QC)

Quality Management Quality management: coordinated activities to direct and control an organization with regard to quality Note : generally includes establishment of the quality policy, and quality objective, quality planning, quality control, quality assurance, and quality improvement. Quality management system: management system to direct and control an organization with regard to quality (ISO 9000:2005) MAFF The laboratory shall establish, implement and maintain a management system appropriate to the scope of its activities. (ISO/IEC 17025:2005) 6

7 Quality Assurance Quality assurance: All those planned and systematic actions necessary to provide adequate confidence that analytical results will satisfy given requirements for quality. GUIDELINES ON ANALYTICAL TERMINOLOGY (CAC/GL 72-2009) part of quality management focused on providing confidence that quality requirement will be fulfilled (ISO 9000:2005)

8 Quality Control Quality Control: part of quality management focused on fulfilling quality requirement (ISO 9000:2005) for chemical analysis laboratories Internal Quality Control and Proficiency testing.

9 Why quality assurance is needed? Prevention of quality problems Basis of cconfidence in the data Risk manager need to make decision based on data obtained from other persons, other laboratories, other countries. International requirement

What is reliable analytical results? Fit for purpose Data produced by a measurement process enables a user to make technically and administratively correct decisions for a stated purpose. Use of validated method If you don t use validated methods, only you will know how good your numbers are. By a laboratory with Quality Management Internal quality control and proficiency testing MAFF Back ground information/data 10

Use of analytical results Conformity assessment Import/export control Risk management activities Occurrence data for estimation of exposure Evaluation of effectiveness of risk management options International activities GEMS/Food, Codex, JECFA, JMPR, etc. MAFF 11

12 Inadequate data (example) 5.3 Distribution The data from Brazil and Uruguay could not be used as they had been obtained by analytical methods with high LODs, ranging from 5 µg/kg to 50 µg/kg, which are inadequate to detect and measure ochratoxin A at 0.94 µg/kg and 0.19 µg/kg, the weighted mean concen-trations found in Europe and the USA in cereals and cereal products, respectively. (JECFA 47, 2001)

Codex guidelines for Import and Export Control testing laboratories Compliance with the general criteria in ISO/IEC 17025 Participation in appropriate proficiency testing schemes Use of validated method Internal quality control procedures MAFF Guidelines for the Assessment of the Competence of Testing Laboratories Involved in the Import and Export Control of Food (CAC/GL 27-1997) 13

Food Control Laboratory Management: Recommendations Protocols and Guidelines adopted in Codex International Harmonized Protocol for the Proficiency Testing Pure & Appl. Chem., 65 (1993) 2132-2144 Protocol for the Design, Conduct and Interpretation of Method Performance Studies Pure & Appl. Chem., 67 (1995) 331-343 Harmonized Guidelines for Internal Quality Control Pure & Appl. Chem., 67 (1995) 649-666 MAFF Food Control Laboratory Management: Recommendations (CAC/GL 28-1995, Rev.1-1997) 14

Guidelines for Settling Disputes over Analytical (Test) Results the following information should be shared between competent authorities of the importing and exporting country validation status of the analytical method used raw data internal quality assurance/control (control charts, sequence of analysis, blank data, recovery data, etc.) performance in relevant proficiency testing or collaborative studies; official accreditation status of the laboratories. MAFF (CAC/GL 70-2009) 15

16 Topic 1 Laboratory Management, ISO/IEC 17025

Why is ISO/IEC 17025 important? Requirements of Codex guidelines ISO an International standard / international recognition Benefit of accreditation for laboratory A marketing advantage for laboratory often required for international trade A bench mark for performance In depth assessment Required periodic reassessment MAFF An indicator of technical competence 17

Requirements of ISO/IEC 17025 Management (15) Organization Management system Document control Review of requests/contracts Subcontracting of tests Purchasing services/suppliers Service to the customer Complaints Control of non-conforming tests Improvement Corrective action Preventive action Control of records Internal audits Management review MAFF Technical (10) General Personnel Accommodation and environmental conditions Test method and method validation Equipment Measurement traceability Sampling Handling of test and calibration items Assuring quality of test and calibration results Reporting the results 18

19 What is Accreditation of ISO/IEC 17025? Formal recognition by an authoritative body of the competence of a laboratory to carry out specific tests or types of tests in a reliable, credible and accurate manner Evidence of Quality, Credibility and Reliability technical competence of staff validity and appropriateness of test methods suitability, calibration and maintenance of equipment appropriate handling of test materials quality control procedure, e.g., use of CRM, Proficiency testing, etc. Actual performance and ethical behavior is critical. Not a guarantee

Notes for Accreditation of ISO/IEC 17025 Accredited to each analytical method each laboratory Need to confirm : For what method the laboratory is accredited? Which laboratory is accredited? If the organization has blanch laboratories. MAFF 20

ILAC, APLAC ILAC: International Laboratory Accreditation Cooperation The international organization on laboratory accreditation with membership of accreditation bodies and affiliated organizations. Maintain a Mutual Recognition Agreement (MRA) among accreditation bodies to facilitate trade and to eliminate the need for duplicate accreditation. Assist developing accreditation programs APLAC: Asia Pacific Laboratory Accreditation Cooperation 21 MAFF

ILAC-MRA, APLAC-MRA Tested once, accepted everywhere A laboratory accredited by one MRA partner has equivalent competence to laboratory accredited by the other MRA partner(s) Accreditation body shall comply with ISO/IEC 17011 and that its accredited facilities are in compliance with ISO/IEC 17025 (laboratories), ISO 15189 (medical laboratories) and ISO/IEC 17020 (inspection bodies). MAFF https://ilac.org/home.html http://www.aplac.org/home.html 22

23 Topic 2 Use of Validated Analytical method

Chemical Analysis Qualitative the presence or absence of a particular substance, but not the mass or concentration. Identification of the substance e.g., structure elucidation of natural product MAFF Quantitative determination of the absolute or relative abundance (concentration) of particular substance(s) present in a sample. 24

Analytical method for Risk Management Analytical Methods Whenever Whoever (analytical specialist) can follow the method procedure and obtain similar results for the same sample. (e.g., variability of results fall in expected interval). MAFF 25

26 Validated method If you don t use validated methods, only you will know how good your numbers are. AOACI Validated Test Method: An accepted test method for which validation studies have been completed to determine the accuracy and reliability of this method for a specific purpose. Reference: ICCVAM Guidelines for the nomination and submission of new, revised and alternative Guidelines on Analytical Terminology (CAC/GL 72-2009)

27 Method Validation Study Validated method is a method tested in many laboratories on the same samples, showing that the method is rugged enough to produce comparable results in different labs, with different operators. Collaboration study, Ring-test- trial

28 Validation Study Protocol and Guidelines IUPAC Protocol for the Design, Conduct and Interpretation of Method- Performance Studies. Pure & Appl. Chem., 67(2), 331-343 (1995) http://www.iupac.org/publications/pac/67/2/0331/ AOAC International AOAC Official Methods of Analysis (2002). Interlaboratory Collaborative Study, Appendix D: Guidelines for Collaborative Study Procedures To Validate Characteristics of a Method of Analysis http://www.aoac.org/vmeth/guidelines.htm ISO 5725-2:1994 Accuracy (trueness and precision) of measurement methods and results -- Part 2: Basic method for the determination of repeatability and reproducibility of a standard measurement method

Comparison of Protocol for the study MAFF * test statistics and application procedure are different between the protocols. 29

30 Single-laboratory Validated Method General Criteria in Codex Procedure Manual single-laboratory validated methods must fulfill the following criteria: (i)the method is validated according to an internationally recognized protocol (e.g. IUPAC Guidelines) (ii) the use of the method is embedded in a quality system in compliance with the ISO/IEC 17025

The method should be complemented with information on accuracy demonstrated for instance with: regular participation in proficiency schemes, where available; calibration using certified reference materials, where applicable; recovery studies performed at the expected concentration of the analytes; verification of result with other validated method where available. MAFF (Codex Procedural Manual) 31

32 Guidelines on Single-Laboratory Validation Harmonized Guidelines for Single-Laboratory Validation of Methods of Analysis IUPAC/ISO/AOAC, 2002 Performance characteristics Applicability Selectivity Calibration and linearity Trueness, Precision Recovery Limit of Detection, Limit of Quantification Sensitivity Ruggedness etc.

33 Recommended methods in Codex General Methods of Analysis for Contaminants (CODEX STAN 228-2001) Recommended Methods of Analysis and Sampling (CODEX STAN 234-1999)

34 Criteria approach Identify method performance characteristics based on existing method validation data. and establish criteria for evaluating acceptable method of analysis laboratory can choose a method meeting criteria (flexibility)

35 Guidelines for Criteria approach <Codex Procedural Manual> Working Instructions for the Implememtation of the Criteria Approach in Codex Guidelines for Establishing Numeric Values for Method Criteria and/or Assessing Methods for Compliance Thereof

Examples for numeric values for the criteria MAFF * The sr should be calculated from the Horwitz / Thompson equation 36

37 Terms related to variability of analytical results

38 Terms related to variability of test results Accuracy Measurement Uncertainty Trueness Precision Repeatability Intermediate precision Reproducibility

A Distribution of results for repeated analysis µ: True value m: mean X-axis: concentration Y-axis: frequency B C µ m µ m D MAFF µ m µ m 39

accuracy, trueness, precision µ (true value); m (mean); σ (standard deviation); c (m+s) A B C s σ m m m accuracy trueness precision MAFF 40

41 Accuracy The closeness of agreement between a test result or measurement result and a reference value. When applied to a test method, the term accuracy refers to a combination of trueness and precision. Guidelines on Analytical Terminology (CAC/GL 72-2009) A reference value is usually provided with reference to a) a certified reference material; b) a reference measurement procedure; c) a comparison of measurement standards.

42 Trueness The closeness of agreement between the average of an infinite number of replicate measured quantity values and a reference quantity value. Measurement trueness is inversely related to systematic measurement error, but is not related to random measurement error. Measurement accuracy should not be used for measurement trueness and vice versa. Guidelines on Analytical Terminology (CAC/GL 72-2009)

Precision The closeness of agreement between independent test/measurement results obtained under stipulated conditions. depends only on the distribution of random errors and does not relate to the true value or to the specified value. usually expressed in terms of imprecision and computed as a standard deviation of the test results. Less precision is reflected by a larger standard deviation. Quantitative measures of precision depend critically on the stipulated conditions. Repeatability and reproducibility conditions are particular sets of extreme conditions. MAFF Guidelines on Analytical Terminology (CAC/GL 72-2009) 43

Repeatability Precision under repeatability conditions. Repeatability conditions: Observation conditions where independent test/measurement results are obtained with the same method on identical test/measurement items in the same test or measuring facility by the same operator using the same equipment within short intervals of time. MAFF Guidelines on Analytical Terminology (CAC/GL 72-2009) 44

Reproducibility Precision under reproducibility conditions. Reproducibility conditions: Observation conditions where independent test/measurement results are obtained with the same method on identical test/measurement items in different test or measurement facilities with different operators using different equipment. MAFF Guidelines on Analytical Terminology (CAC/GL 72-2009) 45

46 Relative standard deviation Relative standard deviation (%) = standard deviation (SD) / mean 100 same as CV (coefficient of variation) SD ( x 1 μ ) 2 + + n 1 ( x n μ ) 2 RSDr : Repeatability relative standard deviation RSDR : Reproducibility relative standard deviation

Horwitz equation Horwitz equation 1-0. 5 log RSD (%) 2 10 R approximation of Horwitz equation C RSDR (%) = 2 C -0.1505 MAFF RSDR (%): reproducibility relative standard deviation C:concentration ratio if concentration ratio =100 %, then C = 1 1 %, C = 0.01 A ppm C = A 10-6 47

相対標準偏差 [%] Graphical expression of Horwitz equation 200 150 100 50 128% 45% 16% 4% 0-50 -100 MAFF RSDR -150-200 1 ppt 1 ppb 1 ppm 1% 48

Horwitz/Thompson equation 22 C < 1.2 10-7 RSDR (%)= 2 C-0.1505 1.2 10-7 C 0.138 C-0.5 C > 0.138 RSDR (%): Reproducibility relative standard deviation C:concentration ratios MAFF Thompson, M., Analyst, 125, 385-386 (2000) 49

Graphical expression of Horwitz/Thompson equation 30 20 16% 10 0 22% -10 MAFF Relative standard deviatio -20-30 1 ppt 1 ppb 1 ppm 1% 0.1ppm Concentration Thompson, M., Analyst, 125, 385-386 (2000) 50

HorRat Value The ratio of the reproducibility relative standard deviation to that calculated from the Horwitz equation, HorRat(R) = RSDR/PRSDR HorRat(r) =RSDr/PRSDR MAFF RSDr, RSDR: observed relative standard deviation (sr/c, sr/c) PRSDR: predicted relative standard deviation calculated from Horwitz/Thompson Normal range of HorRat Guidelines on Analytical Terminology (CAC/GL 72-2009) 51

52 Limit of Detection (LOD) and Limit of Quantification (LOQ)

What is Limit of Detection? Limit of Detection (LOD) : The true net concentration or amount of the analyte in the material to be analyzed which will lead, with probability (1-β), to the conclusion that the concentration or amount of the analyte in the analyzed material is larger than that in the blank material. Critical Value (Lc) : The value of the net concentration or amount the exceeding of which leads, for a given error probability α, to the decision that the concentration or amount of the analyte in the analyzed material is larger than that in the blank material. MAFF Guidelines on Analytical Terminology (CAC/GL 72-2009) 53

LOD and Critical Value (Lc) Results of LOD concentration sample β: Type II error (false negative) β Sample of which concentration > LOD will lead to the decision that the concentration of the material is larger than that in the blank material with probability larger than 1-β LOD If results are lower than Lc, they are N.D. (could not decided to be detected) If results > Lc, then they will lead to the decision that it is not blank, detected with probability larger than 1- α. α: Type I error (false positive) MAFF Results of blank 0 Lc 54

55 The limit of detection LOD is estimated by, LOD 2 t1-αν σo [where α = β], Where t1-αν : Student's-t, based on ν degrees of freedom for a onesided confidence interval of 1-α σo : the standard deviation of the true value (expectation). LOD = 3.29 σo, when the uncertainty in the mean (expected) value of the blank is negligible, α = β = 0.05 Usually estimated by using variance of sample (SD of True value is not known.) LOD depends on probability (α and β) Calculate t-value depending on number of data (LOD should not always equals to 3.29 s (SD. observed))

t-distribution and Standard Normal Distribution t-distribution, df=1 t-distribution, df=3 t-distribution, df =10 Standard normal distribution MAFF α=0.05: Degree of Freedom = 1 t 0.95,1 = 6.31 (2 t-value=12.62) Degree of Freedom = 10 t 0.95,9 = 1.81 (2 t-value=3.62) Degree of Freedom = t 0.95, = 1.64 (2 t-value=3.29) 56

Limit of Quantification Limit of Detection (LOD) : A method performance characteristic generally expressed in terms of the signal or measurement (true) value that will produce estimates having a specified relative standard deviation (RSD), commonly 10% (or 6%). LOQ is estimated by: LOQ = kq σq, Where kq = 1/RSDQ If σ is known and constant (SD of the estimated quantity is independent of concentration), and σq = σo : substitution of kq = 10% LOQ = (10 σq) = 10 σo (3.04 LOD, α = β = 0.05) MAFF Guidelines on Analytical Terminology (CAC/GL 72-2009) 57

Example of Method Criteria at Different Concentrations MAFF (Codex Procedural Manual) 58

59 Topic 3 Internal Quality Control and Proficiency testing

60 3-1 Internal Quality Control (IQC)

61 Guidelines on internal Quality Control Harmonized Guidelines for Internal Quality Control in Analytical Chemistry Laboratories, Pure & Appl. Chem. 67 (1995) 649-666. Introduction Definition Quality assurance practices and internal quality control Internal quality control procedure IQC and within-run precision Control materials in IQC Recommendations Conclusions

62 Definition Internal Quality Control Internal Quality Control: Set of procedures undertaken by laboratory staff for the continuous monitoring of operation and the results of measurements in order to decide whether results are reliable enough to be released.

63 Control materials in IQC Control materials are characterized substances that are inserted into the run alongside the test materials and subjected to exactly the same treatment. appropriate concentration of the analyte same matrix in terms of bulk composition, including minor constituents similar physical form stable and possible to divide the material into effectively identical portions for analysis

Use of Certified reference material (1) Certified reference material (CRM): Reference material accompanied by documentation issued by an authoritative body and providing one or more specified property values with associated uncertainties and traceability, using valid procedures Documentation is given in the form of a certificate (ISO guide 30:1992). Procedures for the production and certification of certified reference materials are given, e.g. in ISO Guide 34 and ISO Guide 35. MAFF GUIDELINES ON ANALYTICAL TERMINOLOGY (CAC/GL 72-2009) 64

Metrological Traceability Metrological Traceability: Property of a measurement result whereby the result can be related to a reference through a documented unbroken chain of calibrations, each contributing to the stated measurement uncertainty. Metrological traceability of a measurement result does not ensure that the measurement uncertainty is adequate for a given purpose or that there is an absence of mistakes. MAFF Guidelines on Analytical Terminology (CAC/GL 72-2009) 65

Metrological Traceability (2) Comparability of measurement result, for quantities of given kind. (metrologically traceable to the same reference) BIPM Definition of unit, SI base National Metrology Institute, Designated Institute Primary national standard Primary national standard (other countries) Calibration organization Competent authority, Industry, Academia End user Reference standards Working standards Measurement result MAFF Larger Measurement Uncertainty in lower stage 66

67 Use of Certified reference material (2) COMAR (International Database) http://www.comar.bam.de/en/ Purchase from reagents manufacturer or reagents sales company Comparison of the measurement results with the certified value

Comparison of the measurement results with the certified value Δ m C C m CRM Δm:absolute difference between mean measured valued and certified value Cm: mean measured value CCRM : Certified Value Δ 2 u + U 2u Δ Δ u m u 2 CRM UΔ: expanded measurement uncertainty um: measurement uncertainty ucrm : uncertainty of the certified value If Δm UΔ then there is no significant difference between the measurement result and certified value. MAFF http://www.erm-crm.org/erm_products/application_notes/pages/index.aspx 68

69 Constraint on the use of CRM For the majority of analysis there is no closely matching CRM available. Cost to stock every relevant CRMs Not applicable to unstable materials Availability of sufficient amounts for IQC to use over extended periods Preparation of House reference material e.g. materials in proficiency testing, spiked control materials, etc.

70 Recovery checks If the use of reference material is not practical, limited check on bias is possible by a test of recovery. applicable unstable analyte/matrices, ad hoc analysis A test portion of the test sample spiked with a known amount of the analyte is analyzed alongside the original test material.

71 Blank determination an analysis of a sample without the analyte, or an analysis without a sample, i.e. going through all steps of the procedure with the reagents only. 1.In many analyses sample results are calculated by subtracting blank readings from sample readings. 2.Blank readings can be excellent monitors in quality control of reagents, analytical processes, and proficiency. 3. They can be used to estimate several types of method detection limits.

72 Insertion of control materials for IQC (1) Recommendation 1.Short (e.g. n<20) frequent runs of similar materials at least once per run analyze in duplicate at least half of the test materials selected at random insert at least one blank determination 2. Longer (e.g. n>20) frequent runs of similar materials at an approximate frequency of one per ten test materials at least once per run analyze in duplicate at minimum of five test materials selected at random insert one blank determination per ten test materials

Insertion of control materials for IQC (2) 3. Frequent runs containing similar materials but with a wide range of analyte concentrations at an approximate frequency of one per ten test materials at least once per run. at least two concentration levels, one close to the median level of typical test materials, and the other is approximately at the upper or lower of decile as appropriate. duplicate a minimum of five test materials. insert one procedural blank per ten test materials. 4. ad hoc analysis (statistical control is not applicable) MAFF duplicate analysis on all of the test materials recovery tests or use of formulated control material blank determination 73

74 Use of Control Charts Control Chart (Shewhart Chart): A statistical tool used to monitor process stability and control. One of the Seven Basic Tools of Quality Control. 1. Understanding current and past process performance and its degree of consistency 2. Establishing a "state of statistical control" by identifying and removing causes of unnatural (or "special cause") variation so as to achieve a consistent and predictable level of process quality over time;

Common Control Charts 75 X - R chart X - X - s chart R chart

76 How to calculate/plot X-Bar and R chart (1)

How to calculate/plot X-Bar and R chart (2) : Calculate the following statistics X : The mean for each subgroup X X 1 + X 2 + + n X n : The number of measurements within a subgroup (here, n=2) n X : The grand mean of all subgroup averages MAFF X X 1 + X 2 + + k k : The number of subgroups (here, k=9) X k 77

78 How to calculate/plot X-Bar and R chart (3) R : The average of the ranges for all subgroups R R 1 + R 2 + + k Ri : The individual range for each subgroup k : The number of subgroups (here, k=9) R k

How to calculate/plot X-Bar and R chart (4) : Calculation of Upper and Lower control limit Upper control limit (UCL): For X chart: UCL X + A2 R For R chart: UCL D4 R Lower control limit (LCL): For X chart: LCL X A2 R MAFF For R chart: LCL D3 R 79

http://www.itl.nist.gov/div898/handbook/pmc/section3/pmc321.htm 80 How to calculate/plot X-Bar and R chart (5) Constants for Calculating Limits for X-Bar and R charts

How to calculate/plot X-Bar and R chart (6) MAFF Time line (Day/Run) 81

82 Example of X-Bar and s chart Time line (Day/Run)

83 3-2 Proficiency testing

84 External Program for Quality Control (Proficiency testing) Proficiency testing (PT) : evaluation of participant performance against pre-established criteria by means of interlaboratory comparisons (ISO/IEC 17043:2010) Participating laboratories analyze same sample distributed from proficiency testing provider. Reported results were compared to the assigned value and statistically evaluated. Participants can confirm their results.

85 How to participate in PT program? Proficiency testing provider Internationally renowned: Fera(FAPAS, FEPAS), AOACI, AOCS, etc. Domestic (In case of Japan): JSAC, FDSC, JAB, etc. Participation Through internet, agency, etc. Check schedule of individual proficiency testing round.

Example of Proficiency Testing Scheme example :FAPAS Round: Nutritional components, Food ingredients contaminants, Pesticides, Veterinary drug residues, Food additives, etc. Participating laboratories analyze distributed sample using their method. Report the results and the method information through website. Results were Statistically evaluated and z-score were returned to participating laboratories. z 2, 2< z <3, z 3 MAFF 86

87 z = (x - X) / σp Result of PT Usually shown in z-score defined as follows: x: the participant s reported result X:the assigned value e.g., robust mean after excluding results that are clearly spurious and outliers by statistical evaluation. σp :the target value for standard deviation e.g., using RSDR from collaborative study, calculation from Horwitz equotion etc.

Graphical expression of z-scores (example) MAFF z-scores for Patulin (80.7 g/l) in Apple Purée Test Material From: FAPAS PROTOCOL FOR THE ORGANISATION AND ANALYSIS OF DATA SIXTH EDITION, 2002 88

89 How to interpret a result of PT? z-scores z 2 is satisfactory? A result of z >2 is not specially rare. Is that a usual operation for the laboratory? Importance of Participating PT Checking bias of the results Review of method procedure and management system of laboratory by periodical participation Information on methods used by other laboratories

90 Topic 4 Measurement Uncertainty

Measurement Uncertainty in Codex (1) Definition Measurement uncertainty: Non-negative parameter characterizing the dispersion of the values being attributed to a measurand, based on the information used. Expanded measurement uncertainty: product of a combined standard measurement uncertainty and a factor larger than the number one MAFF Guidelines on Analytical Terminology (CAC/GL 72-2009) 91

92 What is Measurement Uncertainty? Estimated range in which the true value would be found Showing reliability of the results Estimated by experimental data and statistical evaluation

93 Analytical results and its variability Analytical results of repeated analyses of the same sample can be found in a normal distribution. In case of chemical quantitative analysis, reproducibility, between laboratory variation, depends on concentration of analyte, independent of variety of food or analyte.

Normal Distribution Mean : µ SD : σ 68% 95% 99% 4σ 3σ 2σ σ µ +σ +2σ +3σ +4σ MAFF 68% of the observations have values within the range of µ±1σ 95% of the observations have values within the range of µ±2σ 94

95 Measurement Uncertainty and Bias Y-axis: Density Population distribution (unknown) True Value (Value we want to know) (unknown) Sample distribution Mean X-axis: Concentration Measurement Uncertainty : Estimated range in which the true value would be found. Bias: difference between the result and the true value

96 Measurement Uncertainty in Codex (2) Measurement Uncertainty have to be estimated in Codex. One of the requirements of the ISO/IEC 17025:2005 The measurement uncertainty of a result shall be estimated and then made available if requested. Guidelines CAC/GL 27-1997 require laboratories involved in the import/export of foods to comply with general criteria in ISO/IEC 17025. Guidelines on Measurement Uncertainty (CAC/GL 54-2004)

Measurement Uncertainty in Codex (3) MAFF 97 Reporting the results in a form of a ± U where a: The best estimate of the true value of the concentration of the measurand (the analytical result) u: The Standard uncertainty U: The expanded uncertainty (usually equal to 2u). The range a ± 2u represents a 95% level of confidence in which the true value would be found. Guidelines on Measurement Uncertainty (CAC/GL 54-2004)

Estimating Measurement Uncertainty 1.Bottom-up approach Identification of uncertainty sources Estimation of the size of the uncertainty component associated with each potential source. Calculation of combined uncertainty 2. Top-down approach Use data from collaborative trials, proficiency studies, validation studies or intra-laboratory quality control samples, MAFF Guidelines on Measurement Uncertainty (CAC/GL 54-2004) 98

Typical Value of Expanded Uncertainty MAFF CAC/GL 54-2004 GUIDELINES ON MEASUREMENT UNCERTAINTY 99

MU in Compliance Decisions Result Maximum Level +U -U Situation ii) iii) iv) i) Result ±U above ML Result > ML but ML within U Result < ML but ML within U Result ±U < ML MAFF Guidelines on Measurement Uncertainty (CAC/GL 54-2004) Guidelines on Estimation of Uncertainty of Results (CAC/GL 59-2006) 10

10 Topic 5 Evaluation of Analytical Results

10 Information from Laboratory Require all relevant data (especially when using 3rd Party laboratory) validation status of the methods of analysis used, sample handling, preparation procedures. raw data (including spectral data, calculations, chemical standards used); results of repeat analysis; internal quality control (control charts, blank data, recovery data, uncertainty data, use of RMs); performance in proficiency testing official accreditation status of the laboratory

Typical Problem for the Reported Results (Examples) mistakes e.g., sampling number, classification of commodity Ambiguous data no formula for calculations of LOD, LOQ, recovery, etc. no information on recovery correction Inappropriate significant figures MAFF 103

Check Analytical work was correctly done as specified? MAFF Sample treatment was appropriate? Validated method used? Verified for the commodity? LOD and LOQ were properly calculated? LOQ was lower than expected occurrence level? Calibration was appropriate? reliable standard reagents linearity of calibration:r2 0.99? Recovery corrected? Significant figures were properly calculated? 10

105 Recovery Recovery/recovery factors: Proportion of the amount of analyte, present in, added to or present in and added to the analytical portion of the test material, which is presented for measurement. Guidelines on Analytical Terminology (CAC/GL 72-2009)

Food /Feed Reagents spiked Measurement Results Marginal Recovery (%) C f C C A u Total % Recovery C A C f + C U MAFF Cf : fortified concentration, analytical results of spiked sample Cu : unfortified concentration, analytical results of un-spiked sample CA : added concentration 106

Criteria for Recovery in Codex MAFF 107 (Codex Procedural Manual) 107

Recovery Correction All data, when reported, should (a)be clearly identified as to whether or not a recovery correction has been applied and (b) if a recovery correction has been applied, the amount of the correction and the method by which it was derived should be included with the report. This will promote direct comparability of data sets. Correction functions should be established on the basis of appropriate statistical considerations, documented, archived and available to the client. MAFF Harmonized IUPAC Guidelines for the Use of Recovery Information in Analytical Measurement (CAC/GL 37-2001) 108

109 Recovery check as IQC IQC control charts for recovery should be established during method validation and used in all routine analysis. Runs giving recovery values outside the control range should be considered for re-analysis in the context of acceptable variation, or the results reported as semiquantitative. Harmonized IUPAC Guidelines for the Use of Recovery Information in Analytical Measurement (CAC/GL 37-2001)

Significant Figures(1) Addition and Subtraction The result should have as many decimal places as the measured number with the smallest number of decimal places. (e.g.) 2.54 cm + 15.75 cm = 18.29 cm 1.46 mg + 12.7 mg = 14.2 mg (e.g.) 3.58 kg 0.779 kg = 2.80 kg 1.46 + 12.7 14.16 2 MAFF 110

Significant Figures(2) Multiplication and Division The result should have as many significant figures as the measured number with the smallest number of significant figures. e.g. MAFF 1.42 mg/kg 15 kg = 21 mg 100 m 12 s = 8.3 m/s Divided by 4 pieces 8.43cm 4 = 2.11 cm Exact numbers, such as the number of people in a room, have an infinite number of significant figures. 1.42 15 710 142 2130 111

ISO 80000-1: 2009 112 Rounding of numbers (1) The rounding shall always be carried out in only one step. e.g. 12.254 should be rounded to 12,3 and not first to 12,25 and then to 12,2. Rounding range: 0.1 Rounding range:10

Rounding of numbers (2) If there are two successive integral multiples equally near the given number, two different rules are in use. Rule A: The even multiple Rule B: The greater in magnitude MAFF ISO 80000-1: 2009 (ISO 31-0 1992) JIS Z 8401:1999 113

114 Experience of MAFF Japan

Im fo Roles of MAFF and MHLW for food safety MAFF 115 Food Chain Primary Production Processing, Distribution Farm Fresh produce Raw material Processed Food Consumer MAFF (recommendation) Improvement of production/processing methods of domestically produced food, etc. D p F MAFF (enforcement) Fertilizer, Feed, Veterinary drug the use of Pesticide Exporting Country Cooperation MHLW (enforcement) Setting MRLs Food Inspection, etc. MHLW (enforcement) Import quarantine

Standard Operating Procedure (SOP) for MAFF & MHLW Publication:25 Aug 2005, rev 2006, 2012 Contents Identification of a food safety problem Establishment of a risk profile Ranking of the hazard for risk management priority Establishment of risk assessment policy Consideration of the result of the risk assessment Evaluation of risk management options Implementation of management decision Risk communication etc 116

Example of priority list of Chemical hazards for food safety (MAFF Marine 2006, toxins revised in Metals 2010) Arsenic Cadmium Lead Methyl mercury Mycotoxins Aflatoxins Zearalenone T2-toxin Paralytic shellfish poisoning toxins Diarrhetic shellfish poisoning toxins Ciguatera poisoning toxins Domoic acid Brevetoxin Process contaminants Acrylamide Polycyclic aromatic hydrocarbons Furan 3-monochloropropane-1,2-diol (3-MCPD) etc. 117

Guideline for the evaluation and publication of analytical results of surveillance/ monitoring (2006, MAFF) MAFF 118 Purpose To Guide MAFF officials for assessing the quality of analytical reports from internal/private laboratories, and for relevant considerations for their publication To provide analytical laboratories with the format for reporting the result of analysis

119 Guideline for assessing the result of surveillance/ monitoring (MAFF) Format for reporting the result of analysis by Laboratories Laboratory Lab Name, Address, Person in charge A. Analysis 1. Analyte / Matrix 2. Laboratory Sample Date of receipt, Condition for storage of laboratory sample and replicate sample (temperature etc.) 3.Sample Preparation Procedure for preparation of a portion for the measurement from Laboratory sample

120 4. Information of analytical method Standard operating procedure of the analytical method 5. Information of method validation Multi Lab / Single Lab / Other Matricies used for validation 6. LOD, LOQ Value, Calculation method 7. Calibration Used standard reagent, Purity, Protocol for preparation of standard solution, Linear range

121 8. Recovery Matrix, Analyte, Concentration, Number of replication, Value of recovery rate and RSDr, Raw data or corrected by recovery 9. Measurement Uncertainty Value of measurement uncertainty Method for estimation 10. Analytical results Number of tested samples, Value of measurement of each sample

122 B Quality Control 1. Internal in accordance with GLP guideline by MHLW frequency 2. Proficiency testing provider, matrix, analyte, z-score must participate in the last two years (same or similar matrix and analyte) 3. ISO/IEC17025 Accreditation

123 C Sampling (only when the laboratory itself conducted sampling) Sampling method and it s reference Sample size for each step in the sampling procedure

Change in design of surveillance in MAFF (1) 124 Before establishing guidelines Large variety but small sample number for one commodity e.g. only 5 samples for one commodity No attention to quality of analytical result Poor requirement of internal quality control, proficiency testing, precision/recovery data for the used method Priority to low financial cost

Change in design of surveillance in MAFF (2) 125 After establishing guidelines Statistic based sample number e.g. enough to obtain occurrence data on 95 percentile value with 95% confidence level Use of validated method Requirement of internal/external quality control method validation/verification data participation to proficiency testing program ISO/IEC 17025 accreditation Priority to data quality

126 Challenges for enhancing quality assurance in Japan Continued capacity building of risk assessors/ managers Enhancement of quality assurance of official and private laboratories and research institutes Collaboration for analytical method validation Supply of standard reagents and certified reference materials

127 Thank you for your attention!