GOODSamples: Guidance for Obtaining Defensible Samples

Transcription

1 2015 Webinar Series GOODSamples: Guidance for Obtaining Defensible Samples 3/24/2016 Speaker Nancy Thiex, MS, Consultant, Thiex Laboratory Solutions LLC and AAFCO, Brookings, SD Nancy has a BS in Mathematics from Northern State Univ., MEd in Physical Science and a MS in Chemistry from South Dakota State Univ, Brookings, SD. She managed an Analytical Services Laboratory at SD State U for 35 years that served as the State Ag Lab. She now serves as Cooperative Agreement administrator for the Association of American Feed Control Officials (AAFCO) and chairs the Sampling and Sample Handling Working Group charged with drafting the Guidance on Obtaining Defensible Samples or GOODSamples. Charles Ramsey, MS, Trainer, EnviroStat, Inc., Windsor, CO Charles has a BA in Chemistry from the University of Denver and a MS in Environmental Science from the Colorado School of Mines. He has performed and taught sampling for a wide variety of media for 30 years. He is the subject matter expert for the Sampling and Sample Handling Working Group charged with drafting the Guidance on Obtaining Defensible Samples or GOODSamples. Objectives At the conclusion of this program, participants will be able to: Describe the GOODSamples systematic approach to sampling Define key sampling terms List sources and consequences of sampling error Discuss the role of quality control in sampling Continuing Education Credit CE credit is not available for this program, but you will receive a Certificate of Attendance. Evaluation/Certificate of Attendance Process Continuing education credit is not available for this program. Individuals who successfully complete the program and evaluation by 2/25/2016 will receive a Certificate of Attendance. 1. Go to to complete the evaluation. 2. After you complete the evaluation, you will automatically receive the Certificate of Attendance. 3. Type or print your name on the certificate. Archived Program The archived streaming video will be available within two days. Anyone from your site can register view the program. URL: Comments, opinions, and evaluations expressed in this program do not constitute endorsement by APHL. The APHL does not authorize any program faculty to express personal opinion or evaluation as the position of APHL. The use of trade names and commercial sources is for identification only and does not imply endorsement by the program sponsors. This program is copyright protected by the speaker(s) and APHL. The material is to be used for this APHL program only. It is strictly forbidden to record the program or use any part of the material without permission from the author or APHL. Any unauthorized use of the written material or broadcasting, public performance, copying or re-recording constitutes an infringement of copyright laws. The Association of Public Health Laboratories (APHL) sponsors educational programs on critical issues in laboratory science. For more information, visit

2 1 Guidance on Obtaining Defensible Samples: GOODSamples An overview for a systematic approach to sampling 2 Overview of GOODSamples: Webinar Content Introduction Terms, Definitions, and Acronyms Management Support SQC Overview Material Properties Theory of Sampling Sample Correctness and Tools Evidentiary and Analyte Integrity Laboratory Considerations Quality Control Inference Data Assessment Resources 3 1

3 Introduction How did GOODSamples come to be? Who are the presenters? What GOODSamples is and isn t 4 GOODSamples What GOODSamples is A FDA Cooperative Agreement deliverable An AAFCO led collaborative effort with FDA, APHL, AFDO, and industry participation Objective is to improve analytical data equivalency among agencies Provides a systematic approach to develop sampling protocols for defensible decisions What GOODSamples is NOT Is not a summary of current practice Does not replace current documents Does not provide specific sampling protocols A tool to evaluate current sampling protocols 5 Who are the presenters? Nancy Thiex, AAFCO Co Ag Project Manager, Brookings, SD. Working Group Chair Chuck Ramsey, EnviroStat, Inc., Fort Collins, CO. Working Group Sampling Subject Matter Expert 6 2

4 Terms, Definitions and Acronyms Why terminology is critical How terms were chosen Yes, new terms = new acronyms! Definition of a representative sample 7 Terminology is critical Decision Unit Increments Primary Sample Laboratory Sample Analytical Sample Test Portion Decision Unit: Material from which sample is collected and to which inference is made Increments: Individual portion of material collected by a single operation of a sampling tool and combined with other increments to form a primary sample. Inference: Estimating a concentration or characteristic about a larger amount of material from data derived from a smaller amount of material. Always use an adjective with sample! 8 What is a representative sample A representative sample must answer a question about a decision unit with an acceptable level of confidence. Imprecision controlled by collecting an appropriate mass and number of increments to address heterogeneity Bias is controlled when every element in the decision unit has the same probability of being selected (equiprobable). Correctness is maintained when additional biases are not introduced during sample preparation A representative sample must Be correct - bias has been controlled to a negligible level Have a sufficiently small imprecision 9 3

5 Management Support Success begins with good management! Success improbable without good management 10 Knowledgeable and supportive management essential Must support good sampling practices Must understand basics of theory of sampling Establish priorities related to sampling Direct relationship between sample collection and sample prep and defensible decisions Allocate resources Provide training for qualified staff Supply and maintain equipment and tools Facilitate communication among sampling, lab and management Provide system for feedback and assessment Ensure health and safety procedures are observed 11 Sample Quality Criteria Framework for representative and defensible sampling What is the question? What is the decision unit? What is the desired confidence? 12 4

6 Sample Quality Criteria (SQC) A series of statements that clarify program technical and quality needs to support defensible decisions Stating what you are going to do before you do it All (management, sampling staff, lab, QA, etc.) provide input and agree on what needs to be done Three inputs: Question, Decision unit, Confidence 13 What is the question to be answered? What information is required? What is the analyte? What is the level of concern? What type of data will be collected? Characteristic of the decision unit? Concentration of analyte(s) in the decision unit? How in the inference from the sample to the decision unit going to be made? Direct inference (single result)? Probabilistic inference (single result)? Statistical inference? 14 What is the decision unit? The decision unit is the material from which the primary sample(s) is collected and to which the inference(s) is made. Maybe one or multiple Establishes the scale of observation Example: 15 pallets of 100 containers each are present. The scale of information drives the decision unit. Is the average value of all 15 pallets found within the safe range? All 15 pallets comprise the decision unit (1) Must each pallet must be safe? Each of the 15 pallets is a decision unit (15) Must each container on every pallet be safe? Each container in all 15 pallets in a decision unit (1500) 15 5

7 What is the required confidence? If the risk related to an incorrect decision is high, more confidence in the inference is required. To achieve more confidence in the inference, error must be controlled to a greater extent. Direct inference: Decision unit selected and analyzed in its entirety. No sampling or sample processing error. Confidence a function of analytical/measurement error. Probabilistic inference: Only a portion of the decision unit is selected or analyzed. Requires correct (unbiased) samples and an estimate of global estimation error to determine confidence. Inference based on statistical calculations: Requires correct (unbiased) replicate samples, each meeting the requirements of probabilistic inference. The specific calculation and confidence must be defined. 16 Material Properties Types of materials Finite or Infinite? Types of heterogeneity Compositional and distributional 17 Types of material elements Finite Element Materials: consists of a finite number of discernible elements which can be individually identified and selected at random from the decision unit. Infinite Element Materials: consists of a practically infinite number of indiscernible elements that cannot be individually identified nor collected individually. 18 6

8 Types of heterogeneity Compositional heterogeneity: exists when individual elements exhibit differing concentrations of the analyte of interest. Distributional heterogeneity: results from non-random distribution of elements. 19 Theory of Sampling Fundamental Sampling Error Grouping and Segregation Errror 20 Theory of Sampling Necessary when have an infinite element material With finite element material you can use Theory of Sampling or classical statistics Necessary for infinite element material since individual elements that made up the decision unit cannot be individually selected at random elements are selected in groups, called increments Theory of sampling describes The final sample mass (combination of all increments) How many increments need to be collected The increment shape (correctness) 21 7

9 Compositional Heterogeneity Presence of Compositional Heterogeneity (CH) leads to the Fundamental Sampling Error (FSE) FSE is mitigated by increasing sample mass and reducing particle diameter FSE can be controlled to any level FSE 2 Cd 3 C sampling constant, unique for every material d - diameter of 95% percentile of largest particle m s mass of sample (all increments combined) m s 22 Distributional Heterogeneity Presence of Distributional Heterogeneity (DH) leads to the Grouping and Segregation Error (GSE) GSE is controlled by the number of increments GSE can be controlled to any level Increments must be at random Increments must be the correct shape GSE is typically not measured, but controlled to less than 1/3 of the FSE so it can essentially be ignored 23 Heterogeneity Considerations Compositional heterogeneity is a state of nature. CH cannot be changed by mixing and moving of the material Compositional heterogeneity can be changed by crushing, chopping, and grinding (comminution) the material Distributional heterogeneity is a transient state of nature. DH can be changed by mixing and moving the material Distributional heterogeneity is changed every time a material is stirred, blended, poured, piled, etc. In some cases it is increased (harder sampling problem) and in other cases it is decreased (easier sampling problem) 24 8

10 Sample Correctness and Tools Sample Correctness = Control of Bias 25 Sample Correctness Bias results when sample correctness is not strictly adhered to Correct sampling mandates that all increments at the increment location have an equal (equiprobable) chance of selection Once sample correctness is achieved, it needs to be maintained through the entire measurement process Unlike the laboratory, sampling bias cannot be measured, therefore it is critical to eliminate it to ensure sample representativeness 26 Tools In order to achieve sample correctness The sampling tool must be the correct shape The tool (and operator) must be able to extract correctly shaped increment from the material being sampled To determine the correct increment shape knowledge of the dimension of the material is critical. Dimensions are typically Slices (cross stream cuts) for a one dimensional flowing stream Cylinders (similar to a probe) for a two dimensional layer 27 9

11 Evidentiary and Analyte Integrity Identification and Authentication of Evidence 28 Evidentiary Integrity Communication, policies and procedures to: tie a test result to a specific decision unit demonstrate that the sample has not been adulterated or compromised during any step from primary sample collection through generation of test results assure that analyte integrity has been maintained Requires correct sampling throughout the process Consists of documentation processes chain of custody 29 Analyte Integrity Communication, policies and procedures to ensure that the physical, chemical, biological and/or radiological analyte integrity has been maintained. Considerations: Preservatives (chemical and temperature) Containers (ensure safe arrival without loss or contamination) Holding times (analyte specific and dependent on many factors ) Sampling techniques (aseptic collections, contamination introduced by sampling personnel, loss or contamination introduced by tools)

12 Laboratory Sampling & Preparation Respect the decision unit Ensure that analyte integrity is maintained Obtain representative test portions 31 Laboratory Sampling & Preparation Laboratory SQC The SQC remains unchanged from the field to the laboratory; however the techniques used to reduce the error to achieve the confidence are different. Material Properties Once processed for particle size reduction, both finite and infinite materials become infinite materials CH does not change unless particle size is reduced DH is altered every time the material is physically manipulated For liquids and semi-solids, mixing can be effective; however, stirring solids promote segregation and increase DH. 32 Laboratory Sampling & Preparation & Theory of Sampling Minimum Mass Controlling FSE critical, especially for very small test portions Often the greatest challenge in the laboratory Consequences of FSE often ignored Minimum Number of Increments Controlling DH important in laboratory Collection of random increments easy Taking of a single increment during mass reductions step inexcusable! Sample Correctness Dimension and shape of tools critical and should be considered A one-d or two-d configuration is preferred 33 11

13 Particle Size Reduction (Comminution) Comminution is a critical step in the preparation of analytical samples to control FSE If comminution produces a uniform shape and size, DH will be reduced Equipment considerations Sufficient capacity Reduce particle size sufficiently to control FSE Produce a uniform shape and size to control GSE Inert equipment and ability to clean between samples 34 Laboratory Sampling & Preparation & Analyte Integrity Proper handling and storage for maintain analyte integrity is specific to materials and analytes of concern Factors that may affect analyte integrity Heat generation during particle size reduction Adsorption of analyte onto equipment Volatilization of analyte Loss of fine particles Contamination of equipment Carryover contamination from previously processed samples Degradation of analyte due to environmental conditions 35 Laboratory Sampling & Preparation Validation of Laboratory Sample Preparation Protocols As important as validation analytical methods Demonstrates minimum level of competence and fitness for purpose; does not imply universal applicability Generally not material & analyte specific Does not negate the need for sample/batch specific QC Performance Tests Grind/Blend Quality and Recovery Carryover Mixing Reserved Materials Require collection of additional primary sample mass of sufficient mass to control FSE 36 12

14 Quality Control (QC) Used to assess data quality QC widely implemented in laboratories QC seldom implemented in sampling or sample preparation 37 Quality Control (QC) QC used To estimate global estimation error To determine if a process is in control To validate a method or protocol Estimation of error Error typically categorized as: bias, imprecision, or gross errors Errors relatively small compared with others have little effect on total error. Sampling errors and sample preparation errors typically greater than testing related errors; therefore, critical that QC be incorporated into sampling and sample preparation 38 Bias Sources of bias Factors related to the collection of the sample (use of incorrect tools) Factors external to the sample collection (unsuitable containers, degradation of analyte) Elimination or minimization of bias critical, because Quantification of bias difficult if not impossible No QC events to determine bias from incorrect tool design or use Inconsistent errors, therefore manifest as imprecision Contamination QC = Blanks QC checks for contamination from containers QC checks for environmental contamination QC checks for carryover contamination from tools and equipment 39 13

15 Imprecision Sources of imprecision Fundamental sampling error (FSE) due to compositional heterogeneity (inversely proportional to mass collected) Grouping and segregation error (GSE) due to distributional heterogeneity (inversely proportional to increments collected) Estimated through replication and typically expressed as RSD or %CV Global estimation error (GEE) can (only) be estimated with properly collected replicate primary samples Replicates can be implemented at multiple levels to sort out error contributions from various mass reduction stages 40 Levels of replication using triplicates GEE can be estimated from data resulting from replicated primary samples. Replicating test portions from the same analytical sample provides an estimate of the imprecision associated with selection of the test portion and the test (no information about the preceding stages). 41 Inference The process of estimating (or inferring) a concentration or characteristic about a decision unit based on a sample(s) collected from that decision unit

16 Inference Forms of inference Estimation of an average analyte concentration Function of sampling error and analytical error The more representative the sample, the lower the error, the greater the confidence in the inference Estimation of a percentage of decision units that have some characteristic or concentration Common where there are more decision units than can be sampled Starts with an inference to sampled decision units The greater number of decision units sampled, the greater the confidence 43 Inference to a Decision Unit Inference to a decision unit occurs at every stage of mass reduction in the sampling to test portion pathway and the number of steps is unique to each project Direct inference: The entire decision unit is collected as the primary sample and analyzed in its entirety. Probabilistic or statistical inference: The entire decision unit is collected, but not analyzed in its entirety OR the entire decision unit cannot be collected. In either case, portions selected must be representative. Amount of error affects quality of the inference. Tolerable GEE specified in SQC and sample correctness must be ensured and maintained. 44 Inference from Sampled to Unsampled Decision Units Used to estimate the proportion of decision units that have a specific concentration or characteristic. Sometimes called acceptance sampling 45 15

17 Data Quality Assessment Do data meet the requirements of the SQC? Does the global estimation error support a defensible decision? 46 Data Quality Assessment Review of documentation & Evaluation of Quality Control Is the documentation complete? Does the documentation support the premise that the correct procedure was followed? Were sufficient increments collected at random for each mass reduction stage? Was there any reason to suspect contamination from outside sources? Were correct tools and equipment used? Was there any evidence of breakage or spillage? Evaluation of quality control Do blanks (at all critical stages) support the absence of contamination below a critical level? Are replicates (from all critical stages) within an acceptable range? 47 Data Quality Assessment Evaluation of global estimation error Evaluation of the global estimation error (GEE) Is the GEE below 35% What is the proximity of the actual concentration and GEE to the specification limit? Was data inappropriately rejected? Ultimately, does the GEE support a defensible decision? 48 16

18 Resources For details and a greater understanding, keep studying! 49 Resources GOODSamples: Guidance on Obtaining Defensible Samples Association of American Feed Control Officials, Champaign, IL. Free download GOODSamples Primary Resources Gy, P. (1998). Sampling for Analytical Purposes. New York: John Wiley & Sons. Pitard, F. F. (1993). Pierre Gy s Sampling Theory and Sampling Practice. 2nd ed. Boca Raton, FL: CRC Press. 50 JAOAC Special Guest Editor Section Open access publications Esbensen, K.H. and Paoletti, C. and Thiex, N, Representative Sampling for Food and Feed Materials: A Critical Need for Food/Feed Safety (2015) J. AOAC Int. 98, Kuiper, H. and Paoletti, C., Food and Feed Safety Assessment: The Importance of Proper Sampling (2015) J. AOAC Int. 98, Thiex, N, Esbensen, K.H. and Paoletti, C., Towards a Unified Sampling Terminology: Clarifying Misperceptions (2015) J. AOAC Int. 98, Ramsey, C., & Wagner, C., A Systematic Approach to Representative Sampling (2015) J. AOAC Int. 98, Ramsey, C., & Wagner, C., Sample Quality Criteria (2015) J. AOAC Int. 98, Esbensen, K.H., Materials Properties: Heterogeneity and Appropriate Sampling Modes (2015) J. AOAC Int. 98, Wagner, C., & Esbensen, K.H., Theory of Sampling Four Critical Success Factors Before Analysis (2015) J. AOAC Int. 98, Esbensen, K.H., & Ramsey, C., Quality Control of Sampling Processes A First Foray; From Field to Test Portion (2015) J. AOAC Int. 98, Ramsey, C., Considerations for Inference to Decision Units (2015) J. AOAC Int. 98, Paoletti, C., and Esbensen, K.H., Distributional Assumptions in Agricultural Commodities Development of Fit-for-Decision Sampling Protocols, (2015) J. AOAC Int. 98, Wagner, C., Critical Practicalities in Sampling For Mycotoxins in Feed (2015) J. AOAC Int. 98, Ramsey, C., Considerations for Sampling Contaminants in Agricultural Soils (2015) J. AOAC Int. 98, Ramsey, C., Considerations for Sampling of Water (2015) J. AOAC Int. 98,

19 Additional Resources Esbensen, K.H. (2013) DS 3077, Representative Sampling Horizontal Standard, Danish Standards. Lehotay, S. J., & Cook, J. M. (2015). Sampling and Sample Processing in Pesticide Residue Analysis. J. Agric. Food Chem., 2015, 63 (18), pp DOI: /JF Ramsey, Charles A. The Decision Unit A Lot with Objectives. TOS Forum, 5, Doi: /tosf.75 Ramsey, Charles A. The Role of Inference in Food Safety. TOS Forum, 5, 2015.Doi: /tosf.77 Ramsey, Charles and Thiex, Nancy. Improved Food and Feed Safety through Systematic Planning and the Theory of Sampling (TOS): An Introduction to GOODSamples. TOS Forum, 2, ISO 6498:2012 Animal Feeding Stuffs Guidelines for Sample Preparation, International Organization for Standardization, Geneva, Switzerland Thiex, N., Novotny, L., Ramsey, C., Latimer, G., Torma, L., & Beine, R. (2000) AAFCO Guidelines for Preparing Laboratory Samples, Association of American Feed Control Officials, Oxford, IN Feed Inspector s Manual, 5 th Ed., AAFCO Inspection and Sampling Committee, Association of American Feed Control Officials, Samplers Manual, Petersen, L, C. Dahl, K.H. Esbensen (2004). Representative mass reduction in sampling a critical survey of techniques and hardware. in Special Issue: 50 years of Pierre Gy s Theory of Sampling. Proceedings: First World Conference on Sampling and Blending (WCSB1). Esbensen and Minkkinen (Eds). Chemometrics and Intelligent Laboratory Systems, vol. 74, Issue 1, p Esbensen, K.H., & Wagner, C. Theory of Sampling (TOS) vs. Measurement Uncertainty (MU) a call for integration. (2014) Trends Anal. Chem. 57, Comments/concerns about this program? webinar@aphl.org 18