Distinguishing between analytical precision and assessment accuracy in relation to materials characterisation

Transcription

1 Distinguishing between analytical precision and assessment accuracy in relation to materials characterisation Steven Pearce Principal environmental scientist Perth

2 Presentation overview Heterogeneity, glossing over the elephant in the data room Paradigm of the lab as the provider of accurate data Statistics and data smoothing (professional lies) Materials characterisation, adopting a rational approach Materials characterisation

3 Materials characterisation Geochemical and geophysical properties Defining specific characteristics that are taken to be representative of the bulk properties of the material Classifying materials into groupings based on characterisation process Examples: Landfill waste classification Contaminated sites assessment Mine waste classification Materials characterisation

4 General problems of carrying out material characterisation studies Remote location of sites Time pressure (everyone has a schedule), and ultimately time is money Need an answer quickly, not recommendations for more testing Size of site, volume of material to sample Heterogeneity of materials that require sampling End result: Defining bulk properties using snapshots Materials characterisation

5 Geological controls Weathering Point source effects Geochemical controls Particle size distribution Bulk properties Fractionation Materials Mine characterisation Closure

6 Distribution Concentration Volume Characterisation Materials Mine characterisation Closure

7 Defining bulk properties, a scalar problem Heterogeneity is the measure of the degree of compositional variability of a material. This can be divided into inter sample (macro scale): Concentration may occur within a particular material (for example in a vein of primary mineralisation), at a particular location, or at a particular depth (point source contamination) intra sample (micro scale): e.g. various mineral phases may be present and unequally distributed (for example isolated macro pyrite crystals) Materials characterisation

8 Intra sample variability shown from multiple XRF results from a single sample (approx 50g dry weight) Composite lab result within 10% of mean of XRF results *50g sample split into 3 parts: bulk, <2mm (fine), >2mm (coarse)

9 Intra sample variability (analysis bias) Typically, for analysis the lab will extract a small 1-10g sub sample of the 1000g parent sample on which to complete analysis selection of the sub sample may take place after sieving or crushing of the parent sample. Therefore, if intra sample heterogeneity is significant then clearly it will be unlikely that a single 1-10g sub sample will be representative (chemically or mineralogicaly) of the sample as a whole. Bias introduced at early stage Materials Mine characterisation Closure

10 Inter sample variability Contamination may be concentrated within a particular material, at a particular location, or at a particular depth If random sampling is being employed then it is clear that inter sample heterogeneity will have a potentially significant impact on the ability of the sampling programme to characterise the distribution of contamination on site Materials Mine characterisation Closure

11 Inter sample variability (90 sample data base) Copper results: Note wide inter sample variability probably not captured the full sample population distribution range Very large distribution tail Materials characterisation

12 Accuracy and precision Very different terms although commonly mixed up. Precision is the repeatability of a testing method Accuracy is a reflection of how well the testing characterises the sample composition Testing a 10g sub sample of material in a laboratory may therefore yield high precision result if a duplicate test is completed on the same 10g sub sample however if a separate 10g sample of the material is tested very likely that intra sample variability will be introduced that will result in increased error (i.e. reduce accuracy). Materials Mine characterisation Closure

13 Accuracy and precision No analytical technique is 100% precise and so random and systematic errors will affect the final result Generally, the error introduced by modern analytical instrumentation as analytical bias is likely to be relatively low. A study carried out in Europe (CLAIRE technical bulletin 7) indicates that for a particular case study sampling was by far the greatest cause of uncertainty rather than analysis. Precision was estimated at 83% of the concentration value for the sampling method, but was much lower at 7.5% for analytical method. The overall random component of uncertainty was estimated as being 83.6%, that is to say, the value of any concentration for an individual location was reproduced to within ± 83.6% of the quoted value (at 95% confidence). Given that analytical precision was only 7.5%, then clearly the majority of the overall variability was related to sampling rather than analytical factors. Materials Mine characterisation Closure

14 NAPP (kg H 2 SO 4 /tonne Lithological characterisation, vertical sampling intervals Est NAPP Depth (m) Materials Mine characterisation Closure

15 NAPP (kg H 2 SO 4 /tonne) Elevated heavy metals not related to sulfur Mercury Metal x Est NAPP Depth (m) Materials Mine characterisation Closure

16 Lithological characterisation, vertical sampling intervals Mercury Metal x NAG ph 7 Est NAPP Depth (m) Materials Mine characterisation Closure

17 Traditional collect and analyse Environmental Sampling: Aiming for high precision on limited sample numbers Intra/inter sample variability Intra sample variability 1000 m 3 500g 5g Typical volume of material represented by one site sample Typical weight of sample collected Typical weight of laboratory subsample that is analysed Materials characterisation

18 Heterogeneity testing (used in mining industry for grade control) Coarse fragment ores (stockpile, mill feed etc) individual fragments picked one by one Each assayed to extinction Consecutive results graphed Often can see that removal of top few results will drop mean grade by orders of magnitude Caused by heterogenity, small pockets of high grade material in general low grade background Applies to contaminated sites, similar distributions Materials characterisation

19 mg/kg Heterogeneity testing profile Inclusion of top 5 results considerable increases mean grade HT group Materials characterisation

20 Current paradigm The lab as the provider of accurate information Worldwide sampling and testing standards (e.g. ASTM, USEPA, UK EA, DEC Australia) based on the premise that collecting limited samples from the field and using laboratory to analyse to high precision provides the most accurate data Based on assumption that Only laboratory derived data is acceptable That analytical precision is the cause of most sampling error That statistics can fill the data gaps left by low sampling density. This assumption is flawed however as multiple studies have shown that sample variability (heterogeneity) has the greatest impact on accuracy, and that statistics do a poor job of interpolation (i.e. Data gap filling). The solution is increase to sampling density Materials Mine characterisation Closure

21 Paradigm reflected in guidance docs NEPC 1999: 4.7 FIELD TESTING A variety of field testing devices may be used as a limited contribution to the screening of samples on contaminated sites. The role in providing real-time data needs to be augmented by chemical analysis of soil. Their use as the sole source of analytical data in the assessment of potentially contaminated sites is inappropriate as they may give falsely high or low results. Materials characterisation

22 Embedded assumptions Precision - measures the reproducibility of measurements under a given set of conditions. The precision of the data is assessed by calculating the Relative Per cent Difference (RPD) between duplicate sample pairs. Co Cd RPD (%) 200 C C o d Where Co = Analyte concentration of the original sample Cd = Analyte concentration of the duplicate sample The Environmental Consultant will adopt nominal acceptance criteria of 30% RPD for field duplicates and splits for inorganics, and nominal acceptance criteria of 50% RPD for field duplicates and splits for organics, however it is noted that this will not always be achieved, particularly in heterogeneous soil or fill materials, or at low analyte concentrations. Question: If analytical techniques are precise why such high acceptable RPDs? And why therefore is field testing unacceptable? There is inconsistent logic in this approach which is embedded in the industry Materials Mine characterisation Closure

23 Statistical considerations A given site (or location) will have a given sample population distribution that cant be known (without testing every gram of material) As we can never know the true sample population, taking a few soil samples from the site is therefore is akin to random sampling as nothing prior is known about the population distribution Generally the more samples that are analysed from a given site (or location) the greater the confidence in the overall assessment. The direct relationship between increasing levels of confidence with sample numbers comes down to simple statistics. Materials Mine characterisation Closure

24 Statistical considerations Guidance on the number of samples to be taken on a site for the purposes of contaminated sites assessment based on assumptions that do not generally apply to most sites. The key assumptions include that the occurrence of contamination is described by normal distribution, and that hotspots present are of uniform size, shape and vertical profile. Common for the 95th percentile value to be quoted by assessors and requested by regulators as a representative concentration for a contaminant upon which to base decisions (to portray an illusion of statistical certainty). However, in reality the probability of being able to define anything close to the true 95 th percentile representative concentration for a site is very unlikely when sampling at densities similar to that recommended by published guidance Materials Mine characterisation Closure

25 Statistics (the magic data gap filler) Statistics commonly cited as a method to account for variability and to allow for interpolation to fill data gaps (e.g. US95, outlier tests, non parametric analysis etc) However data created in this way is prone to large error and can introduce bias into interpretation of data sets. Many broad assumptions are made, most commonly analysis techniques assume a normal distributed data set. Problem is most data sets are not normally distributed, and in the majority of instances the data set is to small to define the true mean, median, and minimum/maximum values. Increasing sampling frequency is the only way to accurately fill data gaps and therefore to reduce the error in calculation of descriptive statistics (mean, minimum etc) Materials Mine characterisation Closure

26 Generated concentration profile, however apparent inter sample variability is in fact likely to be intra sample heterogeneity Materials characterisation

27

28

29

30 Increased sampling density using on site screening 1000m3 10 or more samples analysed Intra sample variability assessment completed on sub samples (as little as 1g material required) Materials characterisation

31 2D contour plots produced from XRF data to show distribution of metals

32

33 Inter sample variability (90 sample data base) Copper results: Note wide inter sample variability probably not captured the full sample population distribution range Very large distribution tail

34 Copper [mg/kg] Intra sample variability shown from XRF results from a single sample location Intra sample variability Composite lab result within 10% of mean of XRF results Bulk fraction Fine fraction Coarse fraction Lab Note: Up to 800 mg/kg variance *50g sample split into 3 parts: bulk, <2mm (fine), >2mm (coarse)

35 Characterising the distribution of contamination, the case for more samples rather than higher precision Sample population range: Likely to be larger than defined Intra sample variability >10% of inter sample variability, difficult to differentiate between the two in some samples and therefore determine what is the cause of spatial variation in concentration Laboratory results rely on compositing, unclear at what scale is this acceptable given the level of intra sample variability (>100%) If we had relied on limited lab samples alone the distribution would be even more poorly characterised as a result of smaller data set, and compositing, preventing understanding of intra sample variability Materials characterisation

36 Conclusions Limitations of standard methodology of taking less samples but aiming for higher precision (lab ICP) Logistical and cost implications of taking physical samples and sending to laboratory Poor understanding of intra sample variability Very easy to assume variability between samples is a function of lateral or vertical distribution, could be simply be a function of intra sample variability Low probability of defining the sample population range and true mean, but a high chance of thinking you have Therefore: Not an ideal method for defining areas or volumes of material as contaminated Materials characterisation

37