Review Inter-laboratory studies in analytical chemistry. Edelgard Hund, D.Luc Massart, Johanna Smeyers-Verbeke

Transcription

1 Analytica Chimica Acta 423 (2000) Review Inter-laboratory studies in analytical chemistry Edelgard Hund, D.Luc Massart, Johanna Smeyers-Verbeke Chemo AC, Farmaceutisch Instituut, Vrije Universiteit Brussel, Laarbeeklaan 103, B-1090 Brussels, Belgium Received 18 February 2000; received in revised form 23 June 2000; accepted 17 July 2000 Abstract Inter-laboratory studies are performed with different aims and consequently require different evaluation methods and statistical treatment. The review considers method-performance studies (collaborative trials), laboratory-performance studies (proficiency tests), collaborative bias evaluation, inter-laboratory evaluation of to-be standard methods as well as certification studies for reference materials. Besides the classical evaluation methods using outlier tests, robust statistics and graphical methods are taken into account Elsevier Science B.V. All rights reserved. Keywords: Collaborative trial; Proficiency test; Bias evaluation; Certification of reference materials 1. Introduction In an inter-laboratory experiment, different laboratories determine some characteristic, e.g. the concentration of an analyte in one or various homogeneous samples under documented conditions. Several subtypes of inter-laboratory studies can be distinguished. Collaborative trials or method performance studies are used to test the performance (generally the precision) of a single analytical method [1]. A standard method, which is routinely used in several laboratories, can also be examined collaboratively to test for a possible bias of either the method (method bias) [2] or the laboratories (laboratory bias) that routinely use it [3]. The comparison of different laboratories that perform comparable analyses with their own individual method is often called proficiency testing or laboratory performance studies [4]; sometimes, the Corresponding author. Tel.: ; fax: address: asmeyers@fabi.vub.ac.be (J. Smeyers-Verbeke). term round robin study is used [5]. It is an essential part of the requirements for laboratory accreditation. The participating laboratories have to prove their technical competence regularly, so that proficiency tests are usually performed several times a year for a certain analysis [6,7]. Proficiency tests that are performed only once are sometimes called cooperative trials. Nowadays, different protocols are provided for the different types of experiments, but earlier publications on inter-laboratory studies [8,9] use the same set-up for the different applications mentioned. In addition to the more classical inter-comparison studies mentioned above, study schemes were developed, which allow laboratories to validate new or improved methods by a comparison with a fully validated (international standard) method in an inter-laboratory approach [10]. Moreover, the choice of an international standard method is usually also based on an inter-laboratory study. Further information on these tests, which are sometimes referred to as improvement schemes, can be found in an International Organization for Standardization (ISO) norm [11]. Besides /00/$ see front matter 2000 Elsevier Science B.V. All rights reserved. PII: S (00)

2 146 E. Hund et al. / Analytica Chimica Acta 423 (2000) these types of inter-laboratory tests, the certification of a reference material is generally also performed by collaboration of different laboratories [12]. Testing schemes are designed for a specific analysis in a specific matrix rather than for a general type of analysis. Not only quantitative methods but also the identification of analytes can be subject to interlaboratory testing, especially in proficiency testing [4]. Standard procedures for collaborative trials are specified in the norm :1994 by the ISO [1]. Based on an earlier version of this standard [13], the German standardization authorities prescribe the set-up and evaluation of collaborative trials in their Deutsche Industrienorm (DIN) , parts 41 and 42 [14]. Although meant for water analysis, it applies in a broader field of applications. For the field of pesticides, the Collaborative International Pesticides Analytical Council limited (CIPAC) provided guidelines for collaborative study procedures [15], which are to a large extent based on [13] too. Moreover, the Association of Official Analytical Chemists (AOAC) published a guideline for collaborative studies to validate characteristics of an analysis method [16]. This guideline is more detailed than the one by the International Union of Pure and Applied Chemistry (IUPAC) [17]. Even though there is a large agreement in the requirements for collaborative trials and laboratory-bias tests, the procedures for the latter are specified in a separate ISO standard [3]. The ISO guide 43-1 is intended only as a framework for proficiency tests [4]. Detailed instructions concerning the set-up and evaluation of proficiency testing are given in the International harmonized protocol for proficiency testing of (chemical) analytical laboratories [6]. It is prepared jointly by the ISO, the IUPAC and the AOAC. In most of the cases, proficiency tests allow to use different methods; sometimes, however, standard methods are prescribed. While collaborative trials are usually organized by one of the participating laboratories, proficiency tests operated a few rounds each year are managed by a central body [7], often a governmental organization. To enable easily accessible information on inter-laboratory testing, German, Austrian and Swiss governmental organizations installed an online information system for inter-laboratory testing, which will be extended to the whole European Union [18]. It informs about proficiency tests and collaborative trials that are open for further participants. Moreover, this system also allows announcing inter-laboratory studies with the aim of inviting collaborators. The online system comprises not only the field of chemical analysis but also, for instance, acoustical and mechanical examinations. Online information on proficiency tests is provided by the American Society of Testing and Materials (ASTM) as well [19]. The German and Austrian online information system [18] contains information about the studies for the certification of reference materials, too. Often small deviations from the protocol cause a high between-laboratory deviation. This leads to worse reproducibility estimates in the collaborative studies. In order to find the critical parameters, Youden and Steiner [8] introduced ruggedness tests to analytical chemistry. These tests examine, within one single laboratory, the susceptibility of analytical methods against small changes in the method parameters using factorial experimental designs. Such preliminary tests are less expensive than collaborative studies and allow adjusting the method (and the protocol) before it is studied collaboratively. This approach is also followed by CIPAC [15]. Some ruggedness definitions, e.g. by the United States Pharmacopoeia [20] allow, however, an inter-laboratory evaluation also of the reliability of the method. An example is for instance given in [21]. As can be seen in Fig. 1, improvement schemes are related to all other types of inter-laboratory studies considered. They cover both laboratory and method performance studies and their positive outcome leads to a validated method that can be used in the certification of reference material. In this paper, the different aims and approaches for collaborative studies as well as for proficiency tests are reviewed. Besides the classical approach using outlier tests [1], robust statistics [22] is also considered. The procedures for the certification of reference materials are given as well. Moreover, a short overview on inter-laboratory bias tests and improvement schemes is also included. 2. General considerations Even though the approaches for inter-laboratory studies are rather different, some problems are common to all types of studies.

3 E. Hund et al. / Analytica Chimica Acta 423 (2000) Fig. 1. Overview on current types of inter-laboratory studies and their relation, corresponding ISO standards and guides (adapted from [10]) Preliminary requirements All participants of inter-laboratory studies should have set up beforehand all within-laboratory quality assurance and quality systems possible [10] Definitions The definitions of repeatability, reproducibility [23] and different types of intermediate precision [24] are given by ISO Repeatability conditions These are conditions where independent test results are obtained with the same method on identical test items in the same laboratory by the same operator using the same equipment within short intervals of time Reproducibility conditions These are conditions where test results are obtained with the same method on identical test items in different laboratories with different operators using different equipment Selection of material true value and assigned value of the content The materials for an inter-laboratory test have to be selected in such a manner that the samples are representative for the type of material that is usually analyzed with respect to the matrix and the concentration range. While it is possible to provide such samples for manufactured materials, natural samples are often not available in the concentration range required. Fortification by spiking is an alternative if the concentration in the natural samples is too low [7]. It also allows determining the recovery rate. In the marine monitoring scheme Quality Assurance of Information for Marine Environmental Monitoring in Europe (QUASIMEME) [25], it is stressed that spiking is particularly useful for small numbers of observations and for analysis methods known to be problematic. Often, this alternative technique cannot be applied, since problems with sample homogeneity can occur. A fundamental requirement is the stability of the samples during the inter-laboratory study. If this provides problems, storage instructions are required. Moreover, the time interval for the analyses might be quite limited [4].

4 148 E. Hund et al. / Analytica Chimica Acta 423 (2000) The true value of the analyte content is only known at a high confidence level if certified reference material is used in the inter-laboratory study. Often, however, certified reference material is not available for the matrix under consideration [7]. Moreover, these certified materials are rather expensive, so that some organization committees for inter-laboratory studies generally do not consider them [26]. CIPAC [15] prescribes the use of validated reference material for collaborative trials. For some substances, e.g. blood, so-called quality control materials are available [27]. These materials have no certified reference values but provide documentation for homogeneity and long-term stability. The most important advantage is the considerably lower cost compared to certified reference materials. These materials are very suitable for spiking. As an alternative to certified reference materials and spiking, the true value can be replaced by an assigned value [6]. A determination of the content beforehand using an accepted reference method is suggested by DIN [14]. In contrast, the Analytical Methods Committee of the British Chemical Society (AMC) suggests that a competent laboratory should determine the sample beforehand [7]. ISO relies on one reference laboratory, usually the national calibration laboratory [4]; AOAC recommends a determination by a group of reference laboratories [6]. In the QUASIMEME laboratory performance studies, the number of reference laboratories should be larger than three and not exceed 15% of the number of participants in the proficiency test. This protocol explicitly states that the reference laboratories should be independent of the laboratory performance study [25]. If these approaches are not possible, the average of the results obtained in the participating laboratories can be accepted as the assigned value, even though it could be biased. To derive this accepted value, some protocols prefer to remove outliers (e.g. [14]); others prefer a robust estimator of location, such as a trimmed mean [25,28] or the median [26]. If there is a large uncertainty about the assigned values, the QUASIMEME protocol considers them only as indicative [25]. For more than five observations, a coefficient of variation of more than 50% indicates a large uncertainty, whereas for less observations, the limit is set to 70%. As to the calibration substances, there are two general possibilities. Either one unique calibration substance is distributed together with the samples or the laboratories use their own calibration standards. For the certification of reference materials, some authors advise to use one single standard determined in three laboratories [29]. Other protocols point to the risk of a possible bias if one single standard is used in all laboratories [30]. Horwitz and Albert [31] showed that the reproducibility in collaborative studies is usually smaller than expected if a common standard is used. Therefore, they recommend performing collaborative studies and proficiency tests with individual standard solutions in each laboratory Homogeneity of materials The homogeneity of the test materials is a fundamental requirement for all inter-laboratory studies. Feinberg et al. showed that the results of an inter-laboratory study could be tremendously affected if non-homogeneous material is used [32]. The better the precision of the applied measurement methods, the higher the requirements on the homogeneity of the material [12]. The homogeneity of the material has to be proved. The harmonized protocol prescribes a test for sufficient homogeneity [6]. A number of test samples (n 10) have to be selected randomly and analyzed in duplicate under randomized repeatability conditions by a suitably precise method. Some proficiency test schemes, however, omit homogeneity tests. It is stated in [33] that this should be done only for materials that are technically capable of being truly homogeneous, e.g. stable solutions. Nevertheless, in the first round of a proficiency-testing scheme, the homogeneity of such materials should also be verified. If homogeneity is shown in this way, occasional formal tests can be sufficient. In a case study for the preparation of a reference material, the set-up of a test for homogeneity is reported. This test is performed within one single lab; the material is divided in sub-samples [29]. About eight independent determinations are performed for one single sub-sample to prove the within sub-sample homogeneity. To evaluate the between-sub-sample homogeneity, about 3 4% of the sub-samples are analyzed. In the case of a non-destructive measurement method, they suggest to measure each unit instead of measuring on the basis of a statistical

5 E. Hund et al. / Analytica Chimica Acta 423 (2000) sampling plan [12]. Caroli [34] recommends that a small group of specialists should test the materials for homogeneity using the most up to date analytical technique. Some matrices generally show restricted homogeneity (e.g. metals, textile fabrics); thus, this factor is often considered as an inherent part of the method variance. To overcome the problem, ISO provided a protocol for collaborative trials for heterogeneous materials [35]. The basic idea is to distribute at least two samples to each laboratory, which allows to isolate the between-sample variability and to obtain a corrected reproducibility estimate. Therefore, the design comprises the factors laboratory, level, sample and replicate. The evaluation also deviates from the one given in [1] Replication of experiments Since replicates within one laboratory are subject to possible manipulation, the German division of EURACHEM (EURACHEM/D) recommends not to repeat the experiments [36]. This attitude is shared by AMC, which suggests performing replication only in particular circumstances and prefers to spend the effort on a wider variety of analyses [7]. AOAC [16] suggests to perform either split-level replication, which means that the two samples show a small difference in composition and concentration, or to distribute the duplicated samples in a blindly coded way. In the case of the analysis of more than one material, if possible, the duplicated samples are distributed in such a way that it is not obvious which samples come from the same source. ISO also recommends a blindly coded sample distribution to avoid manipulation of the repeatability estimate. Therefore, a blindly coded distribution of the samples is recommended [1]. The IUPAC protocol [17] gives further specifications concerning single split-level (see above) and double split-level (use of two nearly identical materials, each of them distributed in duplicate, either blindly or not) experiments. In a report on the experience with the IUPAC protocol, Horwitz and Albert [37] recommend, however, to eliminate the double split-level experiments from the protocol, since they lead to a physically non-interpretable term in the analysis of variance Outlier treatment and robust statistics Outlying results can occur in all types of inter laboratory studies. In some of the studies, no replication or only a duplication is performed, which means that, with such a protocol, it is not possible to check if an individual value obtained in a laboratory is an outlier. Therefore, here, we will mainly consider the properties of outlier tests for discordant laboratory standard deviations and laboratory means. When replicate analyses are performed, these tests are always combined. The outlier treatment of course depends on the aim of a specific inter-laboratory study and will therefore be discussed in detail in the respective chapters. Usually, the outliers are treated in the following way, which probably goes back to Gottschalk [38]. Values identified as outliers at a significance level between 1 and 5% are considered as stragglers, and only those values that are significant at a significance level of 1% are considered as outliers and therefore eliminated. Stragglers are only eliminated if a reason for their particular behavior can be found. Nevertheless, they should be marked, since this can facilitate the further data evaluation. With the exception of DIN [14], all protocols that consider replication of the analyses start outlier testing by examining the laboratory variances. Three different tests could be applied for this kind of outliers, namely the tests by Bartlett, Hartley and Cochran. However, for inter-laboratory studies, the two former tests should not be used, as they are extremely sensitive to very small variances. In the case of inter-laboratory studies, however, one or several of the within-laboratory variances might even be zero, leading to problems with these tests [13]. Cochran s test should strictly only be used for data from uniformlevel designs. It can, however, be adopted in the case of small deviations from this design (due to some outliers or missing data) and is therefore quite widespread in inter-laboratory studies [1,13]. A different approach to test for outlying laboratory variances is followed in the German norm [14], which uses an F-test to compare the individual within-laboratory variances with the reproducibility variance. Since the between-laboratory component can be extremely predominant in the reproducibility variance, it is necessary to perform this test only after

6 150 E. Hund et al. / Analytica Chimica Acta 423 (2000) the elimination of outlying laboratory means, as is recommended in the protocol. These tests are always performed as one-sided tests [1,15 17], so that only too large variances will be considered as outlying. Nevertheless, one should be aware that the variance could also be underestimated due to some extremely small laboratory variances. A larger variety of tests is used for outlying laboratory means; all of them consider possible outliers at both ends of the ordered data. Older texts on inter-laboratory studies, e.g. Youden and Steiner [8] as well as the obsolete version of the ISO norm [13], apply the Dixon test for outlying laboratory means. More recent protocols [1] mostly use the Grubbs test. Moreover, in the evaluation of proficiency tests [39], the Graf and Henning test [40] was used, which Barnett and Lewis [41] ascribe to McMillan and David. Furthermore, an outlier test originating from a homogeneity test by Nalimov [42] was applied in a certification study [43]. The latter one was quite widespread in Germany especially in the 1970s. In the meantime, it almost disappeared. The most important criticism on this test is that it based on a table that became obsolete [44]. The Dixon test is easy to calculate and was mainly applied to small data sets. It is based on the comparison of the difference between the suspect value and its direct or a close neighbor with the overall range or a modified range. It suffers, however, from the masking effect if multiple outliers are present [45]. As a consequence of this, the technical committee of ISO replaced the Dixon test by Grubbs test [37]. Grubbs provides different outlier tests, for single outliers (highest or lowest value), and for double outliers (two highest or two lowest values) [46]. As a modification of the latter, the so-called paired test for two outliers, which can either be situated at the same or different ends of the ordered data, has been proposed [47]. Often, the single outlier test is combined with one of the tests to detect two outliers. The main difference between Grubbs single outlier test and the Graf Henning test is that the former considers all data in the preliminary estimate of mean and standard deviation, while the latter omits the suspicious value. Linsinger et al. [39] showed for the evaluation of a proficiency test with a large number of laboratories that the repeated application of Grubbs single outlier test detects more outlying values than the Graf Henning test. Thus, the latter one is more conservative. Simultaneous tests for more than two outliers can be found in [41]. They are, however, not applied in official protocols for inter-laboratory studies. Two different approaches to use robust methods instead of classical outlier tests can be found. On the one hand, they are used to obtain robust estimates of the parameters of the distribution, which later on serve in a test that can lead to the elimination of discordant data. Davies [48] as well as Linsinger et al. [39] show that methods based on the median of absolute deviations from the median of the data (MAD) lead to a less conservative rejection of outliers than the methods of Dixon or Grubbs discussed above. On the other hand, one can also find robust methods that do not reject, but attribute less weight to suspicious data, such as Tukey s biweight function [48] or the robust estimation of the mean and standard deviation proposed by AMC [22], which is based on winsorizing. Such an approach takes into consideration that it is not necessarily true that the majority of laboratories achieve the right result. Moreover, according to the AMC, the down-weighting procedures usually also lead to more real, namely larger estimates for the standard deviation [22]. For a robust variance estimation, AMC first applies a winzorization procedure to accommodate the data. To prevent an underestimation of the variance, correction factors are applied. This approach has been proposed by ISO [35] as an alternative method for the evaluation of the precision of a standard measurement method. Based on a bootstrap method, Thompson et al. [49] showed in a reanalysis of published collaborative trials that the robust methods proposed by AMC to estimate the mean and the standard deviations provide acceptable results. The application of robust statistics does not make sense for the evaluation of paired replicates Graphical interpretation Graphical methods usually facilitate the interpretation of the results. Bar plots with or without acceptance limits are frequently used to show the results. The saw-tooth plot mentioned in the Dutch protocol for proficiency tests [50] can be compared with a control chart. Moreover, the Youden plot, also referred

7 E. Hund et al. / Analytica Chimica Acta 423 (2000) to as XY plot [51], is often applied. It is designed for the interpretation of split-level experiments (see Section 2.5). The results of each laboratory for level A are plotted versus the corresponding result for level B. Addition of the medians for both levels in the plot allows to divide the chart into four parts. It is expected that the points be equally distributed over the quadrants. Deviation from this scheme indicates inconsistency of the results. Moreover, single points that are located far from the intersection point of the median lines are related to outlying laboratories. A modification published later uses the mean instead of the median [52], and can therefore be less robust. A box plot [53] displays the scattering within the individual laboratories as well as the distribution of the laboratory means. It is sometimes even referred to as a visual one-way ANOVA [54]. Minkkinen [55] suggests a principal component score plot to highlight outliers. In the score plot, the laboratories are treated as objects and the concentration levels are considered the variables. Homogeneity of the test materials can also be verified by a modified Youden plot [33]. This modified plot requires in addition a different distribution of the samples. Each sub-sample has to be divided into two halves. A pair of laboratories has to share two pairs of such sub-sub-samples. If the results of these paired laboratories are closer on the Youden plot than those of the unpaired laboratories, this indicates heterogeneity of the samples. This test for homogeneity in general requires more experimental work than the tests mentioned in Section 2.4. Mandel s h and k consistency statistics are a graphical way of describing the variability in a set of data and to look for inconsistencies [56]. The h-statistic expresses for each level the deviation of each laboratory mean from the general mean at that level in standard deviation units. The k-statistic on the other hand compares repeatabilities of the laboratories. 3. Collaborative experiments 3.1. Aim Collaborative trials allow to estimate the precision (expressed as repeatability and reproducibility) and to evaluate the possible bias of an analytical method. They are used as a part of method validation. Some authors accept a method as fully validated only if the method has been examined in an inter-laboratory method performance study [57]. Feinberg [58] suggests the use of an inter-laboratory study to establish the precision of an alternative method, which allows to compare it with a standard method. Both methods are examined in each laboratory Preparation Choice of methods For the selection of the method for a collaborative test, different suggestions can be found. The German standard [14] recommends that only operators familiar with the method should perform the experiments in the collaborative study. This requires that the method be regularly used in all the participating laboratories. CIPAC [15], however, uses the collaborative tests to study whether a certain method either a method from the literature or a method developed in-house by one of the participants performs acceptably so that it can be used as a standard method. This approach was for instance used in a case study for the LC-analysis of taurine in milk and infant formulae [59] Panel/staff Similar requirements on the panel and on the staff are specified by ISO [1] and the German norm [14]. The members have to be familiar with the method. They plan and coordinate the collaborative study, which mainly means that they fix the protocol and decide on the number of laboratories, levels and measurements. Moreover, they appoint a statistical expert as well as an executive officer. After the experiments are performed, they have to discuss the report and decide on the final values. Furthermore, the panel decides whether measures have to be taken to improve the method and with respect to outlying laboratories. An executive officer from one laboratory has to supervise the preparation of the samples and materials and take care of their distribution. It is his/her responsibility that the number of participants is sufficient. Moreover, he/she has to provide uniform spreadsheets for the results for each laboratory and collect the results of the participants. He/she is the contact person for discussion and has to give instructions and advice.

8 152 E. Hund et al. / Analytica Chimica Acta 423 (2000) The statistical expert is either a member of the panel or an external. His/her responsibilities comprise the evaluation of the results as well as a contribution on the final decisions. Within each laboratory, a supervisor is responsible for the organization and the report. He/she has to select a proficient operator who will have to execute all experiments and take very detailed notes. The supervisor has to write a final report to the executive officer Pilot trial After the method has proved to be rugged in a test according to Youden and Steiner [8], a pilot trial including three or four laboratories is suggested by CIPAC to obtain rough estimates of the repeatability and reproducibility of the method [15]. From the results of the pilot trial, the organization committee has to decide whether the method is ready for a collaborative study. This trial also allows finding weak points in the protocol of the method, which then can still be adapted for the extended collaborative study. AOAC [16] also recommends performing a pilot trial with three laboratories, but only if time and resources are available. The AOAC guideline stresses the importance of a thoroughly prepared written version of the method Recruitment of collaborators From the statistical point of view, it would be best to select the participating laboratories randomly [14]. If the method chosen is intended for international use, especially for different climate zones, laboratories representing these countries should participate [15]. In practice, however, only a limited number of laboratories are willing to take part. Mandel [56] states that all laboratories that volunteer for participation in the study are acceptable, provided they are reliable, competent and experienced. According to CIPAC [15], it is necessary that the participants are experienced in the analytical procedures used, but they are not required to be familiar with the method itself. This guideline suggests a certain amount of prior determinations for each laboratory in order to familiarize with the method. ISO [1] also mentions this possibility for some of the laboratories which are less confident with the method. In contrast, DIN [14] recommends that the laboratories are familiar with the method and its applications. Accordingly, this norm doubts whether a familiarization is really necessary Number of laboratories required Youden and Steiner [8] point out that the usual number of 6 10 participants is an inadequate basis to ascertain how laboratories vary among themselves with respect to their systematic errors. Moreover, they prefer a larger number of laboratories rather than a larger number of replicates. The German [14] and the ISO [1] standards recommend that 8 15 laboratories should take part in the study. CIPAC [15] and IUPAC [17] additionally specify an absolute minimum number of five laboratories. Youden and Steiner [8] indicate that the number of laboratories does not necessarily have to be equal for all concentration levels or matrices studied. In contrast, ISO [1] recommends a balanced uniform level experiment, which requires the same number of test results and the same number of levels in each laboratory Number of matrices/concentration levels required The German standard suggests examining at least a low, a mean and a high level of possible concentrations [14]. A larger amount of levels and matrices is recommended by the AOAC and IUPAC guidelines [16,17]. A minimum number of three materials for a single-level specification and of five materials for other cases is prescribed Number of replicates recommended Since the different organizations have a different attitude towards the replication of experiments (see Section 2.5), a different number of replicates is recommended. While ISO [1] does not specify the number of replicates, CIPAC [15] clearly requires a duplication of the experiments. The German standard [14] prescribes to perform at least two parallel determinations, but recommends a larger number (e.g. four). Moreover, this norm requires that the product of the number of laboratories and the number of replicates has to be equal to or larger than 24. While the German standard [14] allows the performance of more replicates than prescribed in the protocol only if an obvious outlier occurs, CIPAC [15]

9 E. Hund et al. / Analytica Chimica Acta 423 (2000) requires registered reasons for the execution of further repetitions Interdependence between the number of laboratories and replicates The uncertainty of the variance estimates obtained depends both on the number of laboratories and the number of replicates. ISO [23] provides a table with the uncertainties of repeatability and reproducibility estimates dependent on the number of laboratories and replicates for a probability level of 95%. The table only requires a preliminary estimate of the ratio of the repeatability and reproducibility variances. Using the ISO table, the uncertainty of the precision estimates can be restricted by selecting the appropriate number of experiments (replicates and laboratories). Since the formulae to derive the required number of experiments from the acceptable uncertainty are included in [23] as well, it is also possible to consider a probability level different from 95% Evaluation The individual collection of the data for a certain concentration level or a certain matrix is followed by a thorough check for outliers and the calculation of the performance characteristics of the method Outlier treatment and robust estimation Most guidelines and standards define three types of outliers: outliers among the replicates within one laboratory (type 1), outliers among the means of the laboratories (type 2) and outliers among the laboratory standard deviations (type 3). For outliers of types 1 and 2, the same tests are applicable. If only a pair of replicates is available, no test for outliers of type 1 is performed. In protocols without replication in the laboratories, only tests of type 2 are considered. The general properties of the different tests as well as some robust methods are discussed in Section 2.6. Among the standards considered, only the German standard [14] recommends to measure more than two replicates. Accordingly, it also suggests to test for all three types of outliers. Grubbs test for single outliers is applied consecutively to the results within one laboratory and to the laboratory means. After the elimination of the outliers, a preliminary variance is calculated based on all retained data from all laboratories. An F-test is applied to test for outlying laboratory variances. The calculation of the preliminary variance is repeated using the retained data and a new series of F-tests is performed until no more outliers can be identified. Afterwards, it is tested whether some of the values that were eliminated as an outlier of type 1 fall in the interval between the lowest and the highest laboratory mean. If such values are found, they are retained in the data. ISO [1] first repeatedly performs an upper-tail Cochran test for the comparison of the laboratory variances. If two or three laboratories show large standard deviations, especially within only one level, conclusions from the Cochran test should be drawn with care. Inconsistent values at different concentration levels or matrices for a certain laboratory indicate that, in general, the within-laboratory variance is too high and that all data from that laboratory should be omitted. Grubbs tests for single and double outliers are then applied to check whether there are outliers among the laboratory means. The outlier algorithms in the ISO standard [1] are accompanied by a graphical outlier test that uses ordered bar plots, and Mandel s h- and k-statistics [56] (see Section 2.7). The AOAC guideline [16] and the IUPAC protocol [17] perform the outlier tests in a different way. Each cycle comprises a one-tailed Cochran test, a two-tailed single Grubbs test and a paired Grubbs test. If no outliers can be found in such a cycle, the outlier removal is complete; otherwise a new cycle follows. Nevertheless, it is also specified that not more than two out of nine of the laboratories should be rejected, since a large number of outliers indicates that there are problems with the analysis method(s) used. The CIPAC guideline [15] also considers only outliers of types 2 and 3. Additionally, a box plot [53] is suggested to highlight deviating values. If a split-level design is applied, Cochran s test [60] is performed using the ranges of the paired replicates instead of their variances. The test compares the maximum range occurring with the sum of all ranges. Outliers among the laboratory means are identified using Grubbs test. Youden and Steiner [8] attribute subordinate importance to replication and test only for outlying laboratories using the Dixon outlier test [45]. Moreover, if a pair of materials, X and Y, is examined, they recommend a Youden plot, which identifies points outside of a cluster as outliers (see Section 2.7). Furthermore,

10 154 E. Hund et al. / Analytica Chimica Acta 423 (2000) they suggest a ranking test for the different laboratories based on a second classification scheme. For each concentration level or matrix considered, the results are replaced by their rank within the results. The ranks are summed over all concentration levels and matrices. Laboratories with an unacceptable performance usually show extreme scores and can be identified using comparison with tabulated values. AMC suggests applying robust statistics as an alternative to outlier tests and elimination [61] Missing values Youden and Steiner [8] suggest to drop the whole pair of replicates if one of the values is missing. ISO [1] offers the choice between discarding or keeping solitary test results. CIPAC [15], suggests the maintenance of solitary test results but only if otherwise the number of laboratories would be lower than the minimum number required Further statistical treatment Mandel [56] presents his h- and k-statistics not only to detect outliers but also as a flexible tool to analyze the results of an inter-laboratory study. Besides these statistics, he suggests to plot the standard deviations versus the concentration levels to check whether a functional relationship exists. ISO [1] also suggests checking for the dependency of the precision estimates on the concentration level. For both repeatability and reproducibility, it is verified whether a straight-line relationship or an exponential relationship exists. A graphical evaluation is preferred to a numerical one for practical reasons since the latter requires weighted regression. It is also pointed out that a functional relationship does not necessarily exist. The IUPAC protocol [17] prescribes a one-way ANOVA at each level to estimate the components of variance as well as repeatability and reproducibility parameters. According to Horwitz and Albert [31], this is a weak point of the protocol, since the one-way ANOVA assumes equal within-laboratory precision for all laboratories, which is rarely true Results All standards and guidelines consider the repeatability and reproducibility estimates as the most important results of the inter-laboratory test. This is independent of the question whether these measures are expressed as repeatability/reproducibility (limits), as repeatability/reproducibility standard deviations or as relative repeatability/reproducibility standard deviations. The corresponding variances are derived either by direct calculation or by an ANOVA. Although it is not always explicitly mentioned, the grand or general mean is an important result of the inter-laboratory study, too. The comparison of the grand mean with the true value of the certified reference material or the accepted value of the sample allows testing whether the method is biased. If spiked material is used for the inter-laboratory test [16], the recovery rate allows to estimate the bias. Moreover, in the extension of the inter-laboratory study to a comparison of two alternative methods, Youden and Steiner [8] evaluate a possible bias and compare the precision of these methods Report and conclusions The final report should contain the results of the single measurements, the laboratory means, the grand mean, the repeatability, the reproducibility and a possible functional relationship between the precision estimates and the concentration levels. The results of the outlier treatment including stragglers should be reported as well. Moreover, the comments of the participants could be included as useful information. Considering more than 7500 method performance studies, Horwitz et al. [62] derived a functional relationship between the reproducibility precision and the concentration of the analyte examined. The reproducibility is doubled for each decrease in concentration by a factor of 100. This relationship turned out to be independent of the analyte, the matrix and the analytical method. Although the results do not exactly follow this relationship, the majority of the points falls in a Horwitz-band between 0.5 and 2 times the value predicted [63]. For concentrations higher than 10 8 mol/l, comparison with this expected reproducibility allows judging whether the reproducibility attained is acceptable or not. Recently, Thompson [64] suggested to establish three different concentration precision relations for high and lower concentrations as well as for trace analysis. For concentrations lower than g/l, he proposes a constant ratio between precision and concentration. AOAC [16] recommends a generally accessible publication of the report to attribute more significance to the study. The report should comprise the specifi-

11 E. Hund et al. / Analytica Chimica Acta 423 (2000) cation of the analyte, the results observed, a listing of a description of all materials, the number of laboratories retained after elimination of outliers, the number of outliers, the grand mean, the true or accepted value (if known) as well as repeatability and reproducibility estimates. The latter two values should also be expressed as standard deviation and relative standard deviation. The report form suggested by the German standard additionally comprises variation coefficients and degrees of freedom [14]. The panel has to decide whether some measures have to be taken in the outlying laboratories. Moreover, if the precision evaluated for the method is not acceptable, possibilities to improve the method have to be discussed [1]. Furthermore, if the aim of the study was to provide a standard method, the panel also has to decide whether the method studied is appropriate to be used as a standard method. 4. Inter-laboratory bias tests 4.1. Aim This type of inter-laboratory study [2,3] allows to evaluate either the bias of a standard measurement method or the bias introduced by the laboratories that use a standard method (i.e. the laboratory bias). For this study, it is essential that an accepted reference value can be established, e.g. by using certified reference materials or materials, whose properties have been established using a measurement method whose bias is known to be negligible Relation to collaborative trials The subject of study in a collaborative trial is the accuracy of a measurement method, whereas the bias tests examine either the bias of a standard method or the bias of the laboratories that use a standard method. There is a large agreement in the experimental set-up of these three types of studies. For instance, within each laboratory, the experiments are performed under repeatability conditions. Moreover, the same requirements are specified for the panel; the experimental design; the recruitment of the collaborators; the outlier treatment explanations for outliers are, however, more important and the application of Mandel s h and k plots [56]. The main differences between inter-laboratory bias tests and collaborative trials are as follows: A standard method is used instead of a selected method. No pilot trial is required since repeatability and reproducibility estimates are available. The determination of the number of laboratories to participate and the number of replicates required (see below). The differences in the interpretation (see below) Evaluation of the method bias Determination of the number of laboratories and replicates The number of laboratories depends on the number of replicates performed per laboratory and vice versa. ISO [2] provides a table from which the number of laboratories and replicates can be derived, which allow with high probability to detect that the bias is not significantly different from zero (α = 0.05) and to detect a beforehand specified bias (β = 0.05) if this really exists. It is suggested to fix the number of replicates to two and to derive the corresponding number of laboratories [2] Interpretation For each participating laboratory, the variance and the mean value of the measurement method are calculated. This allows deriving the grand mean and the repeatability variance, which is defined as the arithmetic mean of the repeatability variances of the laboratories. Cochran s tests [60] and Mandel s h and k plots [56] are recommended to check for outliers. If an estimate of the repeatability variance is available, it has to be tested whether the value observed in the test is in agreement with this a priori estimate (χ 2 -test). If the test is not significant, the already established repeatability variance is used for the further evaluation. Otherwise, it is necessary to investigate the reasons for the difference and possibly to repeat the experiments. The reproducibility variance is calculated, but it is only used for the further evaluation if no value for the reproducibility variance is already available. If an estimate of the reproducibility variance is available, it

12 156 E. Hund et al. / Analytica Chimica Acta 423 (2000) has to be tested whether the value observed in the test is in agreement with this a priori estimate (χ 2 -test). The evaluation of the reproducibility is performed in an indirect way from the variance of the laboratory means. This can be expressed as a function of the reproducibility and the repeatability variance. If the χ 2 -test is significant, a thorough examination of the working conditions in all participating laboratories is required. As a consequence of this investigation, it might be necessary to repeat the experiments in order to obtain the required precision values. The bias of the standard measurement method is estimated as the difference between the grand mean observed and the accepted value. A confidence interval is calculated for the bias to test whether it is significantly different from zero Determination of the laboratory bias The laboratory bias is evaluated by comparing the results in one laboratory with the results obtained from a collaborative study [3]. The number of replicates required to be able to detect a beforehand specified bias with 95% probability depends on the repeatability standard deviation. It can be derived from a formula given by ISO [3]. If the repeatability variance of the standard measurement method is known, the within-laboratory variance is compared with the former by means of a χ 2 -test. If the test is not significant, the established value is used in the bias evaluation. Otherwise, it is necessary to verify whether all steps of the standard measurement method are properly implemented. The bias is estimated as the difference between the mean value observed in the laboratory and the accepted value. A confidence interval is calculated for the bias in order to test whether it is significantly different from zero. 5. Proficiency testing 5.1. Aim Proficiency tests are inter-laboratory experiments to check the testing performance of the participating laboratories [4,6]. Such test schemes are organized individually for the analysis of a certain analyte (or a group of analytes) in a certain matrix. All proficiency testing schemes are run regularly, but each scheme has its own frequency of test rounds. Inter-laboratory studies that are performed only once to compare the ability of different laboratories in a certain analysis are called cooperative trials. A typical occasion to set-up such an experiment is the decision on the outsourcing of an analytical method [65]. Well-established proficiency testing schemes can be used in decisions on accreditation [66,67]. It was shown in a case study [68] that a good within-laboratory quality control is usually related to good results in proficiency testing. King et al. [69] evaluated whether accredited and certified laboratories perform better than others in proficiency tests, but did not find significant differences. The provision of additional confidence to laboratory clients may be a reason for voluntary participation [4]. In a proficiency test, the laboratories are generally free in the selection of the test method, except if authorities prescribe a standard method. Proficiency testing is only applied to methods that are performed in routine. In trace analysis, it seems necessary also to analyze materials containing effectively zero concentration of analyte. This requires, however, that the scheme organizers have at their disposal a method with a detection limit lower than the ones of the methods of the participants [70]. In the last years, even some organizers of proficiency tests have obtained accreditation for their schemes, e.g. the KIWA and the Institute for Inter-laboratory Studies (IIS), both from The Netherlands [71] Preparation Organization committee The online information system for inter-laboratory studies [18] shows that at least in Germany and Austria, the largest amount of proficiency tests are organized by governmental supervision authorities. Governmental or public initiation can also lead to self-supporting proficiency testing organizations, as is for instance the case for the marine monitoring program QUASIMEME [25]. Further tests are initiated by joint organizations of firms producing in a common field. In contrast to collaborative trials, the organization committee should be independent of the participating laboratories. If the result of a proficiency test affects the professional status of the participating