Proficiency test evaluation Performance data evaluation Chapter 8.2.2

Transcription

1 Proficiency test evaluation Performance data evaluation Chapter Thursday 1st September th ISTA seminar on statistics Hohenheim Germany Sylvain Gregoire

2 Statistical evaluations Sylvain GREGOIRE Kirk REMUND Jean Louis LAFFONT Christoph HALDEMANN Bettina KAHLERT and many others ISTA Statistics Committee 2

3 Conditions for ISTA accreditation Chapter Have established method(s) Define scope of accreditation Submit performance data evaluation Have results from ISTA Proficiency test program Have an on site evaluation ISTA Statistics Committee 3

4 Suggested decoding of rates = as for other proficiency tests A No problem has been detected in this test B There are small problems, but no specific look or action is suggested to the participant C Problems, ISTA indicates there might be things to consider by the laboratory to explain or correct things BMP Below Minimum of Performance, ISTA indicates that the results were poor and the laboratory need to find explanations and to improve/correct ISTA Statistics Committee 4

5 Rating as for other proficiency tests Rating is a general feature of ISTA proficiency tests, provided by each technical committee through the test leader and ISTA secretariat, with the assistance of the proficiency test committee and statistics committee The current ISTA system rates : Example1: 5A rating and 1 BMP (Cotton) 5*5points + 1*0point =25 points Overall rating is B Example 2: 4 B =16 points each proficiency test One test rating One test Score Value and a run of 6 tests Overall rating on 6 tests Range on 6 tests A 5 points A points B 4 points B points C 3 points C points BMP 0 points BMP below 16 points ISTA Statistics Committee 5

6 Information available Before to send the samples to the laboratories Protocol (events, sample size, ) Purity check (Conventional and GM seeds) True value for each sample (spiking) Result sheets returned from laboratories Presence/absence Quantification when presence Other information (method, raw data points, ) = better situation than GE,PU,OSD,Vigour, Computational results by laboratory, and from all laboratories (median ) ISTA Statistics Committee 6

7 Rating a laboratory in a given proficiency test We need to compute objectively, and transparently to the participants, criteria to rate laboratories for each proficiency test on: Presence/absence 3 systems are proposed to rate presence/absence Quantification 3 systems are proposed to rate quantification ISTA Statistics Committee 7

8 Rating presence/absence True value is known (purity check + more than 2 seeds are spiked) 2 types of error can occur: Laboratory report as negative a positive sample Laboratory report as positive a negative sample The number of mistakes is used as a basis for the rating computations ISTA Statistics Committee 8

9 Presence/absence 3 suggested rating systems Example with 12 samples in a Test Rate Fixed Number of misclassified samples Rate Percentage of misclassified samples Rate Percentage of misclassified samples A 0 errors A 0% - 5% A 0% - 6% B 1 or 2 errors B >5% - 10% B >6% - 20% C 3 errors C >10% - 20% C >20% - 30% BMP More than 3 errors BMP >20% BMP >30% System1 System2 System3 3 mistakes 3/12=25% ISTA Statistics Committee 9

10 Presence/absence 3 suggested rating systems Rate Fixed Number of misclassified samples Rate Percentage of misclassified samples Rate Percentage of misclassified samples A 0 errors A 0% - 5% A 0% - 6% B 1 or 2 errors B >5% - 10% B >6% - 20% C 3 errors C >10% - 20% C >20% - 30% BMP More than 3 errors BMP >20% BMP >30% System1 System2 System3 Example with 12 samples in a Test : system 2 is little more stringent Rate Fixed Number of misclassified samples Rate Percentage of misclassified samples Rate Percentage of misclassified samples A B C BMP 0 errors 1 or 2 errors 3 errors More than 3 errors A 0 errors A 0 errors B 1 error B 1 or 2 errors C 2 errors C 3 errors BMP More than 2 errors BMP More than 3 errors ISTA Statistics Committee 10 System 1 system 3

11 Presence/absence 3 suggested rating systems Rate Fixed Number of misclassified samples Rate Percentage of misclassified samples Rate Percentage of misclassified samples A B C BMP Rate 0 errors 1 or 2 errors 3 errors More than 3 errors Fixed Number of misclassified samples A 0% - 5% B >5% - 10% C >10% - 20% BMP >20% A 0% - 6% B >6% - 20% C >20% - 30% BMP >30% System1 System2 System3 Example with 20 samples in a Proficiency Test: System 3 is more lenient Rate Percentage of misclassified samples Rate Percentage of misclassified samples A 0 errors A 0 or 1 error A 0 or 1 error B 1 or 2 errors B 2 errors B 2 to 4 errors C 3 errors C 3 or 4 errors C 5 or 6 errors BMP More than 3 errors BMP More than 4 errors BMP More than 6 errors System 1 and system 2 are about the same ISTA Statistics Committee 11

12 Conficence interval are rather wide Many more samples would be necessary to estimate error rates with precision ISTA Statistics Committee 12

13 Presence/absence PT04 50 laboratories Comparaison of the 3 Rating Systems for the Laboratories participated in ISTA Proficiency Test 4 only 50 Number of Laboratories Rating System 1 Rating System 2 Rating System 3 0 A B C BMP ISTA Statistics Score Committee Classes 13

14 Presence/absence rating systems 21 laboratories PT01+PT02+PT03 20 Number of Laboratories Rating System 1 Rating System 2 Rating System A B C BMP Score Classes ISTA Statistics Committee 14

15 Presence/absence rating systems 13 laboratories PT01+PT02+PT03+PT04 Comparaison of the 3 Rating Systems for the Laboratories participated in all 4 ISTA Proficiency Tests Rating System 1 Rating System 2 Rating System A B C BMP Score Classes ISTA Statistics Committee 15

16 Quantification 3 suggested rating systems System 1: true value is number of GM seeds/ total number of seeds in each sample System 2: true value is weight of GM seeds/total weight of seeds in each sample System 3: true value is the median of values reported by participating laboratories (after Cochran test) There is no link between [system1 for presence/absence] and [system1 for quantification] Computations are performed from all sample results returned by the laboratories whatever the method (including sub-sampling and quantitative PCR ) Even with Stacked genes, true value and expected %DNA copy reults, could be agreed upon in the furture ISTA Statistics Committee 16

17 Where does these rating criteria come from? It has been worked out for years (as soon as GMOTF was established) Joint group of STA committee and GMO Task Force In line with strategy position paper Adapted to the successive versions of Chapter 8 rules In line with ISTA proficiency test system Different statistical options were developped and compared (pooling, Bayesian approach, estimation and robustness of error rates, ) Checked for adequacy to data from actual situations Checked for consistency with more sophisticated analysis (ie Mixed models) Checked Selection for of the Consistency criteria, among with appropriate testing plan ones design in routine (ie Seedcalc) transparency, «easiness» to compute and understand (Excel, BMP=pencil) ISTA Statistics Committee 17

18 Rating principles for quantification True value is known (purity check + event + number+ weight), and/or can be estimated from all results received Too many sample results are too far from truth - >BMP Average results by spiking level are not accurate ->C Too many inaccurate sample results are not accurate ->B Otherwise rate A NB:Computations are made on GM spiked samples ISTA Statistics Committee 18

19 BMP Yes Too many sample results are too far from truth No C Yes The average results by spiking level are not accurate No B Yes There are too many inaccurate sample results No A BMP Yes More than ½ of the sample results outside [½ true level; 2 x true level] No C Yes Probability to observe a more extreme value of the sum of absolute spiking levels z-scores when assuming the laboratory provides accurate results is 0.01 No B Yes More than 1/6 (~17%) of the sample z-scores outside [-2; +2] (missing sample values are counted as outside) No A ISTA Statistics Committee 19

20 BMP:too many sample results (many could be ¼ 1/3 ½ 2/3 of data points) are too far from truth (4 possible criteria are shown below) Selected criterium: More than half of the results are outside half and double of truth between half and double (m ax 2%?) plus or minus 0,5% ,5 0,2 0, ,5 0,2 0,1 plus or minus half of level plus or minus half of level, minimum 0,2% ,5 0,2 0, ,5 0,2 0,1 ISTA Statistics Committee 20 Within vertical bar = not too far from truth

21 C: The average results ( from 2-6 samples) by spiking level are not accurate Mean of sample results per spiking level is computed A «spiking level z_score» is computed for each spiking level. A sum of absolute «spiking level z_scores» for the different spiking levels is compared to a statistical threshold This is comparable with C rating used in Germination rating for instance 5.3 for 3 values (germination) 5.25 for 3 spiking levels (GMO) 6.43 for 4 levels ISTA Statistics Committee 21

22 B: There are too many inaccurate sample results Individual «sample z_scores» are computed The number of «big» absolute «sample z_scores» is counted If number of big «sample z_scores» exceeds a pre-defined limit then B rating, otherwise A rating Number of non zero level samples in the proficiency test 1 to to to to to 29 4 Maximum tolerated number of sample z_scores out of the interval [-2,+2] More than 1/6th are inaccurate => rate B Accurate = Not far from true value and Not too much variable ISTA Statistics Committee 22 5 samples in a spiking level :not looked at by C rating, checked by B rating

23 ISTA Statistics Committee 23

24 Graphical outputs 1 line = 1 laboratory Under estimate with small variability Over estimate with Big variability Reference: True level in % seed Q-30-BMP S-4-BMP Q-13-BMP Q-18-BMP S-27-BMP Q-43-BMP Q-44-BMP S-1- Q-11- S-20- Q-49- Q-50- Q-2- Q-3- Q-5- Q-6- S-7- Q-8- Q-9- Q-10- S-12- Q-14- Q-15- Q-16- Q-17- Q-19- S-21- Q-22- Q-23- Q-24- Q-25- Q-26- Q-28- S-29- Q-31- Q-32- Q-33- Q-36- Q-37- Q-38- S-39- S-40- Q-41- S-42- Q-46- Q-47- Q-48- Q-34- Q-35- Q % 0.5 % 1 % BMP Check example Results ISTA Statistics Committee 24

25 Same types of graphs are available for BMP, C, B and A rating, with their specific thresholds Reference: True level in % seed Q-30-BMP S-4-BMP Q-13-BMP Q-18-BMP S-27-BMP Q-43-BMP Q-44-BMP S-1- Q-11- S-20- Q-49- Q-50- Q-2- Q-3- Q-5- Q-6- S-7- Q-8- Q-9- Q-10- S-12- Q-14- Q-15- Q-16- Q-17- Q-19- S-21- Q-22- Q-23- Q-24- Q-25- Q-26- Q-28- S-29- Q-31- Q-32- Q-33- Q-36- Q-37- Q-38- S-39- S-40- Q-41- S-42- Q-46- Q-47- Q-48- Q-34- Q-35- Q % 0.5 % 1 % BMP Check example Results ISTA Statistics Committee 25

26 PT3 rating summary for quantitative and semi-quantitative tests 25 laboratories PT3 PT # of Labs 8 6 Rating System 1 Rating System A B C BMP ISTA Statistics Committee 26

27 PT4 rating summary for quantitative and semi-quantitative tests 50 laboratories PT4 PT # of Labs Rating System 1 Rating System 2 Rating System A B C BMP ISTA Statistics Committee 27

28 Summary on suggested rating systems Follow same ISTA principles as for other types of tests Consistent with computational ratings already used in other types of tests Statistically appropriate for this specific type of test (presence, quantification) Appliance (ad hoc, fair and robust) Checked with non ISTA proficiency tests, and data from companies/laboratories Backed up by more sophisticated statistical models, and carefull look by different types of experts ISTA Statistics Committee 28

29 Use of the detailed results provided by the Labs for the Quantitative Test (flour sub-sample results, measurement replicates - results) The dataset corresponding to the results from a particular laboratory is analyzed using a heteroscedastic linear mixed effects model: Y ijkl = μ i + A j(i) + B k(ij) + E ijkl (1) where:. μ i is the mean for the i th spiking level. Only spiking levels 0., and are retained for this analysis (i= 1, 2, 3).. A j(i) is the random effect of the j th sample (j= 1, 2, 3) with spiking level i. The A j(i) are i.i.d. N(0,σ ² sample ), where i.i.d. is used to indicate that the observations are independently and identically distributed. B is the random effect of the k th flour sub-sample from sample j and spiking level i. The. k(ij) B k(ij) are i.i.d. N(0,σ ² flour ).. E ijkl are the measurement errors: E 1jkl are i.i.d. N(0,σ 1 ²) E 2 jkl are i.i.d. N(0,σ 2 ²) E 3 jkl are i.i.d. N(0,σ 3 ²) cov( E ijkl, E i' j'k'l' ) = 0 for i different from i. ISTA Statistics Committee 29

30 ISTA Ordinary Meeting Results of the ISTA GMO Proficiency Tests For each lab and each spiking level, mean and its associated 95% confidence interval Lab # 2 Lab # 4 Lab # 8 Lab # 9 Lab # Lab # 12 Lab # 14 Lab # 15 Lab # 17 Lab # Lab # 19 Lab # 22 Lab # 23 Lab # 24 Lab # Lab # 28 Lab # 29 Lab # 31 Lab # 32 Lab # Lab # 35 Lab # 37 Lab # 40 Lab # 42 Lab # 44b Lab # 47 Lab # 48 Lab # 56 Lab # 59 Lab # ISTA Statistics Committee

31 For ISTA Ordinary each Meeting lab and Results each of thespiking ISTA GMO Proficiency level, measurement Tests CVs Lab # 2 Lab # 4 Lab # 8 Lab # 9 Lab # Lab # 12 Lab # 14 Lab # 15 Lab # 17 Lab # Lab # 19 Lab # 22 Lab # 23 Lab # 24 Lab # Lab # 28 Lab # 29 Lab # 31 Lab # 33 Lab # Lab # 37 Lab # 40 Lab # 42 Lab # 44b Lab # Lab # Lab # 59 ISTA Statistics Committee Lab # 61 Measurement CVs not displayed for lab 32 and 47 because of their large values for some spiking levels: Lab Spiking level Measurement CV % 189% 47 69%

32 Performance data evaluation For chapter laboratory can choose an established and reliable method Then we use the «performance based approach» instead of «one ISTA validated method» Laboratory needs to demonstrate to ISTA accreditation team its ability to perform well with the method used ISTA Statistics Committee 32

33 Accreditaion documents are on line on the ISTA web site, including performance data evaluation ISTA Statistics Committee 33

34 Performance data evaluation The aim: Demonstrate Ability to detect (presence/absence) Ability to quantify (when presence) Be sure before to perform the evaluation Have ad hoc material (seeds, CRM ) Check PDE protocol towards the scope of accreditation with ISTA secretariat ISTA Statistics Committee 34

35 3 steps and more 1. Define accrediation scope 2. Check with ISTA PDE protocol 3. Obtain seeds 4. Check seeds 5. Prepare samples 6. Perform tests 7. Send back PDE document ISTA Statistics Committee 35

36 Check of seeds Test 30,000 conventional seeds and if all are (-) for GM then we have 95% confidence that true impurity is <0.0 If we test 400 individual GM seeds and if all are (+) for GM then we have 95% confidence that true GM purity is above 99.25% Check of purity of reference material REMUND Kirk, workshop on statistical aspects of GMO detection Aim: be sure of the true value of samples prepared for test ISTA Statistics Committee 36

37 Presence/absence ISTA Statistics Committee 37

38 Quantification evaluation 28 samples Range of 7 levels ISTA Statistics Committee 38

39 Accuracy: departure from true value Repeatability: variability of results by level ISTA Statistics Committee 39

40 Repeatability computation ISTA Statistics Committee 40

41 Example of use of the data sent by the laboratory ISTA Statistics Committee 41

42 Departure from true value ISTA Statistics Committee 42

43 repeatability ISTA Statistics Committee 43

44 Deviation and variability by level ISTA Statistics Committee 44

45 Grading performance ISTA Statistics Committee 45

46 Example of documents on related topic Codex alimentarius such as CX/MAS O4/10 European norms such as CEN/TC 275/WG11 ENGL Method Performance Requirements ISO/DIS pren ISO IUPAC harmonised guidelines for single laboratory validation of methods of analysis ISTA has checked with these documents, and with different laboratories and contacts before to release the ISTA documents, which are fitted for the purpose of seed testing ISTA Statistics Committee 46

47 ISTA Statistics Committee 47

48 Updates to come? Difficulty to obtain reference seeds Management of non event specific accreditations Defining true value as expressed in %DNA copies, taking into account stacked genes Increase checking range (presentl 0.-3%) Check purity of GM seeds (95%...) Etc.. ISTA Statistics Committee 48