In-house germination methods validation studies: analysis

Transcription

1 1 In-house germination methods validation studies: analysis Jean-Louis Laffont - ISTA Statistics Committee

2 Design assumptions for the validation study Based on peer validation guidelines From Table 1 in ISTA Method Validation for Seed Testing document and experience Number of laboratories: minimum of 3 Number of lots: 3 4 reps of 100 seeds tested

3 Which characteristics to use for assessing the performance of the in-house method? Evidence for Repeatability and Reproducibility assessment for which appropriate computations in the context of germination data have been developed within ISTA: Repeatability Let I be the total number of lots, J be the total number of labs, K be the number of reps of m seeds for a given lot in a given lab, pk be the percentage of germinated seeds for lot i, lab j and rep k p... ( 100 p... The repeatability standard-deviation is computed as: Sr fr m. p is the overall average percentage of germinated seeds..... f is an estimate of the dispersion parameter: r 1 var_ obs f r IJ i,j var_ bin (1 var_ obs 1 ( p. ( 100 p. pk p and. var_ bin with p K 1 k m. being the average percentage of germinated seeds in lot i and lab j If f > 1 one speaks of overdispersion because the data have larger variance than r expected under the assumption of a binomial distribution. If f <1, then there may be r evidence of underdispersion which may result from lack of independence between reps within a lab. Consider the following Generalized Linear Model (GLM: y k ~ Binomial(m k, π k π k logit ( π k log µ + αi + β j + ( αβ 1 π k. i 1,,, I j 1,,, J k 1,,, K. y k is the number of germinated seeds out of m k in lot i, lab j and rep k. µ is the general effect. α i is the fixed effect of lot i. β j is the fixed effect of lab j. (αβ is the fixed interaction effect between lot i and lab j. The φ ² factor, characterizing overdispersion, can be estimated by dividing the sum of the squared Pearson residuals after fitting the model by the residuals degrees of freedom ( IJ(K-1 here. For this particular GLM, the algebraic expression of this estimate is: 1 ( yk m ˆ kπ yk k fr IJ ( K 1 ( where ˆ π m ˆ π 1 ˆ π mk ( i,j k k When m k m 100, expression ( simplifies to expression (1. Therefore the simplified calculations in (1 are appropriate and easily implemented in MS Excel. k Reproducibility Consider the following Linear Mixed-effects Model (LMM: p. µ + αi + bj + e. i 1,,, I j 1,,, J. p. is the percentage of germinated seeds out of n in lot i and lab j. µ is the general mean. α i is the fixed effect of lot i. b j is the random effect of lab j. The b j are iid N(0, σ Lab.. e are the residuals. The e are iid N(0, σ. In a context of a LMM, the reproducibility standard-deviation is then defined to be: S ˆ ˆ R σlab + σ where ˆσ and Lab ˆσ are the variance component estimates. When data are perfectly balanced (no missing Lot x Lab combination, we have: 1 ( p. p p. i.. j SR where pi.. I J 1 J i j Assuming a binomial distribution, the variance of p. is: p. ( 100 p. Var ( p. n We then compute the following quantity to characterize overdispersion when Lab and Lot by Lab variations are considered: p. nsr i,j fr where p... p... ( 100 p... IJ The square root of f is then compared to the f value defined by Miles (1963 in R equation AG4 and which is used to develop ISTA tolerance tables for comparing germination results from different labs. 3 3

4 Which characteristics to use for assessing the performance of the in-house method? 4 In addition to Repeatability and Reproducibility, do we need to assess Accuracy (agreement between the value that is adopted, either as a conventional, true or accepted reference value, and the value found? How to define the reference value? Means from the reference method? Statistical test: Mean from the in-house method significantly > Mean from the reference method? 4

5 Analysis process summary 5 Data checking Boxplots ISTA tolerances for germination test replicates Repeatability assessment: f r dispersion factor f r 1 NO In-house method doesn t meet repeatability requirement Reproducibility assessment: f R dispersion factor f R Miles f factor taking into account variation among labs NO In-house method doesn t meet reproducibility requirement Accuracy assessment? 5

6 Excel tool to support the process 6 Ensuring appropriate/harmonized analysis performed 6

7 7 Thank you for your attention 7