Our Experience With Westgard Rules

Transcription

1 Our Experience With Westgard Rules

2 Statistical Process Control Wikipedia Is a method of quality control which uses statistical methods. SPC is applied in order to monitor and control a process. Monitoring and controlling the process ensures that it operates at its full potential.

3 Principle Sampling or monitoring i of the process The results of the process follow some distribution Gaussian Distribution Continuous variable with repeated measures of QC Binomial Distribution Proportion of QA results outside the allowable limits ofperformance. Poisson Distribution Count of misidentified samples. Distribution parameters established based on historical data where process has been stable. Take a measurement Ask a statistical question How likely is the current result to belong to the same distribution? Does a series of results suggest the distribution ib i has changed?

4 Control Chart Observed values plotted against time

5 Binomial Distribution 0.25 Binomial Distribution Trials = 25 p =

6 Control Charts for Continuous Data Mean and Range Chart X and R For n series of measurements plot the mean of the n values and therange (maximum value minium value) Typically n is 4 or 5. Calculate +/ 3s Control Limits Levey Jennings Chart Chart of individual measurements

7

8 Statistical Quality Control Charts for individual measurements Such a chart may be better than nothing, but is much less satisfactory than a conventional Xbar chart based on a subgroup size of 4 or 5. If the limits on a chart for individuals values are set at μ (or Xbar) ±3σ, the chart is relatively insensitive to substantial shifts in process average. Although greater sensitivity to such shifts may be gained by the use of narrower limits, such sensitivity is gained only by increasing the chance of false indications of lack of control (called type I errors). Unless a chart for individual measurements is accompanied by a range chart, it is difficult to discover whether there have been changes in process dispersion. In general, charts for individual measurements are inferior to conventional control charts because they fail to give as clear a picture of changes in a process or as quick evidence of assignable causes of variation Grant, E. L. and R. S. Leavenworth (1988). Statistical Quality Control. 6th ed. New York, McGraw Hill Book Company.

9 If Levey Jennings Charts are less powerful, why do we use them? Each point on the chart is subject to evaluation Go, No go decision making May need to run more quality control in a given time period in May need to run more quality control in a given time period in order to construct a mean and range chart

10 So, having chosen to use the inferior control chart, how can we improve the rates of error detection and reduce the rate of false rejection

11 Westgard Rules 1 3s One control value more than 3 SD from the target s One control value more than 2.5 SD from the target. 2 2s Two consecutive control values more than 2 SD from the target in the same + direction. This rule applies within a single QC level or across QC levels. R 4s One control value more than 2 SD in one direction and another control value more than 2SD from the target in the opposite direction. This may also apply within a single level or across levels. 4 1s Four consecutive control values more than 1 SD from the target in the same direction. 10 x Ten consecutive control values falling in one direction irrespective of magnitude, a rejection rule sensitive to bias error.

12 s 2 2s 4 1s 8x 10x Serum X -3SD -2SD -1SD Mean 1SD 2SD 3SD 142 R 4s

13 Other run rules 7 successive points on the same side of the mean 10 of 11 successive points on the same side of the mean 12 of 14 successive points on the same side of the mean 14 of 17 successive points on the same side of the mean 16 of 20 successive points on the same side of the mean Grant, E. L. and R. S. Leavenworth (1988). Statistical Quality Control. 6th ed. New York, McGraw Hill Book Company.

14 What about the 1 2s rule Quality Control Paper based evaluation 1 2s 1 3s 2 2s R4s 4 1s 10x OK Troubleshoot tthe assay

15 What about the 1 2s rule Quality Control Computer based evaluation 1 3s 2 2s R4s 4 1s 10x OK Troubleshoot tthe assay

16 How good is the QC algorithm?

17 Power Function Chart Power Function Chart (N=2) Erro or Detectio on Systematic Error (SD) 1.3s 1.3s/2.2s/R4s 2.5s 12 2s

18 Power Function Charts Figure 1. Power function charts derived as described in the text, using n = 2. (A), recreation of the power function charts as described on the Westgard website. The values of R are taken from the website. ( ), 13s/22s/14s/10x / / (R = 5); ( ), 13s/22s/41s (R = 2); ( ), 13s/22s (R = 1); (Graphic), 13s (R = 1). (B), the same rules as in A except that cross run run rules add to error detection only on those occasions when a change in bias has not been detected in the QC runs required for application of the crossrun rules. For cross run rules, error detection is shown for the first run in which each rule has sufficient data to operate. ( ), 13s/22s/41s/10x; / / ( ), 13s/22s/41s; ( ), 13s/22s (R = 1); (Graphic), 13s (R = 1). (C), the power of error detection of the individual cross run rules for errors undetected by shorter run rules. ( ), 10x; ( ), 41s. Jones, G. R. D. (2004). "Reevaluation of the Power of Error Detection of Westgard Multirules." Clinical Chemistry 50(4):

19 How good is the assay?

20 Assay Capability C p = USL LSL 6 C pk = USL LSL min, 3 3 Cps = Allowable Limit of S Performance Sigma Metric 6 Sigma process has a C p of 2

21 3 Sigma Process P(x) P(x+1.5) LCL UCL

22 Capability and Error Rates Errors per Errors with Sigma million 1.5s shift

23 AQCStrategy For any quantitative test: Select an appropriate analytical goal. Determine the imprecision in your laboratory. Calculate the capability. Select QC algorithm based on capability index. Aim for probability of error detection ~90% Minimise probability of false rejection Cps < 4 Cps >= 4 and < 6 Cps >= 6 Multirule or 1 2.5s with N = 4 (two levels twice per shift). Multiruleor 1 2.5s with N=2. (two levels once per shift) 1 3s rule with N=2. (two levels once per shift)

24 Reacting to a QC flag Serum X -3SD -2SD -1SD Mean 1SD 2SD 3SD 142

25 Serum X -3SD -2SD -1SD Mean 1SD 2SD 3SD 29/01/ /01/ /01/ /01/2006 1/01/2006

26 Serum X -3SD -2SD -1SD Mean 1SD 2SD 3SD 29/01/ /01/ /01/2006 8/01/2006 1/01/2006

27 QC Failure Corrective action required immediately? Y Any instrument flags? N Maintenance up to date? Y N N Monitor QC performance and correct non-urgent faults as appropriate Correct instrument fault Perform all outstanding maintenance Y Reagents OK? N Correct reagent fault Y QC material OK? Y Calibrators OK? Y Instrument OK? Y N N N Make-up and run correct QC Make-up & run correct calibrator Fix identified instrument t fault N Contact senior scientist if unable to identify fault Re-run QC if fault corrected Rerun QC OK? Y If indicated, contact supplier/ manufacturer Rerun pt samples in batches of 5 'til old & nes results differ by <1 ALE. Correct pt results. Record corrective actions, lab no. & reults in troubleshooting logbook

28 Biggest Problem Lot to lot variation in reagents and calibrators

29 Summary of our initial approach Select an appropriate analytical goal. Determine the imprecision in your laboratory. Cl Calculate lt the capability. Select QC algorithm based on the capability. Aim for probability of error detection ~90% Minimise probability of false rejection Have an action plan for QC failures Staff training and education

30

31 Frequency of QC If just base on P ed and P fr, then increasing the number of QC makes sense.

32 Parvin, C. A. and A. M. Gronowski (1997). "Effect of analytical run length on quality control (QC) performance and the QC planning process." Clinical Chemistry 43(11):

33 Frequency of QC What is the average number of patient results that exceed your performance criteria that you would accept before detecting a problemwith quality control?

34 Definitions M is the number of patients t in an analytical lti lrun. Number of patient samples between QC samples. N is the number of control samples run when performing QC (typically 2 or 3) TE a is the total allowable analytical error Typically the RCPA allowable limit i of performance in our laboratory. ANP TE is the average number of patients that contain unacceptable analytical errors resulting from an out of control situation. ANP E : Increase in number of patient samples with unacceptable errors before the next QC ANP QE : Increase in number of patient samples with unacceptable errors after the first QC tests after the error.

35 5.00 Error Condition ANP TE M = 20 ANP E ANP QE Result QC 3SD 2SD Mean +2SD +3SD N = 2

36 More Definitions P ed is the probability bilit of error detection ti for our QC rule. ARL ed is the average number of runs before an error is detected. ARL ed = 1/P ed ANP ed is the average number of patient before error detection ANP ed ~ M/2 + M(ARL ed 1) Suppose M = 100 patients, and P ed = 90%, then ANP ed = 100/ *(1/0.9 1) = 61 ARL fr is the average number of runs before false rejection ANP fr is the average number of patients before false rejection ANP fr = ARL fr * M 1 3s rule with N = 2, P fr = , ARLfr = 1/ ~ 185, ANP fr =18500

37 Yet More Definitions Let P E (SE) represent the probability that the test result contains an analytical error that exceeds TE a during an out of control condition that causes a systematic error of SE standard deviations. P E (0) is the probability that the test results contains an analytical error when there is no systematic error. ANP TE = ANP E (P E (SE) P E (0)) ANP E = M/2(P E (SE) P E (0)) ANP QE = M(ARL ed 1)(P E (SE) P E (0)) Similar relationship can be shown for random error (RE)

38 Rules 1 ks rule Reject if any of the N control samples in the analytical run are more than k analytical SDs from the target. X(c)/R4s rule Reject if the average z score for the N control samples in the analytical run exceed c standard errors of the mean, or the range of the z scores of the N control observations exceeds 4. X(c)/R 4s

39 M = 80, N = 4 M = 40, N = 2 TE a = 5.0 M = 20, N = 1 Fig. 1. ANP TE (A), ANP E (B), and ANP QE (C) as a function of SE for the 1 ks rule with three different analytical run lengths when TE a 5.0. The short dashed lines represent an analytical run defined as 20 patient specimens followed by one control sample, the medium dashed lines represent 40 patient specimens followed by two control samples, and the long dashed lines represent 80 patient samples followed by four control samples. kset such that the 1 ks rule results in an ANP fr = 2000 Parvin, C. A. and A. M. Gronowski (1997). "Effect of analytical run length on quality control (QC) performance and the QC planning process." Clinical Chemistry 43(11):

40 Fig. 2. ANP TE as a function of SE for the 1 ks rule with three different analytical run lengths and a range of TE a specifications. See Fig. 1 for further details.

41 Lesson Only in rare cases would less frequent testing of a large group of control samples be preferable to testing fewer control samples more frequently

42 Summary of our approach Select an appropriate analytical goal. Determine the imprecision in your laboratory. Cl Calculate lt the capability. Select QC algorithm based on the capability. Aim for probability of error detection ~90% Minimise probability of false rejection For poorly capable assays, more frequent QC rather than just more QC. Have an action plan for QC failures Staff training i and education